Humanity - AI Short Film (RunwayML Gen:48 2024 Edition Entry)

Video Description

This weekend, I decided to dive into a short film competition hosted by Runway, who provided some credits to play with. I joined because one of the main challenges with AI tools is the randomness of the output. With enough credits, however, you can render a lot of material, hoping that some of it turns out to be usable. After two intense days of work, I produced my first serious film. Admittedly, if I had more time, I would have made several changes, but overall, I’m content with the outcome. The film is inspired by Yuval Harari’s talk, "The Future of Humanity." If you haven’t seen it yet, I highly recommend it. Here’s a breakdown of my workflow: 33% of my time was spent experimenting and figuring out how to craft effective prompts. I found myself playing around without any specific direction at first. My takeaway is that the process involves more luck than skill. While text-to-video works reasonably well, don’t expect to achieve intricate details through it. For those, you need tools like Flux or Midjourney, which allow you to focus on action and camera movement without needing to over-explain the image. 10% was dedicated to a unique approach I took. I began building an Airtable base integrated with multiple AI tools to analyze the images I generated. First, I connected GDrive to Airtable and used Claude, via script, to analyze my images and provide detailed descriptions, including possible camera movements. Then, I chained other AIs to generate prompts for Flux and instructions for camera action within Runway. 33% of the time was spent on image and video generation. Unfortunately, I didn’t have an API connection with Flux, and it was only today that Runway announced their API release. This part of the process felt tedious—copying, pasting, and rendering repeatedly—work that could have been automated. It was time-consuming and not particularly rewarding. 24% of my time, which I wish had been more, was dedicated to editing, music, sound design, and final tweaks. This part should have been the most critical, but the earlier stages ate up so much time that I was left with too little for this crucial element. All in all, it was a lot of fun. While I don’t think this project is ready for the "real" world yet, I see its potential when combined with real footage. For low-budget documentaries, the results are already fairly comparable. And if I compare the time spent on this versus traditional 3D animation, it’s remarkably faster.