Runway’s latest AI video generator brings giant cotton candy monsters to life

On Sunday, Runway announced a new AI video synthesis model called Gen 3 Alpha This is still in development, but it appears to create video of similar quality to OpenAI’s Sora Video, which debuted earlier this year (and also hasn’t been released yet). It can create fresh, high-resolution video from text prompts ranging from realistic humans to surreal monsters roaming the countryside.

Unlike the runway Previous best model As of June 2023, which can only create 2-second clips, the Gen-3 Alpha can reportedly create 10-second videos of people, places, and things that have consistency and coherence that easily surpasses Gen-2. If 10 seconds seems short compared to Sora’s full minute of video, keep in mind that the company operates on a shoestring computing budget compared to the generously funded OpenAI — and actually has a history of bringing video creation capability to commercial users.

Gen-3 Alpha does not generate sound to accompany videos, and temporally consistent generations (those that maintain personality consistency over time) are more likely to rely on Similar high-quality training materials. But it’s hard to ignore Runway’s improvement in visual fidelity over the past year.

Video AI heats up

It’s been a busy couple of weeks of AI video installation in the AI ​​research community, including the launch of the Chinese model Kling, created by Beijing-based Kuaishou Technology (sometimes called “Kwai”). Kling can create two minutes of 1080p HD video at 30 frames per second at a level Details and cohesion It is said that he matches Sora.

Gen-3 Alpha prompt: “Hidden reflections of a woman in the window of a train moving at breakneck speed in a Japanese city.”

Shortly after Kling’s debut, people on social media started getting creative Surreal AI videos Using Luma AI Luma Dream machine. These videos were new and weird but overall cool Lacks cohesion; We tested the Dream Machine and were not impressed by anything we saw.

Meanwhile, New York City-based Runway, founded in 2018, a leading text-to-video company, recently found itself the victim of memes that showed its Gen-2 technology was no longer popular compared to video compositing models the most recent. This is probably the motivation behind the announcement of the Gen-3 Alpha.

Gen-3 Alpha prompt: “An astronaut runs through an alley in Rio de Janeiro.”

Creating realistic humans has always been difficult for video compositing models, so Runway specifically showcases Gen-3 Alpha’s ability to create what its developers call “expressive” human characters with a range of actions, gestures, and emotions. However, the company Examples provided It’s not particularly expressive – mostly people staring and blinking slowly – but it looks realistic.

Human examples provided include generated videos of a woman on a train, an astronaut running down the street, a man whose face is illuminated by the glow of a television set, a woman driving a car, and a woman running, among others.

Gen-3 Alpha Prompt: “Close-up of young woman driving a car, looking pensive, blurry green forest visible through rainy car window.”

The demo videos created also include more surreal video installation examples, including a giant creature walking through a rundown city, a man made of rocks walking through a forest, and the giant cotton candy monster shown below, which is perhaps the best video ever on the page.

Gen-3 Alpha Prompt: “A giant human, made of fluffy blue cotton candy, stomps the ground, roaring toward the sky, a clear blue sky behind him.”

The Gen-3 will power several Runway AI editing tools (one of the company’s biggest claims to fame), including Multi action brush, Advanced camera controlsAnd Manager mode. It can create videos from text or image prompts.

Runway says the Gen-3 Alpha is the first in a series of models trained on a new infrastructure designed for large-scale multi-modal training, taking a step toward advancing what it calls “General global models“, which are virtual artificial intelligence systems that construct internal representations of environments and use them to simulate future events within those environments.