Adobe just introduced their new generative AI for video, and we need to talk about it, because this is going to change Premiere Pro forever.

I just got back from the Adobe MAX conference where Adobe showed off all of the new features they’re building for their programs, including AI for Photoshop and Illustrator. That’s cool, but as a filmmaker I was hoping and waiting for them to show something for us video creators. Thankfully they did not disappoint, and they’re working on some really mind blowing stuff.

Firefly Video AI

If you have any experience with generative AI, you’ve probably worked with something like Firefly, where you type into a text box and either generate an image or modify an existing one. This Firefly video AI system is similar, in that Adobe showed you’ll be able to type into a box and generate a video from a simple text prompt.

The possibilities are really endless with something like this, but keep in mind that generating video from text is significantly more complex than generating a still image, because you’re not generating one image, you’re generating at least 24 images per second. That’s a lot more work, and a lot more opportunities for some kind of glitch to occur and the video to look weird.

Ocean Waves Example

In Adobe’s example, they used the prompt “a blue ocean surrounded by rocks,” and it generated multiple clips that resembled ocean waves crashing onto rocks on a seashore.

It was rough, and the footage looked like it was lower resolution and a lower frame rate. But as we talk about this, remember that this is still version 1.0, and it’s only going to improve as time goes on.

Adobe Sneaks

Here’s where things get more mind blowing, because the ocean waves example was not the only time Adobe showed generative AI for video, and their second example was significantly more impressive.

The second time was during what they call Adobe Sneaks, where they preview projects that are a bit further off, that their engineers are working on and that will hopefully be integrated into the software at some point.

The presentation started in an interface that looked more like Photoshop, where there was a woman standing in front of a bridge. The presenter selected some distracting people in the background and used generative AI to remove them. Pretty standard, right?

Well, the engineer then blew my mind, and everyone else’s, when they revealed that this was actually a video clip, and that the AI had removed those unwanted people throughout the entire clip.

I’m sure you’re thinking, Matt, After Effects already has this, it’s called Content-Aware Fill. But this new Fast Fill feature is dramatically more advanced, because the next example is something I have never seen Content-Aware Fill do.

The presenter had footage of a model walking toward the camera down a walkway while the lighting was constantly changing, with the subject going in and out of sunlight and shadow. The model was in a suit but did not have a tie on, so the presenter selected the part of their shirt where a tie would be, just like you’d select something in Photoshop. Then in the AI text box they typed “tie” and pressed Fast Fill.

In a few seconds, the AI had added a tie to this person’s shirt and tracked it throughout the clip. And what was most impressive is that whenever the presenter hit play, the tie realistically stretched and moved like the rest of the fabric the model was wearing would. More importantly, the lighting on the tie matched the lighting on the rest of the model, so as they moved in and out of light and shadow, this generated footage matched the existing footage essentially perfectly.

This is just mind blowing stuff. I think about how many times I’ve wanted to remove something from a clip, or even add something in, and been unable to. With Fast Fill, it looks like that’s now going to be possible.

Still Image To Video

We aren’t done with video features yet, because another thing Adobe announced is the ability for AI to generate a video from a still image.

In their demo, Adobe showed a still image of an elephant and then used the Firefly video model to animate it and make it move in a realistic way.

Of course, this was a very low frame rate video and resembled one of those Instagram boomerangs moving forward and backward. But my mind immediately went to how usable this would be even in its earliest beta form.

As a filmmaker, there have been many times where I’ve needed to show static images, either photos of people or of locations, and animate them somehow. I could print the images out and slowly pan my camera over them, which can look cool. Or I could animate the digital images and slowly zoom in and out using Ken Burns style effects to make them more interesting to watch.

With this, I imagine turning all of your images into videos, so instead of a slow pan or zoom, the entire video is actually video, even the images. That would be far more interesting for your viewers. It’s super cool.

I really think we’re just starting to scratch the surface of what Firefly AI for video is going to offer. If you want a preview of where things are going, I’d recommend looking at the work being done by projects like Runway ML, which seems a bit more advanced than Adobe right now in terms of AI video generation. But considering how much work and money Adobe is throwing at this, I can see them catching up very quickly.

Also, consider that if you’re a Creative Cloud subscriber, Adobe is most likely going to give you access to all of these AI tools as part of your subscription, which would be great. Of course, they could also decide to charge extra for it. Keep that in mind.

Project Res Up

We’re still not done with AI video features. There’s a lot, isn’t there?

The next one is called Project Res Up, and as the name implies, it’s a tool that upscales video files using AI. The most impressive part was that Adobe was working with very low resolution imagery, either heavily zoomed in footage or old footage much lower than HD, and the AI upscaled it to HD and made it look significantly better.

Of course, this was just a demo and I didn’t get to pixel peep any of it, so it remains to be seen exactly how high quality this upscaling is going to be. But considering there are plugins like Topaz Video AI, and that Blackmagic has very good upscaling in DaVinci Resolve, it’s great to see Adobe working on it too.

Project Dub Dub Dub

We need to take a detour now and talk about audio, because Adobe showed another project with huge implications if you’re a filmmaker who makes videos for viewers around the world.

This one is called Project Dub Dub Dub, and it’s essentially a way to dub any of your videos into a foreign language automatically using AI. You upload a clip with audio being spoken, select the languages you want it dubbed into, and the AI translates your audio into those languages while at the same time recreating your voice speaking them.

So when your audio gets dubbed, it doesn’t sound like you hired another actor to speak in another language. It sounds like your voice.

This is really crazy. I think about how many YouTubers out there have separate channels for separate languages, one where they speak English and another where they speak Spanish. Project Dub Dub Dub could make that whole thing much easier by using AI to dub your voice while still keeping it your voice. So cool.

Conclusion

In short, I’m super excited about not just Firefly audio AI, but video AI as well.

Let me know in the comments. Are you excited about this stuff too, or are you scared? Are you nervous about what’s going on?

And because I’d like to help you out with your filmmaking, I’ve created a free guide called Edit Videos Like A Pro. It shows you the biggest and most important things I’ve learned over the years as a video editor, and I know it’ll help you.

Thanks so much for reading, and have a great day!