There are no Quick Fixes to AI Art

It has been a marathon of short movie making for me the last days. Today I feel the air coming out of me. As I explore my next short movie ideas I am reminded that there are no Quick Fixes to AI Art or AI Video.

What do I mean by that? I think for anyone of us getting into this we tend to naively think that we can get exactly what we want by just writing the perfect prompt. Don't get what we want? Just tweak the prompt. Still not what we want? Tweak it more. Ever tried that? Gone through 20 images and still not getting what you want?

In a way I think I was lucky to do this with the crappy early AI models. I learned the hard way through hours and hours of wasted time trying to tweak prompts that a lot of what you want to do simply cannot be done.

Instead you need to fire up your photo editor and get your hands dirty. There really is no other way. Making good stuff requires actually spending a lot of time inpainting, photo editing and video editing. AI is still way too stupid to just give you out of the box a beautiful finished product you want.

The image shown here on this article is example of just getting my hands dirty. I generated an initial image without any people in one AI Art generator good with backgrounds. Then I brought it into a model I find very good at inpainting and start inpainting every little detail. I did railing, staircase, floors, windows, walls, everything separately. Adjusting prompts as I went. I had to lookup what all the different architectural elements were called to describe them right. Otherwise you don't get good details.

Next, I used that image to create a video of a girl walking through the scene. But she didn't look right. Not the kind of curves, bust size etc you know me from. Now if you still prescribe to the "quick fixes" belief you would have kept tweaking that Video AI prompt until you get the right girl. Meanwhile you would have burned an enormous amount of cash and I promise you: You would never have gotten the result you wanted.

Again it is down to doing the work. I snapshot part of the video, blew up the image. Brought it into my image editor. Did masking to blend the girl in while keeping my higher detailed background image. The video generator will blur details in the background.

Of course this is the wrong girl, so now the job is to start inpainting with an AI model suitable to create these busty bimbo girls. So that ends up looking quite good. Except now the light conditions are wrong. She doesn't really fit in. So I need to alter the light, but altering the light, contrast and white balance on the whole scene messes everything up. So I need to mask the effect to only apply to her.

Okay, maybe all of this didn't make sense to you, but I am only trying to get across that this image went through a lot of steps to get to the final result. I could not magically get myself to the result I wanted by just endlessly tweaking a prompt.

All through my AI work history of over two years I have kept making this same mistake. We are all a little lazy and rather just have the AI fix everything for us. The allure of the magic perfect prompt is still there in our minds. But just shove that idea to the side. It is BS. Same applies to throwing in endless number of plugins and LoRAs. You throw in one LoRA to fix one problem and then it breaks something else.

Why am I rambling on about this now? Because I realize that so much of what I want to do going forward requires hunkering down and doing all of that manual labour to get the right images and video clips. It feels daunting. I just want my magic prompt, right now.

Then again I tell myself: If it was that easy, then everyone would have done it and nobody would have cared for your work. That sums up my mixed feelings about AI. It enables us to do amazing work, but how far do we want it to go?

On the one hand I really do want to make a Hollywood quality movie of my world. But on the other hand I don't really want everyone in the movie industry to be rendered obsolete. I think it is good that people can do acting, voices, write movie scripts, direct etc.

Just as I think the limitation of AI Image making is probably good as it means traditional artists are not made entirely obsolete. But the technology keeps improving so where will it end?

I honestly don't think it will become good enough in the next couple of years. How can I say this when I lived through such amazing improvements?

Well, over a year ago when first seeing OpenAI Sora demonstrations, I vexed about all the cool AI movies I was going to make. It turns out that a lot of what I predicted and described there was wrong. It is simply much harder to make movies than I assumed back then. The technology is far more limiting. Short demoes can impress a lot but mask very well how many limits exists with AI movie making.

The fundamental problems I saw over two years ago when I began working with AI still applies: AI doesn't really understand what it is doing. Various artifacts is inevitable. There is no way around extensive human guidance and intervention.

Saying current AI will become like humans soon is a bit like looking at a calculator and saying it will become more like humans merely because we have observed each new calculator model can handle bigger numbers and more math operations.

Look, I still think we have lots of improvements ahead of us. Image quality will get better. Resolution will get better. Things will look more realistic. We will be able to generate longer clips than 10 seconds. But we are not going to get whole movies out with just a cool prompt. Not going to happen.

Sure very far down the road that will happen. There is no reason AI cannot replicate all human behavior in the end. But that will not be with current AI technology. That will require a paradigm shift and we are very far away from that.

Or... okay let me moderate that a bit. I have seen the massive improvements on reasoning LLMs (large language models). They likely will supersede humans according to the intelligence tests they are run through in a few years if not next year. But I am increasingly realizing that intelligence is not all of what humans are about. These AI engines still operate in a very limiting, question-answer paradigm. Sure agent systems moves them closer to what we humans do, but still.

All these AI models are still isolated islands. The AI for working with speech is separate from the ones working with images and audio. We humans incorporate essentially numerous specialized AI models all into one grand model. Our brains can reason about text, voice, sound, video and images all at once.

I am simply skeptical anyone will combine all of this into one grand model like humans any time soon. And frankly that is probably a good thing. I don't think we are ready for general intelligence.