Making video ads with AI, and catching the bad ones
Generating a video with AI is easy now. Generating one that a stranger would watch past the first three seconds, and that says only true things about the product, is still hard. Most of the early videos this pipeline made were not good enough to run. This is what changed that.
One pipeline
Every ad takes the same route. The formats differ a lot on the surface, from a selfie-style testimonial to a staged debate or a retro newscast, but they share the route, so a fix made for one format reaches all of them.
| Stage | Model or tool |
|---|---|
| Script, diagnosis | Claude, through the Anthropic API |
| Stills and character sheets | Nano Banana (Gemini image) |
| Talking takes with native speech | Veo 3.1 |
| Action and b-roll shots | Seedance |
| Voice and music | ElevenLabs |
| Captions, cuts, audio levels | ffmpeg, driven from code |
Scripts from a framework
My first approach to scripts was to have a model write a draft and another model judge it, in a loop, until the judge was satisfied. It was slow, and the scripts converged on the same safe, templated voice.
The version that works goes the other way. The script is built in steps, and the last step is a director's script in three columns, one row per shot:
| Say | See | Knows |
|---|---|---|
| The exact spoken line | What is on screen while it is said | What the viewer understands by the end of the shot |
The third column is the one that matters. If a shot does not change what the viewer knows, it is cut. After that come checks written in code and, at most, one rewrite.
- A list of banned phrases, the ones that make a script sound machine-written.
- Lines must be full spoken sentences. No clipped fragments.
- A line can only play over a shot that matches it.
- Every ad has to show the problem, the product being used, and the result.
- The finished voice track is transcribed and compared with the script word for word.
Three tries, then a person
Video models fail in ways that are obvious to a person and invisible to a retry loop: a hand with the wrong number of fingers, a word that comes out garbled, a product used in a way that is physically impossible. Left alone, a pipeline will keep paying to regenerate the same mistake.
So each ad has a hard limit. This is the function that decides what happens after an attempt:
export const MAX_ATTEMPTS = 3;
export function nextAfterAttempt(s: Attempt): Next {
// mistakes I have repeatedly shipped anyway are noted, not counted
const counted = s.misses.filter((m) => !s.lenient.has(m.check));
if (!counted.length) return { action: "pass", notes: s.misses };
const again = counted.find((m) => s.priorChecks.includes(m.check));
if (again) return { action: "deliver", why: `the same mistake twice (${again.check})` };
if (s.attempt >= MAX_ATTEMPTS) return { action: "deliver", why: `${MAX_ATTEMPTS} tries` };
if (s.spentUsd >= s.capUsd) return { action: "deliver", why: `its $${s.capUsd} budget is spent` };
return { action: "retry", why: counted.map((m) => m.check).join(", ") };
}
"Deliver" means it comes back to me with the specific mistake named. Three things in that function took a while to get right:
- The same mistake twice ends it early. If the second attempt fails the way the first did, a third will too.
- It learns my calls. My verdicts are stored as rows. A mistake I keep approving anyway stops costing retries; one I reject stays strict.
- Some errors never retry. A provider's content refusal comes back identical however often it is asked, so it stops at once.
When the same mistake shows up on three different ads, the system stops treating it as bad luck. It proposes a fix to the rule that caused it, I approve or decline it, and an approved fix is applied to every script written afterwards.
What I took from it
Asking a second model whether the first one did a good job did not raise quality much. What raised it was writing down what a good result has to contain, checking that directly, and putting a ceiling on how much money a bad idea is allowed to spend.