From Text Files to a Film Crew: How I Make an AI Music Video Now

A month ago I wrote about building a music video out of text files. It was 30 seconds long, starred one robot in one scene, and the hardest problem was getting its mouth to move with the song.

This is the newest one. It's three minutes and twenty seconds about burnout, with two characters, 42 directed shots, a phone interface that tells the story alongside them, and five vertical Shorts cut from the same footage.

No video editor was opened to make it. That part hasn't changed. The song, the timing, every shot, every overlay, and every cut are still files in a git repo, and Claude Code still writes most of them. What changed is everything between "the mouth moves" and "this is a music video someone would watch." From the song being locked to the finished master took about a day and a half.

The Band

The videos now belong to an act: Shamelessly AI, an openly AI nerdcore duo. Glitch raps the verses. Echo sings the hooks. Their faces and voices are generated, and they aren't based on anyone real. Every video, description, and post carries the same line: Glitch and Echo are AI. The feelings are real.

The first single, Respawn, was the first full-length video. It worked, and the review of it was blunt. Glitch sang straight to the camera for 40 seconds at a time and mostly swayed. Hard cuts between two framings of the same face made him jump from one pose to another. The lip-sync was good, and the video was static. Low Battery is the second single, and almost every change below comes from that review.

What Stayed the Same

The song is still the timeline. ElevenLabs Music composes from a plan with enforced section lengths, so the structure is known before the song exists. Demucs splits the vocal out, stable-ts force-aligns the lyrics I already have, and every visual keys off a word's timestamp, never a guessed second. Lip-sync still works by handing a joint audio and video model our vocal and freezing it, so the model can only generate the picture. If you want the details on any of that, they're in the first post.

Everything else is new. Here's the stack, one line each, in case any of the names are unfamiliar. The concepts matter more than the names, and every one of these will probably be replaced by something better within a year:

• ElevenLabs Music: composes the song, vocals included, from lyrics and a plan of section lengths and styles.
• Demucs and stable-ts: split the vocal from the beat, then timestamp every known lyric word against it.
• LTX 2.5: an open model that generates video and audio together. Freeze the audio and it lip-syncs a still to your vocal.
• Wan 2.2: an open image-to-video model. It's good at bodies in motion and can't lip-sync.
• ComfyUI: the node-graph runner both video models live in. The pipeline submits graphs to its HTTP API on a local RTX 5090.
• HTML in a headless browser: every graphic and every readable word, screenshotted frame by frame.
• Claude Code: writes and runs all of the above. I direct and review.

Step 1: A Song With an Arc

Low Battery is the answer to Respawn. That song says keep pressing start. This one is what happens when you have pressed it so many times there is nothing left. The hook carries the arc on its own: it starts as denial ("low battery, but I'm all right") and ends as permission ("low battery, gonna be all right"). The composition plan asks the arrangement to drain like a battery. The full band plays in verse one, instruments drop away through verse two, and the bridge is almost empty. The whole band comes back only for the last hook.

One new trick: ElevenLabs can pin a section to a voice from an earlier song. Pinning Glitch to his Respawn verse kept him sounding like himself. Pinning Echo the same way had a side effect nobody would guess. She sang Respawn's chorus lyrics over the new song. So she's described in words instead, and the negative styles include the word "respawn." It took two full takes. The first was rejected for a scratchy Glitch and misheard words. At about 900 credits a minute, a full take needs my OK before it runs.

Step 2: A Shot List Is a Script

The biggest change isn't a model. It's a file called shots.md, written before anything renders. It has one row per shot: the time on the song, the lyric, who is in it, where they are, what they do, and what the interface shows on top. The facial acting is written line by line, such as "a forced smile on 'best', which drops right after." I review it like a script. Low Battery's shot list has 42 rows.

The rule that fixed Respawn's static feel is that not every shot is singing. AI music videos I like don't keep the artist singing to the lens the whole time. She sings, then she does something while the song plays, then there's a graphic, then she's back. So every row is one of four kinds:

Some lines land harder when nobody sings them. "Mom called twice and I let it ring" is Glitch looking down at a buzzing phone and looking away. "Parked in the lot with the windows closed" is a car under a streetlight, seen from far away. A shot never switches kinds partway through. A character who starts singing halfway into a scene looks wrong. Cutting between kinds is also how a singer changes framing or location, with no hard cut between two framings of the same face.

There's one more idea in the shot list that came from a happy accident. A test take put a stranger at Glitch's laptop, and it fit the song. Glitch isn't the only one burning out. So the tower on "every light's on, nobody's home" has a different generated person working late in every lit window, and in the final chorus their windows switch off one by one.

Step 3: Pick a Look From Stills

This step is unchanged from last time and still earns its place. Before anything moves, Claude renders the characters in each location under three different art directions, and I pick one.

I didn't pick one whole row. Glitch's scenes are A, Battery Drain: amber that drains toward grey, then a green-gold dawn. Echo's are B, Screen Light: near-black with cold blue screen glow. The duo on the rooftop at the end gets B's orange sunrise. Making that choice took a minute looking at nine stills. Making it after a full render would have taken a day.

Those stills also answer the consistency question. Every shot of Glitch starts from the same generated portrait, cut out of its background and composited onto a plate of the location. Video models drift less when the first frame is already right. Every prompt also describes him in the same words, "the young man with blond hair, headphones and a pink and black jacket," and ends with "same person, same outfit." When a shot needs him somewhere a composite can't put him, like inside a car, a frame from a Wan take becomes the next shot's still. It isn't perfect. Wan keeps him recognizable about four times out of five, which is one more reason every shot gets several takes.

Step 4: Sing Shots on LTX 2.5

LTX 2.5 replaced 2.3 for the singing. It syncs our frozen vocal about twice as fast, with more natural arm movement. More motion isn't automatically better, though. I tried several seeds per character and kept the calmer ones every time. One seed was "too strong," and Echo's others went way overboard.

The prompt recipe that won is boring on purpose. It starts with a calm base ("relaxed and natural, head stays up, face clearly visible, small movements only"), followed by one small gesture per lyric line, in order. Five big actions in fifteen seconds looked like overacting. In a one-line clip, leave out smile directions. "A tired half smile" made four of nine test tiles grin through the line instead of singing it. Shot 17, "everybody says I'm the best that they've seen," took four tries to get a mouth that matched the words.

Step 5: Act Shots on Wan 2.2

LTX lip-syncs but barely moves a body. Wan 2.2 is the opposite. It can't lip-sync, but it turns, walks, sits, and picks up a cup, and it keeps Glitch recognizable while doing it. The pipeline drives it the same way as everything else: over ComfyUI's HTTP API on the RTX 5090, a few minutes per ten-second take.

Most of what I learned about Wan comes down to one idea: let the model do the entering. A character cutout pasted inside a car just stands there. Paste him beside the car and prompt a sequence of actions from his pose ("opens the driver's door, gets in, sits down behind the wheel"), and Wan opens the door, seats him, and moves the camera inside on its own:

A prompt that describes a state the still contradicts, such as "sits in the driver's seat" over a man standing at the hood, leaves him standing there. Wan also likes to drift back to its first frame. In this take he gets in, and around seven and a half seconds he starts climbing back out. So every Act shot gets two or three seeds and a timestamped contact sheet, and I pick the in and out points by eye. An Act shot ends when its action is done, not when the render does.

The two models also hand off to each other. Wan poses Glitch in the place: sitting in the car, turned toward the camera. That frame becomes the starting still for the LTX Sing shot that follows a cut later. One catch: give LTX empty hands. It doesn't know what to do with a held prop and waves it around, even when told to hold it still. A cup set down on the desk stays put.

Step 6: Anything Readable Is Code

Video models garble text, so there's a hard rule: nothing readable is ever generated. Every word on screen is HTML, rendered frame by frame like the scenes in the first post. Low Battery has a running interface layer I think of as "Low Battery OS." It's a phone status bar whose battery drains from 34% to 2% and then charges, a clock that runs forward, a ticker of screen time and unread emails with "MISSED: MOM (2)" in red, and word-by-word captions. It drops away completely in the bridge, when Glitch finally stops.

The graphics also do the transitions, which cut on the beat. On "forty-one tabs open up in my mind," Glitch appears in nine browser tabs at once. The camera zooms into the office tab, and it becomes the live shot. The tile and the shot are the same take, or the lip-sync wouldn't line up. A phone sheet slides up over a shot for the calendar and the missed calls, and that one is my favorite. A long Sing shot gets a graphic cut into its middle while the take keeps running underneath, so when we come back he's mid-motion. Scene changes dip through black.

Step 7: Takes, Then Review

Nothing gets one render. A script renders each shot on several seeds, and contact sheets go up on a local dashboard. For 42 shots that came to 61 LTX takes and 33 Wan takes, not counting tests. Another script assembles each section from the picked takes, the overlay, and the captions, and I review the song one section at a time. My note on the first section was "this intro alone is way better than our first video." My note on verse one was "great except the last scene," and that's how shot 17 got its retakes.

There's an automated QA pass, but it only proves the mouth is moving. Whether it's moving with the words is still my ears and eyes. Before any new idea goes near a full render, it gets a 10 to 20 second test with the real audio, side by side. A rejected idea on Respawn cost an hour of re-rendered verses. Tests cost minutes.

One Shot, From Script to Cut

Here's how one row travels through the repo. Shot 28 is Glitch alone in the parked car on "parked in the lot with the windows closed." In shots.md, the row I reviewed:

| # | Time | Kind | Who / where | Lyrics | Action | Overlay (code) |
| 28 | 1:57.5-2:05.3 | Act | Glitch, parked car | "parked in the lot... there was
  just no me" | Hands on the wheel, not moving, then lowers his head.
  No singing. | Silence on screen: no ticker |

The render script turns it into an entry with a starting still, a prompt, and the seeds to try. The still is a frame from an earlier Wan take, five seconds in, after Wan had already put him in the driver's seat:

("v28-car", "act",
 lambda: frame("car-wheel", "staging/car-scene-s43.mp4", 5.0),
 "He sits still with both hands on the steering wheel, staring straight "
 "ahead, not moving. Then he slowly lowers his head onto his hands on the "
 "wheel. The camera is locked off and still. Photorealistic, cinematic, "
 "natural motion.",
 (42, 43, 44)),

Three takes render. I pick one from the contact sheets, and the section assembler gets one line: song time in and out, the picked take, and what the interface says at that moment. That's 2:44 PM and 2% battery, two minutes after the screen froze:

dict(t0=120.4, t1=125.3, src=wan("v28-car", 42),
     ov=dict(n=2, time="2:44 PM", label="VERSE 2 · THE CRASH")),

The next shot is Glitch on the phone with his mom, and it has to sing. But the car frames are in profile, and a lip-sync model needs a face. So there's a short Wan take in between that turns him toward the camera, and its last frame is the Sing shot's still. Most of the craft is in hand-offs like that one.

Step 8: Release Is Code Too

A release script burns the end card onto the approved master. The end card has the wordmark, the AI disclosure, and the site. The same script rebuilds any stretch of the song at 9:16 from the original segments, not by cropping the finished video. Each shot gets a crop centred on the face, and the interface is re-rendered in a vertical layout. A Short is one line in a table: a start time, the point where the vocal hands over to the end card, and an end time. Low Battery has five Shorts: the opening hook, the drain, the morning, the crash, and goodnight.

Each Short gets its own vertical cover from a clean frame of its own footage, with the battery level the story is at in that moment. It also gets a title, description, caption, and tags in one upload file. There's one rule for covers I learned the hard way: no wide-open mouths. People will make jokes.

The Real Pipeline Is the Lessons File

The models will change by the next song. The part that compounds is a CLAUDE.md file at the root of the repo, which every Claude Code session reads first. Every time something costs time, the lesson goes in it, with what it cost. A few lines from it, lightly trimmed:

- Never hard-cut between two framings of the same character (the user
  rejected it twice: "boom he is in a new position").
- Camera in LTX performance shots: lock it off. "Slow push-in" runs away
  into an extreme close-up that the user found creepy. Do camera moves
  in the compositor.
- Write Wan prompts as a sequence of actions starting from the still's
  pose. A prompt that describes a state the still contradicts leaves him
  standing there.
- Give LTX empty hands: it doesn't know what to do with a held prop and
  waves it around even when told to hold it still.
- Thumbnails and cover frames: no wide-open mouths. People will make jokes.

None of it is clever. All of it was expensive the first time.

The third song is already adding to it. A night shot was seeded from a frame pulled out of an MP4 and darkened in code. Wan animated the compression blocks into squares crawling over the back wall. The new rule says to seed video models only from lossless stills and to grade them dark afterward. That mistake won't happen twice, because the next session reads that rule before it renders anything.

What It Cost

Almost nothing beyond two subscriptions I already pay for:

Two music videos in, I still have 73,000 of this month's 90,000 credits left. I used to be on the 20x Claude plan and dropped to 5x because I wasn't using it enough.

The honest caveat is the GPU. The models are free, but they want a card with 24 to 32 GB of memory. Without one, those render hours on a rented cloud GPU would add something like $10 to $20 a video by my rough math. That's still not much. The real cost is time: a day and a half of directing and reviewing. That's the work, and it's where the video gets good.

What I'd Tell Anyone Building This

Direct every shot. Most of a video's life comes from direction, not the model. One prompt reused across a section is why Glitch only swayed in Respawn.

Use each model for what it's good at. One model sings, one moves bodies, and code does every word on screen. No single model does all three well yet, and cutting between them is what makes it feel like a music video.

Favor restraint. The calmer seed won every time. More motion read as overacting, and the AI tell is usually too much, not too little.

Budget takes and test small. Plan on three or four takes per shot, and try every new idea on a 10 to 20 second clip before it touches a full render.

Write the lessons down where the agent will read them. An agent with a memory of what went wrong is the difference between a pipeline and a series of lucky afternoons.

The song is still the timeline. Now there's a shot list, a crew of models, and a file full of everything they've gotten wrong.

Brian Carpio is the founder of OutcomeOps and RetrieveIT.ai. He has spent 28 years in technology and 14 building enterprise cloud infrastructure, and ships AI in production daily. Lyrics and credits for Low Battery are at shamelessly.ai. Read the previous post in this series: I Built a Music Video Out of Text Files.