Can AI make a video from a story? Yes, shot by shot

Yes, AI can make a video from a story, but as a sequence of short shots rather than one long generation. Split the story into shots, make a still for each one and approve it, animate it into a clip of a few seconds, give the narration and dialogue voices with text to speech, then join everything in story order under one soundtrack.

On Sume you can hand the story to the agent in the Agents tab, or run each step yourself over the API. The steps below come from Sume's Quick start, Models, Video generation, and Timeline 1.0 docs and the Sume API reference, read on 2026-09-27. Anything described as current behavior is read from Sume's code.

How does a story become an AI video?

Start with a shot list: for each shot, what is on screen, who speaks, and for how long. One generated clip runs 2–30 seconds depending on the model, so a longer scene becomes several shots. Each kind of shot then has its own step:

Why make a still before each clip?

Sume's docs route wordless beats as image, then inspect, then video, and Sume's guidance for its own agent says to accept the still before any video spend. The still sets how the shot opens: its setting, characters, and framing. Send the approved still as the clip's first_frame in frame_images, and the clip opens on it; the prompt then describes the motion, camera, and light.

For a character who returns in many shots, send the same source image each time. Consistent character across AI video shots compares the inputs, and none is documented to guarantee an identical look. The Image API returns stills at signed URLs, so host the approved still at a public HTTPS URL before you send it as a frame.

How do characters talk in an AI story video?

Through a separate lip-sync step. Sume's docs say video models do not lip-sync to generated TTS or to a later voice-over, and they route every on-camera speaking shot to VEED Fabric 1.0: an accepted still plus TTS audio. Voice the line with TTS 1.0: a transcript of up to 20,000 characters, a voice from an avatar or a voice.id, and language set for any non-English line. Then send the still and the Sume-hosted audio to Fabric. Lip sync API covers that request.

How do I put the shots together?

Timeline 1.0 joins them in one render: one audio spine of 1–1,800 seconds under 1–200 clips in story order, with fades, wipes, slides, or dissolves of up to 1 second between them. The spine supplies the sound, and in current code a clip's own audio is not mixed in, so each spoken line must be on the spine too, at the time its speaking clip starts. audio.parts[] joins up to 20 slices of audio, such as narration passages and dialogue lines, into one gapless spine.

Every URL must be a Sume-hosted file; generated clips and TTS audio come back as Sume-hosted artifacts. How to assemble a long-form video covers the request, and Faceless video API shows narration over generated shots.

Can Sume's agent make the whole story video for me?

It can run the steps with you. In the Agents tab you describe the deliverable, not the tool calls: the agent picks the models and asks before it spends, and the thread keeps every artifact and approval. Sume's docs describe its agent composing B-roll, voiceover, and timeline assembly into a post-ready video that a single clip cannot deliver. From your own code, Agent Completions take the same brief as instruction, with a required generation_spend_cap_usd.

What are the limits?

• 2–30 seconds per generated clip, and up to 1,800 seconds of output per Timeline render.
• No input guarantees that a character looks the same in every shot, so review each shot before you join them.
• A talking face needs a still-plus-audio lip-sync step such as Fabric: a generated clip will not lip-sync to a voice added later.
• Timeline takes only Sume-hosted media, and Fabric only Sume-hosted audio.
• Every step is its own job, billed at its own rate on API pricing.