Storyboard to video AI: turn each frame into a shot

To turn a storyboard into a video with AI, treat each frame as one shot. Render a rough sketch into a finished still first, or use a finished frame as it is, then animate it as the first frame of a short clip and join the clips in storyboard order. The board's order becomes the edit, and each approved frame becomes the opening image of its shot.

The steps below follow Sume's Image API, Video generation, Models, and Timeline 1.0 docs, read on 2026-09-27.

Should I animate a rough storyboard sketch directly?

Only if the shot should look like the sketch. A first_frame in frame_images is the image the clip opens on, so a pencil frame opens the clip in pencil. For a finished look, render the sketch into a still first with image-to-image: send it to POST /v1/images as an image_url entry in input_references, with a prompt for the finished style.

The docs suggest aspect_ratio: "auto" on image-to-image calls so the result matches the reference, and models whose input_references descriptor is {"min": 0, "max": 0} reject references. Nothing promises that the sketch's composition survives, so compare each still with its board frame before you animate it. Image generation API with reference images covers the request.

How do I animate each storyboard frame?

Send the approved frame as the first_frame of a POST /v1/videos request, set aspect_ratio to the board's frame shape from the model's supported_aspect_ratios, and put the board's action and camera notes in the prompt. Sume's docs order wordless shots the same way: image, then inspect, then video. If the board also draws where a move ends, send that frame as the last_frame; a last frame needs a first frame, and the model must list both in supported_frame_images.

Frame URLs must be public HTTPS, and signed URLs are rejected. The Image API returns stills at signed URLs, so host each approved still at a public HTTPS URL first.

{
  "model": "seedance-2",
  "prompt": "Shot 4: the courier pushes the door open and steps into the rain. Camera tracks left with her, low angle, blue night light.",
  "frame_images": [
    {
      "type": "image_url",
      "image_url": { "url": "https://example.com/board/shot-04.png" },
      "frame_type": "first_frame"
    }
  ],
  "aspect_ratio": "16:9",
  "duration": 6
}

How do I keep the storyboard's order and timing?

Join the clips with Timeline 1.0, which renders one audio spine and 1–200 ordered clips into one MP4. Every clip must already be this workspace's media.sume.com file, which clips generated on Sume are; your own frame files cannot go in directly, and the public API documents no upload route. Its fields map onto a board:

Can Sume's agent take the whole storyboard?

Up to 30 frames at a time. Agent Completions accept up to 30 images as attachments that the agent can see and use, with your brief as instruction and a required generation_spend_cap_usd that caps generation spend. Send images to the agent API covers the attachment format.

What are the limits?

• One clip per frame, 2–30 seconds depending on the model.
• At most two frames per clip: a first frame and an optional last frame.
• No promise that a sketch's composition or a character's look carries through rendering and animation.
• Timeline takes only Sume-hosted media: up to 200 slots and 1,800 seconds per render, with output edges of 256–2160 pixels.