Image to video vs reference to video: what's the difference?

Image-to-video starts the clip from your picture: the image becomes its first frame, and on most models another image can set its last. Reference-to-video uses your images, clips, or audio only as guidance, so no frame of the output is pinned to them. Text-to-video starts from words alone, and video-to-video edits a clip you already have.

On Sume you don't name the mode: the fields you send to POST /v1/videos decide it. The field rules come from the Video generation docs and per-model support from the catalog behind GET /v1/videos/models, read on 2026-09-27.

How do the four modes differ?

Each mode starts from a different input, and not every model offers every mode:

What is image-to-video?

Image-to-video animates a still. Your image is the opening frame, and the model generates the motion after it from your prompt. Send it in frame_images with frame_type: "first_frame". A last_frame entry sets where the clip ends, and current code refuses a last frame sent without a first frame. Frame images must be public HTTPS URLs.

This is the docs' image-to-video example. The first and last frame rules per model are in Image-to-video API: first and last frame.

{
  "model": "seedance-2",
  "prompt": "A character walking through a forest",
  "frame_images": [
    {
      "type": "image_url",
      "image_url": { "url": "https://example.com/first-frame.png" },
      "frame_type": "first_frame"
    }
  ],
  "resolution": "1080p"
}

What is reference-to-video?

Reference-to-video treats your inputs as hints. In the docs' words, the model uses them "as visual guidance rather than exact frames", so the clip is not required to open on, or contain, your exact picture. input_references takes three types: image_url, video_url, and audio_url. A model accepts only the types its supported_input_references lists.

The docs' reference example sends one image. Counts and per-model rules are in Reference-to-video API.

{
  "model": "seedance-2",
  "prompt": "A colossal solar flare beside a planet",
  "input_references": [
    {
      "type": "image_url",
      "image_url": { "url": "https://example.com/style-ref.png" }
    }
  ],
  "resolution": "1080p"
}

What happens if I send a frame and references together?

The frame wins. When a request carries both frame_images and input_references, Sume treats it as image-to-video, and in current code the references are dropped rather than passed to the model. Choose one mode per request.

Which one should I use?

Start from what you have and what must stay fixed:

• No picture: text-to-video, from the prompt alone.
• The clip must open on your picture, such as a product shot or a photo you want to move: image-to-video. Add a last_frame to fix the ending as well. Keep a product exact in AI video covers packshots.
• You want the look of a character, product, or style from pictures, or want to guide the clip with a reference video or audio track, without fixing any frame: reference-to-video. Consistent characters across shots weighs a first frame against references.
• You have a clip and want to change something in it: video-to-video on gemini-omni-flash-1.1, covered in Edit a video with a prompt.