Reference video shot breakdown API: cuts, keyframes and timings

To break a reference video into shots with an API, send the clip to POST /v1/reference-ingest. Sume answers with a manifest whose shots[] list holds each cut's start and end, one sharp keyframe per shot, a palette, a brightness reading and a motion class, and the shots tile the clip from 0 to its duration with no gap.

Read this first: reference ingest is listed only where the SUME_COM_REFERENCE_INGEST_ENABLED flag allows it. The Reference ingest page, read 2026-09-29, says it is automatically on in development and opt-in in production, so check GET /v1/catalog or your tool list before you build on it.

What does one call give me?

The whole preflight comes back in one manifest, called a ReferenceVideoManifest. Shot facts are the measurement; the strip is only orientation.

How do I call it?

The clip must already be a media.sume.com file your workspace owns, and Idempotency-Key is required. The default mode is sync: the call waits up to 30 seconds and answers 200 with the manifest, otherwise 202 with a queued job to poll.

curl -X POST https://api.sume.com/v1/reference-ingest \
  -H "Authorization: Bearer $SUME_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: ref-shots-001" \
  -d '{
    "video_url": "https://media.sume.com/artifacts/artf_demo/reference.mp4",
    "purpose": "reference_remix"
  }'

What are the limits?

• One clip of at most 300 seconds. Longer clips fail with source_too_long_for_reference_ingest; the docs point to video inspect instead.
• The file must have a video stream, or the call fails with source_no_video_stream.
• There is no vf, filtergraph, codec or argv field. Sending one returns 400 ffmpeg_fields_rejected.
• purpose (reference_remix, brief_format, face_swap or qa) is stored, not interpreted.

Does it cost anything?

The manifest itself is unbilled: it is CPU work on the media runtime, like frame extraction. Only the optional transcript is billed, and only if you set speech.allow_billed_stt and the track has speech. To turn a shot list into new footage, hand the timings to a generation call; reference to video covers that step.

What do I do with the shots afterwards?

Use the timings to cut or sample the source. POST /v1/video-trim cuts a [start, end) range for $0.02 per job, and POST /v1/video-frames pulls up to 24 stills at chosen times and is unbilled, per the Video trim and Video frames docs. Then write your own version of each shot and generate it. Through hosted MCP the same call is the reference_ingest tool, which returns text; the Sume Agent host's variant also attaches the strip and up to four low-confidence crops as images.