Media tools

Video inspect

Probe, sample stills from, and optionally transcribe one Sume-hosted clip. Default clip inspection on dest and prod.

Note

Current SoT. For new clip inspection, use this surface and not video analyses.

Dest and prod: POST /v1/video-inspect (MCP video_inspect). Sume charges probe + stills in credits by their compute time. The default is mode: sync.
Not typed scenes. Inspect returns probe facts, stills, and optional STT. On dest, use video_analyze / video_segment for semantic questions / intervals, but only when those names are in tools_list.
Legacy POST /v1/video-analyses: dest returns 410 video_analysis_retired. On prod, it stays available until #5953 PR-C2.

Video inspect 1.0 reads one media.sume.com clip that the workspace already owns. It never re-encodes the source, and it never makes an MP4. The job ID is the resource ID (sume/video-inspect-1.0, type video_inspect).

text

POST /v1/video-inspect
GET  /v1/video-inspect/:id

Hosted MCP: video_inspect (packages/mcp-server/src/mcp.ts). A write must have idempotency_key (and mcp:write under OAuth). There is no GET MCP wrapper. When the submit returns 202, poll with jobs_wait and then jobs_result.

Create an inspect

You must send video_url (a media.sume.com artifact or asset of this workspace). The API does not fetch from the open internet. First, import the clip (POST /v1/media-imports). You must send Idempotency-Key.

Optional fields: frames, transcribe, and (only with transcribe: true) language_code, segmentation, duration_seconds. You can also send the usual mode / webhook_url / wait_timeout_seconds communication fields.

The default mode is sync. The handler waits a maximum of 30 seconds. If the inspect completes in that time, the handler returns 200 with the result. If not, it returns 202 with the queued job, and you poll that job.

bash

curl -X POST https://api.sume.com/v1/video-inspect \
  -H "Authorization: Bearer $SUME_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: video-inspect-001" \
  -d '{
    "video_url": "https://media.sume.com/artifacts/artf_demo/talk.mp4"
  }'

If the submit is successful, the API returns request_id = video_inspect_id (the job ID) and a video_inspect object. Read the object later:

bash

curl https://api.sume.com/v1/video-inspect/$REQUEST_ID \
  -H "Authorization: Bearer $SUME_API_KEY"

When the inspect is complete, the resource contains probe and frames[{t,url,width,height}] as durable media.sume.com image artifacts. If the request included a transcript, the resource also contains transcript (text, words[], optional sentence segments[], audio_url). It also contains warnings[].

Frames program

`frames`	Effect
not sent	8 mid-bin stills (1 fps if the clip is shorter than 8 s)
`false`	Only the probe, with no stills
`{ at: [seconds…] }`	Explicit timestamps, 1–24 values, each ≥ 0
`{ fps: n }`	Sample rate, `0 < n ≤ 2`, mid-bin, with a limit of 24

An object must contain only one of at[] or fps. Optional fields on that object: format jpeg (default) or png, and max_edge 64–2160 (default 768). For a first-frame restage, send the source edge.

seek on that object selects how Sume finds each still:

`seek`	Effect
`precise` (default)	Decodes to the accurate instant. Programs that you already use do not change.
`fast`	Moves each still to the keyframe at or before its instant, and does not do the decode. The still can be earlier by a maximum of one GOP (approximately 0–5 s on typical sources), but it is never later. The quality, the resolution, and the transcript do not change.

Use fast for a quick look at a clip. Use precise when the timestamp must be accurate (explicit at[] instants, the default 8-still midpoints). With fast, each grid has seek: "fast". sample_times are the instants that the tiles show. requested_times are the instants in the program.

Limits (from packages/api-contract/src/index.ts): source ≤ 1800 s, and 24 stills for each call.

For accurate frames at the source size at one t, use video frames, not this route.

Transcript (optional, billed)

transcribe: true runs Sume STT 1.0 on the audio of the clip. Each inspect reserves its compute ceiling. Sume charges the job in credits for its own compute time, never more than the hold it reserved. The transcript adds its per-minute credits to that reservation.

Credits scale with the audio minutes. The API rate card lists the current credits.
Without duration_seconds, Sume reserves 1 minute. Maximum hint: 600 s.
language_code (for example en or ko) is an STT hint. If you do not send it, STT uses auto-detect.
segmentation.mode: "sentence" also returns sentence segments[] with no gaps (in the shape of caption lines). You can also set silence_split_seconds in the range 0.2–3.
If you send language_code / segmentation / duration_seconds without transcribe: true, the API returns 400 video_inspect_transcribe_required.
A silent clip gives inspect_source_has_no_audio. First, examine probe.has_audio (a frames: false inspect is sufficient).

Refusals (stable codes)

Code	When
`ffmpeg_fields_rejected`	The client sent `vf` / `filter` / `ffmpeg` / `cmd` / `codec` / `crf` / related fields. The server compiles ffmpeg.
`video_inspect_frames_program_conflict`	The request has `frames.at[]` and `frames.fps` together.
`video_inspect_frames_program_required`	`frames` is an object that has no `at[]` and no `fps`.
`video_inspect_transcribe_required`	The request has STT-only fields without `transcribe: true`.
`source_not_found`	The `media.sume.com` URL is dead or is not from this workspace.
`frame_time_out_of_range`	An `at` value is outside `[0, duration)`. The error gives the duration.
`inspect_source_has_no_audio`	The request has `transcribe: true` for a clip with no audio track.

The API rejects off-host URLs (https://example.com/…) at admit. Import the clip first.

Not this surface

Need	Use
Typed `scenes[]` / TwelveLabs Pegasus	Legacy video analyses (prod until PR-C2, dest `410`)
Dest semantic Q&A / intervals	`video_analyze` / `video_segment`, when they are in the list on `mcp.dev.sume.com`
The accurate frame at `t`, at the source size	Video frames
Burn captions onto a public URL	Video captions
A new MP4 cut	Video trim
Audio track as wav / mp3	Audio detach
Pixel pass (dim / crop)	Video filter
Put several clips in a sequence	Timeline 1.0

Video analyses (legacy vana_ resource)
Video frames
Video trim
Audio detach
Video filter
Timeline 1.0
Video captions
Media inputs
Jobs and results
MCP tools and gates

Media tools

Video inspect

Probe, sample stills from, and optionally transcribe one Sume-hosted clip. Default clip inspection on dest and prod.

Note

Current SoT. For new clip inspection, use this surface and not video analyses.

Dest and prod: POST /v1/video-inspect (MCP video_inspect). Sume charges probe + stills in credits by their compute time. The default is mode: sync.
Not typed scenes. Inspect returns probe facts, stills, and optional STT. On dest, use video_analyze / video_segment for semantic questions / intervals, but only when those names are in tools_list.
Legacy POST /v1/video-analyses: dest returns 410 video_analysis_retired. On prod, it stays available until #5953 PR-C2.

text

POST /v1/video-inspect
GET  /v1/video-inspect/:id

Create an inspect

bash

curl -X POST https://api.sume.com/v1/video-inspect \
  -H "Authorization: Bearer $SUME_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: video-inspect-001" \
  -d '{
    "video_url": "https://media.sume.com/artifacts/artf_demo/talk.mp4"
  }'

If the submit is successful, the API returns request_id = video_inspect_id (the job ID) and a video_inspect object. Read the object later:

bash

curl https://api.sume.com/v1/video-inspect/$REQUEST_ID \
  -H "Authorization: Bearer $SUME_API_KEY"

Frames program

`frames`	Effect
not sent	8 mid-bin stills (1 fps if the clip is shorter than 8 s)
`false`	Only the probe, with no stills
`{ at: [seconds…] }`	Explicit timestamps, 1–24 values, each ≥ 0
`{ fps: n }`	Sample rate, `0 < n ≤ 2`, mid-bin, with a limit of 24

seek on that object selects how Sume finds each still:

`seek`	Effect
`precise` (default)	Decodes to the accurate instant. Programs that you already use do not change.
`fast`	Moves each still to the keyframe at or before its instant, and does not do the decode. The still can be earlier by a maximum of one GOP (approximately 0–5 s on typical sources), but it is never later. The quality, the resolution, and the transcript do not change.

Limits (from packages/api-contract/src/index.ts): source ≤ 1800 s, and 24 stills for each call.

For accurate frames at the source size at one t, use video frames, not this route.

Transcript (optional, billed)

Credits scale with the audio minutes. The API rate card lists the current credits.
Without duration_seconds, Sume reserves 1 minute. Maximum hint: 600 s.
language_code (for example en or ko) is an STT hint. If you do not send it, STT uses auto-detect.
segmentation.mode: "sentence" also returns sentence segments[] with no gaps (in the shape of caption lines). You can also set silence_split_seconds in the range 0.2–3.
If you send language_code / segmentation / duration_seconds without transcribe: true, the API returns 400 video_inspect_transcribe_required.
A silent clip gives inspect_source_has_no_audio. First, examine probe.has_audio (a frames: false inspect is sufficient).

Refusals (stable codes)

Code	When
`ffmpeg_fields_rejected`	The client sent `vf` / `filter` / `ffmpeg` / `cmd` / `codec` / `crf` / related fields. The server compiles ffmpeg.
`video_inspect_frames_program_conflict`	The request has `frames.at[]` and `frames.fps` together.
`video_inspect_frames_program_required`	`frames` is an object that has no `at[]` and no `fps`.
`video_inspect_transcribe_required`	The request has STT-only fields without `transcribe: true`.
`source_not_found`	The `media.sume.com` URL is dead or is not from this workspace.
`frame_time_out_of_range`	An `at` value is outside `[0, duration)`. The error gives the duration.
`inspect_source_has_no_audio`	The request has `transcribe: true` for a clip with no audio track.

The API rejects off-host URLs (https://example.com/…) at admit. Import the clip first.

Not this surface

Need	Use
Typed `scenes[]` / TwelveLabs Pegasus	Legacy video analyses (prod until PR-C2, dest `410`)
Dest semantic Q&A / intervals	`video_analyze` / `video_segment`, when they are in the list on `mcp.dev.sume.com`
The accurate frame at `t`, at the source size	Video frames
Burn captions onto a public URL	Video captions
A new MP4 cut	Video trim
Audio track as wav / mp3	Audio detach
Pixel pass (dim / crop)	Video filter
Put several clips in a sequence	Timeline 1.0

Video analyses (legacy vana_ resource)
Video frames
Video trim
Audio detach
Video filter
Timeline 1.0
Video captions
Media inputs
Jobs and results
MCP tools and gates