Lip sync AI translate: how to dub an avatar video

AI lip sync translation, or dubbing, replaces a video's speech with a translated voice and moves the mouth to match the new words. Sume does it by regenerating rather than by editing the original footage: its lip sync animates a still image or an avatar to new audio. So you translate the script, speak it in the target language with TTS 1.0, and lip sync the same face to that speech with VEED Fabric 1.0.

Facts come from Sume's Models overview, Timeline 1.0, and Video frames docs and the TTS and lip sync schemas in the Sume API reference, read on 2026-09-27; anything called current behavior is read from Sume's code. To swap the voice without new lip movement, see Translate a video's voiceover by API.

Can Sume re-sync the lips in a video I already have?

No. Both lip sync endpoints, VEED Fabric 1.0 and MiniMax H3 Max Lip Sync, take one still (image_url) or a ready avatar plus Sume-hosted audio; neither takes a video. Avatar 1.0 can't re-render its talking video in the new language either, because its clip prompt allows English speech only in current code. The dubbed version is a new clip of the same face.

How do I dub an avatar video, step by step?

• Get the lines. If Sume made the video, you already have its script. For a clip already on media.sume.com, the voiceover guide shows how to transcribe it with timings.
• Translate them, yourself or with an agent run. TTS speaks the transcript you send.
• Speak the translation with POST /v1/tts-1.0/generate: a voice, language set to the target code, and timestamps.words: true, so that current code reports the audio's duration_seconds.
• Lip sync with POST /v1/veed/fabric-1.0: the same avatar_handle, the TTS audio_url, and that duration. One job takes 1–300 seconds of audio in a file of at most 10 MB, more than the 60-second cap on an Avatar 1.0 video.
• Past 300 seconds, lip sync the speech in parts and lay the clips over the full TTS track in one Timeline 1.0 render, as Lip sync AI for long videos shows.

curl -X POST https://api.sume.com/v1/veed/fabric-1.0 \
  -H "Authorization: Bearer $SUME_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: dub-es-lipsync-001" \
  -d '{
    "avatar_handle": "product_host",
    "audio_url": "https://media.sume.com/artifacts/example/script-es.mp3",
    "duration_seconds": 41.2,
    "resolution": "720p"
  }'

Which face should the dubbed clip start from?

Fabric takes exactly one visual source:

• avatar_handle or avatar_id: Sume resolves the avatar's identity still. Use it to dub a video you made with that avatar.
• image_url: any public HTTPS still, such as a frame of the original. For a clip on media.sume.com, POST /v1/video-frames returns durable stills at the times you name, unbilled.

What changes compared with the original?

• The motion. Fabric animates one still to the new audio, so the original's gestures, camera moves, and product shots are not kept.
• The timing. A translation can run longer or shorter than the original line, so time any cuts to the new TTS timings.
• The voice. Use the avatar's own voice or a Voices library voice made for the target language; in current code, each avatar's own voice is cloned as English. What languages can an AI avatar speak? covers the language rules.

What does a dubbed version cost?

Each step bills on its own, plus a 5.5% agent fee by default. Fabric counts audio seconds rounded up, so lip syncing 60 seconds of translated speech at 720p comes to $11.25 before the fee.