AI avatar language: which languages can an avatar speak?

Which languages an AI avatar speaks depends on how its speech is made: a lip-synced avatar speaks whatever language its audio track is in, while an avatar whose speech is generated with the video speaks only what that generator allows. On Sume, the Avatar 1.0 talking video speaks English only in current code; for Spanish, Hindi, Korean, or another language, you generate the speech with TTS 1.0 and a language code, then lip sync the avatar to it with VEED Fabric 1.0.

Facts come from Sume's Generate avatar video and Models overview docs and the TTS and Fabric schemas in the Sume API reference, read on 2026-09-27. The English-only rule, the Voices languages, and anything called current behavior are read from Sume's code.

Can a Sume avatar video speak Spanish or Korean?

Not the one-call talking video. In current code, the prompt behind each spoken Avatar 1.0 clip says "Spoken language: English only." and asks for the exact English dialogue, Sume's hosted MCP guidance says the script must be English, and each avatar's own voice is cloned as English. POST /v1/avatar-1.0/talking-video has no language field, so send it English scripts only.

How do I make an avatar speak another language?

Build the speech first, then put the face on it:

• Write the line in the target language. TTS speaks the transcript you send; translating is your step, as in Translate a video's voiceover.
• Speak it with POST /v1/tts-1.0/generate, a voice, and language set to the target code; left out, language defaults to English. The voice can be the avatar's own (avatar_handle) or one made for that language in Assets → Voices (voice.id).
• Lip sync the avatar with POST /v1/veed/fabric-1.0: its avatar_handle, the TTS audio_url, and the audio's duration_seconds. Fabric has no language field; it animates the avatar's identity still to the audio.

curl -X POST https://api.sume.com/v1/tts-1.0/generate \
  -H "Authorization: Bearer $SUME_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: avatar-es-line-001" \
  -d '{
    "transcript": "Hola, soy Ana. Hoy te muestro la nueva app en un minuto.",
    "avatar_handle": "product_host",
    "language": "es",
    "timestamps": { "words": true }
  }'

Which languages can I pick?

TTS 1.0 takes one BCP-47 / ISO-639 code per request, with ko, ja, and en as the documented examples; the schema publishes no full list, so synthesize one short line in a new language and listen before a long script. When you create a voice in Assets → Voices, the Sume app today has you tag it with one of 16 languages: English (en), Korean (ko), Japanese (ja), Chinese (zh), Spanish (es), French (fr), German (de), Portuguese (pt), Italian (it), Hindi (hi), Dutch (nl), Polish (pl), Russian (ru), Swedish (sv), Turkish (tr), and Tagalog (tl). The tag records the voice's language; it is not a promise about how each one sounds.

What happens if the voice and the language don't match?

In current code, TTS checks a Voices library voice's language tag against the requested language. If they differ, the submit stops with 409 tts_voice_language_mismatch, before any job or charge, until you resend the same request with confirm_language_mismatch: true. Translate a video's voiceover by API walks through that double-check.

What are the limits?

• One language per TTS request, so give a script that switches languages one request per language.
• TTS takes up to 20,000 characters per request. Fabric takes 1–300 seconds of Sume-hosted audio, at most 10 MB.
• Fabric starts from the avatar's identity still, so an Avatar 1.0 request's scene and product don't carry over.
• On-screen text is a separate caption job. POST /v1/video-captions takes a language hint; its documented styles draw in Latin or Hangul faces, and Korean needs a Hangul style (Korean captions).
• TTS costs $0.0475 per 1,000 characters and Fabric $0.1875 per audio second (720p), each plus a 5.5% agent fee by default.