Text to speech with highlighted words: sync word timings

To highlight words while text to speech plays, get word timestamps together with the audio, then have your player code mark the word whose start and end times contain the audio's current playback time. With Sume, timestamps: { "words": true } on POST /v1/tts-1.0/generate returns words[] with start and end seconds for the file it generated.

Sume facts come from the TTS 1.0 and STT 1.0 schemas in the Sume API reference, the OpenAPI document behind the API reference docs, read on 2026-09-27. Browser behavior comes from MDN's pages on `timeupdate`, `currentTime`, and `requestAnimationFrame()`. Sume returns the audio and the timings; the highlighting runs in your own page code.

How do I get word timings with the audio?

Add timestamps to the TTS request; the full request is in Text to speech API. The completed job result then carries monotonic words[] with start and end seconds. Make the call from your server, which keeps the API key, and pass the page only the file URL and the timings.

• In the current code each entry is { word, start, end }, in seconds from the start of the file, and the result also holds audio_url and the file's duration_seconds, as in the excerpt below.
• Add segmentation: { "mode": "sentence" } to also get gapless segments[], one per sentence, for highlighting the current line.

{
  "audio_url": "https://media.sume.com/artifacts/artf_demo/tts.mp3",
  "duration_seconds": 1.52,
  "words": [
    { "word": "Read", "start": 0.08, "end": 0.34 },
    { "word": "along", "start": 0.34, "end": 0.71 },
    { "word": "with", "start": 0.71, "end": 0.9 },
    { "word": "me.", "start": 0.9, "end": 1.35 }
  ]
}

How do I highlight the current word in the browser?

Render one <span> per entry in words[]. On every tick, read the player's currentTime, the current playback time in seconds, and mark the last word whose start has passed. The sample joins tokens with spaces; adjust that for languages written without them.

• timeupdate is the simplest tick, but MDN says it fires between about 4 Hz and 66 Hz depending on system load, so at the slow end a short word can pass between two ticks.
• A requestAnimationFrame() loop runs about once per display refresh, most commonly 60 times a second, and most browsers pause it in background tabs.
• Setting currentTime seeks, so a click on a word can jump the audio to that word's start.

// words: result.words, passed down by your server
const audio = document.querySelector("audio");
const box = document.querySelector("#transcript");
const spans = words.map((w) => {
  const span = document.createElement("span");
  span.textContent = w.word + " ";
  span.onclick = () => { audio.currentTime = w.start; };
  box.append(span);
  return span;
});

let active = -1, frame = 0;
function highlight() {
  const t = audio.currentTime;
  let i = active > -1 && t >= words[active].start ? active : -1;
  while (i + 1 < words.length && words[i + 1].start <= t) i++;
  if (i !== active) {
    spans[active]?.classList.remove("current");
    spans[i]?.classList.add("current");
    active = i;
  }
}
function loop() { highlight(); frame = audio.paused ? 0 : requestAnimationFrame(loop); }
audio.addEventListener("play", () => { if (!frame) frame = requestAnimationFrame(loop); });
audio.addEventListener("seeked", highlight);

What if the words don't match my text exactly?

Build the spans from words[] itself rather than by splitting your original string, so the list and the timings always line up. If you must keep your own markup, walk both lists in order and pair the tokens one by one instead of looking words up by text, since the same word can appear many times.

The timings belong to the generated file. If you edit the audio afterwards, for example by adding an intro, shift every start and end by the same offset.

Can I highlight narration I didn't generate?

Yes, with speech-to-text. POST /v1/stt-1.0/transcribe takes a public HTTPS audio_url and always returns word timings, so there is no flag to set. Each entry has word, start, and end in seconds, plus a type such as word or spacing when one is supplied; skip the spacing entries. Speech-to-text API with word timestamps covers the call.

What does it cost?

Each job bills at its published rate, plus a 5.5% agent fee by default. TTS is priced per transcript character, and in the current code the estimate counts characters only, so asking for word timings adds nothing to it.