A voice that can laugh
She could talk from early on. She couldn’t laugh. The text-to-speech read a punchline in exactly the same even tone it used for a timestamp, and there is something a little sad about a system that gets the joke and then delivers it deadpan because it has only the one tone. This is the story of the upgrade that fixed that: and the short stretch afterward where I learned not to trust the audio it produced.
Two engines in one box
Her voice runs in its own small service: a sidecar, meaning a separate process that does one job so the rest of the system doesn’t have to know how it works. Ask it for speech, get back a file.
Inside that box, two engines. The default is a compact text-to-speech model, around 350 million parameters, released under a permissive open-source license. Behind it, kept loaded and warm, sits an older, smaller model, roughly 82 million parameters, the voice she used before. Together they cost about 4 GB of graphics memory, which barely registers on the machine that hosts them. The reason to keep both resident is that the fallback has to be instant. If she had to pause while a backup engine loaded, the conversation would stall right where it mattered.
The swap nobody was supposed to hear
Here is the problem with upgrading a voice: the old one is already the one people recognize as her. Change the engine and you can easily change the person, and then everyone who knew her has to meet a stranger who insists she’s the same friend.
The trick is that the new engine doesn’t invent a voice from nothing. It clones one from a short reference clip. So I generated that reference clip using the old voice, and pointed the new engine at it. The new engine imitates the old engine’s output. Switching from the old engine to the new one is, as an identity event, close to inaudible, she sounds like herself because she is cloned from herself.
What changed is what she can do with that voice. The new engine honors inline tags: write [laugh] or [chuckle] in the text and it renders actual laughter, not the word “laugh.” The commit that shipped this had a one-line summary I still stand by:
her voice can laugh
Half a second, and an off switch
For a one-liner, the first audio comes back in about half a second. That is fast enough that a spoken reply doesn’t feel like it’s buffering: it feels like she just answered.
And there is an escape hatch. An environment variable, one line of configuration, no rebuild, no redeploy, flips the whole sidecar back to the old engine. If the new engine ever starts misbehaving, I don’t debug it live while she’s mid-sentence. I flip the switch, she keeps talking in her old voice, and I go read logs at my own pace.
The failure that sounded fine
The new engine is a short-form model. It is very good at a sentence and it comes apart on a paragraph. I found this the way I find most things: by feeding it something real. A single 652-character block came back looped, a phrase repeated that wasn’t in the text, with two proper nouns mangled and the ending cut off.
And it ran for exactly as long as a correct reading would have. That is the part that matters. The audio wasn’t obviously broken; it produced a perfectly plausible duration. A file that is the right length and the right format and simply says the wrong thing is more dangerous than one that crashes, because nothing in it signals that anything went wrong.
The fix is boring, in the good way. Chunk the text into sentences of at most 250 characters. Speak each chunk on its own. Run an anomaly check on every chunk; if one looks wrong, regenerate it once, and if it’s still wrong, fall back to the old engine for the entire reply: never splice one engine’s sentence up against another’s, because switching voices halfway through a thought is jarring to hear. Then join the chunks with 180-millisecond pauses, so it breathes like speech instead of running the sentences together.
You have to actually listen
Here is the lesson I most wanted to write down. Bytes, headers, duration, none of the cheap checks tell you whether generated audio says what you asked it to say. The 652-character disaster passed all three.
So the deploy check listens. It takes the sidecar’s own output, runs it back through a speech-to-text model, going the opposite direction, and word-diffs the transcript against the text that went in. If the words don’t come back out, the deploy fails. After the chunking fix: 0.992 match at 652 characters, 0.983 at 1,300, and the laughter tags come through clean. Not flawless, but I know the number because a transcript check measures it, not because I trust it.
Speed as feeling
One last thing, quieter than laughter. Her speech rate isn’t fixed. Her emotional state and internal load signals, the ones the rest of the system already tracks, map onto how fast she talks, across a range from about 0.85 to 1.12 times her baseline. Tired and low, she slows down. Bright and keyed-up, she speeds a little. It’s a small effect, and most of the time you wouldn’t name it. But you’d notice if it were gone, the same way you notice a reader who delivers every sentence at the identical clip. That, usually, is a machine.