Originally published at ai.bedvibe.studio.

I built a text-to-speech product and kept getting burned by the same thing. On normal sentences the model sounded great. Then it would hit a number, a date, an acronym or a name, and quietly mangle it.

Worse, the metric everyone reaches for — Word Error Rate — was lying to me in both directions. It flagged perfectly good audio as broken because the script said 3:30 PM and the transcript said "three thirty pee em." And it missed real failures on short tokens, where the speech recogniser is as unreliable as the TTS.

So I wrote the QA framework I wished I had, packaged it as ttsproof, and then ran it as a blind study against a production TTS service so the results would be more than an opinion.

The two failures WER cannot see

A TTS pipeline breaks in two different ways, and a single WER number blurs both.

Structural defects. The clip is empty, truncated, three times too long, stuck in a repeated-chunk loop, clipping, or has a click at the tail. These have nothing to do with pronunciation — you can catch them with no model at all, straight from the waveform.

Pronunciation and content errors on the hard cases: numbers, decimals, dates, clock times, acronyms, single letters, URLs, names.

ttsproof splits them apart and handles each one honestly:

• Structural checks, no model needed — empty or truncated audio, duration explosions, long internal silences, clipping, loop detection, end-of-clip artifacts. numpy and soundfile, nothing else.
• Equivalence-aware WER/CER — the expected text and the ASR transcript are both canonicalised to spoken form before scoring, so 3:30 PM against "three thirty" stops counting as an error.
• ASR-uncertainty quarantine — when the audio is structurally clean but the recogniser disagrees on a very short utterance, the sample is set aside for a human instead of being auto-failed. At that length the ASR is as likely to be wrong as the TTS.

The study: 390 samples, and a blind human check

I evaluated the method against a production neural TTS service — 130 edge cases × 3 voices = 390 samples — and published it as a citable technical report (DOI 10.5281/zenodo.20757553, CC-BY-4.0).

• Zero structural audio-integrity defects across all 390 clips. That matters: it means every failure that did exist was pronunciation, exactly the kind WER mislabels.
• Exact-match rate 0.769.
• Then the honest part — a blind human review of the ASR-uncertain quarantine zone. Of 42 uncertain clips, 23 (55%) were ASR false negatives (the TTS said it right and the recogniser misheard) and 19 (45%) were genuine TTS mispronunciations. Fifteen control clips came back 15/15 correct, so the rater was reliable.

That 45/55 split is the entire argument for having a quarantine verdict at all. Auto-passing that zone ships 19 real mispronunciations. Naive ASR-WER auto-failing it wrongly kills 23 correct clips. Neither is acceptable, so ttsproof refuses to guess there.

What the real failures looked like

All 19 genuine failures were short isolated letters and acronyms, and the pattern is oddly specific:

The structural detectors did not fire on any of these. "GPUB" is intelligible speech, not a click. Structural checks and ASR quarantine are complementary; neither alone catches everything.

Benchmark any engine in one command

Beyond the study, ttsproof ships a corpus of 817 curated edge cases across 39 categories — numbers, currencies, dates, ISO timestamps, phone numbers, URLs, file paths, pronunciation-torture words (Worcestershire, synecdoche), proper names (Reykjavík, Nguyễn), Greek, Norwegian and more. The corpus is versioned independently of the software, so published scores stay comparable across tool updates.

It is engine-agnostic — point it at any TTS via a command template, or at a folder of audio you already generated:

ttsproof benchmark --cmd "mytts --text {text} --wav {out}"

You get a category scoreboard, a self-contained report.html with waveforms, an audio player and what the ASR actually heard, and a CI regression gate. Closed-source engines work too through a SpeechSDK wrapper — an integration a user suggested after the first release.

Try it

pip install ttsproof            # structural checks + metrics + corpus
pip install "ttsproof[asr]"     # + faster-whisper for pronunciation gating

• Repo: github.com/Mormolykos/ttsproof (MIT)
• The study: doi.org/10.5281/zenodo.20757553

It has already had its first outside contribution, a community fix for a real number-formatting bug, which is exactly what I hoped for. If your TTS breaks on something, open an issue with the case — the corpus grows from real failures.

The text-to-speech platform this came out of is live at tts.bedvibe.studio — ttsproof exists because I needed to QA that, and the model's engineering write-up covers what it is built on.