{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Panagiotis Gkilis",
  "title_listed": "Your TTS Model Sounds Great — Until It Says \"GPUB\"",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/bedvibe_studios/your-tts-model-sounds-great-until-it-says-gpub-1d77",
  "publish_date": "2026-08-25",
  "corpus_stage": "",
  "notes": "",
  "id": "f55521e0d24f",
  "collected_at": "2026-10-02T13:52:18",
  "title": "Your TTS Model Sounds Great — Until It Says \"GPUB\" - DEV Community",
  "body_markdown": "Originally published at\nai.bedvibe.studio\n.\nI built a text-to-speech product and kept getting burned by the same thing. On normal sentences the model sounded great. Then it would hit a number, a date, an acronym or a name, and quietly mangle it.\nWorse, the metric everyone reaches for — Word Error Rate — was lying to me in\nboth\ndirections. It flagged perfectly good audio as broken because the script said\n3:30 PM\nand the transcript said \"three thirty pee em.\" And it\nmissed\nreal failures on short tokens, where the speech recogniser is as unreliable as the TTS.\nSo I wrote the QA framework I wished I had, packaged it as\nttsproof\n, and then ran it as a blind study against a production TTS service so the results would be more than an opinion.\nThe two failures WER cannot see\nA TTS pipeline breaks in two different ways, and a single WER number blurs both.\nStructural defects.\nThe clip is empty, truncated, three times too long, stuck in a repeated-chunk loop, clipping, or has a click at the tail. These have nothing to do with pronunciation — you can catch them with no model at all, straight from the waveform.\nPronunciation and content errors\non the hard cases: numbers, decimals, dates, clock times, acronyms, single letters, URLs, names.\nttsproof splits them apart and handles each one honestly:\nStructural checks, no model needed\n— empty or truncated audio, duration explosions, long internal silences, clipping, loop detection, end-of-clip artifacts. numpy and soundfile, nothing else.\nEquivalence-aware WER/CER\n— the expected text and the ASR transcript are\nboth\ncanonicalised to spoken form before scoring, so\n3:30 PM\nagainst \"three thirty\" stops counting as an error.\nASR-uncertainty quarantine\n— when the audio is structurally clean but the recogniser disagrees on a very short utterance, the sample is set aside for a human instead of being auto-failed. At that length the ASR is as likely to be wrong as the TTS.\nThe study: 390 samples, and a blind human check\nI evaluated the method against a production neural TTS service —\n130 edge cases × 3 voices = 390 samples\n— and published it as a citable technical report (\nDOI 10.5281/zenodo.20757553\n, CC-BY-4.0).\nZero structural audio-integrity defects\nacross all 390 clips. That matters: it means every failure that did exist was pronunciation, exactly the kind WER mislabels.\nExact-match rate\n0.769\n.\nThen the honest part — a\nblind human review\nof the ASR-uncertain quarantine zone. Of 42 uncertain clips,\n23 (55%) were ASR false negatives\n(the TTS said it right and the recogniser misheard) and\n19 (45%) were genuine TTS mispronunciations.\nFifteen control clips came back 15/15 correct, so the rater was reliable.\nThat 45/55 split is the entire argument for having a quarantine verdict at all. Auto-passing that zone ships 19 real mispronunciations. Naive ASR-WER auto-failing it wrongly kills 23 correct clips. Neither is acceptable, so ttsproof refuses to guess there.\nWhat the real failures looked like\nAll 19 genuine failures were short isolated letters and acronyms, and the pattern is oddly specific:\nFailure mode\nExamples\nA-vowel substitution\nNATO\n→ \"NITO\",\nUSA\n→ \"USI\",\nCIA\n→ \"CII\"\nTrailing appended phoneme\nGPU\n→ \"GPUB\",\nEU\n→ \"EUU\"\nEarly truncation\nR\nchopped short\nDoubling\nX\nsaid twice\nOther substitution\nCEO\n→ \"CEE\",\nZ\n→ \"SZ\"\nThe structural detectors did\nnot\nfire on any of these. \"GPUB\" is intelligible speech, not a click. Structural checks and ASR quarantine are complementary; neither alone catches everything.\nBenchmark any engine in one command\nBeyond the study, ttsproof ships a corpus of\n817 curated edge cases across 39 categories\n— numbers, currencies, dates, ISO timestamps, phone numbers, URLs, file paths, pronunciation-torture words (Worcestershire, synecdoche), proper names (Reykjavík, Nguyễn), Greek, Norwegian and more. The corpus is versioned independently of the software, so published scores stay comparable across tool updates.\nIt is engine-agnostic — point it at any TTS via a command template, or at a folder of audio you already generated:\nttsproof benchmark\n--cmd\n\"mytts --text {text} --wav {out}\"\nEnter fullscreen mode\nExit fullscreen mode\nYou get a category scoreboard, a self-contained\nreport.html\nwith waveforms, an audio player and what the ASR actually heard, and a CI regression gate. Closed-source engines work too through a SpeechSDK wrapper — an integration a user suggested after the first release.\nTry it\npip\ninstall\nttsproof\n# structural checks + metrics + corpus\npip\ninstall\n\"ttsproof[asr]\"\n# + faster-whisper for pronunciation gating\nEnter fullscreen mode\nExit fullscreen mode\nRepo:\ngithub.com/Mormolykos/ttsproof\n(MIT)\nThe study:\ndoi.org/10.5281/zenodo.20757553\nIt has already had its first outside contribution, a community fix for a real number-formatting bug, which is exactly what I hoped for. If your TTS breaks on something, open an issue with the case — the corpus grows from real failures.\nThe text-to-speech platform this came out of is live at\ntts.bedvibe.studio\n— ttsproof exists because I needed to QA that, and the\nmodel's engineering write-up\ncovers what it is built on.",
  "metadata": {
    "title": "Your TTS Model Sounds Great — Until It Says \"GPUB\" - DEV Community",
    "description": "Word Error Rate lies in both directions on text-to-speech. A 390-sample blind study: 55% of the uncertain clips were the recogniser's fault, 45% were real mispronunciations. Tagged with python, machinelearning, testing, ai.",
    "keywords": "python, machinelearning, testing, ai, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/bedvibe_studios/your-tts-model-sounds-great-until-it-says-gpub-1d77",
    "og:title": "Your TTS Model Sounds Great — Until It Says \"GPUB\"",
    "og:description": "Word Error Rate lies in both directions on text-to-speech. A 390-sample blind study: 55% of the uncertain clips were the recogniser's fault, 45% were real mispronunciations.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzlkxxelz29zc9zf5k7qg.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Your TTS Model Sounds Great — Until It Says \"GPUB\"",
    "twitter:description": "Word Error Rate lies in both directions on text-to-speech. A 390-sample blind study: 55% of the uncertain clips were the recogniser's fault, 45% were real mispronunciations.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzlkxxelz29zc9zf5k7qg.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/f55521e0d24f/code/body-000.txt",
      "chars": 59,
      "language": "text"
    },
    {
      "file": "attachments/f55521e0d24f/code/body-001.txt",
      "chars": 147,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://ai.bedvibe.studio/ttsproof/",
      "label": "ai.bedvibe.studio",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.5281/zenodo.20757553",
      "label": "DOI 10.5281/zenodo.20757553",
      "type": "a"
    },
    {
      "href": "https://github.com/Mormolykos/ttsproof",
      "label": "github.com/Mormolykos/ttsproof",
      "type": "a"
    },
    {
      "href": "https://tts.bedvibe.studio/",
      "label": "tts.bedvibe.studio",
      "type": "a"
    },
    {
      "href": "https://tts.bedvibe.studio/portfolio/bedvibe-tts/",
      "label": "model's engineering write-up",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:44Z",
  "source_url": "https://dev.to/bedvibe_studios/your-tts-model-sounds-great-until-it-says-gpub-1d77",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 5102,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/f55521e0d24f/pages/source.html",
  "local_html": "attachments/f55521e0d24f/pages/article.html",
  "body_base_path": "attachments/f55521e0d24f/pages/",
  "body_markdown_path": "attachments/f55521e0d24f/pages/article.md"
}