{ "source_file": "0930", "source_platform": "Dev.to", "author": "Panagiotis Gkilis", "title_listed": "Three of Twenty Decoders Actually Stream, and My Quality Metric Was Beaten by an Algorithm from 1984", "keywords": "", "domain": "大模型算法", "content_type": "工程实践", "is_paid_flag": "否", "url": "https://dev.to/bedvibe_studios/three-of-twenty-decoders-actually-stream-and-my-quality-metric-was-beaten-by-an-algorithm-from-1984-5e77", "publish_date": "2026-09-17", "corpus_stage": "", "notes": "", "id": "1b950b7ca435", "collected_at": "2026-10-02T13:52:13", "title": "Three of Twenty Decoders Actually Stream, and My Quality Metric Was Beaten by an Algorithm from 1984 - DEV Community", "body_markdown": "If you are building streaming text-to-speech, you have to pick a decoder — the thing that turns a representation into a waveform. I tried to pick one from the literature and could not, so I measured twenty of them under one frozen protocol on identical audio.\nThree things came out of it. Two are about the decoders. The most useful one is about my own measuring stick, which failed a test I had written in advance specifically to catch it.\nThe problem with the published record\nReal-time factors quoted without naming the hardware. \"Streaming\" claimed for architectures that need the whole utterance before they emit anything. Quality figures from different corpora, at different sample rates, through different measurement paths. Each number is probably fine on its own. None of them compose into a decision.\nSo: one GPU, one corpus, one protocol, twenty arms.\nRepresentation → waveform only.\nGenerator and autoregressive timing are a different question, and a decode-only number reported as end-to-end TTS performance is a failed report.\nOne framing decision changed the candidate set before any measurement happened. I nearly excluded several codecs for not being causal — then realised\nonly the decoder needs to be causal for a TTS build\n. The encoder runs at training time on complete utterances; at inference the generator emits tokens directly and no encoder sits in the serving path. NVIDIA's NanoCodec makes it concrete: non-causal encoder, causal HiFi-GAN decoder. \"Is this codec causal?\" is the wrong question.\nFive gates, each frozen before it ran\ngate\nquestion\n0 — candidate\nwhich decoders are runnable, licensed and comparable at all?\n1 — functionality\ndoes it produce structurally valid audio?\n2 — streaming\ndoes it\ntruly\nstream, or is it stateless chunking?\n3 — performance\nlatency, real-time factor, first-audio, jitter, memory\n4 — quality\nwhat does the round trip cost the signal?\nQ5 — identity\ndoes the speaker survive, and do they stay one person?\nThe protocol for each gate was written and hashed before the data it governs existed. Two later freezes were not prospective — Gate 4's analysis layer and three revisions of the Q5 spec were post-measurement repairs, and the records say so rather than pretending otherwise. That distinction matters more than it sounds like it does, and I got it wrong in the first published version of this work.\nGate 2: \"streaming\" is a property almost nothing has\nAn arm counts as truly incremental only if\nboth\nhold:\nstateful chunked decode reproduces full-context output to\nmax|err| ≤ 1e-3\n,\nand\nstate is load-bearing — stateless error is at least 10× the stateful error.\nThe second condition exists because the first can be satisfied by an arm that ignores state completely. If you only check condition 1, a stateless chunker passes and you ship it believing it streams.\nOf twenty arms:\nthree\nare truly incremental, all three explicitly causal FocalCodec configurations.\nFifteen\nare stateless chunking. One streams its network but never emits audio from partial input. One was blocked on platform and reports no number at all rather than a fabricated one.\nThe three that pass do so with state ratios of\n3,530 to 36,740\nagainst a threshold of 10. State is not marginally load-bearing there; it is decisive.\nThe negative controls are what make that mean anything.\nThe non-causal FocalCodec configurations run through the\nidentical\nstateful code path. Their ratio is exactly\n1\n— carrying state changes nothing. If a negative control had passed, the experiment would have been void, not the arm promoted.\nSeveral of the fifteen never claimed to stream. The classification measures what they do; it is not an accusation about what they advertised.\nGate 3: speed is not validity\nFirst-audio latency (TTFA) across the arms that have one spans\n1.37 ms to 27.64 ms\nat the ~80 ms chunk anchor. MelFlow does not have one: it emits spectrogram frames, and producing playable PCM from a partial stream needs an overlap-add stage this study records as not established. Its 375.91 ms is a\nsteady-state median\n, a different quantity, reported separately and never mixed into the TTFA range. (I mixed them in v1.0.0. More on that below.)\nReal-time margin spans\n64×\ndown to\n0.2×\n.\nBut two arms produce streamed output that fails the study's own validity gate at\nevery\nchunk size tested. A third fails at the anchor and passes at larger sizes — its smallest viable configuration is\n640.8 ms\n, not 80 ms.\nThe sharpest case:\nvocos_mel24\nhas the\nfastest first-audio in the whole study and no achievable streaming configuration at all\n. Its timing numbers are real. What they timed is not a valid streaming configuration. If your benchmark reports latency without a validity gate, this is the arm that wins it.\nOne more result governs how any of these numbers may be read. A controlled probe — byte-identical tokens, same GPU, one arm run in every environment — found a\n50.2% difference attributable purely to the\ntransformers\nversion\n, device-side, at identical torch. So one arm in this study may not be compared numerically against the others at all. That is not a statement about whether it is fast. It is a statement that the comparison does not exist.\nGate 4: the metric failed the test I wrote to catch it\nThis is the part worth your time even if you never touch audio.\nGate 4 ranks on one metric, mel-cepstral distance, chosen after nine variants were implemented and attacked. Alongside it, the pre-registration installed a\nvalidity instrument\n: put Griffin-Lim in the panel. Griffin-Lim is a phase-retrieval algorithm from 1984 with\nzero trained parameters\n. The pre-registration stated in advance what it would mean if Griffin-Lim scored close to the trained neural codecs —\na finding about the metric, not about the decoders\n.\nIt did not score close. It scored\nbest, in six states of six\n.\nThe obvious explanation was tested and refuted. The hypothesis that the metric's low-level frame floor favours a magnitude-matching algorithm failed: splitting each cell at its own median energy, Griffin-Lim leads in\nboth\nhalves, and by\nmore\nin the loud half.\nWhat remains is an explanation consistent with the result rather than a demonstrated mechanism, and the difference is worth being careful about. Mel-cepstral distance here is derived from the magnitude spectrum and is blind to phase by construction. Griffin-Lim iterates toward consistency of the\nlinear STFT magnitude\nand optimises nothing else, accepting whatever phase error that leaves. Aligned in domain — but\nnot the same quantity\n, and nothing in this package demonstrates that Griffin-Lim minimises the released cepstral distance exactly.\nWhat\nis\nestablished does not depend on the mechanism:\na metric whose ranking puts an untrained phase-retrieval algorithm first is not, on its own, a sufficient authority for cross-decoder quality.\nThat is the whole lesson, and it generalises well past audio. The pre-registration naming this outcome in advance is the only reason it reads as a finding rather than an embarrassment — and the reason no metric was swapped in afterwards to produce a tidier table. If you are choosing anything on a published quality number, the question worth asking first is whether that metric has ever been shown a floor it should beat.\nQ5: does the speaker survive?\nA single similarity score conflates two different failures, so this measures them separately.\nRetention\n: how close a reconstruction sits to its own source.\nDispersion\n: how tightly\ndifferent\nreconstructions of the same speaker cluster. A decoder can move a voice consistently — a stable new identity — or make the speaker wander. Those are not the same problem and they do not have the same fix.\nThree separately calibrated speaker encoders, chosen mechanically from a previously frozen fourteen-encoder study before any of these numbers existed.\nCosine values are encoder-specific and never averaged.\nThe three encoders show broadly similar ordering with some pairwise reversals, which is exactly why a bare cosine without its encoder name is not a result.\nThe streaming detector compares each arm against\nits own offline decode\non the same recording, so no ceiling or floor enters and no cross-arm ranking is implied. Two distinct quantities, never quoted as one range:\nS1\n, the offline↔streamed cosine, and\nS2\n, the paired retention delta.\nFor the three causal arms, across all three encoders: S1\n0.999866–0.999946\n, S2\n−3.9×10⁻⁵ to −5.3×10⁻⁶\n. A near-zero median additional change, under the tested encoders, supported recordings and imposed chunking regime. Small, and consistently negative — this study establishes no minimum detectable change and no equivalence threshold, so that is not a claim that the difference is below what the instrument can see.\nThe other thirteen lose\n−0.078 to −0.742 on S2\n, which is nowhere near that scale. For one arm the offline-versus-streamed cosine itself reaches\n−0.013\n: the streamed output is essentially orthogonal to that arm's own offline decode\nin this embedding space\n. That says nothing about waveform or linguistic relationships, which were not measured.\nThis agrees with Gate 4's separate streaming detector — an aligned maximum-error detector on the waveform — which found the same three arms at\n0.69–0.74%\nerror where every other arm sat at 105–225%. Two measurements sharing no implementation, the same three survivors.\nThey do share their input: the same reconstructions of the same corpus. So the agreement is\ncorroborating evidence\n, not proof. Two implementations agreeing makes an independent error in both less likely; it cannot rule out an error in either, and it says nothing about an artefact of the material both were computed on.\nWhat this does not establish\nPerceptual quality was never measured.\nNo listening test was run. Which decoder\nsounds\nbest is not established and no sentence here implies it. Two metrics ranked a phase-blind algorithm first; both are blind to the same thing.\nThe identity result is about\nspeaker-encoder representations\n, not about how audio sounds or about cloning quality.\nA large majority of cells fall outside the quality metric's validated support domain. Support is carried as a covariate, never as a silent exclusion.\nFor one decoder family, dispersion and band-limiting\ncannot be separated\nin this design, so no causal claim is made about it.\nSeven speakers, one studio, one recording chain, English. Absolute values are not comparable to VoxCeleb-scale benchmarks.\nThirteen arms were driven with chunked context they were never designed for. The result is the cost of streaming a decoder that was not built to stream — not a defect report.\nThe part I got wrong, and why it is in the article\nSix rounds of independent adversarial review ran against this work. Four before publication found three defects that\nchanged conclusions\n— a reference that admitted pairs the measurement forbade, a ceiling measured on the wrong duration, and a pair-grouping convention that was order-dependent.\nThen a fifth round ran\nafter publication, against the published package\n, and found what four earlier rounds structurally could not: they had all audited the private research tree, not what shipped from it.\nAll three of my advertised public reproduction commands failed from a clean copy of the release. The scripts resolved their data relative to where they sat in my private tree — one directory above the artifacts — and the release ships them flat. The claim audit opened a deliberately withheld file unconditionally. And the verifier I pointed readers at was the manifest of the\nprivate\ntree, which cannot verify a redacted package and duly reported 26 of 32 artifacts as drift. A first-audio range had borrowed a steady-state number for its upper end. Three sentences claimed more than the measurements carry.\nA sixth round then attacked the correction itself and found five more, two of them claims in the\nrepair\nthat outran their evidence in the same way as the originals.\nThe pattern has now recurred six times in different disguises:\nverification that checks what is present rather than what is required.\nThe estimator without its reference. The instrument without its evidence. The manifest without its inventory. A package that verified its own private origin rather than what it shipped. A privacy scanner that listed the paths it knew about instead of declaring what counts as private. A verifier that printed its own integrity fingerprint without ever comparing it to anything.\nThat last one is my favourite, because it looks like working code. It computed the hash. It printed the hash. It printed the recorded hash right underneath. It never compared them, so corrupting the fingerprint exited 0.\nPrinting a number beside another number is not a check. Only a comparison that can fail is one.\nAll of it is corrected in v1.0.1, itemised with the original wording beside the corrected wording.\nNo measurement changed\n— 51,495 non-string scalar leaves were compared across every JSON artifact common to both versions, and none differ. v1.0.0 stays permanently available, because deleting the version that was wrong would be the same mistake in a different disguise.\nReproducing it\nThree commands run from the public package with no private input: regenerate every table, re-trace every headline claim to the artifact it came from, and re-hash the bytes that shipped.\nThat is a\nconsistency check\n, and the README now says so rather than overclaiming. It is not independent reconstruction — rebuilding the aggregates and their confidence intervals needs the withheld per-recording rows and speaker embeddings, which are not published because an embedding of a named consenting human is biometric data. One published claim is marked\nWITHHELD\nrather than\nPASS\nin the public audit for exactly that reason. Source audio, embeddings and the 35.65 GB of reconstructions stay private; the reconstructions are retained because human listening and adversarial detectability are declared later phases.\nArticle (canonical):\nhttps://ai.bedvibe.studio/decoder-benchmark/\nPaper + code + full audit trail:\nhttps://github.com/Mormolykos/decoder-vocoder-benchmark\nRelease:\nhttps://github.com/Mormolykos/decoder-vocoder-benchmark/releases/tag/v1.0.1\nDOI (v1.0.1):\nhttps://doi.org/10.5281/zenodo.22811349\n— all versions:\nhttps://doi.org/10.5281/zenodo.22798415\nCorrections:\nhttps://github.com/Mormolykos/decoder-vocoder-benchmark/blob/main/CORRECTIONS_v1.0.1.md\nCC BY 4.0.", "metadata": { "title": "Three of Twenty Decoders Actually Stream, and My Quality Metric Was Beaten by an Algorithm from 1984 - DEV Community", "description": "Twenty neural audio decoders through five pre-registered gates. Only three truly stream. The sole quality metric was beaten by a zero-parameter algorithm from 1984 — which the pre-registration had declared in advance would be a finding about the metric. Tagged with machinelearning, python, audio, testing.", "keywords": "machinelearning, python, audio, testing, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/bedvibe_studios/three-of-twenty-decoders-actually-stream-and-my-quality-metric-was-beaten-by-an-algorithm-from-1984-5e77", "og:title": "Three of Twenty Decoders Actually Stream, and My Quality Metric Was Beaten by an Algorithm from 1984", "og:description": "Twenty neural audio decoders through five pre-registered gates. Only three truly stream. The sole quality metric was beaten by a zero-parameter algorithm from 1984 — which the pre-registration had declared in advance would be a finding about the metric.", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmo2yifyid488ng79n2aw.png", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "Three of Twenty Decoders Actually Stream, and My Quality Metric Was Beaten by an Algorithm from 1984", "twitter:description": "Twenty neural audio decoders through five pre-registered gates. Only three truly stream. The sole quality metric was beaten by a zero-parameter algorithm from 1984 — which the pre-registration had declared in advance would be a finding about the metric.", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmo2yifyid488ng79n2aw.png" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://ai.bedvibe.studio/decoder-benchmark/", "label": "https://ai.bedvibe.studio/decoder-benchmark/", "type": "a" }, { "href": "https://github.com/Mormolykos/decoder-vocoder-benchmark", "label": "https://github.com/Mormolykos/decoder-vocoder-benchmark", "type": "a" }, { "href": "https://github.com/Mormolykos/decoder-vocoder-benchmark/releases/tag/v1.0.1", "label": "https://github.com/Mormolykos/decoder-vocoder-benchmark/releases/tag/v1.0.1", "type": "a" }, { "href": "https://doi.org/10.5281/zenodo.22811349", "label": "https://doi.org/10.5281/zenodo.22811349", "type": "a" }, { "href": "https://doi.org/10.5281/zenodo.22798415", "label": "https://doi.org/10.5281/zenodo.22798415", "type": "a" }, { "href": "https://github.com/Mormolykos/decoder-vocoder-benchmark/blob/main/CORRECTIONS_v1.0.1.md", "label": "https://github.com/Mormolykos/decoder-vocoder-benchmark/blob/main/CORRECTIONS_v1.0.1.md", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:48:44Z", "source_url": "https://dev.to/bedvibe_studios/three-of-twenty-decoders-actually-stream-and-my-quality-metric-was-beaten-by-an-algorithm-from-1984-5e77", "papers": [], "actual_video_files": [], "body_chars": 14450, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/1b950b7ca435/pages/source.html", "local_html": "attachments/1b950b7ca435/pages/article.html", "body_base_path": "attachments/1b950b7ca435/pages/", "body_markdown_path": "attachments/1b950b7ca435/pages/article.md" }