{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Panagiotis Gkilis",
  "title_listed": "Fourteen Speaker Encoders Heard the Same Voice. Their Error Rates Differed Five-Fold.",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/bedvibe_studios/fourteen-speaker-encoders-heard-the-same-voice-their-error-rates-differed-five-fold-1pmn",
  "publish_date": "2026-08-31",
  "corpus_stage": "",
  "notes": "",
  "id": "eea2a4879b77",
  "collected_at": "2026-10-02T13:52:16",
  "title": "Fourteen Speaker Encoders Heard the Same Voice. Their Error Rates Differed Five-Fold. - DEV Community",
  "body_markdown": "A speaker encoder turns a few seconds of speech into a vector, and the distance between two vectors is treated as an answer to \"are these the same person.\" That number gets used for authentication, for voice-cloning pipelines, and increasingly as the yardstick in papers asking whether a synthetic voice preserved someone's identity.\nThe number is not neutral. It is a reading from an instrument, and I wanted to know how much the instrument was contributing.\nSo I ran fourteen of them over an identical, frozen list of\n11,935 trials\n.\nThe corpus is why the question is answerable\nMost emotional-speech corpora use different sentences for different emotions. A model can then learn angry\nvocabulary\ninstead of angry\ndelivery\n, and you cannot separate them afterwards.\nThis one can't make that mistake.\nEight speakers each recorded the same 1,360 sentences in six phonation states\n— neutral, happy, angry, scared, shouting, whisper. Lexical content is fixed while phonation varies.\nEnrolment is neutral speech. The test is whether the encoder still recognises the person while they shout or whisper. The trial list was frozen in writing before a single model loaded, and every encoder scored the identical list — so every comparison is\npaired\n.\nThe headline\nEqual error rate spans 0.047 to 0.233 across the panel\n— same trials, same audio, nothing varying but the encoder.\n29 of 91\npairwise comparisons survive Holm–Bonferroni correction.\nIf you are choosing an encoder for expressive audio, that five-fold spread is your decision. It is larger than most architectural differences people argue about.\nCAMPPlus is last, without exception\nHighest error of the fourteen, all 13 of its comparisons surviving family-wise correction\n— for every one of the eight speakers and every one of the six conditions. Margins run from\n+0.087 [+0.064, +0.127]\nagainst ECAPA-TDNN to\n+0.185 [+0.150, +0.215]\nagainst the lowest-error encoder.\nThe mechanism is the useful part. Its\ngenuine\nscores are unremarkable; its\nimpostor\nscores are the problem. It places different people at cosine\n0.28–0.34\nwhere ECAPA-TDNN and ReDimNet place them near\n0.034\n. That is a compressed embedding space — different speakers land close together.\nIt is visible at neutral speech too (0.031 EER where several ReDimNet checkpoints reach 0.000). Expressive speech doesn't cause it. Expressive speech makes it visible.\nA widely used open-source TTS system conditions on CAMPPlus. To be precise: that is a verified fact about its source code,\nnot\na measurement of its audio. I measured encoders in isolation on human recordings. Whether that choice degrades output is a separate experiment I have not run.\nParameter count does not order the panel\nSpearman ρ(parameters, expressive EER) =\n+0.135\nover fourteen encoders. The 20.8M model ranks 13th of 14. A 4.81M model ranks 1st. Published VoxCeleb1-O position doesn't transfer either — that leaderboard is computed on calm read speech, which is not the condition failing you.\nI retracted my claim about the winner, incidentally. It survives as an argmin (selected in 89.4% of speaker resamples, leading all eight leave-one-speaker-out refits) but after correction it is separated from only\n7 of 13\n. The defensible wording is\nlowest observed error\n.\nA clean pre-registered negative\nSpectral denoising does not materially change the measurement:\n|ΔEER| ≤ 0.039 over 10,765 twinned trials\n, and whispered speech — where removing aspiration noise was the stated worry — is among the\nleast\naffected.\nTen retracted claims, published on purpose\nAn 11.7 dB spectral effect, asserted with an instrument whose F0 artifact budget is 12.09 dB.\nSweeping F0 with the spectral envelope held fixed moved the descriptor further than the claimed effect. It was never a measurement.\n\"Shouting does not transfer.\"\nJudged on RMS after peak-normalising every clip, so the figure was crest factor, not loudness.\nA tie-break defect in the EER estimator.\nargmin\nreturns the first minimiser rather than the balanced operating point — invisible at unit weights, common under the integer speaker multiplicities a bootstrap runs at. Fixing it moved one comparison across Holm,\n28 → 29 of 91\n. The correction\nfavoured\nthe paper, which is why it is published rather than quietly kept.\nThe rule I have now paid for three times:\nvalidate the instrument before measuring with it.\nStatistics\nThe trial list is a directed graph on 8 speakers — genuine trials are self-loops, impostor trials the 56 ordered edges. The independent unit is the speaker, on both sides. So the resample is a\nvertex bootstrap\n: draw 8 vertices with replacement, take the induced sub-multigraph, weight the impostor edge by the product of multiplicities.\nAn earlier scheme resampled enrolment speakers only, dropping a speaker as enrollee while leaving them in as impostor. Fixing it widened marginal intervals (width ratio median 1.11, max 1.58) and\ncost two already-written claims\n. Paired differences barely moved, because both encoders score the same impostor clips and the dependence cancels in the difference.\nB = 20,000, seed fixed, families declared before testing, Holm–Bonferroni at α = 0.05.\nThe companion result, which is the cheap fix\nEnrolling on expressive material instead of neutral-only recovers the loss\non all 14 encoders\n, every one surviving Holm.\nThe mechanism is the opposite of the intuitive one.\nEnrolling on a\nsingle\nexpressive state helps that state and slightly harms the others — transfer-matrix diagonal median\n−0.082\n, off-diagonal median\n+0.010\n. Guessing which state a user will be in, and guessing wrong, is worse than enrolling on calm speech.\nA pre-committed control ruled out the competing explanation: acoustic diversity alone delivers\n+0.002\nwhere composition delivers\n−0.065\n.\nIt is coverage, not expressiveness.\nSpread enrolment thinly across states.\nWhat it does not claim\nNothing about any TTS system's output quality. Eight speakers, one studio, one recording chain. Enrolment and test share a session, so absolute error rates are optimistic and not comparable to VoxCeleb — only relative comparisons on identical trials are licensed. Architecture is fully confounded with training corpus; separating them requires training.\nThe generalisable part\nAn established metric is not automatically a neutral one. An encoder, an evaluator, a classifier, an LLM judge — any of them can become part of the experiment rather than a window onto it.\nBefore asking whether a system preserves identity under some transformation, ask whether your instrument has been validated under that transformation. Otherwise you attribute the encoder's behaviour to the speaker, and the number looks perfectly reasonable while you do it.\nPaper:\nSpeaker Encoders Disagree About Who You Are When You Shout\n— Zenodo, CC-BY-4.0.\nDOI:\n10.5281/zenodo.22158030\nCode, data and the full decision record:\ngithub.com/Mormolykos/speaker-encoders-disagree\nThe eval library the analysis checks itself against:\npip install spkproof",
  "metadata": {
    "title": "Fourteen Speaker Encoders Heard the Same Voice. Their Error Rates Differed Five-Fold. - DEV Community",
    "description": "A speaker encoder turns a few seconds of speech into a vector, and the distance between two vectors... Tagged with machinelearning, python, datascience, ai.",
    "keywords": "machinelearning, python, datascience, ai, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/bedvibe_studios/fourteen-speaker-encoders-heard-the-same-voice-their-error-rates-differed-five-fold-1pmn",
    "og:title": "Fourteen Speaker Encoders Heard the Same Voice. Their Error Rates Differed Five-Fold.",
    "og:description": "A speaker encoder turns a few seconds of speech into a vector, and the distance between two vectors...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdnreeo8sb3jd28qr8jvy.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Fourteen Speaker Encoders Heard the Same Voice. Their Error Rates Differed Five-Fold.",
    "twitter:description": "A speaker encoder turns a few seconds of speech into a vector, and the distance between two vectors...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdnreeo8sb3jd28qr8jvy.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://doi.org/10.5281/zenodo.22158030",
      "label": "10.5281/zenodo.22158030",
      "type": "a"
    },
    {
      "href": "https://github.com/Mormolykos/speaker-encoders-disagree",
      "label": "github.com/Mormolykos/speaker-encoders-disagree",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:44Z",
  "source_url": "https://dev.to/bedvibe_studios/fourteen-speaker-encoders-heard-the-same-voice-their-error-rates-differed-five-fold-1pmn",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 6982,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/eea2a4879b77/pages/source.html",
  "local_html": "attachments/eea2a4879b77/pages/article.html",
  "body_base_path": "attachments/eea2a4879b77/pages/",
  "body_markdown_path": "attachments/eea2a4879b77/pages/article.md"
}