{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Panagiotis Gkilis",
  "title_listed": "One Integer Deleted the Stop Token From My Loss. The Curve Never Noticed.",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/bedvibe_studios/one-integer-deleted-the-stop-token-from-my-loss-the-curve-never-noticed-2dll",
  "publish_date": "2026-08-31",
  "corpus_stage": "",
  "notes": "",
  "id": "dbd1b99aa7bd",
  "collected_at": "2026-10-02T13:52:14",
  "title": "One Integer Deleted the Stop Token From My Loss. The Curve Never Noticed. - DEV Community",
  "body_markdown": "For a long time I had a model that trained cleanly and produced nothing usable. The loss fell. Gradients were finite. Nothing crashed. It simply never learned to stop — every generation ran to the length cap and got truncated.\nThe cause was two lines of my own source that were individually correct.\nThe collision\nIn a neural codec language model the audio vocabulary has a fixed size, and the stop token is one extra class on top. So the output layer is one wider than the codebook:\nnn.Linear(d_model, NUM_AUDIO_TOKENS + 1)   # 1025 classes: 0..1023 audio, 1024 = EOS\neos_id = NUM_AUDIO_TOKENS                  # 1024\nEnter fullscreen mode\nExit fullscreen mode\nCorrect. EOS is the last class and the layer has room for it.\nThen, elsewhere, the loss:\nF.cross_entropy(logits, targets, ignore_index=NUM_AUDIO_TOKENS)\nEnter fullscreen mode\nExit fullscreen mode\nAlso reasonable on its own.\nignore_index\nis how you skip padding.\nBut the sentinel and the stop token are the same integer.\nEvery position whose target was \"stop\" was discarded before the loss was computed. Not down-weighted — removed. The model was never once shown an example of stopping, across every epoch it ever ran.\nPyTorch defaults\nignore_index\nto\n-100\nprecisely because it must be a value that can never be a real class. Replace it with a real vocabulary constant and that guarantee is gone, silently: shapes valid, loss finite, run healthy.\nThe curves are identical\nMinimal reproduction, two arms differing only in the sentinel value.\nThe broken arm finished at 0.0035. The fixed arm finished at 0.0034.\nSame curve to any human, any dashboard, any threshold you would write. One has a working objective and one has an objective with a hole in it, and the loss cannot distinguish them — because the loss is computed\nover what survived the mask\n. A metric cannot report on examples it never received.\nThe second half: lowest loss, worse model\n200 epochs at lr = 2e-5, evaluated every 50 on a fixed split held out by utterance (n = 32).\nRank\nis the stop token's position among 1025 classes at the true terminal frame.\nepoch\ntraining loss\nmean P(stop)\nargmax = stop\nself-terminated\n50\n2.199\n0.4218\n16/32\n2/8\n100\n1.628\n0.4655\n18/32\n6/8\n150\n1.302\n0.2159\n8/32\n3/8\n200\n1.429\n0.1818\n5/32\n3/8\nBetween epochs 100 and 150 the training loss\nimproved by 20%\nwhile mean P(stop) fell\n54%\n, top-1 stop accuracy went 18/32 to 8/32, and autonomous termination halved.\nSelecting by lowest training loss returns epoch 150. The model that terminates reliably is epoch 100.\nThe reversal was observed independently in a shorter run, which is why I am willing to state it.\n\"Save the checkpoint with the lowest validation loss\" is the default in more or less every training script in existence, including mine. On this run it was actively the wrong rule, and the number it optimised looked better the whole way down.\nFor completeness, what fixing the collision bought on the real model, same held-out split:\ncheckpoint\nmean P(stop)\nargmax = stop\nrank\nrandom initialisation\n0.001848\n0/32\n111.6\nafter correction\n0.4655\n18/32\n2.1\nEnd-to-end synthesis then terminated on its own at frame 203 against a 350-frame ceiling. Before the fix that was impossible by two independent mechanisms: the training-time collision above, and an inference-time mask that set the logit of every index at or beyond the codebook size — including end-of-sequence — to negative infinity before sampling.\nOne honest loose end: P(stop) plateaus in the range\n0.35–0.47\nacross two learning rates and a 5.9x increase in training data (224 to 1313 utterances, with speaker, language, emotion and the held-out split all held constant). So the plateau is not a data-quantity limit. I have no confirmed explanation for it. Terminal timing in speech is genuinely ambiguous, and hedging with the stop token ranked second of 1025 may simply be correct behaviour.\nThe same integer, safe one stage over\nThe detail I find most instructive: the identical line is harmless in the next stage of the same model.\nThe autoregressive stage predicts the first codebook plus EOS —\n1025 classes\n, so\n1024\nis a real class and using it as a sentinel is fatal. The non-autoregressive stage predicts audio codes only —\n1024 classes\n, so\n1024\nis out of range and the exact same\nignore_index\nis correct.\nOne\n+ 1\nin a different file decides whether that line destroys your objective. Both stages read identically at the call site. That is not a mistake you catch by reading carefully; it is a mistake you catch by checking a relationship between two numbers that never appear together.\nWhich is what a linter is for\nThis is now two rules in\ntrainproof\n, my linter for training runs:\nSentinel collision.\nCompare the output layer's class count against\nignore_index\n. If the sentinel is a valid class, fail. If it sits exactly one past the end, say so explicitly — because the same integer is fatal one class earlier, and that distinction deserves to be visible rather than silently passed.\nDead class.\nAccumulate which classes ever reach the loss as a positive target during the first epoch, then flag any class the output layer can emit but that never once appears as an answer.\nMy own two stages are the regression fixture — the fatal case and its safe twin, one integer apart. Not a synthetic example.\nThe design decision worth stating: the dead-class rule only fires when coverage is already broad and few classes are missing. One unseen class out of 1025 is a structural exclusion. Nine hundred unseen is a small sample. Without that guard the check screams on every short run and gets switched off — which is how good checks die.\nIf you take one thing from this\nI do not think this is rare. Any codebase where a padding sentinel, an end-of-sequence id and a vocabulary size are all defined as named constants in different files can produce it, and none of your instrumentation will complain.\nIf you fine-tune anything with a custom\nignore_index\n, go and check it against your output layer's width right now. It takes thirty seconds and the failure mode is completely silent.\nPaper:\nThe Loss Curve Is Not a Sufficient Statistic — Silent Objective Failures from Sentinel-Class Collisions in Neural Codec Language Models\nDOI:\n10.5281/zenodo.21864658\npip install trainproof\n—\nGitHub\n, MIT",
  "metadata": {
    "title": "One Integer Deleted the Stop Token From My Loss. The Curve Never Noticed. - DEV Community",
    "description": "For a long time I had a model that trained cleanly and produced nothing usable. The loss fell.... Tagged with machinelearning, python, deeplearning, debugging.",
    "keywords": "machinelearning, python, deeplearning, debugging, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/bedvibe_studios/one-integer-deleted-the-stop-token-from-my-loss-the-curve-never-noticed-2dll",
    "og:title": "One Integer Deleted the Stop Token From My Loss. The Curve Never Noticed.",
    "og:description": "For a long time I had a model that trained cleanly and produced nothing usable. The loss fell....",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgdaampfbvedgkoxjaqd1.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "One Integer Deleted the Stop Token From My Loss. The Curve Never Noticed.",
    "twitter:description": "For a long time I had a model that trained cleanly and produced nothing usable. The loss fell....",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgdaampfbvedgkoxjaqd1.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/dbd1b99aa7bd/code/body-000.txt",
      "chars": 135,
      "language": "text"
    },
    {
      "file": "attachments/dbd1b99aa7bd/code/body-001.txt",
      "chars": 64,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://doi.org/10.5281/zenodo.21864658",
      "label": "10.5281/zenodo.21864658",
      "type": "a"
    },
    {
      "href": "https://github.com/Mormolykos/trainproof",
      "label": "GitHub",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:44Z",
  "source_url": "https://dev.to/bedvibe_studios/one-integer-deleted-the-stop-token-from-my-loss-the-curve-never-noticed-2dll",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 6272,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/dbd1b99aa7bd/pages/source.html",
  "local_html": "attachments/dbd1b99aa7bd/pages/article.html",
  "body_base_path": "attachments/dbd1b99aa7bd/pages/",
  "body_markdown_path": "attachments/dbd1b99aa7bd/pages/article.md"
}