{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Panagiotis Gkilis",
  "title_listed": "I spent months training a 730M model with no tool that could tell me it was already dead",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/bedvibe_studios/i-spent-months-training-a-730m-model-with-no-tool-that-could-tell-me-it-was-already-dead-220l",
  "publish_date": "2026-08-03",
  "corpus_stage": "",
  "notes": "",
  "id": "4c08514db8d3",
  "collected_at": "2026-10-02T13:52:24",
  "title": "I spent months training a 730M model with no tool that could tell me it was already dead - DEV Community",
  "body_markdown": "I spent months trying to train a 730M-parameter TTS model on my own hardware. It wouldn't converge, and nothing in my stack would tell me why — not the loss curve, not TensorBoard, not the checkpoints. Every tool I had showed me numbers. None of them would say\nthis run is already dead, stop paying for it.\nThat situation isn't rare. A fine-tune quietly goes to NaN after step 300 and trains on garbage the rest of the night. A run trains at learning-rate-\nzero\n, updating nothing, for six hours. A dataset carries a few hundred broken rows you find at the end. Every time the pattern is identical: the run\nlooks\nfine, and nothing tells you otherwise until the hours are spent.\nSo I built the linter I needed.\npip\ninstall\ntrainproof\ntrainproof doctor\n.\n# discovers and judges every training log it finds\nEnter fullscreen mode\nExit fullscreen mode\nMIT, and the interesting part isn't the rules — it's the constraint.\nNo ML judging ML\nThere is no model inside trainproof. No \"87% confidence this looks unhealthy.\" Every verdict is a deterministic rule that fires or doesn't, and every finding prints the number it fired on.\nThat's not minimalism for its own sake.\nA reliability tool that hallucinates is worse than no tool\n, because now you don't trust your own alarms — and an alarm you don't trust is just noise. When trainproof can't be sure, it is required to say so rather than guess.\nThat requirement has a concrete shape. Severity and exit code are separate axes:\nSeverity\nMeaning\nExit\nFAIL\nYour run is broken\n1\nWARN\nSomething is worth your attention\n0\nNOT-CHECKED\nI could not judge this\n2\nPASS\nChecked, and fine\n0\nNOT-CHECKED\nis the one that matters. A checking tool can say two very different sentences —\n\"your run failed\"\nand\n\"I couldn't read your file\"\n— and a tool that can't tell them apart is lying to your CI in a quiet, expensive way. An unreadable log reports no verdict rather than inventing a failure.\nValidating the rules: controlled fault injection\nYou validate a detector by feeding it faults whose answer you already know, then measuring what it catches. One Qwen2.5-3B QLoRA on an RTX 5080, run six ways, three seeds each — 18 runs:\nhealthy\n— the control\nlr_hot\n— learning rate 100× too high\nlr_zero\n— learning rate at zero\nfp16_nan\n— an fp16 overflow that NaNs the loss\nbad_labels\n— labels shuffled into pure noise\noverfit\n— 64 samples, many epochs, with a held-out eval set to prove it\nFour of the five failures got caught.\nlr_hot\nspiked gradient norm to\n2,650 — about 4,900× its own median\n, flagged in seconds. NaN and flatline were trivial.\nOne beat every loss rule I had.\nThe one that fooled it\nbad_labels\nis a dataset that\ncannot be learned\n— the labels no longer correspond to the inputs. That run\nreduced its loss by 69.8%\n(18.9 → 5.7).\nOn its own curve it looked like textbook-healthy training. A clean downward slope. It was learning nothing; it was memorizing the statistics of noise, which any sufficiently large network will happily do.\nBe precise about the failure, because you can run this yourself:\nbad_labels\ndoesn't come back clean, it comes back\nWARN\n— and the only rule that fires is\nTP-GRAD-SPIKE\n, an unrelated gradient blip.\nNot one loss rule fired.\nThe tool noticed something twitched and completely missed that the dataset was noise.\nThe lesson is bigger than the tool:\na loss curve going down is not evidence that anything real is being learned.\nI wrote that straight into the README rather than papering over it. It's also\nwhy\nthere's a\ncompare\nmode — put the run next to a known-good baseline and the relative floor gives it away instantly. One run in isolation lies. Two runs side by side tell the truth.\nThen I pointed it at my own real work\nInjected faults are a lab. Eventually I ran the shipped rules against real fine-tunes I had actually paid for, on my own GPU. Two results, both in the repo under\nevidence/\n:\nReal run\nSteps\nVerdict\nCoqui XTTS v2\n125,000\nFAIL\n—\nTP-DIVERGE\n,\nTP-THROUGHPUT\nFish Speech LoRA (Lightning)\n2,049\nWARN\n—\nTP-OVERFIT\nTP-OVERFIT\nfires when eval-loss climbs past 1.2× its own minimum while train-loss keeps falling. It warns rather than fails, because early stopping is your call — but it means\nyour best checkpoint has already gone by\n, and if you're keeping only the last one, you kept the wrong one.\nThe XTTS run is logged twice, by two entirely separate readers — Coqui's text log and its TensorBoard event file, same run. They produce the\nsame verdict and the same rule set\n, which is the check I care about most: agreement between independent readers is how you find out your parser is inventing things.\nThe part I'd defend hardest: it was wrong, and I shipped the fix\nRunning against real logs also made trainproof report a\nfalse FAIL on a perfectly healthy run\n.\nTP-ZERO-GRAD\nfired whenever every finite gradient norm was exactly\n0.0\n, and reported a severed backward graph. Coqui writes\navg_grad_norm\nas\n0.0\nwhen gradient clipping is off. So a healthy 125,000-step XTTS fine-tune whose loss reached\n0.017\nwas reported\nFAIL\nby my own tool.\nThe fix is a piece of reasoning, not a threshold tweak:\na run cannot both learn and receive no gradient.\nThe check now stands down when the loss improved beyond a minimum threshold — and records\nwhy\nit stood down as an explicit skip, rather than silently not running.\nNo test caught that. A real run did. That's the entire argument for shipping evidence logs in the repo instead of only fixtures.\nWhat it actually does\nEvery stage exits with a status code, so it drops into CI as-is:\nBefore a GPU-second —\npreflight\nand\nenv\n.\nDataset and tokenizer: malformed JSONL\nwith the line number\n, empty rows, duplicates, missing\neos_token\n,\npad == eos\n. Then the environment: does the training entrypoint import (\nin a subprocess\n, so a segfaulting extension can't take the linter with it), is the checkpoint intact, is there RAM and disk.\nDuring — the guardian.\nA one-line Hugging Face callback. Warns by default; opt into\nstop_on_fail\nand it aborts a doomed run itself. It killed a diverging demo run at\nstep 20 of 300\n.\nAfter —\nepoch\nand\ndoctor\n.\nDiverged, flatlined, NaN'd, spiked, overfitting.\nAgainst a baseline —\ncompare\n.\nThe relative-floor rules that catch shuffled labels.\nReads what you already produce: Hugging Face\ntrainer_state.json\n, Coqui,\nTensorBoard event files\n, JSONL, CSV. Nothing to instrument.\nOne note on the TensorBoard reader, since it's the part I enjoyed most: it's written from the wire format — TFRecord framing plus the\nEvent\n/\nSummary\n/\nTensorProto\nfields it needs. It imports no tensorflow, no tensorboard, no protobuf, no torch. It's validated byte-exact against\nEventAccumulator\non a real 2,049-step Lightning run: all 13 tags, all point counts, all values. And a\ntruncated\nevent file — the normal state of a killed run — is read up to the cut instead of raising, because the killed run is the run most in need of judging.\nCheckpoints are read without unpickling\ntorch.load\nexecutes arbitrary code by design. That's the reason torch 2.6 flipped\nweights_only\nto\nTrue\n.\nA tool whose job is to tell you a checkpoint is safe to use, which must\nrun\nthat checkpoint to find out, is not a safety tool. So trainproof reads\n.pt\n/\n.pth\n/\n.ckpt\nas the ZIP archives they are — entry table, storage count, CRC, straight from the archive directory. It distinguishes missing, zero-byte, truncated, CRC-corrupt, and legacy pre-1.6 pickle, which it reports as\nNOT-CHECKED\n, because refusing to unpickle is correct behaviour rather than an error.\nThe truncated case is the one that earns its keep. A save killed mid-write leaves a ZIP header with no central directory, and\nzipfile.is_zipfile()\nreturns\nFalse\nfor it — so the most common real checkpoint failure was being reported as \"not a checkpoint at all.\" That's the difference between resuming from your previous checkpoint and hunting for a file that was never written.\nWhere it is now\n17 releases.\n84 stable rule IDs, 230 tests,\nschema_version\n3.\nRule IDs are stable and the JSON schema has a written contract in\nCONTRACTS.md\n— what each exit code means, when the format may change, that verdicts won't silently drift.\nWriting that document found a bug I had already shipped:\ndoctor --baseline\nprinted\n[FAIL]\nfindings and still\nexited 0\n, because the exit code was computed from single-run verdicts and never consulted the comparison. The terminal had been telling the truth; the exit code had been lying. No test caught it, because no test had ever written down which of the two was authoritative.\nEvery verdict the tool produces on its example runs is frozen in 38 golden snapshots. A rule that silently stops firing fails the build — and so does one that starts firing spuriously.\nTry it\npip\ninstall\ntrainproof\ntrainproof doctor\n.\nEnter fullscreen mode\nExit fullscreen mode\nRepo\n(with the fault-injection logs, the real 125k-step evidence, and every locked verdict):\ngithub.com/Mormolykos/trainproof\nPyPI\n:\npypi.org/project/trainproof\nFull write-up\n:\nai.bedvibe.studio/trainproof\nSibling\n:\nttsproof\n— failure-mode QA for text-to-speech\nMore of what I've built\n:\ntts.bedvibe.studio/portfolio\nMIT.\nIf you fine-tune, you've hit at least one of these. I'd genuinely like to know\nwhich failure mode has burned your GPU hours\n— if a deterministic check would have caught it, open an issue and it goes in, with credit.",
  "metadata": {
    "title": "I spent months training a 730M model with no tool that could tell me it was already dead - DEV Community",
    "description": "A deterministic linter for ML training runs — no model judging your model. It caught a real failure in my own 125,000-step fine-tune, and a false positive in itself. Tagged with machinelearning, python, opensource, mlops.",
    "keywords": "machinelearning, python, opensource, mlops, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/bedvibe_studios/i-spent-months-training-a-730m-model-with-no-tool-that-could-tell-me-it-was-already-dead-220l",
    "og:title": "I spent months training a 730M model with no tool that could tell me it was already dead",
    "og:description": "A deterministic linter for ML training runs — no model judging your model. It caught a real failure in my own 125,000-step fine-tune, and a false positive in itself.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsfekt1hk1rlyoup1fatm.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "I spent months training a 730M model with no tool that could tell me it was already dead",
    "twitter:description": "A deterministic linter for ML training runs — no model judging your model. It caught a real failure in my own 125,000-step fine-tune, and a false positive in itself.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsfekt1hk1rlyoup1fatm.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/4c08514db8d3/code/body-000.txt",
      "chars": 96,
      "language": "text"
    },
    {
      "file": "attachments/4c08514db8d3/code/body-001.txt",
      "chars": 43,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/Mormolykos/trainproof",
      "label": "github.com/Mormolykos/trainproof",
      "type": "a"
    },
    {
      "href": "https://pypi.org/project/trainproof/",
      "label": "pypi.org/project/trainproof",
      "type": "a"
    },
    {
      "href": "https://ai.bedvibe.studio/trainproof/",
      "label": "ai.bedvibe.studio/trainproof",
      "type": "a"
    },
    {
      "href": "https://pypi.org/project/ttsproof/",
      "label": "ttsproof",
      "type": "a"
    },
    {
      "href": "https://tts.bedvibe.studio/portfolio/",
      "label": "tts.bedvibe.studio/portfolio",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:45Z",
  "source_url": "https://dev.to/bedvibe_studios/i-spent-months-training-a-730m-model-with-no-tool-that-could-tell-me-it-was-already-dead-220l",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 9308,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/4c08514db8d3/pages/source.html",
  "local_html": "attachments/4c08514db8d3/pages/article.html",
  "body_base_path": "attachments/4c08514db8d3/pages/",
  "body_markdown_path": "attachments/4c08514db8d3/pages/article.md"
}