{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Panagiotis Gkilis",
  "title_listed": "I audited my own ML linter and had to withdraw its best evidence",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/bedvibe_studios/i-audited-my-own-ml-linter-and-had-to-withdraw-its-best-evidence-1g96",
  "publish_date": "2026-09-21",
  "corpus_stage": "",
  "notes": "",
  "id": "044f16e5be47",
  "collected_at": "2026-10-02T13:52:11",
  "title": "I audited my own ML linter and had to withdraw its best evidence - DEV Community",
  "body_markdown": "I maintain\ntrainproof\n, a deterministic linter for ML training runs. No model scores your run — every verdict is a rule that either fires or doesn't, and every finding prints the numbers behind it.\nBefore releasing 0.22.0 I put it through two independent forensic audits, one of them working blind.\nThey took away the single best piece of evidence I had.\nThe claim\nEvery fault in my validation gallery is injected on purpose — learning rate cranked 100×, fp16 overflow, shuffled labels. Injected faults are clean, and that's the known weakness of any fault-injection study. So the one log I was proudest of was the one nobody touched: a 9.8-hour Coqui XTTS fine-tune from my own work that, I said, \"diverged on its own.\"\nIt was on the README. It was in two places in my long-form write-up. It was the thing that made the study stand on something real.\nIt was computed from a truncated prefix\nThe log I shipped stops at\nstep 72,900 of a run that went to 125,039.\nI had truncated it, published the verdict computed from the truncation, and cited the trainer's own bookkeeping as corroboration — the last\nBEST MODEL\nline in the prefix is\nbest_model_49880.pth\n, which does sit a long way behind the end of the prefix.\nThe rest of the run was still on disk. In the complete log:\nthe trainer promotes\nbest_model_124700.pth\n—\nstep 124,700 of 125,039, 99.7% of the way through\nall six retained held-out evaluations improve, the last one included:\n4.8813 → 2.5894\nThe bookkeeping I cited as corroboration was corroborating a sentence about the first 58% of a run.\nWhat replaces it, exactly\nThe temptation here is to overcorrect, and that would be the same mistake pointing the other way. So, precisely:\nWhat the evidence supports:\nthe per-micro-batch training display loss ended above its own minimum. That is a property of that one series.\nWhat it does not support:\nthat the model got worse. And it does\nnot\nsupport that the model got better either — no audio, MOS or perceptual evaluation of that run was kept. Held-out\nloss\nimproved. That is all that was measured.\nThe part that matters more than my mistake\ntrainproof still returns FAIL on that run, and 0.22.0 does not fix it.\nTP-DIVERGE\nreads one training series. That run's event file carries two:\nseries\npoints\nends\nper-micro-batch loss\n1,251\n1.88× above its minimum\nepoch-aggregated loss\n5\nat\nits minimum\nThe reader resolves competing series by density. The 1,251-point series wins; the 5-point series is discarded before any rule sees it. An improving held-out curve sitting in the same file cannot overrule the survivor.\nI could have changed the threshold until this particular run went green. That is tuning a rule against a single example until it produces the answer I now prefer, and it would have been the worst available option. So the run still FAILs, the limitation is written into the README, the rules document and the release's machine-readable record, and it is the top item for the next architecture instead of something quietly smoothed over.\nWhat else the audits found\nSame principle decided each one:\nremove a claim the tool can't support, or make an advertised behaviour real — never strengthen a claim to close a gap.\nA missing optional dependency was a verdict about your work.\nNo\nsentencepiece\ninstalled?\ntrainproof tokenizer\nexited 1 with a FAIL — the tool reporting\nits own\nmissing package as a fault in your tokenizer. Now\nNOT-CHECKED\n, exit 2. FAIL has to mean something about your run, never \"the tool had a problem.\"\nThe HuggingFace callback could change the run it was watching.\nIt read the first batches of labels at\non_train_begin\n, which means opening a fresh iterator over your training dataloader. With a map-style loader and a random sampler drawing from a generator, that consumes generator state and shifts your batch order. A measurement tool perturbing its own experiment.\nobjective_check\nnow defaults to\nFalse\n.\nFour evidence strings described something other than what was measured\n— including a zero-learning-rate finding that printed\n100.0% of steps have lr=0\nfor a run where every value was\n-1e-4\n.\nNo rule ID, threshold or detection predicate changed in 0.22.0. It adds nothing. 497 tests, each correction pinned by a test that fails against the pre-repair code.\nWhat I'm not claiming\nNo rule in trainproof is calibrated.\nThere is no false-positive rate for any check, because nothing here has been run against a population where the answer is independently known. The gallery is a set of faults I injected on purpose — a regression suite, not a sample.\nWhat the tool offers is determinism and evidence: the same log gives the same verdict, every finding prints the numbers behind it, and a check that could not run says\nNOT-CHECKED\ninstead of\nPASS\n.\nRead a finding's\nevidence\nas the measurement and its\nmessage\nas an interpretation that may exceed it. That sentence is in the README now because the audits showed it needed to be.\npip\ninstall\ntrainproof\ntrainproof doctor\n.\n# discovers and judges every training log it finds\nEnter fullscreen mode\nExit fullscreen mode\nRelease 0.22.0 is archived at\n10.5281/zenodo.22882964\n. The full write-up, including what the audits found that I haven't covered here, is\non my notebook\n.\nIf a deterministic check would have caught a failure that burned your GPU hours, tell me on the repo — it goes in, with credit.\nPanagiotis (Panos) Gkilis\n— founder, BedVibe Studios, the operating brand of BED VIBE GKILIS, a sole proprietorship registered in Norway (org. no. 935 267 897).\nMain hub\n·\nPortfolio\n·\nWork\n·\nResearch & articles\n·\nAbout\n·\nLinkedIn\n·\nGitHub\n·\nORCID",
  "metadata": {
    "title": "I audited my own ML linter and had to withdraw its best evidence - DEV Community",
    "description": "I maintain trainproof, a deterministic linter for ML training runs. No model scores your run — every... Tagged with machinelearning, python, testing, devops.",
    "keywords": "machinelearning, python, testing, devops, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/bedvibe_studios/i-audited-my-own-ml-linter-and-had-to-withdraw-its-best-evidence-1g96",
    "og:title": "I audited my own ML linter and had to withdraw its best evidence",
    "og:description": "I maintain trainproof, a deterministic linter for ML training runs. No model scores your run — every...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4eel1q0u471zfznqym7k.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "I audited my own ML linter and had to withdraw its best evidence",
    "twitter:description": "I maintain trainproof, a deterministic linter for ML training runs. No model scores your run — every...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4eel1q0u471zfznqym7k.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/044f16e5be47/code/body-000.txt",
      "chars": 96,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/Mormolykos/trainproof",
      "label": "trainproof",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.5281/zenodo.22882964",
      "label": "10.5281/zenodo.22882964",
      "type": "a"
    },
    {
      "href": "https://ai.bedvibe.studio/trainproof/",
      "label": "on my notebook",
      "type": "a"
    },
    {
      "href": "https://bedvibe.studio/",
      "label": "Main hub",
      "type": "a"
    },
    {
      "href": "https://tts.bedvibe.studio/portfolio/",
      "label": "Portfolio",
      "type": "a"
    },
    {
      "href": "https://ai.bedvibe.studio/work/",
      "label": "Work",
      "type": "a"
    },
    {
      "href": "https://ai.bedvibe.studio/",
      "label": "Research & articles",
      "type": "a"
    },
    {
      "href": "https://bedvibe.studio/panagiotis-panos-gkilis/",
      "label": "About",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/panagiotis-gkilis-57995117b/",
      "label": "LinkedIn",
      "type": "a"
    },
    {
      "href": "https://github.com/Mormolykos",
      "label": "GitHub",
      "type": "a"
    },
    {
      "href": "https://orcid.org/0009-0007-3805-170X",
      "label": "ORCID",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:43Z",
  "source_url": "https://dev.to/bedvibe_studios/i-audited-my-own-ml-linter-and-had-to-withdraw-its-best-evidence-1g96",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 5593,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/044f16e5be47/pages/source.html",
  "local_html": "attachments/044f16e5be47/pages/article.html",
  "body_base_path": "attachments/044f16e5be47/pages/",
  "body_markdown_path": "attachments/044f16e5be47/pages/article.md"
}