{ "source_file": "0930", "source_platform": "Dev.to", "author": "Seth Wheeler", "title_listed": "Ten Packages, One Rule: A Check Must Be Able to Fail", "keywords": "", "domain": "大模型算法", "content_type": "工程实践", "is_paid_flag": "否", "url": "https://dev.to/megapixel99/ten-packages-one-rule-a-check-must-be-able-to-fail-60f", "publish_date": "2026-09-20", "corpus_stage": "", "notes": "", "id": "e88f4986c330", "collected_at": "2026-10-02T13:51:45", "title": "Ten Packages, One Rule: A Check Must Be Able to Fail - DEV Community", "body_markdown": "Between 20 August and 31 August I published ten small packages, eight of them on the last day. The first of the eight repositories was created at 06:05 UTC and the last at 18:34. They are small verification and measurement tools, each in Python or JavaScript or both, installable from PyPI and npm, and each encodes one way a green check can mean nothing. Nothing about the code was written in a day. The packages are extractions from the measurement work behind much of what is on this site. The recurring failure there was almost never the thing being measured; it was the instrument reporting success without having checked anything.\nA memory profiler that watched the wrong process for eight months\nand\na rate limiter that reported requests it never sent\nare both that failure. The research codebase these packages fell out of is, like the ones in those posts, not public. The packages are the public form of what it kept teaching.\nThe failure has a family resemblance wherever it appears.\ngo test ./...\non a tree with no test files prints\n[no test files]\nand exits 0. A pytest suite in which every test is skipped prints\n2 skipped\nand exits 0 (measured on pytest 9.1.1). A JUnit report can read\ntests=\"50\" skipped=\"50\"\n, which is a green run of nothing wearing a total. A lint rule disabled by a config merge is green forever, and green is what everyone was looking for. In each case a check stopped checking, and nothing about the repository looks wrong.\nThe rule all ten packages are built on is stated in\ndidrun\n's README: a check must answer separately whether it ran, whether it failed, and whether the failure was the right one. Collapsing any two of those three is how every defect in this family happens. The corollary is the title of this post. A guard that has never failed may be incapable of failing, and the only way to know is to give it something it must fail on.\nSo here is the claim, and a stranger can check it one repository at a time:\nevery one of the ten commits at least one test that would report the package's own premise wrong.\nNot a test that the tool works; a test that the naive alternative fails where the tool succeeds, or that fixtures can force the tool's verdict in both directions. If any repository below lacks such a test, this post is wrong about it. Two terms recur, because the family shares its probing machinery: a ladder is a fixed list of probe inputs a tool walks in order, and a witness is an input that produced two different answers. And assay-checks appears twice, because it carries two unrelated questions in one binary and each needs its own control, so the table has eleven rows for ten packages.\npackage\nthe way a check lies\nthe committed control\nassay-checks\n, a duplicate-function finder that executes code\ntwo functions maintained separately answer identically\npairs come from executed outcome vectors, never names, and functions that genuinely differ must not group\nnondet\n, a determinism prober\nan in-process repeat calls a function deterministic when hash order only moves between processes\nthe in-process check must find nothing in 20 calls while fresh processes find the witness\nassay's\nrunners\nhalf\n, an auditor of mutation harnesses\nno failures reported and no test executed look identical; a crash scored as a catch\nseven properties, each shipped as a mutation its own runner must catch\nrestore-verified\n, a guard for tools that break files on purpose\na restore that ran is read as a restore that worked\na control asserts\ntry/finally\nreally does leave the tree broken on SIGTERM\ndidrun\n, a command wrapper demanding evidence of work\nexit 0 is read as evidence that something happened\n0 passed\nmatching the expected pattern must still score as did-not-run\ncanfail\n, a breaker of guarded files\na CI guard is green because it cannot go red\nthe example config must yield a catch, a blind guard and two refusals in one run, and CI asserts that tally line\nundetermined\n, a curve fitter that can refuse\na constant is fitted to a drift and reported without the error bar that decided it\nthe demo's second observable must come back UNDETERMINED while the first does not\nzerocase\n, a reader of the reports runners already write\na zero denominator reports clean\na full report and an empty one from the same command shape must get opposite verdicts\ncountfn\n, an operation counter for scaling questions\na complexity class is named because a curve fit looks close\nthree functions must produce three answers at once: n², log n, and a refusal\nladderpin\n, a freezer of measured behaviour\nbehaviour drifts under tests that stay green; a flaky pin blames the pinning tool\nwith the determinism gate off, a pin on an unchanged tree must go red\nlexindex\n, a code completion engine over repository statistics\na completion accuracy is quoted with no baseline beside it\nthe shipped harness must exit 2 when its scorer was never observed producing both a hit and a miss\nThree of the controls are worth reading in full, because each one tests the reason its package exists rather than any feature of it.\ntest_in_process_repetition_would_have_missed_it\nasserts that calling a function 20 times in one interpreter finds no variation, and that\nnondet\nfinds it anyway in fresh processes, where Python's per-interpreter hash randomisation actually varies. If in-process repetition ever catches it, fresh processes are expensive theatre and the test says so.\ntest_SIGTERM_leaves_a_try_finally_harness_broken\nspawns a real child running an ordinary\ntry/finally\nharness, kills it, and asserts the mutated file stayed mutated.\nrestore-verified\nexists because\nfinally\ndoes not run on SIGTERM, and this is the test that would report that premise obsolete. And\ntest_without_the_gate_the_same_function_is_pinned_and_the_pin_is_flaky\nturns off\nladderpin\n's determinism gate, pins a function whose output depends on hash order, changes not one byte of the tree, and asserts the next check reports a change that nobody made.\nThe tools were then turned on themselves, and the results that mattered were the failures. Seven of the ten READMEs report a deliberate-mutation pass over their own source, from five mutations in\nrestore-verified\nto 193 in\nassay\n, and the survivors were worth more than the catches. In\ncanfail\n, the mutation that re-enabled mtime restoration survived, which proved that an earlier fix did nothing. mtime invalidation has one-second granularity, and a tool that edits, runs and restores in milliseconds defeats it whichever way the flag is set; the load-bearing guard is\nPYTHONDONTWRITEBYTECODE\n, and the README had to be corrected. In\nzerocase\n, two mutations survived their first run. One exposed a branch reachable only through\n--allow-stale\nthat nothing exercised. The other exposed that no fixture held a test case carrying both a skip and a failure, a shape real runners emit. Two packages also hit the same third category: a mutation that cannot change behaviour at all still scores as survived and reads as a test gap. A mutation suite can lie in the flattering direction as well as the other one.\nBuilding the packages against each other found defects none of them found alone.\ncountfn\nis built on\nundetermined\n, and its parity suite compares the rendered report between the Python and JavaScript halves. On the first run after that comparison was added it caught the two halves of\nundetermined\nformatting any integer of one million or more differently,\ntruth=1e+06\nagainst\ntruth=1000000\n.\nundetermined\n's own parity suite asserts those strings agree, and had passed forever, because its test ladder tops out at 512. The checker's checker needed a checker, and the general lesson is the one\nthe mutation-testing post\nreached from the other side: an instrument is only as trustworthy as the thing that has been shown to break it.\nThe same discipline shaped the dependency graph.\ncanfail\noriginally carried 78 lines of inline restore logic, about a quarter of the module, so that it could claim no dependencies. That was right while\nrestore-verified\nwas unpublished and wrong afterwards: a second copy of a guarantee is a second thing to get wrong, and the copy is the one that does not get the upstream's tests.\nundetermined\nwent the other way and rejected a proposed dependency on\nnondet\n, since its observables are closures with no file-and-name address for\nnondet\nto probe. A dependency that looks like a guarantee and is not is worse than no dependency.\nThe cost of all this refusing is coverage, and the numbers are small and printed rather than hidden.\nnondet\n, pointed at a real 283-function tree, probed 127 of them with the safety gate on when that census was taken, and found 2 genuinely nondeterministic.\nassay\n's census on a 41-function tree probed 9, and\nladderpin\ncan only pin what\nassay\nprobes.\nlexindex\n, the one package here that is a product rather than an instrument, states the condition under which it is worth installing as a single number, the recital rate (how often a short context in a held-out file already appears in the index), which ran from 13.5% to 72.9% across the nine corpora measured. Its harness reports its own nulls instead of suppressing them. Every census (each tool's printed count of what it refused and why) names its refusals, because a report that stays quiet about what was never examined is reporting we never looked as we found none.\nWhat generalises is the asymmetry the whole family is built around. A witness is a fact: an input and two different answers to it, a file that did not come back, a zero where a denominator belongs. Agreement is only the absence of a witness, and a check that has never disagreed with anything has not yet demonstrated that it can. Before trusting any green light, mine included, the question worth a minute of anyone's time is the one in the title: what, concretely, would make this check fail, and has that ever been watched happening?", "metadata": { "title": "Ten Packages, One Rule: A Check Must Be Able to Fail - DEV Community", "description": "Eight of the ten went up in one day, extracted from a year of measurement work in which the thing lying was usually the instrument, not the code. Each package encodes one way a green check can mean nothing, and each commits a named test that would report its own premise wrong. Tagged with verification, measurement, python, node.", "keywords": "verification, measurement, python, node, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/megapixel99/ten-packages-one-rule-a-check-must-be-able-to-fail-60f", "og:title": "Ten Packages, One Rule: A Check Must Be Able to Fail", "og:description": "Eight of the ten went up in one day, extracted from a year of measurement work in which the thing lying was usually the instrument, not the code. Each package encodes one way a green check can mean nothing, and each commits a named test that would report its own premise wrong.", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgaefj5hrl2cgb8u7kjk3.png", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "Ten Packages, One Rule: A Check Must Be Able to Fail", "twitter:description": "Eight of the ten went up in one day, extracted from a year of measurement work in which the thing lying was usually the instrument, not the code. Each package encodes one way a green check can mean nothing, and each commits a named test that would report its own premise wrong.", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgaefj5hrl2cgb8u7kjk3.png" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://sethwheeler.dev/blog/wrong-process/", "label": "A memory profiler that watched the wrong process for eight months", "type": "a" }, { "href": "https://sethwheeler.dev/blog/limiter-self-report/", "label": "a rate limiter that reported requests it never sent", "type": "a" }, { "href": "https://github.com/Megapixel99/didrun", "label": "didrun", "type": "a" }, { "href": "https://github.com/Megapixel99/assay-checks", "label": "assay-checks", "type": "a" }, { "href": "https://github.com/Megapixel99/nondet", "label": "nondet", "type": "a" }, { "href": "https://github.com/Megapixel99/restore-verified", "label": "restore-verified", "type": "a" }, { "href": "https://github.com/Megapixel99/canfail", "label": "canfail", "type": "a" }, { "href": "https://github.com/Megapixel99/undetermined", "label": "undetermined", "type": "a" }, { "href": "https://github.com/Megapixel99/zerocase", "label": "zerocase", "type": "a" }, { "href": "https://github.com/Megapixel99/countfn", "label": "countfn", "type": "a" }, { "href": "https://github.com/Megapixel99/ladderpin", "label": "ladderpin", "type": "a" }, { "href": "https://github.com/Megapixel99/lexindex", "label": "lexindex", "type": "a" }, { "href": "https://github.com/Megapixel99/nondet/blob/master/tests/test_nondet.py", "label": "test_in_process_repetition_would_have_missed_it", "type": "a" }, { "href": "https://github.com/Megapixel99/restore-verified/blob/master/python/tests/test_guard.py", "label": "test_SIGTERM_leaves_a_try_finally_harness_broken", "type": "a" }, { "href": "https://github.com/Megapixel99/ladderpin/blob/master/tests/test_ladderpin.py", "label": "test_without_the_gate_the_same_function_is_pinned_and_the_pin_is_flaky", "type": "a" }, { "href": "https://sethwheeler.dev/blog/mutation-testing-timeouts/", "label": "the mutation-testing post", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:48:38Z", "source_url": "https://dev.to/megapixel99/ten-packages-one-rule-a-check-must-be-able-to-fail-60f", "papers": [], "actual_video_files": [], "body_chars": 9878, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/e88f4986c330/pages/source.html", "local_html": "attachments/e88f4986c330/pages/article.html", "body_base_path": "attachments/e88f4986c330/pages/", "body_markdown_path": "attachments/e88f4986c330/pages/article.md" }