{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Seth Wheeler",
  "title_listed": "Breaking CI Guards on Purpose to Prove They Can Fail",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/megapixel99/breaking-ci-guards-on-purpose-to-prove-they-can-fail-271k",
  "publish_date": "2026-09-24",
  "corpus_stage": "",
  "notes": "",
  "id": "d051245d8216",
  "collected_at": "2026-10-02T13:51:43",
  "title": "Breaking CI Guards on Purpose to Prove They Can Fail - DEV Community",
  "body_markdown": "Code:\nMegapixel99/canfail\nA CI guard that has never failed may be incapable of failing. A lint rule disabled by a config merge, a type check whose glob stopped matching, a schema validation step pointed at the wrong directory, a security scanner with an empty ruleset. Every one of them is green forever, and green is what you were looking for. For code guarded by tests, mutation tools like Stryker generate the breaks automatically and you should use them.\ncanfail\nis for the guards they do not cover, where the thing being guarded is YAML, Terraform, a Dockerfile, or anything else without a function to mutate. You declare a break (a file, an anchor string, its replacement, and what failure you expect). The tool applies it, runs your check, and reports whether the check noticed.\npip\ninstall\ncanfail\ncanfail canfail.json\nEnter fullscreen mode\nExit fullscreen mode\n{\n\"checks\"\n:\n[{\n\"name\"\n:\n\"unit tests\"\n,\n\"run\"\n:\n[\n\"python3\"\n,\n\"example/test_prices.py\"\n,\n\"-q\"\n],\n\"breaks\"\n:\n[{\n\"name\"\n:\n\"total: multiply -> add\"\n,\n\"file\"\n:\n\"example/src/prices.py\"\n,\n\"replace\"\n:\n\"item[\n\\\"\nprice\n\\\"\n] * item[\n\\\"\nquantity\n\\\"\n]\"\n,\n\"with\"\n:\n\"item[\n\\\"\nprice\n\\\"\n] + item[\n\\\"\nquantity\n\\\"\n]\"\n,\n\"expect\"\n:\n\"assert|Error\"\n}]\n}]\n}\nEnter fullscreen mode\nExit fullscreen mode\nIt is a mutation harness, a very small one aimed at CI configuration, so it inherits the rules that make one trustworthy. The check must pass on the clean tree first; a check that was already red tells you nothing when you break something. A failure is not a catch: it has to fail for the reason you named. A break that just makes the file unparseable is scored\nwrong-failure\n, and a check that hit its timeout exits 124, which looks exactly like going red and settles nothing. The anchor must match exactly once. Zero matches means the break never happened, and two means the check was asked about code nobody was thinking of. And the file must come back, verified by digest, which is\nrestore-verified\n's whole job.\ncanfail\nused to carry 78 lines of that guard inline, about a quarter of the module, so it could claim no dependencies. Deleting the copy once the real package existed was the right trade: a second copy of a guarantee is a second thing to get wrong, and the copy is the one that never gets the upstream's tests.\nThe example configuration in the repository declares four breaks against one check and yields four different outcomes in a single run, closing with the tally\n4 declared break(s): 1 caught, 1 not caught, 2 not settled\n. CI asserts that tally line rather than just the exit code, because exit 1 alone would be satisfied by finding the wrong thing.\nThe paragraph I most want to keep is about a fix of mine that measurement killed. The first working version reported a genuinely blind guard as catching, but only when its break ran second. After the first break, Python had written\n__pycache__\nbytecode from the broken source, and the second break's run executed that stale bytecode, failing for the previous break's reason. My fix was to stop restoring mtime on the guarded file. Then\ncanfail\ngot the treatment it gives other people's guards, deliberate breaks to its own source with its suite as the check, and that mutation pass showed the fix does nothing. Re-enabling bytecode caching breaks the test that pins the ordering bug whether or not mtime is restored. mtime invalidation has one-second granularity, this tool edits, runs and restores in milliseconds, and a\n.pyc\nwritten from the broken source therefore looks fresh either way.\nPYTHONDONTWRITEBYTECODE\nis the load-bearing guard; not restoring mtime is a cheap belt beside it. The general form is worth carrying to any tool with a fast edit cycle: anything keyed on mtime (bytecode caches, make, ninja, file watchers) is blind on sub-second edits.\nblind\nalso means two different things unless you make it say which: the check ran and did not notice, or the check never ran at all. The first says your guard is weak and the second says it is missing. They send you to opposite ends of the CI file. Declaring\nevidence\non a check hands that question to\ndidrun\n, so a break that stops the check from running becomes a refusal rather than a finding. Omit it and the report says outright that it cannot tell the two apart. An\nevidence\nobject naming no known predicate is a config error rather than a no-op. A misspelled key silently downgrading the check is this package's own failure mode pointed at itself.\nThat self-directed suspicion earned its keep. The current README carries a section titled \"Things this got wrong about itself\", and every entry is a verdict the tool reported without having earned it. A killed check scored as a catch, because exit 124 is non-zero. A\nwrote\nevidence path resolved in the tool's directory while the check ran in\n--cwd\n, so a check that wrote its report perfectly was reported as never having run. An uncompilable\nexpect\nregex reached the interpreter as a traceback and exited 1, the code that means a guard is blind. A break applied to a CRLF file came back as a whole-file diff, because universal-newline reading stripped the returns. Each is now pinned by a test that fails without the fix.\nThe score so far is six mutations against the original properties: five caught, and the sixth (restoring mtime) survived, which is how the paragraph above got corrected. Six more were applied against the later fixes, and all six were caught. The adoption cost is honest too: you have to write the breaks yourself. For Python code guarded by a Python suite, the README points you at PyPI's\nmutation-testing\ninstead, which swaps\n__code__\nobjects and never touches the disk at all. Everything\ncanfail\ncarries about restores and signals is apparatus it needs only because it edits real files. That is what buys it the config formats nothing else covers.",
  "metadata": {
    "title": "Breaking CI Guards on Purpose to Prove They Can Fail - DEV Community",
    "description": "A fix I believed in was proved a no-op by the tool auditing itself: mtime invalidation has one-second granularity and the edit cycle is milliseconds, so stale bytecode looks fresh either way. PYTHONDONTWRITEBYTECODE is the guard actually holding; the surviving mutation corrected the README. Tagged with verification, python, measurement.",
    "keywords": "verification, python, measurement, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/megapixel99/breaking-ci-guards-on-purpose-to-prove-they-can-fail-271k",
    "og:title": "Breaking CI Guards on Purpose to Prove They Can Fail",
    "og:description": "A fix I believed in was proved a no-op by the tool auditing itself: mtime invalidation has one-second granularity and the edit cycle is milliseconds, so stale bytecode looks fresh either way. PYTHONDONTWRITEBYTECODE is the guard actually holding; the surviving mutation corrected the README.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F44xyga3vmbyx25m8ciwo.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Breaking CI Guards on Purpose to Prove They Can Fail",
    "twitter:description": "A fix I believed in was proved a no-op by the tool auditing itself: mtime invalidation has one-second granularity and the edit cycle is milliseconds, so stale bytecode looks fresh either way. PYTHONDONTWRITEBYTECODE is the guard actually holding; the surviving mutation corrected the README.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F44xyga3vmbyx25m8ciwo.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/d051245d8216/code/body-000.txt",
      "chars": 41,
      "language": "text"
    },
    {
      "file": "attachments/d051245d8216/code/body-001.txt",
      "chars": 347,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/Megapixel99/canfail",
      "label": "Megapixel99/canfail",
      "type": "a"
    },
    {
      "href": "https://sethwheeler.dev/blog/restore-verified/",
      "label": "restore-verified",
      "type": "a"
    },
    {
      "href": "https://sethwheeler.dev/blog/didrun/",
      "label": "didrun",
      "type": "a"
    },
    {
      "href": "https://pypi.org/project/mutation-testing/",
      "label": "mutation-testing",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:37Z",
  "source_url": "https://dev.to/megapixel99/breaking-ci-guards-on-purpose-to-prove-they-can-fail-271k",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 5811,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/d051245d8216/pages/source.html",
  "local_html": "attachments/d051245d8216/pages/article.html",
  "body_base_path": "attachments/d051245d8216/pages/",
  "body_markdown_path": "attachments/d051245d8216/pages/article.md"
}