{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Debashish Ghosal",
  "title_listed": "1,558 Tests Green and No Auth: The Tests That Never Actually Ran",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/debashish_ghosal/1558-tests-green-and-no-auth-the-tests-that-never-actually-ran-nkk",
  "publish_date": "2026-09-19",
  "corpus_stage": "",
  "notes": "",
  "id": "586a26e3a809",
  "collected_at": "2026-10-02T14:05:58",
  "title": "1,558 Tests Green and No Auth: The Tests That Never Actually Ran - DEV Community",
  "body_markdown": "A test named\ntest_all_adapters_importable\nasserted nothing. It would pass forever, even if every adapter was broken.\n57 of 65 assertion files were in the wrong format, and the harness returned\n0 / 0\nwithout a whisper.\nIf a green suite makes you relax, this post is going to un-relax you.\nThree Times a Green Suite Hid a Real Failure\n1. The test gutted to\npass\n.\nIn\nplanner-critic-engine\n, a test literally had a\npass\nbody:\ndef\ntest_all_adapters_importable\n():\n# if it imported, it's fine — except this proves nothing\npass\nEnter fullscreen mode\nExit fullscreen mode\nIt was caught in code review, not by CI, and only before the LLM sweep because a human read it. Issue #236.\n2. The harness that returned\n0 / 0\nand called it green.\nIn the same repo, 57 of 65 assertion files were in the wrong format. The harness parsed them, found zero assertions to run, and returned\n0 / 0\n— which it treated as success. The suite was green because it had executed nothing.\n3. The auth guard that never ran.\nIn\nCauterRule\nv0.3.0 the suite reported\n1,558 tests green\n. The MCP HTTP bearer-auth guard never ran, because an import was swallowed. Every unit test passed. The only reason we found the unauthenticated store read was a\nDocker field test\nrunning the real thing in a real container.\nThe Dangerous Thing Wasn't the Bug\nIn all three cases the bug mattered, but it wasn't the scariest part. The scariest part was the\nconfidence\n.\nWe treat \"tests pass\" as evidence. Sometimes it's evidence that the harness silently skipped a module, or that an import was swallowed, or that a fixture file was parsed into an empty set. The suite stays green and the system ships with an unauthenticated read path.\nThe Fix Is a Meta-Test, Not More Tests\nThe fix is not to write more tests. It's to assert that your tests actually ran and actually asserted.\ndef\ntest_suite_is_not_empty\n():\nresults\n=\nrun_assertion_files\n(\n\"\ntests/assertions/\n\"\n)\nassert\nresults\n.\nexecuted\n>\n0\n,\n\"\nharness ran zero assertions\n\"\nassert\nresults\n.\nassertions\n>\n0\n,\n\"\nassertions parsed to empty set\n\"\nEnter fullscreen mode\nExit fullscreen mode\nTwo rules fall out of this:\nCI must fail when a module produces zero results.\n0 / 0\nis an error state, not a pass.\nDistinguish \"did not run\" from \"ran and passed.\"\nA swallowed import and a skipped module should be loud failures, not silence.\nThat single distinction is the difference between a flaky gate and a strict one.\nThe Honest Limitation\nMeta-tests add process, and process can rot — a meta-test that stops checking is just another green checkmark. And no amount of test discipline catches the false negatives you never thought to test for. This reduces the class of \"green but broken.\" It does not eliminate it.\nBut it does close the worst category: the test that never ran and told you everything was fine.\nWhat's the last green build you caught lying to you? I now trust a green suite about as far as I can read its raw output.\nRepos and receipts:\nCauterRule\n·\nCauterRule v0.3.0 field test report\n·\nPlannerCritic failure-mode register\n·\nagent-tooltrust design decisions\n— all MIT, all public.",
  "metadata": {
    "title": "1,558 Tests Green and No Auth: The Tests That Never Actually Ran - DEV Community",
    "description": "A test named test_all_adapters_importable asserted nothing. It would pass forever, even if every... Tagged with ai, security, programming, testing.",
    "keywords": "ai, security, programming, testing, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/debashish_ghosal/1558-tests-green-and-no-auth-the-tests-that-never-actually-ran-nkk",
    "og:title": "1,558 Tests Green and No Auth: The Tests That Never Actually Ran",
    "og:description": "A test named test_all_adapters_importable asserted nothing. It would pass forever, even if every...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fuwekohcmpj1j6sr0opj5.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "1,558 Tests Green and No Auth: The Tests That Never Actually Ran",
    "twitter:description": "A test named test_all_adapters_importable asserted nothing. It would pass forever, even if every...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fuwekohcmpj1j6sr0opj5.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/586a26e3a809/code/body-000.txt",
      "chars": 106,
      "language": "text"
    },
    {
      "file": "attachments/586a26e3a809/code/body-001.txt",
      "chars": 217,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/deghosal-2026/planner-critic-engine",
      "label": "planner-critic-engine",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule",
      "label": "CauterRule",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/FIELD_TEST_REPORT.md",
      "label": "CauterRule v0.3.0 field test report",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/planner-critic-engine/blob/main/docs/reference/failure-modes.md",
      "label": "PlannerCritic failure-mode register",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/agent-tooltrust/blob/main/docs/design/design-decisions.md",
      "label": "agent-tooltrust design decisions",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:12Z",
  "source_url": "https://dev.to/debashish_ghosal/1558-tests-green-and-no-auth-the-tests-that-never-actually-ran-nkk",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 3092,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/586a26e3a809/pages/source.html",
  "local_html": "attachments/586a26e3a809/pages/article.html",
  "body_base_path": "attachments/586a26e3a809/pages/",
  "body_markdown_path": "attachments/586a26e3a809/pages/article.md"
}