{ "source_file": "0930", "source_platform": "Dev.to", "author": "Seth Wheeler", "title_listed": "When a Test Suite Rejects a Correct Program", "keywords": "", "domain": "大模型算法", "content_type": "工程实践", "is_paid_flag": "否", "url": "https://dev.to/megapixel99/when-a-test-suite-rejects-a-correct-program-45n", "publish_date": "2026-09-17", "corpus_stage": "", "notes": "", "id": "063e3c7a1975", "collected_at": "2026-10-02T13:51:47", "title": "When a Test Suite Rejects a Correct Program - DEV Community", "body_markdown": "I started with a narrow question: could something without a GPU replace a language model for writing small programs? The answer arrived by elimination, and then the question turned into a different one, because the part I assumed was weak turned out to be fine and the part I had never measured turned out to be broken.\nThe setup is twenty-one small operations and a property oracle for each, written before any implementation existed. The operations are what an exercise list holds: reverse a string, build a minimum spanning tree, validate a card number with the\nLuhn checksum\n(double every second digit from the right, sum the digits of the results, and a valid number totals a multiple of ten), print the first eight\nhappy numbers\n(replace a number by the sum of the squares of its digits, repeat, and see whether it reaches 1 or falls into a cycle).\nA property oracle is the part worth being precise about, because everything below turns on it. It is not a stored list of expected outputs, which would only work for an implementation somebody had already run; it is a program that asserts\nproperties\nof whatever it is handed, so it can score an implementation it has never seen. The ones here mix four kinds, and the mix matters later. A\nknown value\n, where the answer is external and settled.\neulerian\nis checked against the actual\nSeven Bridges of Königsberg\n: four nodes, seven edges, every vertex of odd degree, and an answer that has not moved since Euler settled it in 1736. A\nmetamorphic\nproperty, which needs no correct answer of its own: negating a complex number twice returns the original, so the check holds whatever the program computes. A\nstructural\none, where the answer is re-checked rather than its cost: the oracle re-walks a printed spanning tree edge by edge, so a program printing the right total weight beside a set of edges that is not a tree fails. And an\nindependent brute force\nover every spanning tree, on graphs small enough for that to be exhaustive. A second copy of the same algorithm would agree with a wrong first copy about anything the two misunderstood together.\nA blind generator gets the task statement and never sees the oracle. Then the oracle scores it. In the first round of that, the generator passed 9 of 12 operations against frozen suites, where retrieval from a pool of 48 human solutions had managed 2 of 12; a behavioural index did worse still, recognising 0 of 11 operations even when relevance was guaranteed by construction. So the generator was never the scarce thing.\nWhat stayed scarce was the residue. Across four independent generation runs,\n11 of 21 operations never passed in any run\n, and the failing checks were identical run to run for nine of the eleven. I wrote that up twice as capability: graph algorithms are genuinely harder than reversing a string, so of course\nmst\nand\neulerian\nsat in the residue.\nThat was an inference from\nwhich\noperations failed, and it was wrong.\nThe code lives in a private research repo, so there is no link to it here, and nothing below names a file inside it. Every figure comes from a recorded run rather than from memory.\nA wrapper that only re-expresses stdout\nThe test that settled it is blunt. I wrapped each candidate program in a shim that runs it, rewrites its standard output into the notation its oracle turned out to demand, and hands that to the unmodified oracle. The shim inherits stdin, so it never reads the input, it opens no files, and it computes nothing about the problem. Every transform in it is a rename, a token map, or a re-serialisation of a value the program had already printed.\nIf a check goes from failing to passing under that wrapper, the program had computed the answer and the contract could not carry it.\nFor the five deepest failures in the residue, across all four runs:\noperation\nrun1\nrun2\nrun3\nrun4\ncomplex_algebra\n3/9 →\n9/9\n3/9 →\n9/9\n3/9 →\n9/9\n3/9 →\n9/9\nconnected\n2/7 →\n7/7\n2/7 →\n7/7\n2/7 →\n7/7\n2/7 →\n7/7\neulerian\n6/11 →\n11/11\n4/11 →\n11/11\n6/11 →\n11/11\n6/11 →\n11/11\nmst\n4/9 →\n9/9\n4/9 →\n9/9\n4/9 →\n9/9\n4/9 →\n9/9\ngraph_from_links\n6/9 → 8/9\n6/9 → 8/9\n4/9 → 4/9\n4/9 → 4/9\ntotal\n21/45 →\n44/45\n19/45 →\n44/45\n19/45 →\n40/45\n19/45 →\n40/45\n78 of 180 checks passing became 168 of 180.\n90 failures repaired, 12 still failing, and none broken by a wrapper.\nThe remaining two operations in the residue's middle band went the same way: all 16 of their failing checks repaired without touching a line of any program, 12 by re-formatting and 4 by lifting an interpreter limit I will come back to.\nEach operation failed for its own reason, and every reason is the same kind of thing.\nconnected\npublishes the key. The oracle demands\nconnected=yes\n; all four generations printed\nconnected=true\n. The vocabulary was never stated, and the only value check that passes is the one reading a different key.\neulerian\ndoes not publish the key at all. The oracle scrapes a\npath=\nprefix and splits on whitespace; the task statement named\nodd_degree\nand\nresult\n. Across four generations the trail arrives as\ntrail=A,B,C,D,A\n,\nroute=A->B->C->D->A\nand\ntour=A D C B A\n: four key names and three separators, none published, every trail correct. One run also answers Königsberg correctly and calls it\nresult=impossible\nwhere the oracle reads\nnone\n.\nmst\npublishes the key and not its type. The oracle parses\nedges\nas a comma-separated\nA-B\nlist; every generation used it for a count and put the tree under\ntree=\nor\nmst_edges=\n. The same key means a\ncount\nin another operation's oracle in the same suite, so the contract is not merely silent, it is internally inconsistent. The four checks that pass are the ones reading\nweight\n, which includes an independent brute-force optimality check: the oracle could already prove the tree was minimal while being unable to read it.\nThat spread across four independent runs is the part I would not have got from one. One program using an unpublished notation is a coincidence; four programs inventing four different notations for the same unpublished value is what an underspecified contract does.\nThe complexity check that field order decides\nThe sharpest case is\nfast_expt\n, whose task is to output\na^b\nin O(lg n) time. Two of its six checks failed in every run, and both are complexity claims, which is a different kind of thing from a notation: if a program computes the right answer by naive repeated multiplication, no re-expression of its output can make the count logarithmic. I predicted, in writing and in advance, that at least one of those two would survive the wrapper for exactly that reason.\nIt survives. The reason has nothing to do with complexity.\nThe task statement published\nvalue\nas the only key the checker reads. The oracle reads\nmults\n. All four generations print their multiplication count under\nsteps=\nor\nmultiplications=\n, and every count is comfortably inside the bound (5, 7 and 10 to 15, against bounds of 11, 15 and 21). Renaming the key repairs one of the two checks in all four runs.\nThe other check also probes\n3**65536\n. That number is\n31,269 decimal digits\n, and CPython refuses to render an integer past\n4,300\ndigits by default, a limit added as a\ndenial-of-service mitigation\nand configurable through\nsys.set_int_max_str_digits\n. Every generation prints\nvalue\nbefore its count:\na=2 b=10 value=1024 steps=4\nbase=2 exponent=10 value=1024 multiplications=5\nEnter fullscreen mode\nExit fullscreen mode\nSo every generation computes the right answer with correct square-and-multiply and then dies inside\nprint\n, and the count the check wants is never emitted. Setting\nPYTHONINTMAXSTRDIGITS=0\nand changing no line of any program, one of them emits\nsteps=17\nagainst a bound of 35.\nThe consequence is worse than the failure. A program printing its count\nbefore\nits value would pass this check while crashing in exactly the same place, and would pass it whatever its algorithm did, because the count is read and the crash is ignored.\nThe one check in this registry that tests a complexity bound is decided by the order of fields in the output.\nIt is also interpreter-dependent: on CPython 3.10.6 that row never fails, which means the pass rates I published are tied to the interpreter they ran on.\nCounting the whole residue, 11 of 21 operations failed on the specification and not one on capability. Two of the twelve unrepairable checks are still the specification rather than the program: every generation collapses two identical input links into one edge, because multigraph support was never published, and two generations refuse the oracle's own weighted\nA B 4\nline as malformed, because the task required a malformed-line refusal without ever publishing the input grammar that refusal is the complement of.\nSo I built a linter, and it did not survive contact\nIf a checker demands things it never publishes, and those demands are visible in the checker's source, then the fix is mechanical: read each oracle and emit the contract it actually enforces. Keys, value shapes, vocabularies, the separator a list is split on, the largest magnitude probed. Publish shape and never a value bound to an input, so an alphabetic token like\nyes\nis fair game (which one is correct still depends on the input) while anything carrying a digit appears only as its shape.\nThat works well enough to be worth showing. As a detector for \"this operation's contract is missing something load-bearing,\" measured against which operations actually failed, it scored\n91% precision and 62% recall\non the twenty-one: ten true positives, one false positive, six missed.\nThen I wrote down why that number could not be trusted, which is the part I would want a reader to check hardest. The taxonomy of contract defects was derived by examining those same operations, so 91% is a fit statistic. Worse, the discovery curve had not flattened: the first round of four operations found four defect classes, the second round of five found seven more, and the third round of two still found one new class. Twelve classes from eleven operations, with a new one in the final round, means recall on a suite the taxonomy was not fitted to is unknown and probably much lower.\nThere was no holdout in the repo to settle it. The other oracle suites there check HTTP routes and database state, so a stdout-contract linter cannot apply; the operations never examined are all single-value positional ones that cannot exercise the classes that matter; and per-check ground truth exists only for the same twenty-one.\nOne thing worth saying before that number gets any weight: the\nprompt\nside of this is already well studied, and there is a literature on\nambiguous, contradictory and incomplete task descriptions\nin exactly these benchmarks. What I had been looking at is the other side of the same seam, where the task statement is fine and the checker is the thing demanding something unstated.\nAn external one exists, and it is somebody else's work. Datacurve's\nDeepSWE audit\nran ten agent configurations three times over thirty sampled tasks from SWE-Bench Pro, had an independent judge review every rollout against the task definition, reference solution and verifier output, and published the labels. Their finding is that the verifier\nrejected 24% of correct solutions\nwhile accepting 8.5% of wrong ones, and their headline example is the defect class above in another costume: a test suite carrying an import that never appeared in the task prompt, so a functionally identical patch failed to compile and was rejected.\nTheir labels give 14 tasks with at least one checker-side false negative and 10 with only genuine failures. Before running anything I wrote down the number that decides the experiment, because it is the easiest one to leave out of a favourable table: 14 of 24 tasks are positive, so\na predictor that flags every task scores 58% precision at 100% recall.\nBeating that by ten points at 50% recall or better was the bar.\nAgainst the published spec for each task (the problem statement plus the\nrequirements\nand\ninterface\nfields that\nSWE-bench Pro\nalready ships, since diffing against the prose alone would measure a gap its authors have closed), and demands taken from the added lines of each test patch:\npredictor\nprecision\nrecall\nagainst baseline\nflag every task\n58%\n100%\nn/a\nany identifier absent from the spec\n58%\n100%\n+0 points\nnewly imported symbols only\n50%\n36%\n−8 points\nnewly called public symbols only\n53%\n57%\n−5 points\nimports or called symbols\n53%\n64%\n−5 points\nNothing clears the bar, and two of the three restrictions score below chance. The naive predictor landing\nexactly\non the base rate is the result in one number: in a real repository every test patch names fixtures, helpers and internal APIs that no prose statement mentions, so \"something here is unstated\" is true of all 24 tasks and separates none of them.\nThe single thing that explains the gap between 91% and 58% is what the linter was allowed to assume. On the twenty-one operations it could read\nkv()\n, the one function every oracle used to parse output, and learn that values are split on whitespace, which is precisely what made the\nluhn\ndefect statable (a correct\nvendor=American Express\ncannot survive that parser, so the right answer is unrepresentable in the published format). Twenty-four repositories in three languages have no single parser to read. An earlier round had already noticed that all 34 oracle functions in the repo take exactly one program path, and that stereotypy is what the 91% was measuring.\nA per-harness implementation, one that parses each repository's test framework and knows its assertion conventions, is not ruled out by this. It is also exactly the cost that made a general tool doubtful in the first place, and there is now no cheap-version evidence to justify paying it.\nWhat generalises\nThe claim I would carry elsewhere is not \"check your test suites,\" which is advice rather than a finding. It is narrower and it has a mechanism.\nA pass rate against a fixed checker measures the checker's format discipline and the generator's capability together, and nothing in the number says which.\nFour independent programs solved\neulerian\ncorrectly and all four scored between 5 and 7 of 11, entirely on where they put the answer. Had I stopped at the pass rate, the write-up would have said graph algorithms are hard, which is both plausible and false here.\nThe instrument that made it visible is worth stating too, because it is cheap and it is not a language model:\nlet something generate blind, then repair only the notation and re-score.\nA person writing to an underspecified contract silently patches it from context and the defect never surfaces; a blind generator does exactly what it is told, which is what makes the gap between what a checker demands and what it publishes measurable at all.\nTwo things I could not settle. Whether a derived contract actually raises a blind pass rate is unanswerable by me, because it needs a session that has never read these oracles, and having spent two rounds reading them line by line I am no longer that; anything I generated now would be teaching to the test. And the whole series rests on twenty-one operations of a size where the hard case is a minimum spanning tree over six nodes, with one model. The failure mode it found is real and independently corroborated at a much larger scale by the audit above. The rate at which it occurs, on anything bigger than this, I have not measured.", "metadata": { "title": "When a Test Suite Rejects a Correct Program - DEV Community", "description": "Eleven of twenty-one operations never passed my frozen checks, and I wrote that up twice as a limit of the generator. A wrapper that only re-expressed each program's own stdout repaired 90 of 102 failures, changing no algorithm. Tagged with verification, codegeneration, measurement, llm.", "keywords": "verification, codegeneration, measurement, llm, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/megapixel99/when-a-test-suite-rejects-a-correct-program-45n", "og:title": "When a Test Suite Rejects a Correct Program", "og:description": "Eleven of twenty-one operations never passed my frozen checks, and I wrote that up twice as a limit of the generator. A wrapper that only re-expressed each program's own stdout repaired 90 of 102 failures, changing no algorithm.", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftuli57siogxr2115iqhl.png", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "When a Test Suite Rejects a Correct Program", "twitter:description": "Eleven of twenty-one operations never passed my frozen checks, and I wrote that up twice as a limit of the generator. A wrapper that only re-expressed each program's own stdout repaired 90 of 102 failures, changing no algorithm.", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftuli57siogxr2115iqhl.png" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/063e3c7a1975/code/body-000.txt", "chars": 88, "language": "text" } ], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://en.wikipedia.org/wiki/Luhn_algorithm", "label": "Luhn checksum", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Happy_number", "label": "happy numbers", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Seven_Bridges_of_K%C3%B6nigsberg", "label": "Seven Bridges of Königsberg", "type": "a" }, { "href": "https://docs.python.org/3/library/stdtypes.html", "label": "denial-of-service mitigation", "type": "a" }, { "href": "https://docs.python.org/3/library/sys.html", "label": "sys.set_int_max_str_digits", "type": "a" }, { "href": "https://arxiv.org/abs/2507.20439", "label": "ambiguous, contradictory and incomplete task descriptions", "type": "a" }, { "href": "https://deepswe.datacurve.ai/blog/deepswe", "label": "DeepSWE audit", "type": "a" }, { "href": "https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro", "label": "SWE-bench Pro", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:48:39Z", "source_url": "https://dev.to/megapixel99/when-a-test-suite-rejects-a-correct-program-45n", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/2507.20439", "bytes": 1675501, "reused": false, "local_path": "papers/repair/910ea6c504b78e6beab6.pdf" } ], "actual_video_files": [], "body_chars": 15405, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/063e3c7a1975/pages/source.html", "local_html": "attachments/063e3c7a1975/pages/article.html", "body_base_path": "attachments/063e3c7a1975/pages/", "body_markdown_path": "attachments/063e3c7a1975/pages/article.md" }