{ "source_file": "0930", "source_platform": "Dev.to", "author": "Seth Wheeler", "title_listed": "Searching for a Program Instead of Generating One", "keywords": "", "domain": "文本处理", "content_type": "工程实践", "is_paid_flag": "否", "url": "https://dev.to/megapixel99/searching-for-a-program-instead-of-generating-one-20gc", "publish_date": "2026-09-18", "corpus_stage": "", "notes": "", "id": "1e91c9a253a7", "collected_at": "2026-10-02T13:51:46", "title": "Searching for a Program Instead of Generating One - DEV Community", "body_markdown": "The question I started with was whether you could get small programs written without a language model and without a GPU. Not as a matter of principle, just as a matter of cost: the tasks are small, the answers are short, and it seemed likely that something cheaper would cover a useful fraction of them.\nTwo candidates are obvious enough that anyone would name them. Search the space of programs and keep the one that matches your examples, which is\nprogram synthesis\nin its oldest form. Or find code somebody has already written and check whether it does what you want. I priced both, and both closed, and in each case the wall was somewhere other than where I expected to find it.\nThe code lives in a private research repo, so there is no link to it here, and nothing below names a file inside it. Every figure comes from a recorded run rather than from memory.\nCandidate one: search the program space\nThe setup is an exercise list,\nkaran/Projects\n, which is a well-known list of practice problems and a fair source of tasks nobody wrote for me. Thirty-eight entries from three complete sections, of which\n28 could be specified as a function from a string to a string\nand became the corpus. The searcher is a typed enumerator over 62 primitives, budgeted at 300,000 candidate terms and 20 seconds per entry, with a maximum term size of 7. Everything about it was frozen in advance, before a single primitive existed: the corpus, the split, the admissibility rules, the budget, and five predictions.\nSplit by position parity into calibration and held-out halves, so the entries I could look at while building primitives were not the entries it was scored on. The blind arm found terms consistent with the given examples for\n6 of 28 entries, and 4 of those were right on held-out inputs\n. Giving the searcher the entry's title as a hint (a bag of words to bias primitive selection) took it to 7 and 5.\nRead across the split rather than down it, and the shape is clearer. On calibration the blind arm got 4 example-consistent and 3 right; on held-out entries it got\n2 and 1\n. The overfit rate, meaning terms that satisfied every example and still failed on inputs they had not seen, was 0.25 on calibration and\n0.5 on held-out\n.\nThen the finding that ended the arm. All five entries either arm solved already had hand-written implementations in the same repository. Running the existing tool over the same 28 gives 13 transforms, 10 record-storing apps and 5 refusals, so 13 entries covered; the union of that tool with the searcher is also 13.\nThe marginal coverage of a 1,052-line searcher over the code already in the tree is zero entries\n, and its honest ceiling is one:\nFast Exponentiation\n, which the biased arm reaches, and which existed in the repo already and was merely not wired up.\nWhat the examples could not tell it\nThe part worth carrying elsewhere is not the count, it is why the count cannot be improved by searching harder.\nSorting\nis the case. The searcher returns\nsort_(toks(in))\n, which is consistent with all four examples it was given and right on 8 of 12 held-out inputs. It sorts the tokens as text. So\n-3 5 -1 0\ncomes back as\n-1 -3 0 5\nwhere the answer is\n-3 -1 0 5\n, and every input with a negative number or a two-digit number is wrong. The correct term is one primitive away,\nsort_(nums(in))\n, and it is unreachable, because the wrong one is observationally identical on everything the search was shown and banks first.\nCheck if Palindrome\nfails the same way on the empty string.\nTwo of the six example-consistent terms are wrong, and nothing in the examples could have said which.\nThat is not a budget problem and more compute does not touch it. It is a property of specifying a function by examples: the examples admit a family of terms, the search returns whichever member it reaches first, and the ones it reaches first are the short ones. More compute buys wrong answers faster than right ones.\nA generate-and-check loop needs a checker that can reject an observationally-identical impostor, which means a property rather than a set of examples. That is a whole other problem, and it is\nthe one I ended up spending most of the project on\n.\nCandidate two: find the code somebody already wrote\nThe second candidate has a much better reputation, and it is the thing people mean when they ask whether a vector database of code would help.\nThe mechanism I already had is a behavioural index. You take a function, run it against a fixed ladder of inputs, and record what comes back; two functions with the same outcome vector are candidates for being the same function. It works inside one repository, which is what it was built for: it found a helper duplicating another function's logic under a different name.\nPointed outward it has to survive foreign code, and an earlier round had already priced that badly. Its adapter had to import the containing package and call a positionally compatible function, and\n81 of 96 fetched candidates never ran at all\n. The newer prober never imports the module: it parses with\nast\n, keeps functions passing a purity gate, and executes the function's own source. So the reachable fraction was genuinely open, and I predicted it would reach at least 25 of exp 160's 48 recorded candidates.\nIt reached\n14\n. That prediction was wrong in the generous direction, which was the third method prediction to miss that way in a single session.\nReach was not the interesting number though. Recognition was: across 48 candidates,\none\nwas recognised as the same function as ours (\ntriflescure/is_palindrome\n, one of the ones the older importing adapter could not even load). Forty-six of the 48 fetched, and none of the 46 archives had drifted since the earlier round recorded them, despite being pinned only at\nHEAD\n.\nThe mechanism behind that one-in-forty-eight is the part I would want anyone considering a code index to read. Of\n746 functions seen, 683 (91.6%) never reach a vector at all\n, and it is the purity gate rather than the ranking that stops them:\nshare of the 683 skipped\nreason\n24.9%\nneeds a third-party import (numpy 50, one project's own package 37, yolo, cv2, tensorflow)\n17.7%\na free name resolving from module scope: a class, a config, a compiled regex\n16.8%\ntakes no arguments at all\n6.9%\narity above 3\n6.9%\nnot discriminated by the ladder\n5.9%\ndecorated\n5.6%\nstar or keyword-only arguments\n5.0%\nimpure\n3.1%\nreads module state\nA behavioural index keyed on returns-for-inputs has nothing whatever to say about a nullary function, and a sixth of foreign Python is that shape. An embedding index would see the same 746 functions and would rank the ones it cannot run, which is a different failure and not a better one.\nAnd where two functions did get compared, the comparison was mostly about how they break. Over 497 differing pairs the outcome was an answer disagreement 39.6% of the time, a mix 49.1%, and pure failure 11.3%, so\n60.4% of pairs have at least one side raising\n. The case that made me pre-register that category: our\ncount_vowels\n, written as\nc in \"aeiouAEIOU\"\n, against a stranger's\ncountNumber\n, written as\nchar.lower() in vowels\n. Both are correct on every string. They are separated by\n([1, 2, 3],)\n, where one raises\nTypeError\nand the other raises\nAttributeError\n.\nInside one repository that divergence is a real inconsistency worth reading, which is why the tool exists. Across foreign code it is noise, at 60%.\nThe control, and the fix that died in it\nThe obvious objection to all that is the search. The candidates came from a code search that returned PIN brute-forcers for \"pi digits\", so low recognition might just be low relevance, and\na relevance-improved candidate set is exactly what a vector database supplies.\nThat objection is right; it is testable without building one.\nSo: hold everything constant except relevance. The prober, the twelve reference functions, the verdicts and the classifier are all imported from the previous round rather than copied, so a difference in the numbers cannot be a difference in the instrument. The candidates become\nkaran/Projects-Solutions\n, which is other people's own solutions to the same exercise list: 66 Python links across 11 operations, with relevance guaranteed by construction rather than by ranking.\nRecognised: 0 of 11.\nTwo things about the corpus first, because they are honest limits on that zero. Of the 66 links, 49 fetched and the other 17 are dead; a decade-old link list to other people's repositories has rotted about a quarter through. And\n37 of the 49 fetched files (76%) define no function at all.\nThey are top-level scripts,\nn = int(input())\nand a loop and a\nprint\n, because they are solutions to an exercise list rather than libraries. Reach went down, not up, to 8 of 49.\nBut the substance is in the 11 comparisons that did happen, and it is not about ranking or reach.\nThe disagreement is the return contract.\nOur\npi_digits\nreturns a string and a stranger's\ncompute_pi\nreturns a\nDecimal\n. Our\ncollatz\nreturns the sequence and a correct outsider's\ncollatz\nreturns the number of steps, disagreeing at 21 of 31 ladder positions. Our\nis_palindrome\nreturns a bool and the neighbouring\nlongest_subpalindrome\nreturns a pair of indices. Nothing there is wrong; there is simply no agreed shape for the answer.\nSeven of the 11 comparisons were against a helper rather than the file's actual answer (a\nfactorial\ndefined inside a program that computes pi), because the index never reads names. That property is exactly what let it find a duplicate inside one repo, and across foreign files it means the function you wanted is often not the function being compared.\nThen the part I would most want to keep. Every witness pointed at the first degenerate input on the ladder:\npi_digits(0)\nis\n''\nin our implementation and\nDecimal('3')\nin theirs, and both are defensible. That suggested a clear fix, restricting the ladder to a function's actual domain, and I would have shipped it on the strength of those witnesses. Instead I ran it post-hoc as a labelled probe: drop every degenerate position, 26 of the 31 remain, hold the same discrimination floor.\nRecognised: 0, unchanged.\nThe fix I believed in bought nothing, and the only reason I know is that it was run rather than reasoned about.\nWhat generalises\nNeither of these is a statement about search or retrieval in general, and the numbers are small: 28 entries for one, 48 and 66 candidates for the other, one language throughout, and a candidate list old enough to have rotted. What generalises is where each wall sat relative to where I would have tuned.\nFor search, I would have tuned the budget and the primitive set. The wall is that\nexamples cannot distinguish the right term from an observationally identical wrong one\n, and 2 of 6 consistent terms were wrong with nothing in the examples able to say which. Tuning either knob moves the search deeper into a family it cannot discriminate inside.\nFor retrieval, I would have tuned the ranking, which is what a vector database improves. The wall is that\n91.6% of functions never arrive\n, and then that two people solving one problem do not produce the same artifact: with the right file in hand, correct, and written for the same exercise, recognition was zero. Agreeing on what the answer looks like is the work, and it is the work whether or not you found the file.\nThat is the sentence I would carry out of the whole exercise. Both of these are ways of avoiding the question of what the answer should look like, and both of them ran into it anyway, one round later and with a worse instrument for handling it.", "metadata": { "title": "Searching for a Program Instead of Generating One - DEV Community", "description": "A primitive algebra searched from examples solved five of twenty-eight exercises, and every one already had a hand-written implementation in the same repo. A behavioural index recognised one function in forty-eight, and zero of eleven when handed other people's correct solutions to the same exercises. Tagged with codegeneration, measurement, llm, python.", "keywords": "codegeneration, measurement, llm, python, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/megapixel99/searching-for-a-program-instead-of-generating-one-20gc", "og:title": "Searching for a Program Instead of Generating One", "og:description": "A primitive algebra searched from examples solved five of twenty-eight exercises, and every one already had a hand-written implementation in the same repo. A behavioural index recognised one function in forty-eight, and zero of eleven when handed other people's correct solutions to the same exercises.", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1ubaa22v1i6qzfbrkauq.png", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "Searching for a Program Instead of Generating One", "twitter:description": "A primitive algebra searched from examples solved five of twenty-eight exercises, and every one already had a hand-written implementation in the same repo. A behavioural index recognised one function in forty-eight, and zero of eleven when handed other people's correct solutions to the same exercises.", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1ubaa22v1i6qzfbrkauq.png" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://en.wikipedia.org/wiki/Program_synthesis", "label": "program synthesis", "type": "a" }, { "href": "https://github.com/karan/Projects", "label": "karan/Projects", "type": "a" }, { "href": "https://sethwheeler.dev/blog/rejecting-correct-programs/", "label": "the one I ended up spending most of the project on", "type": "a" }, { "href": "https://docs.python.org/3/library/ast.html", "label": "ast", "type": "a" }, { "href": "https://github.com/karan/Projects-Solutions", "label": "karan/Projects-Solutions", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:48:39Z", "source_url": "https://dev.to/megapixel99/searching-for-a-program-instead-of-generating-one-20gc", "papers": [], "actual_video_files": [], "body_chars": 11574, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/1e91c9a253a7/pages/source.html", "local_html": "attachments/1e91c9a253a7/pages/article.html", "body_base_path": "attachments/1e91c9a253a7/pages/", "body_markdown_path": "attachments/1e91c9a253a7/pages/article.md" }