{ "source_file": "0930", "source_platform": "Dev.to", "author": "Seth Wheeler", "title_listed": "An App Generator That Verifies Everything Except Its Parser", "keywords": "", "domain": "文本处理", "content_type": "工程实践", "is_paid_flag": "否", "url": "https://dev.to/megapixel99/an-app-generator-that-verifies-everything-except-its-parser-42dk", "publish_date": "2026-08-16", "corpus_stage": "", "notes": "", "id": "d49ce21e76fc", "collected_at": "2026-10-02T13:52:09", "title": "An App Generator That Verifies Everything Except Its Parser - DEV Community", "body_markdown": "appgen\nis a tool I built to answer a narrow question: can you get from a sentence to a running, verified application without a language model doing the generating? You type\n\"I want a support desk system with priorities, comments, search and closing tickets\"\nand it writes a single dependency-free Python file, starts it on a private port, exercises every feature you asked for over real HTTP, and only then shows it to you. It takes about 2 ms of CPU and no GPU.\nIt works. That is the uncomfortable part, because the way it reports that it works is misleading, and I wrote the reporting.\nThe tool's README says the verification sweep covers \"all 810 domain × feature × family cells,\" and that this \"is why the 100% claims above are trustworthy.\" Both halves of that are true in a narrow sense and wrong in the sense a reader will take. This post is about the gap, which I found by asking a question I should have asked much earlier: what is a cell, exactly?\nThe code lives in a private research repo, so there is no link to it here. Every figure below comes from a file or a command I ran, and I have named which.\nWhat a cell is made of\nThe sweep lives in\nexperiments/028-relations/exhaustive.py\n, and the whole thing turns on a six-line generator:\ndef\ncells\n():\nfor\ndom\nin\nDOMAINS\n:\nyield\ndom\n,\nfrozenset\n()\nfor\nf\nin\nFEATURES\n:\nyield\ndom\n,\nfrozenset\n([\nf\n])\nyield\ndom\n,\nfrozenset\n(\nFEATURES\n)\nEnter fullscreen mode\nExit fullscreen mode\nDOMAINS\nhas 10 entries (blog, booking, contacts, events, expenses, inventory, library, recipes, support_desk, todo) and\nFEATURES\nhas 7 (category, comments, delete, edit, search, stats, toggle). So each domain contributes one empty set, seven single-feature sets and one full set: 9 configurations, 90 cells. Nine families of emitter (single-file unstyled, single-file styled, api, cli, project, relational, node_web, node_api, c_cli) multiply that to 810. I checked the arithmetic rather than trusting the README, and 810 is right.\nNow look at what a cell\nis\n. It is a tuple of a dictionary key and a frozen set of strings. A cell is not a request. Nothing in\ncells()\nproduces English, and nothing downstream of it consumes English. The emitters are handed a schema and a feature set and asked to produce a program.\nThe sweep is explicit about this where it would otherwise be tempting to use the real entry point. In\nsweep_project\n, at line 96:\n# build_project parses a request; drive the emitters directly instead\nEnter fullscreen mode\nExit fullscreen mode\nI wrote that comment for a good reason (a sweep should test one thing, and\nbuild_project\nwould drag the parser into an emitter test), and it is the right call for the sweep. The problem is what happened to the claim afterwards.\nappgen\nis a tool you drive with a sentence, and its verification is exhaustive over everything downstream of that sentence while touching the sentence zero times. When the README says 810 cells are why the 100% is trustworthy, the 100% it is vouching for belongs to the composer, and a user's request has to survive a different component first.\nThe component that reads English, measured on its own\nThat component is a hand-written keyword planner, and to its credit I did measure it, in a separate experiment (041) built specifically to ask whether a learned model should replace it.\nMeasuring an intent classifier fairly is harder than it looks, because I wrote the test phrasings, so a random train/test split leaves near-duplicates on both sides and everything scores well. Instead the corpus is grouped into eight\nphrasing families\n(imperative, desiderative, interrogative, polite, declarative, need, terse, wordy) and whole families are held out, so a model is tested on a way of speaking it has never seen. Slot vocabularies (entities, features, languages, kinds, shapes) are disjoint across the split too, so nothing wins by memorising\nbug tracker\n. That is 146 training clauses and 32 test clauses per fold, 8 folds, over 7 operations.\nI re-ran\ncompare.py\nwhile writing this. It reproduces:\nheld-out family\nrules\nnb\nnb-func\nknn\nhybrid\nprior\nimperative\n0.78\n0.31\n0.50\n0.34\n0.78\n0.22\ndesiderative\n0.38\n0.29\n0.43\n0.52\n0.38\n0.29\ninterrogative\n0.79\n0.21\n0.34\n0.55\n0.79\n0.24\npolite\n0.90\n0.19\n0.38\n0.62\n0.90\n0.38\ndeclarative\n0.65\n0.17\n0.22\n0.43\n0.65\n0.35\nneed\n0.75\n0.25\n0.15\n0.25\n0.75\n0.25\nterse\n0.68\n0.32\n0.37\n0.58\n0.58\n0.32\nwordy\n0.64\n0.18\n0.27\n0.50\n0.64\n0.32\nmean\n0.70\n0.24\n0.33\n0.48\n0.68\n0.29\nThe rules win, decisively, against a Naive Bayes classifier (0.24, barely above the always-guess-the-most-common floor of 0.29) and against nearest-neighbour retrieval (0.48). That was the experiment's finding, and it is why the planner is still hand-written; there was nothing worth grafting on, since the hybrid (0.68) does not beat the rules alone.\nBut read the winning column as a user rather than as a comparison.\n0.70 means that on a phrasing the rules were not written for, roughly three requests in ten are read wrong.\nThe per-operation breakdown says which:\noperation\nrules\nport\n1.00\nshape\n0.89\nupdate\n0.88\nunsupported\n0.86\nverify\n0.85\ncreate\n0.52\nkind\n0.32\nkind\nis how the tool decides whether you asked for an HTML app, a JSON API or a CLI. On held-out phrasing it gets that right 32% of the time.\nI wrote a paragraph here claiming the obvious consequence: that the composer then builds the wrong kind of program, verifies it exhaustively, and reports\nALL CHECKS PASSED\non an app nobody asked for. Then I went and checked, and it does not follow.\nOf the 17 misread\nkind\nclauses,\n15 are read as\nupdate\n, and they all share one shape: the clause names a kind\nand\na feature (\n\"turn it into a web app with search\"\n). Driven end to end against a real app, that label produces the right program every time:\ngoal\nplanned as\nresult\nturn it into a JSON API with search\nupdate\napi, with search\nmake it a JSON API with a stats page\nupdate\napi, with stats\ncan you make it a command line tool keeping comments\nupdate\ncli, with comments\nThe\nupdate\nop hands the whole sentence to the updater, where one function converts the kind and another applies the features. It is the only operation that carries both; a bare\nkind\nop carries the kind alone. So \"improving\" that 0.32, which means relabelling those 15 clauses, would convert the program and silently drop the feature asked for in the same breath. The labels are wrong for compound clauses, not the planner.\nThat is worth stating plainly because it cuts against the post:\nan op-label metric is not a behaviour metric\n, and I reached for the scary reading of a low number before driving the tool. The number that describes what a user gets is 4 out of 4 on those goals, not 0.32.\nWhat survives is narrower and still the point: 0.70 and 0.32 are real measurements of a component that the 810 cells do not touch, and neither figure appears next to the 100%. The two numbers describing this tool measure different things, and only one of them is on the front page.\nBuilt is not the same as reachable\nThe second shortcoming is the one I find hardest to defend, because the repo diagnosed it, wrote a fix, and then left the original offenders alone.\nExperiment 045 audited every transform in the repo against what the shipped tool could actually reach, and found\nsix built, verified, written-up transforms that no user could ask for.\nPagination, SMS, CSV and email had all been explicitly requested. Asking returned \"not supported\" while the implementation sat in the repo, tested and documented.\nThe mechanism is worth naming: reaching a feature meant wiring it at five separate sites (the grammar, the kinds table, the emit path, the verify path, the manifest stickiness, plus the agent's tokens and its goal self-check). Exp 045 wired pagination by hand. It worked, and it did not scale; CSV was still unreachable thirty rounds later.\nThe same failure appears twice more, in comments I left in\nappgen.py\nitself:\nExp 034d built and verified the Node one (18/18 with the SAME oracle); it was simply never wired up, so porting an authenticated app silently removed its guard.\nThree lines down, the same story for ranked search: a Node transform verified at 10/10 that \"was never wired up, so porting a ranked app to JavaScript silently downgraded it to LIKE.\" A verified transform that nothing dispatches to is a verified transform that does not exist, and in both of those cases the failure was silent and security-relevant.\nExperiment 040 built the structural fix, and it is the right one: a feature\ndeclares\nits own tokens, flag, transforms, oracle and applicable cells, and generic dispatch reads a registry. Discovery is a glob over\nfeature_*.py\n, because naming the modules in an import list would recreate the defect being removed; a glob cannot forget. Wiring that in cost about 120 lines once, after which SMS, email and validation each cost a single declaration file with no edit to\nappgen.py\n.\nHere is what I did not do. The three transforms that motivated the whole exercise are still hand-wired. As of today\nappgen.py\n(1,586 lines) references the\nAUTH_KINDS\n,\nRANKED_KINDS\nand\nPAGED_KINDS\ntables\n21 times\n, and the registry those tables were supposed to be replaced by holds four features:\nfeature\nflag\nreachable in\ncsv\n--csv\npython / web\nsms\n--sms\npython / web\nemail\n--email\nnode / web\nvalidate\n--validate\nnode / web\nEach is available in exactly one of the six language × kind cells the tool supports. The fix works, and it was applied to the new features rather than to the old ones, which means the class of bug that produced six unreachable transforms is still live in the three places where it did the most damage.\nThe sweep became a constraint on the design\nThis is the part I did not expect to find, and it is the strongest form of the problem.\nRanked search should probably have been an eighth feature in the grammar. It is not, and the comment explaining why sits above\nRANKED_WORDS\n:\nbaking an eighth feature into the grammar would change the corpus from 2^7 to 2^8 subsets and invalidate exp 028's 810 cells\nThat is a verification suite dictating a design decision. Ranked search became a transform layered over the emitted app, rather than a feature the composer knows about, in order to keep a number from going stale. The reasoning is locally sensible (a 2^8 corpus would invalidate published results across a dozen experiments) and it is still the tail wagging the dog.\nIt also gives the 810 figure a shape worth stating plainly. 810 cells is exhaustive over domain ×\nsingle\nfeature, plus the empty and full sets. It is not exhaustive over feature\ncombinations\n: 7 features admit 2^7 = 128 subsets, and the sweep visits 9 of them. The sweep's own docstring is honest about this (\"every domain x feature pair covered at least once\"), and pairwise coverage is a real, defensible standard that catches interaction bugs cheaply. The README's \"all 810 cells\" is the phrase that quietly promises more.\nWhy I trust the 810 anyway, and what that is worth\nNone of this makes the sweep theatre. It replaced random sampling, and it exists because of a bug that random sampling hid.\nEvery behavioural result in the repo before experiment 028 came from a random sample of roughly 10 cells. That sample missed\ncontacts\n×\ncategory\n, where the generated column was named\ngroup\n, a SQL reserved word. Those apps compile cleanly and crash on the first query.\n235 of 4,412 corpus apps (5.3%) never worked, and the defect survived from experiment 012 to experiment 028\n, so sixteen experiments were trained and scored on a corpus containing 5.3% crash-on-start code. Exhaustive sweeping along the axis that mattered found it immediately.\nI re-ran the whole thing while writing this, all six flags, and it is green:\nsingle-file, unstyled (corpus path): 90 cells ALL PASS\nsingle-file, styled (tool path): 90 cells ALL PASS\napi + cli kinds: 180 cells ALL PASS\nmulti-file projects: 90 cells ALL PASS\nrelational projects: 90 cells ALL PASS\nnode + C emitters: 270 cells ALL PASS\n\ntotal failures: 0 (810 cells)\nEnter fullscreen mode\nExit fullscreen mode\nSo the headline is not inflated. 810 of 810 really do pass, today, on my machine. That is what makes this a reporting problem rather than a correctness one, and it is why I would not remove the number. Usefully, the script also prints its own limits when run partially, labelling the run a\nPARTIAL sweep\nand naming the flags the full 810 requires. An instrument that reports its own coverage was already doing something the README was not.\nSo the composer is genuinely exact inside its grammar, and the grammar is genuinely narrow: three program kinds and three languages, but only 6 of the 9 cells exist (Python does web, api and cli; Node does web and api; C does cli only), because each cell is a hand-written emitter of 104 to 287 lines with no transfer between them.\n--project\nand\n--relations\nare Python-only. C stores flat records, so it drops\ncomments\n,\nstats\nand\ncategory\nwith a printed note. Coverage is exactly the union of the emitters someone wrote, and it grows only by someone writing more.\nThat narrowness is not a footnote to the 100%; it is the reason for it.\nWhat generalises\nThe thing I would carry to another project is not \"sweep exhaustively,\" though the\ngroup\nbug argues for it. It is sharper than that:\nAn exhaustive sweep proves a property of the component it drives, and its credibility then transfers silently to the whole tool.\n810 is a real number about the emitters. It became, in my own README, a claim about\nappgen\n. Nobody lied; the sweep simply had a much better publicist than the parser did, because the parser's number lives in an experiment write-up and the emitters' number lives on the tool's front page.\nThe tell was available the whole time, and I had written it myself: the sweep bypasses the entry point, in a comment, on line 96. A verification suite that cannot be driven the way users drive the tool is measuring something adjacent to what you are claiming, and the honest fix is not to weaken the sweep but to give the uncovered component its own number and print it next to the other one.\nFor\nappgen\nspecifically, the shortcomings I would fix in order: add a sweep that starts from request\nstrings\n, so the parser is measured where the emitters already are; migrate\nAUTH_KINDS\n,\nRANKED_KINDS\nand\nPAGED_KINDS\ninto the registry, so the unreachable-transform bug cannot recur where it already has; and stop printing 100% without naming the component it belongs to.\nThe correction above is the reason that list is in that order. I had the sweep second and the number's interpretation settled. Building the sweep first is what showed the interpretation was wrong, and a request sweep would have caught a defect the 810 cells never could: the tool read\na script tracker with search\nas a request for a command-line program, because six of its sixteen kind keywords are ordinary nouns and\nscript\nwas simultaneously the entity it had resolved.\nThe composer was never the risky part. It was just the part with a number.", "metadata": { "title": "An App Generator That Verifies Everything Except Its Parser - DEV Community", "description": "A natural-language app generator whose verification sweep is exhaustive over everything except the part that reads English. The composer scores 100% on 810 cells built from feature sets; the parser that has to produce those feature sets scores 0.70, and nothing prints that number next to the other one. Tagged with measurement, codegeneration, verification, llm.", "keywords": "measurement, codegeneration, verification, llm, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/megapixel99/an-app-generator-that-verifies-everything-except-its-parser-42dk", "og:title": "An App Generator That Verifies Everything Except Its Parser", "og:description": "A natural-language app generator whose verification sweep is exhaustive over everything except the part that reads English. The composer scores 100% on 810 cells built from feature sets; the parser that has to produce those feature sets scores 0.70, and nothing prints that number next to the other one.", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5y2yvy99m1a0s1igcnu3.png", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "An App Generator That Verifies Everything Except Its Parser", "twitter:description": "A natural-language app generator whose verification sweep is exhaustive over everything except the part that reads English. The composer scores 100% on 810 cells built from feature sets; the parser that has to produce those feature sets scores 0.70, and nothing prints that number next to the other one.", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5y2yvy99m1a0s1igcnu3.png" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/d49ce21e76fc/code/body-000.txt", "chars": 172, "language": "text" }, { "file": "attachments/d49ce21e76fc/code/body-001.txt", "chars": 70, "language": "text" }, { "file": "attachments/d49ce21e76fc/code/body-002.txt", "chars": 380, "language": "text" } ], "images": [], "audio": [], "videos": [], "links": [], "repair_version": 2, "repaired_at": "2026-10-14T10:48:42Z", "source_url": "https://dev.to/megapixel99/an-app-generator-that-verifies-everything-except-its-parser-42dk", "papers": [], "actual_video_files": [], "body_chars": 15070, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/d49ce21e76fc/pages/source.html", "local_html": "attachments/d49ce21e76fc/pages/article.html", "body_base_path": "attachments/d49ce21e76fc/pages/", "body_markdown_path": "attachments/d49ce21e76fc/pages/article.md" }