{ "source_file": "0930", "source_platform": "Dev.to", "author": "xbill", "title_listed": "Count It or Compute It: When a Tool Returns Rows, the Models That Count Them Right Spend the Tokens", "keywords": "", "domain": "大模型算法", "content_type": "工程实践", "is_paid_flag": "否", "url": "https://dev.to/gde/count-it-or-compute-it-when-a-tool-returns-rows-the-models-that-count-them-right-spend-the-tokens-2hae", "publish_date": "2026-09-28", "corpus_stage": "", "notes": "", "id": "58920ea6fcc7", "collected_at": "2026-10-02T13:51:06", "title": "Count It or Compute It: When a Tool Returns Rows, the Models That Count Them Right Spend the Tokens - DEV Community", "body_markdown": "This is a submission for the\nKaggle Benchmarking Challenge\nThis article reports a Kaggle benchmark for a step every agent performs and few people test, counting what a tool returns, and ends with a step by step guide to reproducing it.\nAn agent's search, database or API tool usually hands back a list of records. When the user asks how many, the model does the counting. That step passes every quick test with a handful of rows, and nothing flags it when it goes wrong: the model sends the right query and quotes a confident number.\nThis benchmark asks ten models the same 68 counting questions with two versions of one tool:\ncount_ids\nreturns the exact count,\nlist_ids\nreturns the matching ids.\nWith the count, every model but one answered every question about 330 ids correctly. With the ids, the models split in two. The ones that spent 2,700 to 8,200 output tokens per question counted 15 to 21 of 21 lists correctly. The ones that answered in under 600 tokens counted 0 to 10.\nThe surprise was who landed where: Claude Opus 5, the most expensive model in the lineup, answered in 579 tokens and counted 9 of 21. Gemma 4 26B, a 26B open-weight model, spent 8,153 and counted 20.\nhttps://github.com/xbill9/devto-kaggle\nhttps://www.kaggle.com/benchmarks/xbillwork/count-it-or-compute-it\nWhat I Benchmarked\nThe itch is eleven ids:\n0, 2, 3, 20, 21, 22, 23, 10, 11, 12, 13\nEnter fullscreen mode\nExit fullscreen mode\nHow many are 10 or more? The answer is 8, and every model here gets it. Eleven ids say nothing about three hundred, and eleven is about the size of the lists most quick tests use.\nLanguage models' trouble with counting is familiar from examples like counting the letters in a word. This benchmark measures it where agents meet it: inside tool use, with every miss traced to the query or the count, and every answer priced in tokens.\nThe benchmark holds everything fixed except who does the arithmetic:\nTask\nWhat the model gets\nWho counts\ncount-engine\ncount_ids(where)\n, which returns the exact count, minimum and maximum\nThe tool\ncount-rows-tool\nlist_ids(where)\n, which returns the matching ids\nThe model, from the returned list\ncount-python-tool\nEvery id in the prompt, plus\nrun_python\nwith\nids\nalready defined\nThe model's code, if it writes any\nIn the first two tasks the ids never appear in the prompt, so the only difference is whether the tool returns a number or a list. Both tasks check every filter the model sends by the ids it selects, so\nid > 9\ncounts as right for \"10 or more\", and every wrong answer is traced to either a wrong query or a wrong count.\nEach task asks 68 questions: lists of 11, 110 and 330 ids, seven phrasings of the threshold (\"10 or more\", \"no less than\", \"under\", \"between 5 and 9 inclusive\" and so on), three seeds each, and the original eleven ids five times. Every threshold is an id in the list, so\n>\nand\n>=\nalways give different answers.\nModels Tested\nVendor\nModels\nGoogle\nGemini 2.5 Flash, Gemini 3.7 Flash, Gemini 3.8 Flash, Gemma 4 26B A4B\nAnthropic\nClaude Haiku 4.5, Claude Sonnet 5, Claude Opus 5\nOpenAI\nGPT-5.4 nano, GPT-5.4 mini, gpt-oss-20b\nThe lineup takes a small, a mid-sized and a large model from each vendor, plus the open-weight models from Google and OpenAI, so the comparison covers price, size and what anyone can run. Each model runs with the settings Kaggle's model proxy serves it with. Some of those reason before answering and some answer straight away, and that setting tracks the result more closely than size or price.\nGPT-6 Astra is refused function tools by the proxy (\nFunction tools with reasoning_effort are not supported for gpt-6-astra in /v1/chat/completions\n). Gemini 3.5 Flash-Lite and both Qwen 3 Next 80B models returned\n429\nor\n503\non most calls.\nFindings\nAt 330 ids, the size where the models separate:\nModel\nRows tool correct\nOutput tokens per question, rows tool\nEngine correct\nRows-tool cost vs engine\nGemma 4 26B A4B\n20/21\n8,153\n21/21\n18.6x\nGemini 3.7 Flash\n20/21\n2,785\n21/21\n13.5x\nGemini 3.8 Flash\n21/21\n2,742\n21/21\n14.3x\ngpt-oss-20b\n15/21\n2,684\n19/21\n5.2x\nGemini 2.5 Flash\n0/21\n580\n21/21\n2.6x\nClaude Opus 5\n9/21\n579\n21/21\n2.4x\nClaude Sonnet 5\n10/21\n401\n21/21\n1.9x\nClaude Haiku 4.5\n5/21\n143\n21/21\n1.5x\nGPT-5.4 mini\n5/21\n39\n21/21\n1.9x\nGPT-5.4 nano\n0/21\n35\n21/21\n1.7x\nThe Claude rows-tool figures come from their 2026-09-25 run, which asked the same 68 questions with the same tool; their 2026-09-28 rows-tool run stopped on the daily quota. Every other figure comes from the 2026-09-28 runs.\n1. With the Count in the Tool, Every Model Is Right at a Flat Cost\nEvery model but one answered all 21 engine questions about 330 ids correctly, at 35 to 451 output tokens per question, and each model's tokens stayed about the same from 11 ids to 330. Turning \"no less than 244\" into\nid >= 244\nand quoting the number back is something every model here does reliably. The exception, gpt-oss-20b, answered 5 of 68 engine questions with a number other than the count it was given, all of them 0, 1 or 2, and answered once without calling the tool.\n2. With the Rows, Quick Tests Pass and Larger Lists Fail\nEight of the ten models counted all 26 questions about 11 ids correctly. At 330 ids, five of those eight counted 10 or fewer of 21 correctly. None of them returned an error or a hedge: each answer was a single confident number.\n3. The Models That Count Right Spend the Tokens\nThe models split into two groups by how many tokens they spend, and model size and price do not predict the split. That was the surprise. The models that count correctly spend more tokens as the list grows: Gemma 4 26B went from 394 output tokens per question at 11 ids to 8,153 at 330. The models that miscount spend about the same at every size: GPT-5.4 nano spent 35 tokens per question at 11, 110 and 330 ids, which leaves no room to count anything. Claude Opus 5, the most expensive model here, spent 579 output tokens per question at 330 ids and counted 9 of 21 correctly; Gemma 4 26B, an open-weight model, spent 8,153 and counted 20.\nThe same models landed in the same group in every rows-tool run, three or four runs per model between 2026-09-25 and 2026-09-28. The order inside a group moves by a few questions between runs.\nCounting right by reasoning costs 5 to 19 times what the engine costs for the same answer. An earlier run with 1,100 ids in the prompt shows the same link from the other side: with output capped at 8,192 tokens, Gemini 3.7 Flash counted 1 of 21 lists correctly, against 19 and 15 of 21 without the cap, and still gave a number every time.\n4. The Query Was Right Every Time\nNo answer on either task rested on a filter that selected the wrong ids. Every miss on the rows tool was the model counting the correct list wrong, apart from 3 answers given without calling the tool and 1 that could not be read as a number. The failure is in the arithmetic.\n5. Python Works When the Model Uses It\nWith every id in the prompt and a Python tool available, 7 of 10 models scored 68 of 68; Gemini 2.5 Flash called the tool on 1 of 42 questions about 110 and 330 ids and scored 37.\nWhat It Changed About How I Think About These Models\nIn these runs, counting a list was work the model did in tokens, and the models that answered straight away had not done it. A tool that returns rows moves that work onto the model and makes its accuracy depend on a setting the caller may never have looked at. The count belongs in the tool: return the count, the minimum and the maximum, and every model in this lineup answers correctly at a fraction of the tokens.\nCompare and Contrast\nEngine\nRows tool\nPython tool\nWho counts\nThe tool\nThe model, from the returned list\nThe model's code, if it writes any\nOutput tokens per question at 330 ids\n35 to 451\n35 to 8,153\n—\nWhat went wrong\nQuoting a number other than the count\nMiscounting the right list\nSkipping the tool, no\nprint()\nAnswers built on a wrong filter\n0\n0\n—\nSo, Which One?\n🟢\nEngine\n— put the count, minimum and maximum in the tool. Every model is right at a flat cost.\n⚠️\nPython tool\n— reliable when the model uses it, prints the result and quotes it.\n❌\nRows tool\n— correct only with a model that reasons through the list, at 5 to 19 times the cost per question.\nWhat I'd Measure Next\nThe same model with reasoning on and off.\nClaude Opus 5 with extended thinking against its default. The runs here show that tokens and accuracy move together across models; this would show whether switching reasoning on is enough to fix one model.\nOther arithmetic.\nSums, averages and the largest value over returned rows, which the same tool-shape question applies to.\nHarder filters.\n\"At least 10 but under 20\" and \"outside 5 to 9\", where a wrong filter with an exact count would show up.\nHow to Reproduce It\nThese steps rebuild the benchmark from the repository; the tasks, the runs and every table above come from these commands.\nAt This Point You Should Have…\nA Kaggle account, with the Kaggle CLI installed (\npip install kaggle\n, version 2.2.4 here) and logged in with\nkaggle auth login\nPython 3 for the local checks and the report; the\nkaggle_benchmarks\nlibrary the tasks import is already installed on Kaggle\nThe repository cloned:\ngit clone https://github.com/xbill9/devto-kaggle\nand\ncd devto-kaggle\nStep 1 — Check the Questions Locally\nEvery question is generated in code, and its expected count comes from running a filter over the ids.\ncheck.py\nbuilds all 68 questions with no model calls and no Kaggle library:\npython3 tasks/check.py\nEnter fullscreen mode\nExit fullscreen mode\nok: 68 rows per task, by size {11: 26, 110: 21, 330: 21}\nEnter fullscreen mode\nExit fullscreen mode\nStep 2 — Push Each Task\nEach file in\ntasks/\nis one Kaggle task. The first\n@kbench.task\nin the file returns the share of questions answered correctly, because Kaggle names the task and reads its score from the first one it finds. A second task answers one question and runs once per question:\n@kbench.task\n(\nname\n=\n\"\ncount-engine\n\"\n)\ndef\ncount_engine\n(\nllm\n)\n->\nfloat\n:\nruns\n=\ncount_engine_row\n.\nevaluate\n(\nllm\n=\n[\nllm\n],\nevaluation_data\n=\npd\n.\nDataFrame\n(\nROWS\n),\nn_jobs\n=\n4\n,\non_failure\n=\n\"\ncontinue\n\"\n)\nreturn\nsummarize\n(\nruns\n,\nlen\n(\nROWS\n),\n\"\nengine\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nEach question's result keeps the answer, every filter the model sent, and the tokens and cost Kaggle's model proxy reported for it. The push slug must match the task name:\nkaggle b t push count-engine\n-f\ntasks/count_engine.py\n--wait\nkaggle b t status count-engine\nEnter fullscreen mode\nExit fullscreen mode\nTask: count-engine\nVersion: 13\nStatus: Completed\nCreated: 2026-09-28 16:11:45\nEnter fullscreen mode\nExit fullscreen mode\nStep 3 — Run the Models\nkaggle b auth -y\nfetches the short-lived key for Kaggle's model proxy, and\nkaggle b t models\nlists the model slugs. A run takes one\n-m\nper model:\nkaggle b t run count-rows-tool\n-m\ngemini-2.5-flash\n-m\ngemini-3.8-flash\n-m\ngemma-4-26b-a4b-it\n-m\ngpt-oss-20b\n-m\ngpt-5.4-nano-2026-03-17\n-m\ngpt-5.4-mini-2026-03-17\n--wait\nEnter fullscreen mode\nExit fullscreen mode\nAll runs completed:\n gpt-5.4-mini-2026-03-17: COMPLETED\n gpt-5.4-nano-2026-03-17: COMPLETED\n gpt-oss-20b: COMPLETED\n gemma-4-26b-a4b-it: COMPLETED\n gemini-3.8-flash: COMPLETED\n gemini-2.5-flash: COMPLETED\nEnter fullscreen mode\nExit fullscreen mode\n🔎 Tip: Cap the Output to Stay Inside the Quota\nKaggle's model proxy reserves the worst-case cost of a call, based on the output-token limit, and refuses the call when that exceeds what is left of the day's quota. With no limit set, GPT-6 Astra reserved $6.40 per call and Claude Opus 5 $3.20. Passing a limit keeps the reservation to cents:\nllm\n.\nprompt\n(\nprompt\n,\nschema\n=\nint\n,\nextra_api_params\n=\n{\n\"\nmax_completion_tokens\n\"\n:\n8192\n})\nEnter fullscreen mode\nExit fullscreen mode\nStep 4 — Download the Runs and Build the Tables\nreport.py\nbuilds every table in this article from the downloaded runs, including the tokens and cost per question:\nfor\nt\nin\ncount-engine count-rows-tool count-python-tool\n;\ndo\nkaggle b t download\n$t\n-o\nresults\n;\ndone\npython3 tasks/report.py results\nEnter fullscreen mode\nExit fullscreen mode\n| Model | Size | Engine correct | Engine out tokens | Engine $ | Rows correct | Rows out tokens | Rows $ |\n|---|---|---|---|---|---|---|---|\n| Gemini 3.8 Flash | 330 | 21/21 | 140 | 0.0010 | 21/21 | 2,742 | 0.0146 |\n| GPT-5.4 nano | 330 | 21/21 | 35 | 0.0002 | 0/21 | 35 | 0.0003 |\nEnter fullscreen mode\nExit fullscreen mode\nStep 5 — Group the Tasks Into a Benchmark\nA benchmark is created in the Kaggle web UI. On the benchmark page,\nAdd Tasks\nadds the tasks and\nAdd Models\nthe models. Under\nSettings\n, set the overall score to\nAverage of task scores\n: the default,\nPercentage of tasks passed\n, ignores tasks that return a number. Setting the visibility to\nPublic\nis permanent.\n🔎 Tip: The Leaderboard Shows Each Model's Latest Run\nEach leaderboard cell shows the model's most recent run of that task, so a run that fails on the quota replaces a complete one before it. Check the quota before a run, and rerun the pair if one fails.\npending.py\nlists pairs whose latest run is incomplete.\nMy Benchmark\nhttps://www.kaggle.com/benchmarks/xbillwork/count-it-or-compute-it\n(the benchmark: the three tasks and their leaderboard)\nhttps://www.kaggle.com/benchmarks/tasks/xbillwork/count-engine\nhttps://www.kaggle.com/benchmarks/tasks/xbillwork/count-rows-tool\nhttps://www.kaggle.com/benchmarks/tasks/xbillwork/count-python-tool\nSummary\nThe goal of this article was to measure whether models count correctly what their tools return. The key to the solution was changing only what the tool returns, a count or a list, and recording every filter and every token.\nThe results were:\n🟢 With the count in the tool, every model but gpt-oss-20b answered all 330-id questions correctly.\n⚠️ With the rows, the models that spent 2,700 to 8,200 output tokens per question counted 15 to 21 of 21 lists of 330 correctly.\n❌ The models that answered in under 600 tokens counted 0 to 10 of 21, Claude Opus 5 among them, with no error to show it.\nEach model ran each task on Kaggle's model proxy with its default settings and output capped at 8,192 tokens; the engine and rows-tool figures come from runs on 2026-09-28, apart from the Claude rows-tool figures from 2026-09-25, the Python-tool figures from 2026-09-28, and the token-cap result from earlier in-context runs with lists of 1,100 ids.\nThe strategy for benchmarking counting across 10 models was validated with an incremental step by step approach.\nReferences\nThis benchmark's code:\nhttps://github.com/xbill9/devto-kaggle\nKaggle Benchmarking Challenge:\nhttps://dev.to/challenges/kaggle-2026-09-23\nKaggle Benchmarks:\nhttps://www.kaggle.com/benchmarks\nThe tasks are built on Kaggle's\nkaggle-benchmarks\nlibrary:\nhttps://github.com/Kaggle/kaggle-benchmarks\nKaggle's benchmark-writing skill, used as the reference for the CLI workflow:\nhttps://github.com/Kaggle/kaggle-skills/blob/main/write-kaggle-benchmarks/SKILL.md", "metadata": { "title": "Count It or Compute It: When a Tool Returns Rows, the Models That Count Them Right Spend the Tokens - DEV Community", "description": "A Kaggle benchmark of the step agents rarely test: counting what a tool returns. Ten models, 68 questions, one tool that returns the count and one that returns the rows. With the count, every model is right at a flat cost. With the rows, models that reason through the list count 330 ids right and spend 6 to 26 times the tokens doing it; models that answer straight away get 0 to 10 of 21. Tagged with devchallenge, kagglechallenge, ai, machinelearning.", "keywords": "devchallenge, kagglechallenge, ai, machinelearning, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/gde/count-it-or-compute-it-when-a-tool-returns-rows-the-models-that-count-them-right-spend-the-tokens-2hae", "og:title": "Count It or Compute It: When a Tool Returns Rows, the Models That Count Them Right Spend the Tokens", "og:description": "A Kaggle benchmark of the step agents rarely test: counting what a tool returns. Ten models, 68 questions, one tool that returns the count and one that returns the rows. With the count, every model is right at a flat cost. With the rows, models that reason through the list count 330 ids right and spend 6 to 26 times the tokens doing it; models that answer straight away get 0 to 10 of 21.", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fxbill9%2Fdevto-kaggle%2Fmain%2Farticle%2Fdevto-cover.d50932c1.jpg", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "Count It or Compute It: When a Tool Returns Rows, the Models That Count Them Right Spend the Tokens", "twitter:description": "A Kaggle benchmark of the step agents rarely test: counting what a tool returns. Ten models, 68 questions, one tool that returns the count and one that returns the rows. With the count, every model is right at a flat cost. With the rows, models that reason through the list count 330 ids right and spend 6 to 26 times the tokens doing it; models that answer straight away get 0 to 10 of 21.", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fxbill9%2Fdevto-kaggle%2Fmain%2Farticle%2Fdevto-cover.d50932c1.jpg" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/58920ea6fcc7/code/body-000.txt", "chars": 40, "language": "text" }, { "file": "attachments/58920ea6fcc7/code/body-001.txt", "chars": 23, "language": "text" }, { "file": "attachments/58920ea6fcc7/code/body-002.txt", "chars": 57, "language": "text" }, { "file": "attachments/58920ea6fcc7/code/body-003.txt", "chars": 240, "language": "text" }, { "file": "attachments/58920ea6fcc7/code/body-004.txt", "chars": 92, "language": "text" }, { "file": "attachments/58920ea6fcc7/code/body-005.txt", "chars": 86, "language": "text" }, { "file": "attachments/58920ea6fcc7/code/body-006.txt", "chars": 169, "language": "text" }, { "file": "attachments/58920ea6fcc7/code/body-007.txt", "chars": 211, "language": "text" }, { "file": "attachments/58920ea6fcc7/code/body-008.txt", "chars": 81, "language": "text" }, { "file": "attachments/58920ea6fcc7/code/body-009.txt", "chars": 132, "language": "text" }, { "file": "attachments/58920ea6fcc7/code/body-010.txt", "chars": 282, "language": "text" } ], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://dev.to/challenges/kaggle-2026-09-23", "label": "Kaggle Benchmarking Challenge", "type": "a" }, { "href": "https://github.com/xbill9/devto-kaggle", "label": "https://github.com/xbill9/devto-kaggle", "type": "a" }, { "href": "https://www.kaggle.com/benchmarks/xbillwork/count-it-or-compute-it", "label": "https://www.kaggle.com/benchmarks/xbillwork/count-it-or-compute-it", "type": "a" }, { "href": "https://www.kaggle.com/benchmarks/tasks/xbillwork/count-engine", "label": "https://www.kaggle.com/benchmarks/tasks/xbillwork/count-engine", "type": "a" }, { "href": "https://www.kaggle.com/benchmarks/tasks/xbillwork/count-rows-tool", "label": "https://www.kaggle.com/benchmarks/tasks/xbillwork/count-rows-tool", "type": "a" }, { "href": "https://www.kaggle.com/benchmarks/tasks/xbillwork/count-python-tool", "label": "https://www.kaggle.com/benchmarks/tasks/xbillwork/count-python-tool", "type": "a" }, { "href": "https://www.kaggle.com/benchmarks", "label": "https://www.kaggle.com/benchmarks", "type": "a" }, { "href": "https://github.com/Kaggle/kaggle-benchmarks", "label": "https://github.com/Kaggle/kaggle-benchmarks", "type": "a" }, { "href": "https://github.com/Kaggle/kaggle-skills/blob/main/write-kaggle-benchmarks/SKILL.md", "label": "https://github.com/Kaggle/kaggle-skills/blob/main/write-kaggle-benchmarks/SKILL.md", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:48:23Z", "source_url": "https://dev.to/gde/count-it-or-compute-it-when-a-tool-returns-rows-the-models-that-count-them-right-spend-the-tokens-2hae", "papers": [], "actual_video_files": [], "body_chars": 14946, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/58920ea6fcc7/pages/source.html", "local_html": "attachments/58920ea6fcc7/pages/article.html", "body_base_path": "attachments/58920ea6fcc7/pages/", "body_markdown_path": "attachments/58920ea6fcc7/pages/article.md" }