{ "source_file": "0930", "source_platform": "Dev.to", "author": "Seth Wheeler", "title_listed": "The Embedding Table Was 72% of the Model", "keywords": "", "domain": "文本处理", "content_type": "工程实践", "is_paid_flag": "否", "url": "https://dev.to/megapixel99/the-embedding-table-was-72-of-the-model-2ghm", "publish_date": "2026-09-11", "corpus_stage": "", "notes": "", "id": "8496bd7713ad", "collected_at": "2026-10-02T13:51:54", "title": "The Embedding Table Was 72% of the Model - DEV Community", "body_markdown": "An earlier experiment in this series had established something slightly deflating about a small transformer: quantising the whole network to int8 is free, and the bytes you save are better spent on count tables than on network precision. That is a useful result and it invites a sharper question, which is where the bytes actually were.\ntok.weight (embedding) 1,024,000 params 71.7%\neverything else 404,992 params 28.3%\nEnter fullscreen mode\nExit fullscreen mode\nNearly three quarters of that \"neural network\" is a lookup table. So the dial worth turning is not the model's precision, it is the\nembedding's\nprecision, quantised independently of the transformer body. The research repo is not public, so what follows comes from the experiment's own sweep and its recorded results rather than from anything you can clone.\nTwo axes, and only one of them is the obvious one\nThe sweep varies two things. Width is int8, int4, int3 and int2 on the embedding, with the body held at int8.\nGranularity\nis one scale for all 8,000 rows, which is what the earlier experiment used, against one scale per token.\nPer-row scales cost V extra fp16 values, which is 16 KB, under 2% of the table. They should matter more for an embedding than for a weight matrix. Token embedding norms differ by orders of magnitude between frequent and rare words, so a single tensor-wide scale spends most of its levels on a handful of outlier rows; the rest of the vocabulary shares a coarse grid.\nEverything is reported on the same evaluation as the earlier frontier work, with the count table and document cache mixed in at their tuned weights, so the numbers land on a comparable curve. No retraining anywhere.\nembedding / body\nMB\nnetwork alone\nmix top-1\nOOD top-1\nfp32 / fp32\n5.72\n0.351\n0.384\n0.185\nint8 per-tensor\n1.43\n0.350\n0.381\n0.185\nint8 per-row\n1.44\n0.350\n0.381\n0.185\nint4 per-tensor\n0.92\n0.325\n0.366\n0.185\nint4 per-row\n0.93\n0.347\n0.384\n0.185\nint3 per-row\n0.80\n0.313\n0.371\n0.177\nint2 per-row\n0.68\n0.110\n0.276\n0.143\nGranularity is the entire reason int4 works\nint4 per-tensor loses 2.5 points of network top-1, from 0.350 down to 0.325. int4 per-row loses 0.3, landing at 0.347; those 16 KB of extra scales, 1.7% more bytes, recover 2.2 of the 2.5 points.\nAt int8 the two granularities are identical, both 0.350. The distinction does not exist until quantisation levels get scarce, which is why it is easy to miss. You test granularity at the width where it does not matter; you find no effect, and drop it from the sweep.\nThat is the part worth taking away. A sweep over width alone would have measured int4 at 0.325, concluded that four bits is where this model breaks, and stopped at int8. The conclusion would have been wrong and it would have looked well-supported, because the width axis really does show a cliff; it is just not at four bits. The cliff is between four and three: 0.371 then 0.276 in the mixture. At four bits with per-row scales there is nothing to pay. The mixture scores 0.384, which is fp32's 0.384 exactly, at 0.93 MB against 5.72 MB. A 6.2x reduction for zero measurable loss.\nTwo axes where one only reveals itself in the tail of the other is not a rare shape, and the only defence I know is to sweep the cross product when it is cheap. Here it was seven runs.\nThe budget arithmetic, done honestly\nThose mixture numbers use the full count table, about 25 MB, so none of the rows above describe a small system. The follow-up redoes the allocation properly: int4-per-row as the network, and the largest pruned count table that fits in whatever the budget leaves.\nbudget\nsystem\nactual MB\nin top-1\nOOD top-1\n0.75 MB\ncounts only\n0.68\n0.289\n0.136\n0.75 MB\nint4 hybrid\nnetwork does not fit\n1.0 MB\ncounts only\n0.68\n0.289\n0.136\n1.0 MB\nint4 + pruned counts\n0.96\n0.356\n0.175\n1.5 MB\ncounts only\n1.28\n0.305\n0.140\n1.5 MB\nint4 + pruned counts\n1.28\n0.366\n0.175\n2.0 MB\ncounts only\n1.28\n0.305\n0.140\n2.0 MB\nint4 + pruned counts\n1.61\n0.372\n0.179\nThe earlier experiment had reported that at 1 MB nothing but counts fits, giving 0.289. A real 1 MB system now scores 0.356, which is 6.7 points better in distribution and 3.9 better out of it. That 0.96 MB system also beats the full-precision 5.72 MB network outright, 0.356 against 0.351, at six times fewer bytes.\nI want to be careful about what that comparison is and is not. It is not \"quantisation improved the model.\" It is that the bytes freed by quantising bought count table and cache, and those were worth more than the precision they replaced. The network alone got slightly worse (0.351 to 0.347). The system got better.\nThe thing I did not expect\nThe best 1 MB configuration prunes the count table down to contexts seen at least 400 times, which is a\n24 KB\ntable, and weights the document cache at 0.40.\nAt that budget, almost all of the non-neural contribution is coming from statistics of the document currently being processed, which cost nothing to store because they are derived from text that is already in memory. An earlier finding in this series was that the cache earns a higher weight the tighter the budget gets. This sharpens it: at the tightest budgets the cache is the only count model you can afford, and it is enough.\nOut of distribution the story is flatter and it makes the same point from the other side. OOD top-1 sits at 0.185 all the way down through int4 per-tensor, and only moves at int3. Out of distribution the mixture's accuracy is dominated by counts and cache, so the network's precision barely registers, which is consistent with everything else in this series: the cheap components matter most exactly where the trained one is weakest.\nWhat this does not show\nQuantisation shrinks a table that was\ntrained\nat full width. A factorised embedding attacks the same 72% by never materialising the wide table at all, and could plausibly go below int4-per-row's 0.93 MB floor, but it requires training, so it is a separate experiment. The open question there is whether factorisation and 4-bit quantisation compose, or whether a low-rank bottleneck has already removed the redundancy that per-row scaling was exploiting. I would guess the latter. Guessing is all that is on offer until it is run.", "metadata": { "title": "The Embedding Table Was 72% of the Model - DEV Community", "description": "Nearly three quarters of my small transformer was a lookup table, so the dial that mattered was embedding precision, not network precision. int4 with one scale per row costs 16 KB more than one scale per tensor and recovers 2.2 of the 2.5 points per-tensor int4 loses. Tagged with ai, machinelearning, deeplearning, llm.", "keywords": "ai, machinelearning, deeplearning, llm, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/megapixel99/the-embedding-table-was-72-of-the-model-2ghm", "og:title": "The Embedding Table Was 72% of the Model", "og:description": "Nearly three quarters of my small transformer was a lookup table, so the dial that mattered was embedding precision, not network precision. int4 with one scale per row costs 16 KB more than one scale per tensor and recovers 2.2 of the 2.5 points per-tensor int4 loses.", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5v9g0y0e377vogqawsid.png", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "The Embedding Table Was 72% of the Model", "twitter:description": "Nearly three quarters of my small transformer was a lookup table, so the dial that mattered was embedding precision, not network precision. int4 with one scale per row costs 16 KB more than one scale per tensor and recovers 2.2 of the 2.5 points per-tensor int4 loses.", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5v9g0y0e377vogqawsid.png" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/8496bd7713ad/code/body-000.txt", "chars": 100, "language": "text" } ], "images": [], "audio": [], "videos": [], "links": [], "repair_version": 2, "repaired_at": "2026-10-14T10:48:40Z", "source_url": "https://dev.to/megapixel99/the-embedding-table-was-72-of-the-model-2ghm", "papers": [], "actual_video_files": [], "body_chars": 6182, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/8496bd7713ad/pages/source.html", "local_html": "attachments/8496bd7713ad/pages/article.html", "body_base_path": "attachments/8496bd7713ad/pages/", "body_markdown_path": "attachments/8496bd7713ad/pages/article.md" }