{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "xbill",
  "title_listed": "Plain Gemma 4 26B vs Jev on One EC2 L4: 2.1 Points Behind Overall, Level on Yes/No, 4.5 Behind on Multiple Choice",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/gde/plain-gemma-4-26b-vs-jev-on-one-ec2-l4-21-points-behind-overall-level-on-yesno-45-behind-on-15k6",
  "publish_date": "2026-09-24",
  "corpus_stage": "",
  "notes": "",
  "id": "20e142602715",
  "collected_at": "2026-10-02T13:51:16",
  "title": "Plain Gemma 4 26B vs Jev on One EC2 L4: 2.1 Points Behind Overall, Level on Yes/No, 4.5 Behind on Multiple Choice - DEV Community",
  "body_markdown": "This article provides a step by step guide to measuring Gemma 4 26B as a Jev-style decision model on an AWS EC2 L4 GPU, and compares a plain read of its label probabilities against DiffusionGemma's one-step read and against Jev's published results. The measurement was pre-registered, and every per-item output is committed.\nOn Bespoke Labs' 3,880-record public suite, plain Gemma 4 26B trails Jev 1.13.0 by 2.1 points overall, shows no measurable difference on yes/no questions, and trails by 4.5 points on multiple choice. Jev is better calibrated out of the box, and one temperature fitted on 50 labels brings Gemma's median calibration error within 0.01 of Jev's. Against DiffusionGemma, the plain read is level on accuracy, worse calibrated before fitting and similar after, and 1.9 to 5.1 times faster per decision.\nhttps://github.com/xbill9/gemma4-dev/tree/main/jev\nWhy Measure This?\nA Jev-style decision model answers a typed question with a probability for each allowed option, in one forward pass, with no generated text. TypeSafe's Jev does this as a hosted service. Any open model can do it: end the prompt where the answer starts, read the scores of the allowed label tokens, and apply a softmax over those.\nA\ncompanion review of the independent evidence on Jev\nfound two open questions for Gemma. Plain Gemma read this way had no published accuracy or calibration result. DiffusionGemma, read through vLLM PR #57250, had been described by Google's Gemma account on September 18 as \"yielding well-calibrated decision distributions\", with no published measurement behind it.\nThis run answers both on the same GPU, the same prompts, the same label tokens and the same scoring code, using community 4-bit (AWQ) builds of both models. A third run puts both, and Gemma 4 E4B, on the public suite where Bespoke Labs has published results for Jev, so the Gemma numbers sit beside Jev's on identical records.\nAt This Point You Should Have…\nAn AWS account with quota for one G-family instance, and the AWS CLI logged in\nA security group, subnet and instance profile with\nAmazonSSMManagedInstanceCore\nfor remote commands\nPython 3 with\ntransformers\nand\npybase64\nfor the client side\nThe repository cloned:\ngit clone https://github.com/xbill9/gemma4-dev\nand\ncd gemma4-dev/jev\nStep 1 — Pre-Register the Measurement\nThe models, image, serving flags, data, metrics and comparisons are written down and committed before any model call. The file is\nPREREGISTRATION.md\n.\ngit show\n-s\n--oneline\n3c67f62 d94471d a5cda09 894323e adf0c16\nEnter fullscreen mode\nExit fullscreen mode\n3c67f62 jev: pre-registration for the L4 AWQ run; option-order variant, label-count calibration curve, EC2 user-data\nd94471d jev: run 2026-09-23-l4-awq — Gemma 4 26B label logits vs DiffusionGemma one-step reads, both AWQ 4-bit on one EC2 L4; per-item outputs, summary, host evidence\na5cda09 jev: pre-registration addendum (latency pass on-instance, off-label tokens, E2B/E4B bf16 arms); 20-split calibration analysis; archived run log and throughput arithmetic\n894323e jev: plain arm builds its prompt from the served model's own chat template (26B unchanged, checked); E4B/E2B deviation recorded\nadf0c16 jev: Bespoke Labs public-suite run — rebuild with checksum checks, runner via the proxy's Jev parser, scorer matched to Nimble's definitions, instance driver, generic launcher; pre-registration addendum\nEnter fullscreen mode\nExit fullscreen mode\nThe latency and small-model runs were added in one pre-registration addendum after the first run, and the public-suite run in another; each was committed before its own first call, and each deviation from the plan is recorded there. The pre-registration commits to publishing every result, including any where either arm does worse.\nStep 2 — Pick a Matched Pair of Checkpoints\nAn NVIDIA L4 has 24 GB of memory, so both 26B models run at 4 bits. Both checkpoints come from the same uploader and apply the same quantization settings to both models, so what differs is the model and how it is read.\nPlain arm\nDiffusion arm\nCheckpoint\ncyankiwi/gemma-4-26B-A4B-it-AWQ-4bit\ncyankiwi/diffusiongemma-26B-A4B-it-AWQ-INT4\nFormat\ncompressed-tensors, 4-bit, group 32, symmetric\nsame\nSize\n17.19 GB\n17.22 GB\nBoth arms share one set of serving flags.\nStep 3 — Launch One L4 on EC2\nThe security group opens port 8000 to one address only.\naws ec2 create-security-group\n--region\nus-east-1\n--group-name\njev-eval-sg\n\\\n--description\n\"jev eval: vLLM 8000 from one IP\"\n--vpc-id\n<vpc-id>\n--query\nGroupId\n--output\ntext\naws ec2 authorize-security-group-ingress\n--region\nus-east-1\n--group-id\n<sg-id>\n\\\n--protocol\ntcp\n--port\n8000\n--cidr\n<your-ip>/32\nEnter fullscreen mode\nExit fullscreen mode\nThe instance boots the regional GPU Deep Learning AMI and runs\naws/user-data.sh\n, which pulls a vLLM nightly that contains PR #57250 and serves one model at a time.\naws ec2 run-instances\n--region\nus-east-1\n--image-id\n<gpu-dlami>\n--instance-type\ng6.xlarge\n\\\n--subnet-id\n<subnet-id>\n--security-group-ids\n<sg-id>\n\\\n--iam-instance-profile\nName\n=\n<profile>\n--user-data\nfile://aws/user-data.sh\n\\\n--block-device-mappings\n'DeviceName=/dev/sda1,Ebs={VolumeSize=100,VolumeType=gp3,DeleteOnTermination=true}'\n\\\n--tag-specifications\n'ResourceType=instance,Tags=[{Key=Name,Value=jev-eval},{Key=ManagedBy,Value=jev}]'\nEnter fullscreen mode\nExit fullscreen mode\ni-0c5b14e913b4c1019 g6.xlarge   us-east-1a\nEnter fullscreen mode\nExit fullscreen mode\nA\ng6.xlarge\nhas the same L4 as the larger sizes with 16 GB of host memory, so\nuser-data.sh\nadds a 16 GB swapfile on hosts under 30 GB; both 17 GB checkpoints loaded with it in place.\nStep 4 — Serve the Plain Arm\nOne flag set serves both arms: Triton attention, eager mode, prefix caching, 2048-token context, 16 sequences and 32 logprobs. The diffusion arm adds a 64-token canvas.\n/opt/jev/serve.sh ar\nEnter fullscreen mode\nExit fullscreen mode\nThe client waits on\n/v1/models\nuntil the model answers.\nREADY after ~480s\n['cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit']\nEnter fullscreen mode\nExit fullscreen mode\nThat covers the image pull, the 17 GB download and the load. The host record from the instance:\nNVIDIA L4, 595.91.07, 23034 MiB\nimage: vllm/vllm-openai:nightly-e9757321527ca1ecd514c07c1418dd2c53da3d19 digest: sha256:f75fec992c293dd41042745f8aa71bfdf12b188b6a01e0baf075e4bca5d297e5\nvllm 0.29.1rc1.dev573+ge97573215 torch 2.13.0+cu130\nEnter fullscreen mode\nExit fullscreen mode\nStep 5 — Check the Read Before the Full Run\nBoth arms use the prompt, answer template, label tokens and probability code from the PR's own\nstructured_server.py\n.\nrun_eval.py\nconfirms both chat templates render the identical prefix before it sends anything. A five-example run checks the path end to end.\npython3 run_eval.py\n--arm\nautoregressive\n--upstream\nhttp://<host>:8000\n\\\n--model\ncyankiwi/gemma-4-26B-A4B-it-AWQ-4bit\n--run\nsmoke\n--limit\n5\n--concurrency\n2\nEnter fullscreen mode\nExit fullscreen mode\nautoregressive sst2: 5/5 (5s)\nautoregressive ag_news: 5/5 (1s)\nautoregressive emotion: 5/5 (0s)\nautoregressive irony: 5/5 (0s)\nEnter fullscreen mode\nExit fullscreen mode\nEach record stores the label probabilities,\nlabels_returned\n, which confirms every label came back with a real probability, and\nlabel_mass\n, the share of the model's probability that landed on the allowed labels.\nStep 6 — Run Both Arms\nThe data is 300 labelled examples each from sst2, AG News, DAIR Emotion and tweet_eval irony, stratified with a fixed seed. sst2 and irony carry annotators' labels, AG News its source's news categories, and DAIR Emotion labels taken from the hashtags on each tweet. Each arm runs once as written, and the three choice tasks run once more with their options listed in reverse order.\npython3 run_eval.py\n--arm\nautoregressive\n--upstream\n$U\n--model\n$PLAIN\n--run\n2026-09-23-l4-awq\n--concurrency\n8\npython3 run_eval.py\n--arm\nautoregressive\n--upstream\n$U\n--model\n$PLAIN\n--run\n2026-09-23-l4-awq\n--concurrency\n8\n--variant\nreversed\n/opt/jev/serve.sh diffusion\npython3 run_eval.py\n--arm\ndiffusion\n--upstream\n$U\n--model\n$DIFF\n--run\n2026-09-23-l4-awq\n--concurrency\n4\npython3 run_eval.py\n--arm\ndiffusion\n--upstream\n$U\n--model\n$DIFF\n--run\n2026-09-23-l4-awq\n--concurrency\n4\n--variant\nreversed\nwc\n-l\nresults/2026-09-23-l4-awq/diffusion-\n*\n.jsonl\nEnter fullscreen mode\nExit fullscreen mode\n300 results/2026-09-23-l4-awq/diffusion-ag_news.jsonl\n    300 results/2026-09-23-l4-awq/diffusion-ag_news--reversed.jsonl\n    300 results/2026-09-23-l4-awq/diffusion-emotion.jsonl\n    300 results/2026-09-23-l4-awq/diffusion-emotion--reversed.jsonl\n    300 results/2026-09-23-l4-awq/diffusion-irony.jsonl\n    300 results/2026-09-23-l4-awq/diffusion-sst2.jsonl\n    300 results/2026-09-23-l4-awq/diffusion-sst2--reversed.jsonl\n   2100 total\nEnter fullscreen mode\nExit fullscreen mode\nThe diffusion arm reads each example four times with different noise in the answer slots, as the PR's proxy does. The DiffusionGemma server came up in about 300 seconds with the image already pulled.\nStep 7 — Score It\npython3 score.py\n--run\n2026-09-23-l4-awq\nEnter fullscreen mode\nExit fullscreen mode\nscore.py\nwrites\nSUMMARY.md\nand\nsummary.json\nbeside the records. Every figure below comes from those two files. Readouts:\nplain\nis one read of plain Gemma.\ndiffusion\nis the proxy's automatic readout, the mean of four DiffusionGemma reads on the 96% to 100% of examples where its rule re-reads and one read on the rest; the label-count tables use the four-read mean throughout.\nHow Accurate Is Each One?\nLevel on three tasks, plain ahead on AG News.\nTask\nAlways-majority\nPlain Gemma\nDiffusionGemma\nDifference (95% range)\nsst2\n51.0%\n95.0%\n93.3%\n−1.7 (−4.0 to +0.7)\nAG News\n25.0%\n🥇 86.7%\n83.0%\n−3.7 (−6.7 to −0.7)\nDAIR Emotion\n35.0%\n58.3%\n60.3%\n+2.0 (−1.0 to +5.0)\ntweet_eval irony\n60.3%\n89.7%\n87.3%\n−2.3 (−5.7 to +1.0)\nThroughout, level means the 95% range of the difference includes zero, and 🥇 in an accuracy table marks a lead whose range excludes it; in Compare and Contrast 🥇 marks the better value. Only the AG News difference excludes zero.\nAre the Raw Probabilities Calibrated?\nDiffusionGemma's are closer. Plain Gemma's are strongly overconfident.\nTask\nPlain Gemma ECE\nDiffusionGemma ECE\nDifference (95% range)\nsst2\n0.051\n0.045\n−0.005 (−0.021 to +0.020)\nAG News\n0.129\n0.119\n−0.010 (−0.030 to +0.020)\nDAIR Emotion\n0.386\n0.261\n−0.125 (−0.154 to −0.089)\ntweet_eval irony\n0.098\n0.043\n−0.055 (−0.077 to −0.014)\nECE is expected calibration error over 15 bins: the gap between how confident a model says it is and how often it is right. DiffusionGemma is better calibrated on emotion and irony, and level on the other two.\nThe fitted temperatures show the size of the gap. On the pre-registered single split, one temperature that best fits the labels was 3.48 to 7.24 for plain Gemma once 25 or more labels were used, and 0.73 to 2.36 for DiffusionGemma. A temperature above 1 means the model's probabilities are too extreme.\nWhat Do 50 Labels Buy?\nMost of the correction for plain Gemma; a smaller gain for DiffusionGemma, and none on irony.\nEach task's examples split into a fitting half and a held-out half. One temperature is fitted on the first N labels of the fitting half and scored on the held-out half. On the pre-registered single split, 50 labels took plain Gemma from 0.040, 0.139, 0.440 and 0.088 to 0.029, 0.064, 0.102 and 0.042 on sst2, AG News, DAIR Emotion and irony, and DiffusionGemma from 0.047, 0.132, 0.311 and 0.055 to 0.025, 0.061, 0.115 and 0.054.\nOne split of 150 held-out examples moves ECE by several hundredths, so the same fit was repeated over 20 random splits, an analysis added after the run. Means, with the lowest and highest split in brackets:\nTask\nPlain, 0 labels\nPlain, 50 labels\nDiffusion, 0 labels\nDiffusion, 50 labels\nsst2\n0.048\n0.039 (0.021–0.061)\n0.050\n0.042 (0.025–0.081)\nAG News\n0.130\n0.066 (0.037–0.132)\n0.121\n0.085 (0.060–0.138)\nDAIR Emotion\n0.392\n0.089 (0.047–0.161)\n0.266\n0.111 (0.062–0.155)\ntweet_eval irony\n0.099\n0.060 (0.035–0.130)\n0.054\n0.066 (0.031–0.116)\nWith no labels, DiffusionGemma is better calibrated on emotion and irony. With 50, plain Gemma's mean is at or below DiffusionGemma's on all four tasks, and the split-to-split ranges overlap throughout.\nWhat Does DiffusionGemma's Read Do?\nAbout half of its probability lands outside the allowed labels. The median share on the allowed labels per task:\nTask\nPlain Gemma\nDiffusionGemma\nsst2\n1.000\n0.503\nAG News\n0.999\n0.370\nDAIR Emotion\n1.000\n0.392\ntweet_eval irony\n1.000\n0.593\nAt the answer slot the server returns the allowed labels plus the single most likely token. In the latency run that token was\n<eos>\n, the end-of-sequence token, in 40% of reads on sst2, 53% on AG News and 64% on DAIR Emotion, and\nthe\nin 24% of reads on irony. The proxy rescales the label probabilities to sum to one, so the answer reads as confident either way;\nlabel_mass\nis the field that shows it, and it is worth logging in any deployment.\nThe proxy re-reads when the answer slot's entropy is above 0.1. It computes that entropy from the returned tokens' full-vocabulary probabilities without rescaling them, so a label holding half the probability on its own already scores 0.35. It re-read 100% of sst2 and AG News examples, 99.0% of DAIR Emotion and 96.3% of irony.\nThe re-reads barely change the answer: one read and the mean of four are within 0.6 points of accuracy on every task (93.3% and 93.3% on sst2, 83.3% and 83.0% on AG News, 60.7% and 60.3% on DAIR Emotion, 86.7% and 87.3% on irony). How much the four reads disagree separates right from wrong well on sst2 and irony, AUROC 0.891 and 0.873, and weakly on AG News and DAIR Emotion, 0.685 and 0.635.\nDoes Option Order Change the Answer?\nFor both models, by similar amounts. Listing the choice options in reverse order changed:\nTask\nPlain Gemma\nDiffusionGemma\nsst2\n6 of 300 (2.0%)\n7 of 300 (2.3%)\nAG News\n14 of 300 (4.7%)\n21 of 300 (7.0%)\nDAIR Emotion\n27 of 300 (9.0%)\n26 of 300 (8.7%)\nHow Fast Is Each One?\nA second run timed both arms with the client on the instance itself, against\nlocalhost\n, one request at a time, over the first 100 examples of each task. Same image, flags and checkpoints, on a\ng6.4xlarge\n: the same L4 with a larger host.\npython3 run_eval.py\n--arm\nautoregressive\n--upstream\nhttp://localhost:8000\n--model\n$PLAIN\n--run\n2026-09-23-l4-latency\n--limit\n100\n--concurrency\n1\npython3 run_eval.py\n--arm\ndiffusion\n--upstream\nhttp://localhost:8000\n--model\n$DIFF\n--run\n2026-09-23-l4-latency\n--limit\n100\n--concurrency\n1\n--reads\n4\n--keep-top\n5\nEnter fullscreen mode\nExit fullscreen mode\n| Task | Plain, median | Plain, 90th pct | Diffusion one read, median | Diffusion one read, 90th pct | Diffusion 4 reads (first + 3 parallel), median |\n|---|---|---|---|---|---|\n| sst2 | 61 | 62 | 118 | 119 | 306 |\n| ag_news | 61 | 62 | 119 | 120 | 307 |\n| emotion | 61 | 62 | 119 | 120 | 308 |\n| irony | 61 | 61 | 121 | 122 | 312 |\nEnter fullscreen mode\nExit fullscreen mode\nTimes are in milliseconds. A plain Gemma 26B decision takes 61 ms on one L4 for these prompts, which run 105 to 299 tokens with a median of 132; longer inputs such as full tickets or logs take longer. One DiffusionGemma read takes 1.9 to 2.0 times as long, and the automatic rule's four reads 5.0 to 5.1 times, which is the path it takes on nearly every call. At 61 ms one request at a time, one L4 answers about 16 plain decisions a second before any batching.\n🔎 Tip: Fit One Temperature Before You Trust a Threshold\nBoth models give usable probabilities after one fitted temperature.\nscore.py\ndoes it with a grid search on log loss:\ndef\nfit_temperature\n(\nprobs\n,\ngold_idx\n):\nreturn\nmin\n(\nT_GRID\n,\nkey\n=\nlambda\nt\n:\nnll\n([\ntemper\n(\np\n,\nt\n)\nfor\np\nin\nprobs\n],\ngold_idx\n))\nEnter fullscreen mode\nExit fullscreen mode\nLabel 50 real decisions from the task, fit on them, and set any act-or-escalate threshold on the rescaled probabilities. Plain Gemma needs the larger correction, and both reach similar calibration once it is applied.\nWhat About the Smaller Gemma 4 Models?\nThe label read works with any Gemma 4. DiffusionGemma ships only at 26B-A4B, so the smaller models get the plain arm alone, run in bf16 on the same L4 with the same flags. Their chat templates end at the model turn without the empty thought block the 26B template adds, so their prompt follows their own template. These two arms were added in a pre-registration addendum after the first run, and are exploratory. The 26B column is 4-bit and the others bf16, so the gaps mix model size with precision. All three read letter labels (A to F) with the meanings listed in the prompt, the format of the PR's proxy; smaller models may do better with the label words as the answers themselves.\nTask\n26B, 4-bit\nE4B, bf16\nE2B, bf16\nsst2\n95.0%\n94.3%\n88.7%\nAG News\n86.7%\n83.7%\n30.3%\nDAIR Emotion\n58.3%\n54.0%\n53.3%\ntweet_eval irony\n89.7%\n84.3%\n72.7%\nE4B is 0.7 to 5.4 points behind the 26B, needs a smaller correction (fitted temperature 1.89 to 3.77 at 50 labels, against 4.65 to 7.05, both means over 20 splits), and reaches similar calibration once corrected: 0.042, 0.071, 0.089 and 0.081 against the 26B's 0.039, 0.066, 0.089 and 0.060. It changed more answers when the options were reversed on AG News, 35 of 300 against 14.\nE2B answered \"world\" on 277 of 300 AG News examples, and on 263 when the options were listed in reverse order, so the same option wins whatever its position, and its topic accuracy sits near the 25% majority rate. Its fitted temperatures reach 7.99, the top of the pre-registered search range, so its corrected calibration figures understate what a wider search would reach.\nHow Does It Compare With Jev?\nBespoke Labs, which makes the open Nimble-9B decision model, ran Jev 1.13.0 through its API on a 13-subset public suite and published the results per subset, with the converters and record ids. The suite holds 3,880 human-labelled records: yes/no questions from BoolQ, PAWS, SQuAD 2.0, Civil Comments and Aegis 2.0; multiple choice from MultiNLI, PubMedQA, VitaminC and MASSIVE intents in English and German; and five-level ratings from HelpSteer2 and SummEval.\nA third run, on a\ng6.xlarge\nwith the same image and flags, rebuilt the suite from the public sources and read it with all three Gemma arms. All 13 rebuilt subsets matched the published checksums, so the Gemma figures and Bespoke Labs' Jev figures come from the same records.\nEach record is a Jev request, which the PR's proxy converts with its own Jev parser, so the Gemma arms see the same prompt format as the first run: yes/no questions answered with the word yes or no, multiple-choice options lettered A, B, C with the answer read as a letter, and rating levels numbered from 1. Gemma was read in that format as published, with no prompt tuning, while Jev takes the request in its own format; a different prompt could move the multiple-choice gap in either direction. Scoring uses Bespoke Labs' definitions, and\ntests/test_suite_scoring.py\nchecks it against Bespoke Labs' own code.\nbash nimble_suite/build.sh <workdir>\npython3 nimble_suite/run_suite.py\n--arm\nautoregressive\n--upstream\nhttp://localhost:8000\n\\\n--model\n$PLAIN\n--records\n<workdir>/public\n--run\n2026-09-24-l4-suite\npython3 nimble_suite/suite_stats.py\n--run\n2026-09-24-l4-suite\n--run\n2026-09-24-l4-suite-e4b\nEnter fullscreen mode\nExit fullscreen mode\nAccuracy pooled over records, with the 95% range in brackets:\nQuestions\nRecords\nJev 1.13.0, published\nNimble-9B, published\nPlain Gemma 26B\nDiffusionGemma 26B\nGemma 4 E4B\nAll\n3,880\n77.3% (76.0–78.6)\n75.9%\n75.3% (73.9–76.6)\n75.9% (74.5–77.2)\n73.3% (71.9–74.6)\nYes/no\n1,399\n84.6% (82.6–86.4)\n80.1%\n84.8% (82.8–86.6)\n84.5% (82.5–86.3)\n81.9% (79.8–83.8)\nMultiple choice\n1,848\n82.8% (81.1–84.5)\n81.1%\n78.3% (76.4–80.1)\n77.3% (75.4–79.2)\n74.8% (72.8–76.8)\nFive-level rating\n633\n45.2% (41.3–49.1)\n51.2%\n45.5% (41.7–49.4)\n52.8% (48.9–56.6)\n49.6% (45.7–53.5)\nJev leads plain Gemma by 2.1 points over all records (95% range 0.2 to 4.0) and by 4.5 on multiple choice (2.0 to 7.1). Jev's per-record answers are unpublished, so these ranges compare two independent proportions: pairing would narrow them, and records that share a passage or article widen them, most of all for ratings. The overall gap's range starts at 0.2 points, so it is the least secure of the three; the multiple-choice gap is the firm one. PubMedQA and VitaminC each account for 33 of the 84 records behind it, and PubMedQA has the largest single-subset gap, 77.2% against 64.0%.\nOn yes/no questions pooled, plain Gemma shows no measurable difference from Jev, 84.8% against 84.6%, a difference anywhere from 2.8 points ahead to 2.5 behind. Within them Jev is ahead on BoolQ by 5.7 points and plain Gemma on Civil Comments and SQuAD 2.0 by 6.0 and 6.4; PAWS, 5.2 points to Jev, is within its range.\nOn ratings DiffusionGemma leads both Jev, by 7.6 points (2.1 to 13.1), and plain Gemma, on exact-level accuracy; Bespoke Labs advises reading that beside the error of the probability-weighted level, which this article does not report. Plain Gemma and DiffusionGemma are level over all 3,880 records: 187 right only for plain, 211 right only for DiffusionGemma, exact McNemar p = 0.25. On ratings DiffusionGemma was right alone on 97 records against 51, p = 0.0002, and the whole gap comes from SummEval, whose 384 records come from 24 news articles. Records from one article move together, so that p-value overstates the evidence.\nBespoke Labs' open Nimble-9B scores 75.9% over all records, level with the two 26B reads. Gemma 4 E4B trails the 26B by 2.0 points over the whole suite, with single subsets ranging from 9.6 points ahead to 8.0 behind, and trails Jev by 4.1 (2.2 to 6.0).\nMedian calibration error over the 13 subsets, with Bespoke Labs' 10 bins:\nECE as shipped\nECE after 50 labels\nBrier as shipped\nJev 1.13.0, published\n0.071\n0.267\nNimble-9B, published\n0.109\n0.314\nPlain Gemma 26B\n0.180\n0.080\n0.359\nDiffusionGemma 26B\n0.114\n0.074\n0.290\nGemma 4 E4B\n0.173\n0.077\n0.359\nAs shipped, Jev has a lower calibration error than plain Gemma on all 13 subsets, than DiffusionGemma on 11 and than E4B on 12. Brier score, which rewards accuracy and calibration together, favours Jev too: plain Gemma and E4B beat it on 2 of 13 subsets, DiffusionGemma on 4. DiffusionGemma's raw calibration error is lower than plain Gemma's on all 13, and its median share on the allowed labels was 66.5% per read on this suite.\nAfter one temperature per subset, fitted on 50 labels from that subset, the Gemma medians sit 0.003 to 0.009 above Jev's as-shipped median. Per subset, fitted plain Gemma is still above Jev on 8 of 13, by up to 0.049, and the Gemma arms are at or below Jev on 4 to 6. The fitted figures are scored on the held-out half of each subset, 72 to 300 records, and ECE reads higher on fewer records, which works against the Gemma columns. Jev's figures are as shipped, and a temperature fitted on Jev's own output could lower them as well.\nMatt Mastracci, who wrote vLLM PR #57250, compared DiffusionGemma with Jev on 201 hand-built items on September 17, and Google's Gemma account shared the thread the next day. DiffusionGemma answered 198 correctly and Jev 191; he called the two \"roughly tied\" and DiffusionGemma \"the winner, I think.\" Seven of Jev's ten errors fell in one set, 89 words drawn from five sentences, and the items, code and per-item outputs are unpublished. On the public records DiffusionGemma is 1.4 points behind Jev, anywhere from 0.4 ahead to 3.3 behind, so both measurements find the two close overall; by question type they part, level on yes/no, 5.5 points behind on multiple choice (3.0 to 8.1) and 7.6 ahead on ratings. Plain Gemma 26B, read the same way without the diffusion step, lands within 0.6 points of DiffusionGemma over the suite (p = 0.25), so this suite shows no accuracy gain from the diffusion step.\nHis timings show the same cost for the re-reads: 1.9 to 4.6 times one read on his eight sets, against 2.6 times here. One read beat Jev's API on seven of his eight sets and the automatic re-reads lost on all eight; his single reads were timed warm and the re-reads cold, which exaggerates that second gap.\nCompare and Contrast\nJev 1.13.0, published\nPlain Gemma 4 26B, label read\nDiffusionGemma 26B, one-step read\nAccuracy, public suite\n🥇 77.3%\n75.3%, 2.1 behind Jev; level with DiffusionGemma\n75.9%, 1.4 behind Jev, a range that includes a tie\nYes/no, public suite\n84.6%\n84.8%\n84.5%\nMultiple choice, public suite\n🥇 82.8%\n78.3%\n77.3%\nFive-level rating, public suite\n45.2%\n45.5%\n🥇 52.8%\nMedian ECE, public suite, as shipped\n🥇 0.071\n0.180\n0.114, lower than plain on all 13 subsets\nMedian ECE, public suite, after 50 labels\n0.080\n0.074\nMedian Brier, public suite, as shipped\n🥇 0.267\n0.359\n0.290\nAccuracy, four tasks\n🥇 level on three, ahead on AG News\nlevel on three, 3.7 points behind on AG News\nRaw calibration, four tasks\noverconfident, fitted temperature 3.5 to 7.2\n🥇 closer, fitted temperature 0.7 to 2.4\nProbability on the allowed labels\n🥇 about 100%\n37% to 59%, 66.5% on the public suite\nTime per decision\n🥇 61 ms on the instance\n118 to 121 ms for one read, 306 to 312 ms with the automatic rule\nOption-order changes, four tasks\n2.0% to 9.0%\n2.3% to 8.7%\nPrice per million decisions\n$5.54 at 132 input tokens\nat most $5.43 on a\ng6.xlarge\nat full load\nat most $31.72\nServing\nTypeSafe's hosted API\nany vLLM\nvLLM with PR #57250\nSo, Which One?\nBetween the two Gemma 4 26B reads, for a Jev-style decision service on one L4: plain Gemma read by its label probabilities, plus one temperature fitted on about 50 labels. It is at least as accurate on the four tasks and level with DiffusionGemma over the public suite, faster, and places all of its probability on the answers you allowed.\nDiffusionGemma's lower raw calibration error has a range excluding zero on two of four tasks and holds on all 13 public-suite subsets, and it matters when no labels exist at all. With 50 labels the difference is no longer detectable on the four tasks, and on the suite the two split 8 to 5, a count within chance (sign test p = 0.58). On five-level ratings DiffusionGemma scored higher than both Jev and plain Gemma, on few source articles; a rating task is the one place to try both.\nAgainst Jev, on the same public records: plain Gemma 4 26B trails Jev by 2.1 points overall, shows no measurable difference on yes/no questions pooled, and trails it by 4.5 points on multiple choice. Its median calibration error comes within 0.01 of Jev's as-shipped median once one temperature per subset is fitted on 50 labels from that subset; per subset it stays higher on 8 of 13. Jev is the better calibrated with no labels at all.\nWhere a smaller model has to do, Gemma 4 E4B in bf16 gives up 0.7 to 5.4 points against the 26B on the four tasks and 2.0 points over the public suite. E2B falls apart on four-way topic classification with this prompt.\nWhat Does a Decision Cost?\nThe run logs give a throughput. Plain Gemma answered 2,100 decisions in 51 seconds, 41.2 a second, at client concurrency 8. DiffusionGemma, reading each example four times, answered 2,100 in 298 seconds, 7.0 a second, at concurrency 4. The client was on a home connection, so these are lower bounds on what the L4 serves.\nAt the\ng6.xlarge\non-demand price of $0.8048 an hour, that is at most $5.43 per million decisions for plain Gemma and $31.72 for DiffusionGemma with four reads, which ran at half the concurrency, so its figure is the looser bound. TypeSafe prices Jev at $0.042 per million input tokens, which is $5.54 per million decisions at this run's median of 132 input tokens and $12.56 at the longest, 299: about the same as the L4 at full load. The L4 is charged by the hour whether busy or idle, so its per-decision cost holds only at full load. The public suite's longer prompts would raise both the L4's cost and Jev's.\nThe first run, from launch to termination in under 1.2 hours, cost at most $0.97 of instance time. The second, on a\ng6.4xlarge\nat $1.3232 an hour for 1.19 hours, cost $1.57. The public-suite run, on a\ng6.xlarge\nfor 0.82 hours, cost $0.66, including rebuilding the suite and reading 11,640 records across three models. All three, $3.20 together, exclude the prorated 100 GB volume. All ran on demand; G-family spot capacity in us-east-1 was unavailable at launch time.\nTeardown\nEvery command on the instance went through AWS Systems Manager, so no SSH port was ever open, and port 8000 was open to one address.\naws ec2 terminate-instances\n--region\nus-east-1\n--instance-ids\ni-0c5b14e913b4c1019\nEnter fullscreen mode\nExit fullscreen mode\nshutting-down\nEnter fullscreen mode\nExit fullscreen mode\nThen delete the security group, and confirm that nothing tagged\nManagedBy=jev\nremains.\nSummary\nThe goal of this article was to measure Gemma 4 26B as a Jev-style decision model, read two ways, for accuracy and calibration on labelled data, and to set it beside Jev's published results on a public suite. The key to the solution was a matched pair of 4-bit checkpoints on one EC2 L4, identical prompts and label tokens, and a measurement pre-registered before any call. The results were:\n⚠️ On Bespoke Labs' 3,880-record public suite, plain Gemma 4 26B trails Jev 1.13.0 by 2.1 points overall (range 0.2 to 4.0) and 4.5 on multiple choice (2.0 to 7.1)\n🟢 No measurable difference from Jev on yes/no questions pooled, 84.8% against 84.6%\n⚠️ Jev is the best calibrated with no labels: median ECE 0.071 against 0.114 to 0.180 for the Gemma arms\n🟢 One temperature fitted on 50 labels brings the Gemma medians to 0.074 to 0.080; per subset plain Gemma stays above Jev on 8 of 13\n⚠️ DiffusionGemma 1.4 points behind Jev over the suite (0.4 ahead to 3.3 behind): 5.5 behind on multiple choice, 7.6 ahead on ratings; Mastracci's hand-built items had it 198 against 191, both close overall\n🟢 Plain Gemma and DiffusionGemma level on accuracy: plain ahead only on AG News of the four tasks, 75.3% against 75.9% over the suite, p = 0.25\n🟢 DiffusionGemma better calibrated as shipped, on emotion and irony and on all 13 suite subsets; after 50 labels the difference is no longer detectable\n⚠️ DiffusionGemma places only 37% to 59% of its probability on the allowed labels, and the proxy's rescaling hides it\n⚠️ The automatic re-read rule fires on 96% to 100% of examples, costs 2.6 times one read, and moves accuracy by at most 0.6 points\n🟢 61 ms per plain decision on the instance, and at most $5.43 per million at full load, level with Jev's $5.54 at this run's prompt length\n🟢 Gemma 4 E4B in bf16 0.7 to 5.4 points behind the 26B on the four tasks and 2.0 over the suite\n❌ Gemma 4 E2B answered \"world\" on 277 of 300 AG News examples\nScope: one NVIDIA L4 in us-east-1 on three instances, a\ng6.xlarge\nfor the accuracy run, a\ng6.4xlarge\nfor the latency and small-model run and a\ng6.xlarge\nfor the public suite, vLLM\n0.29.1rc1.dev573+ge97573215\n, the 26B models 4-bit from the same uploader with the same quantization settings and E4B and E2B in bf16, 300 examples per task plus the 3,880-record public suite, one run per arm. Google's reference checkpoints are bf16, and 4-bit quantization may affect a diffusion model differently from an autoregressive one, so results at bf16 may differ for either arm. Latency comes from 100 examples per task with the client on the instance; throughput comes from the first run's client on a home connection. The latency, E4B and E2B arms were added in a pre-registration addendum after the first run, and E4B and E2B are exploratory; the 20-split calibration and the public suite's ranges, paired test, medians and per-subset counts were added after their runs, and the per-subset and pooled suite figures are the pre-registered ones. The suite's E4B arm ran under its own run name,\n2026-09-24-l4-suite-e4b\n, with the same instance, image, flags and records. All four test sets and all 13 suite datasets were published before Gemma 4 and may be in its training data, and the high irony scores may indicate it there; whether they are in Jev's is unknown. No Jev call was made: the Jev and Nimble-9B figures are Bespoke Labs' published results on the same records, from one run of Jev 1.13.0 by a company that publishes a competing model, counting an invalid Jev response as wrong, with Jev's probabilities rounded to two decimals by its API. Code, pre-registration and every per-item output are in the repository. Parts of the analysis and writing were done with AI assistance (Claude); every figure comes from the committed output files.\nThe strategy for using label probabilities to run Gemma 4 as a decision model was validated with an incremental step by step approach.\nReferences\nCode, pre-registration and per-item results:\nhttps://github.com/xbill9/gemma4-dev/tree/main/jev\nCompanion review of the independent evidence on Jev:\nhttps://dev.to/gde/jev-after-eight-days-of-independent-tests-level-with-mid-price-llms-behind-the-frontier-1kln\nvLLM PR #57250, DiffusionGemma structured reads:\nhttps://github.com/vllm-project/vllm/pull/57250\nPlain checkpoint:\nhttps://huggingface.co/cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit\nDiffusion checkpoint:\nhttps://huggingface.co/cyankiwi/diffusiongemma-26B-A4B-it-AWQ-INT4\nDiffusionGemma model card:\nhttps://huggingface.co/google/diffusiongemma-26B-A4B-it\nGoogle's Gemma account on DiffusionGemma's calibration:\nhttps://x.com/googlegemma/status/2101069861598482817\nMatt Mastracci, Jev against DiffusionGemma on hand-built items:\nhttps://x.com/mmastrac/status/2100626193943052784\nGuo et al., On Calibration of Modern Neural Networks:\nhttps://arxiv.org/abs/1706.04599\nAmazon EC2 G6 instances:\nhttps://aws.amazon.com/ec2/instance-types/g6/\nTypeSafe Jev:\nhttps://docs.typesafe.ai/concepts/system-one\nBespoke Labs, Nimble public-suite results for Jev 1.13.0 and Nimble-9B:\nhttps://github.com/bespokelabsai/nimble/blob/0e67403/docs/PUBLIC_BENCHMARKS.md",
  "metadata": {
    "title": "Plain Gemma 4 26B vs Jev on One EC2 L4: 2.1 Points Behind Overall, Level on Yes/No, 4.5 Behind on Multiple Choice - DEV Community",
    "description": "Plain Gemma 4 26B read by its label probabilities against DiffusionGemma's one-step read, both as community 4-bit (AWQ) builds on one EC2 L4, on 1,200 labelled examples and on the 3,880-record public suite where Jev 1.13.0 has published results. Pre-registered, with accuracy, calibration, calibration after 0 to 150 labels, latency and cost. Tagged with gemma, aws, machinelearning, llm.",
    "keywords": "gemma, aws, machinelearning, llm, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/gde/plain-gemma-4-26b-vs-jev-on-one-ec2-l4-21-points-behind-overall-level-on-yesno-45-behind-on-15k6",
    "og:title": "Plain Gemma 4 26B vs Jev on One EC2 L4: 2.1 Points Behind Overall, Level on Yes/No, 4.5 Behind on Multiple Choice",
    "og:description": "Plain Gemma 4 26B read by its label probabilities against DiffusionGemma's one-step read, both as community 4-bit (AWQ) builds on one EC2 L4, on 1,200 labelled examples and on the 3,880-record public suite where Jev 1.13.0 has published results. Pre-registered, with accuracy, calibration, calibration after 0 to 150 labels, latency and cost.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fxbill9%2Fgemma4-dev%2Fmain%2Fjev%2Fdevto-gemma4-l4-banner.cb16905f.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Plain Gemma 4 26B vs Jev on One EC2 L4: 2.1 Points Behind Overall, Level on Yes/No, 4.5 Behind on Multiple Choice",
    "twitter:description": "Plain Gemma 4 26B read by its label probabilities against DiffusionGemma's one-step read, both as community 4-bit (AWQ) builds on one EC2 L4, on 1,200 labelled examples and on the 3,880-record public suite where Jev 1.13.0 has published results. Pre-registered, with accuracy, calibration, calibration after 0 to 150 labels, latency and cost.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fxbill9%2Fgemma4-dev%2Fmain%2Fjev%2Fdevto-gemma4-l4-banner.cb16905f.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/20e142602715/code/body-000.txt",
      "chars": 62,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-001.txt",
      "chars": 809,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-002.txt",
      "chars": 304,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-003.txt",
      "chars": 445,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-004.txt",
      "chars": 43,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-005.txt",
      "chars": 21,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-006.txt",
      "chars": 59,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-007.txt",
      "chars": 237,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-008.txt",
      "chars": 158,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-009.txt",
      "chars": 127,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-010.txt",
      "chars": 544,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-011.txt",
      "chars": 442,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-012.txt",
      "chars": 41,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-013.txt",
      "chars": 307,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-014.txt",
      "chars": 333,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-015.txt",
      "chars": 121,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-016.txt",
      "chars": 289,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-017.txt",
      "chars": 82,
      "language": "text"
    },
    {
      "file": "attachments/20e142602715/code/body-018.txt",
      "chars": 14,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/xbill9/gemma4-dev/tree/main/jev",
      "label": "https://github.com/xbill9/gemma4-dev/tree/main/jev",
      "type": "a"
    },
    {
      "href": "https://dev.to/gde/jev-after-eight-days-of-independent-tests-level-with-mid-price-llms-behind-the-frontier-1kln",
      "label": "companion review of the independent evidence on Jev",
      "type": "a"
    },
    {
      "href": "https://github.com/vllm-project/vllm/pull/57250",
      "label": "https://github.com/vllm-project/vllm/pull/57250",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit",
      "label": "https://huggingface.co/cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/cyankiwi/diffusiongemma-26B-A4B-it-AWQ-INT4",
      "label": "https://huggingface.co/cyankiwi/diffusiongemma-26B-A4B-it-AWQ-INT4",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/google/diffusiongemma-26B-A4B-it",
      "label": "https://huggingface.co/google/diffusiongemma-26B-A4B-it",
      "type": "a"
    },
    {
      "href": "https://x.com/googlegemma/status/2101069861598482817",
      "label": "https://x.com/googlegemma/status/2101069861598482817",
      "type": "a"
    },
    {
      "href": "https://x.com/mmastrac/status/2100626193943052784",
      "label": "https://x.com/mmastrac/status/2100626193943052784",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.04599",
      "label": "https://arxiv.org/abs/1706.04599",
      "type": "a"
    },
    {
      "href": "https://aws.amazon.com/ec2/instance-types/g6/",
      "label": "https://aws.amazon.com/ec2/instance-types/g6/",
      "type": "a"
    },
    {
      "href": "https://docs.typesafe.ai/concepts/system-one",
      "label": "https://docs.typesafe.ai/concepts/system-one",
      "type": "a"
    },
    {
      "href": "https://github.com/bespokelabsai/nimble/blob/0e67403/docs/PUBLIC_BENCHMARKS.md",
      "label": "https://github.com/bespokelabsai/nimble/blob/0e67403/docs/PUBLIC_BENCHMARKS.md",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:26Z",
  "source_url": "https://dev.to/gde/plain-gemma-4-26b-vs-jev-on-one-ec2-l4-21-points-behind-overall-level-on-yesno-45-behind-on-15k6",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.04599",
      "bytes": 1349691,
      "reused": true,
      "local_path": "papers/repair/4f4f470ce76b55bd8bc5.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 33746,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/20e142602715/pages/source.html",
  "local_html": "attachments/20e142602715/pages/article.html",
  "body_base_path": "attachments/20e142602715/pages/",
  "body_markdown_path": "attachments/20e142602715/pages/article.md"
}