{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "xbill",
  "title_listed": "Jev After Eight Days of Independent Tests: Level With Mid-Price LLMs, Behind the Frontier",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/gde/jev-after-eight-days-of-independent-tests-level-with-mid-price-llms-behind-the-frontier-1kln",
  "publish_date": "2026-09-24",
  "corpus_stage": "",
  "notes": "",
  "id": "6afb6e675ab0",
  "collected_at": "2026-10-02T13:51:14",
  "title": "Jev After Eight Days of Independent Tests: Level With Mid-Price LLMs, Behind the Frontier - DEV Community",
  "body_markdown": "This article provides a review of the independent evidence on TypeSafe's Jev, the open models built to replace it, and the prior art behind both, as of September 23, 2026. Every figure below is traced to a primary source, and re-scored from committed per-item outputs wherever the author published them. This is a snapshot eight days after launch, and the arXiv preprints it cites are days old and unrefereed.\nOn accuracy, Jev sits level with mid-price LLMs and 6.5 to 11.5 points behind the frontier in the cleanest comparison. Out of the box its probabilities are the best calibrated of the models measured on familiar English tasks and are wrong in both directions off them, and one temperature fitted on 50 to a few hundred labels fixes most of the error.\nhttps://github.com/xbill9/gemma4-dev/tree/main/jev\nWhat Is Jev?\nJev is a hosted model from TypeSafe AI, launched on September 15, 2026. It answers typed questions about a piece of text:\nchoice\npicks one option from a list,\nscore\nplaces the text on an ordered scale, and\nnoul\nreturns the probability that a statement is true. It writes no text.\nThe API is\nPOST /v1/systemone\nat\napi.typesafe.ai\n. The price is $0.042 per million input tokens, and output is free. Access is through a waitlist, OpenRouter and Vercel's AI Gateway.\nTypeSafe calls this category a System One model, after Kahneman's fast and intuitive system. The company raised a $40M seed round led by DCVC, and its co-founders are Diogo Almeida (CEO), Sasha Sheng (COO) and Erik Gafni (CTO).\nThe read itself is an old technique: end the prompt where the answer would start, take the model's scores for only the allowed label tokens, and apply a softmax over them. Any open model can be read this way; vLLM exposes it through\nlogprob_token_ids\n, SGLang through\n/v1/score\n, and Featherless AI's SimpleJev wraps it as a Jev-compatible server.\nWhat Counts as Evidence Here?\nThe review covers 14 arXiv preprints read in full, 104 GitHub evaluation repositories, 33 dev.to and Medium posts, TypeSafe's own pages and documentation, and press and forum coverage. Each figure carries one of three levels of support.\nRe-scored:\nrecomputed from committed per-item outputs, and it matched\nChecked:\nthe source's arithmetic recomputed from its published tables, with no per-item data available\nReported:\nthe author's statement, which cannot be checked from outside\nStudies are graded A (human or public labels, at least 200 items, stated method, committed outputs), B (real but small, synthetic, LLM-labelled or unreproducible) or C (a demo or a restatement).\nBy that rubric, the posts split sharply. Of 33 dev.to and Medium posts read in full, 5 are careful measurements on human or public labels, 11 are small, synthetic or scored by agreement between models, and 17 restate vendor claims. Of the 104 GitHub repositories, 33 grade A, 57 grade B, 5 grade C, and 9 are tools. Of the 14 arXiv preprints, 10 call TypeSafe's hosted Jev.\nWho ran a study matters too. The studies with committed per-item outputs and no product in the race are Ibrahim and Zaki, Janardhan, sanand0, SamuelSacco and OmarMujahid, the last the most favourable to Jev; Rafe and Das also has none, and withholds its per-item outputs. Every's review was written by a launch partner. LangChain, which ships the\nlangchain-typesafe\nintegration, reported 100% on 500 judgments that are 5 test cases repeated 100 times (not re-checked). And the \"15.9% faster pipeline\" headline came from a TypeSafe employee's three-case demo. Bespoke Labs sells Nimble, and the replica authors test their own models. Matt Mastracci wrote the DiffusionGemma PR he compares with Jev, and Google's Gemma account shared his thread. heiko-hotz, whose SNIPS comparison includes Gemma 4 31B, works at Google. synthorai, which found Jev behind flash LLMs, runs a gateway that sells access to the models it compared.\nWhere Does Jev Land on Accuracy?\nLevel with mid-price LLMs and behind the frontier, at a fraction of the frontier's price. That trade is the product's design, and the numbers below show its size.\nThe largest pre-registered study, Ibrahim and Zaki's replication of a social-science annotation suite with 7,977 human-labelled items (arXiv 2609.24574), has Jev behind the best of 19 LLMs on 14 of 15 tasks, by a median 11.6 macro-F1 points. That comparator is the best model per task, picked after the results; within Jev's price band, Gemma 4 31B and Qwen3 235B score 3 and 2 F1 points above it. Its one per-item file checked, the 266-item dialect task, re-scores exactly to the paper's table.\nManjunath Janardhan's 200-item, six-model comparison on BANKING77, BoolQ, Yelp and ChaosNLI re-scores exactly from its committed logs.\nModel\nAccuracy\nCalibration error (ECE)\n🥇 Claude Fable 5.1\n84.0%\n0.064\n🥈 GPT-6 Astra\n79.0%\n0.119\n🥉 DeepSeek V4.1 Flash\n76.0%\n0.138\nMiniMax M3\n75.5%\n0.112\nKimi K3\n74.5%\n0.119\nJev\n72.5%\n0.161\nThe two frontier leads (+11.5 and +6.5 points) exclude zero in their 95% intervals. Kimi K3, MiniMax M3 and DeepSeek V4.1 Flash sit within noise of Jev.\nBanking77 is the one task many authors ran independently. Eight runs, one of them JevBench's own first-party run, put Jev at 0.753 to 0.840, with a median of 0.809. In every study that ran LLMs beside it, the best of them was ahead.\nWhat Does Jev Do Well?\nBinary and few-class decisions, fast, with answers that always fit the schema.\nSpam:\n98.33% on 18,514 emails, level with TF-IDF logistic regression at 98.39%, and it held up on newer mail where the trained baseline fell to 72.5%; the question wording was tuned on labelled errors from the same sets\nReranking:\nlevel with a dedicated reranker on eight retrieval sets, 0.692 against 0.691\nMonitoring code for backdoors:\nAUROC of 0.976 and 0.970, catching about 90% of backdoors at 2% false positives; a best-of-5 attack cut detection to 60%\nSchema compliance:\nzero invalid answers across 23,703 decision-model calls, Jev's included, in the largest study\nRepeatability:\n1.33% of answers changed between identical passes in one study, 2.2% in the phishing benchmark, and 0 of 96 repeats in another\nLanguage:\nGerman cost 0.5 points on MASSIVE\nSpeed and price:\nabout 0.1 s of server time, at $0.042 per million input tokens\nAre Its Probabilities Calibrated?\nThey rank answers usefully; out of the box they beat most alternatives on familiar English tasks and miss badly off them.\nTypeSafe publishes no calibration error, reliability plot, Brier score or log loss for Jev on any dataset. Its only confidence-to-accuracy figure is one cookbook of 60 SEC filings: 90% right at confidence 0.9 or above, 40% below.\nThe independent picture depends on the comparison.\nOn Bespoke Labs' 13-subset public suite, Jev's median ECE is 0.071, lower than Nimble-9B's on 11 of 13 subsets, and lower than every Gemma 4 read in the companion article (0.114 to 0.180) before any fitting.\nAgainst LLMs that write their confidence as a number, Jev's own probabilities usually win. Its median ECE of 0.157 on the social-science tasks beats 16 of 19 LLMs.\nOnce every model gets one fitted temperature, 15 of those LLMs beat it.\nAgainst LLMs that return full probability distributions, Jev's calibration error is the highest in Janardhan's run (0.161) and in sanand0's nine-model Banking77 pilot (0.138).\nThe direction of the error changes with the data. On public multi-class sets Jev is overconfident: on GoEmotions, labels it scored between 0.80 and 0.95 matched the human label 15% of the time. On crash narratives and synthetic items it is under-confident. Within one judge study, refit temperatures ran from 0.65 to 4.45, and its authors conclude that no single temperature fits.\nThe API also rounds every probability to 0.01. In one committed sample, 70.4% of\nchoice\nprobabilities came back as exactly 0, and\nnoul\nis clamped between 0.01 and 0.98.\n🔎 Tip: Fit a Temperature on 50 to a Few Hundred of Your Own Labels\nA small refit fixes most of the calibration error, for Jev and for any open model read the same way.\nA two-parameter refit on civil_comments took Jev's ECE from 0.16–0.21 to under 0.025 on a held-out half\nTransferring a fitted slope and refitting only the intercept on 50 labels cut mean ECE by 74%\nOut-of-fold Platt scaling cut Jev's ECE on crash narratives by 3.35x\nIn the companion article, one temperature fitted on 50 labels brought plain Gemma 4 26B's median ECE from 0.180 to 0.080 on the public suite, against Jev's 0.071 as shipped\nThe open replicas report the same order of correction: Nimble fitted 2.179, Luce 2.5 to 3.3 on unseen tasks. Luce's rule is to ship with 100 to 300 real labelled items and fit the temperature on them.\nHow Much Faster and Cheaper Is It?\nSpeed runs from 0.5x, slower than a local Gemma, to 12.1x faster; cost runs from 0.6x, dearer, to 478x cheaper, depending on the comparison model, the network path and where the clock runs.\nComparison\nSource\nFaster by\nCheaper by\nGPT-5.6 Luna, reasoning off, client clock\nOmarMujahid\n1.04x\n4.3x\nDeepSeek, Banking77\nsanand0\n—\n1.18x\nHaiku 4.5, phishing verdict\njev-phishing-bench\n2.9x\n12x\nMid-size open model, 47 account reviews\ndevopsdaily\n12.1x\n7.4x\nClaude Fable 5.1, via OpenRouter\nJanardhan\n9.9x\n478x\n19 LLMs, social-science tasks (median)\narXiv 2609.24574\n—\n18.6x\nDeepSeek, 6G edge orchestration\narXiv 2609.23136\n1.3x\n0.6x (Jev dearer)\nDeepSeek, edge orchestration, fees per correct completion\narXiv 2609.22753, same lab\n—\n3.2–3.4x\nLocal Gemma 4 26B-A4B Q4 on a mini-PC\nikkun1222\n0.5–0.65x (Jev slower)\n—\nThe server itself is fast: about 105 ms in one run, and about 76 ms once the network round trip is subtracted in another. Most of the cost gap comes from output being free. In devopsdaily's workload 83% of the old bill was output, and per input token the two prices differed by 1.31x. In the companion article, plain Gemma 4 26B on one EC2 L4 at full load cost at most $5.43 per million decisions, against Jev's $5.54 at the same 132-token prompts.\nWhere Does It Break?\nOn wording, language, and questions whose answer is absent from the input.\nOption names move answers. With the question, input, rubric and option set held fixed, swapping which rubric sits behind \"no\" and \"yes\" changed 32.5% of Jev's answers, against about 2% with neutral names, and AUROC fell from 0.81 to 0.58 (arXiv 2609.26758, n = 1,200; reported). The paper is graded C as evidence about Jev: it names no dataset or Jev version, its gold labels likely come from a teacher LLM, its code is unreleased, and no generative LLM was given the same swap.\nBare labels mislead it. A router given option names with no descriptions sent all 40 hard tasks to the cheap model, at a median confidence of 0.96. One-line option descriptions fixed 37 of 40.\nLanguage shift costs accuracy. Russian XNLI dropped from 88.3% to 77.3% with ECE tripling, and Spanish cost 3 to 6 points.\nJev is most confidently wrong where the answer does not follow from the input. With no \"unknown\" option, every answer on KoBBQ bias questions is wrong by design, and Jev gave them 0.79 confidence. On a fair die its\nchoice\nprobabilities averaged 83% against a true 17%, while\nnoul\ngave 19%. On heart-risk data its probabilities ran about 3x too high, and a chat LLM on the same 5,000 people ranked risk slightly better (AUC 0.79 against 0.77).\nQuestions batched in one request cannot see each other's answers. One reviewer's request returned \"suspend\" for an account the same response classified as a developer sending tests.\nTypeSafe's own list of known weaknesses names literal reading, arithmetic and counting, dates, long irrelevant input, adversarial content and phrasing: \"refund\" scored 0.72 and \"not a refund\" 0.47 on the same ticket.\nWhat Does TypeSafe Claim, Checked?\n193.6x faster and 444.6x cheaper.\nThe source is TypeSafe's four in-house workflow evals, which the launch post calls \"on the higher end of real world gains\". TypeSafe discloses that the workflows \"were made by individuals on our model capabilities team, so some bias could exist\", and that its LLM comparators ran through its own probability adapter, which it says is \"slower and more expensive\". The comparator is never named. Checked against the published summary points, 444.6x fits only Jev against Opus 5 in the workflow setup. Averaged over all eight workflow setups, the figures are 97.8x faster and 149.2x cheaper. Timings came from laptops on the US West Coast, with no case counts, run counts or variance.\nEval accuracy.\nReference labels are \"an average of the responses of GPT-6 Astra and Claude Fable 5.1, both at high thinking\", so accuracy there means agreement with two LLMs. Jev scores 67.8%, equal to Sonnet 5, with GPT-5.6 Sol leading at 74.1%. On invoice processing Jev scores 61.8% against Sol's 79.1%. TypeSafe states that it chose to publish no public-benchmark results.\n\"0% hallucination\".\nTypeSafe's launch post: \"Our number is not empirical. Schema matching is guaranteed.\" Every answer is a valid option; the wrong-but-valid rate is the accuracy figures above.\nRLCD.\nReinforcement learning for calibrated decisions has no paper, patent or method description. The CEO said on Hacker News that the architecture is \"close to the chest for now, but we have talked about writing a paper.\" On Latent Space he agreed that all of Jev's training data is synthetic. Model size and base are undisclosed.\nPricing.\nThe launch post says \"We can't prove it isn't subsidized\", and the home-page FAQ says \"We can serve Jev profitably at our current prices.\" How long the launch price holds is an open question for anyone building on it.\nThe frontier comparison.\nOn Hacker News a commenter described Jev as \"basically a zero-shot classifier\" that classifies \"as accurately (they claim) as a frontier-level LLM\", and the CEO replied \"exactly right!\" The independent accuracy results above are the test of the second half.\nThe founder.\nTypeSafe's pages say Diogo Almeida \"co-invented RLHF\". He is not an author of Christiano et al. (2017), which introduced it; he is a primary author of InstructGPT (arXiv 2203.02155), which applied it to instruction following, and on Latent Space he separates the two.\nHow the claims travelled:\nClaim as circulated\nPrimary source\n194x faster and 445x cheaper against GPT-6 Astra (Tom's Hardware)\nAstra is a reference labeller with no scored setup; 444.6x fits Opus 5\nCoverage of arXiv 2609.26758: Jev flips 70.4 of 100 answers\nThe paper gives 70.4 for an open ModernBERT head; Jev's is 32.5%\n\"Its own employee measured 15.9% on a real pipeline\" (Cherry Creek News)\n3 cases of a demo support ticket; costs modelled from an estimated price\nSeveral self-published evaluations also state more than their own data supports, in both directions. The full report lists the corrections for Luce, Laya and NanoJev and uses the corrected figures from every study.\nWhat Predates Jev?\nEvery piece of the mechanism.\nPiece\nPrior art\nScore options by label likelihood\nGPT-3 §2.4 (2020); MMLU §4.1 (2021); lm-evaluation-harness\nmultiple_choice\nMap each label to one token\nVerbalizers, Schick & Schütze (arXiv 2001.07676)\nLabel probabilities carry prior bias\nCalibrate Before Use, Zhao et al. (2021)\nOne fitted temperature fixes overconfidence\nGuo et al. (2017); Kadavath et al. (2022), a temperature of 2.5 for an RLHF policy, beside Jev's 2.66, Nimble's 2.179 and Luce's 2.5 to 3.3\nPost-training worsens calibration\nGPT-4 report, Fig. 8: MMLU ECE 0.007 to 0.074 after PPO\nCross-entropy rewards reporting true probabilities\nGneiting & Raftery (2007): the log score is strictly proper\nCheap model first, defer when unsure\nSelective classification (2017); FrugalGPT\nThe visible contribution is the packaging: a typed API, one hosted endpoint, output priced at zero, and many questions evaluated against one input in a single request. Whether RLCD adds more can be judged once it is published; Jev's native probabilities beating most LLMs' self-reported confidence suggests the training does something.\nWhat About the Open Alternatives?\nEach trained replica is built on Qwen or an encoder, and each publishes its own evaluation.\nProject\nBase\nOwn headline\nIndependent evidence\nNimble-9B (Bespoke Labs)\nQwen3.5-9B + LoRA\n75.9% vs Jev 77.3% on 13 public sets\nnone; run by the seller, with raw probabilities (fitted temperature 2.179); its own model trails Jev\nKev (Jared Palmer)\nQwen3.5 + LoRA + pointer head\nKev-9B 0.852 test vs Jev 0.857 dev\nnone\nLuce (scienthoon)\nQwen3-4B + LoRA\n86.9% vs Jev 84.7% on 500 kubernetes issues\nnone\nLaya (Convai)\nModernBERT-large 421M\n0.766 vs 0.727 after training on the benchmark\nikkun1222: 29.2 / 60.0 / 40.0 on three Japanese tasks; the card's own zero-shot 0.362 sits below its 0.461 majority baseline\nSimpleJev (Featherless)\nany HF model; demo serves Gemma 4 26B\nno accuracy published\nnone\nDiffusionGemma via vLLM PR #57250\none denoise step of\ndiffusiongemma-26B-A4B-it\n198/201 vs Jev 191/201, \"roughly tied\"\nywchiu: 32.2% vs Jev 61.4% on synthetic routing; companion article: 75.9% vs Jev 77.3% on the public suite\nSimpleJev's README states that its probabilities \"are not calibrated probabilities of correctness\". The DiffusionGemma headline comes from 201 items built by the PR's author; publishing the items would let others reproduce it.\nLabels beat zero-shot wherever they were tried. A 310M Japanese encoder trained on 200 rows beat Jev on news topics by 12 points, 88.8% against 76.8%, and tied it on two sentiment tasks, and fine-tuned models led by 2 to 15 points on five public splits. On phishing, Luce trained on 1,000 labels reached 97.4% against Jev's 62.6% on the same benchmark, though on different items. On that benchmark, a two-line regex scored 91.6%, five narrow Jev questions combined by logistic regression reached 95.0% on a held-out half, and Haiku 4.5 asked the same five questions reached 93.2%, a difference too small to be significant (p = 0.063).\nthejevai.com\n, registered on September 20 and promoted on Hugging Face as the endpoint for API keys, has no stated connection to TypeSafe. TypeSafe's API is\napi.typesafe.ai\n.\nCompare and Contrast\nJev\nTrained open replica\nPlain open model, label scores read\nLabels needed\nnone to run; about 50 to fit a temperature\nhundreds to thousands, plus a training run\nnone to run; about 50 to fit a temperature\nAccuracy on human labels\nlevel with Kimi K3, MiniMax M3 and DeepSeek V4.1 Flash; 6.5 to 11.5 behind the frontier\ntrails or matches Jev generically; beats it when trained on the task's own labels\nGemma 4 26B, 4-bit: 75.3% vs Jev 77.3% on the 3,880-record suite; level on yes/no, 4.5 behind on multiple choice\nCalibration\n🥇 median ECE 0.071 as shipped on the public suite; none published by the vendor\nown evaluations\nmedian ECE 0.180 as shipped, 0.080 after 50 labels\nWhere it runs\nTypeSafe, OpenRouter, Vercel\nyour hardware\nyour hardware\nCost\n$0.042 per million input tokens; $5.54 per million decisions at 132 tokens\nyour hardware\nat most $5.43 per million decisions on an L4 at full load\nSo, Which One?\nFor binary and few-class decisions in English, at high volume, with no infrastructure to run and no labels yet, Jev is fast and cheap, its accuracy sits with mid-price LLMs, and its probabilities are the best calibrated out of the box on familiar English tasks. Plan on labelling 50 to a few hundred decisions to set thresholds and fit its probabilities.\nFor multi-class work with many options, decisions where the rule lives outside the input, or non-English text, where Jev loses accuracy, a model trained on your own labels won wherever one was tried; for non-English that evidence is one Japanese study.\nFor data that cannot leave your network, any open model read by label scores gives the same typed interface. In the companion article plain Gemma 4 26B trails Jev by 2.1 points overall and 4.5 on multiple choice, and needs about 50 labels to come within 0.01 of Jev's calibration.\nWhat Is Still Unmeasured?\nThe published Gemma data points so far:\nGemma 4 31B beats Jev's median F1 in the pre-registered social-science study, 0.611 against 0.581\nGemma 4 31B scores 98.86% on SNIPS against Jev's 97.14% after its output's code fences were stripped; 80.57% before\nGemma 4 26B-A4B trails Jev by 11.6 points on Japanese news topics and ties it on two sentiment tasks\nGemma 4 E4B, stating its confidence as text, scores 81.0% against Jev's 95.5% on synthetic emails\nPlain Gemma 4 26B read by its label probabilities, in the companion article: on four public tasks (sst2, AG News, DAIR Emotion, tweet_eval irony) and on Bespoke Labs' 3,880-record, 13-subset suite, 2.1 points behind Jev overall\nStill unpublished: Jev's behaviour across model versions, calibration across difficulty on real data, and whether a generative LLM shows the same option-name sensitivity.\nSummary\nThe goal of this article was to establish what independent evidence shows about TypeSafe's Jev and its open alternatives. The key to the solution was tracing every figure to its primary source and re-scoring from published per-item outputs wherever they existed. The results were:\n🟢 Jev returns a valid option on every call, and its server time is about 0.1 s\n⚠️ Accuracy sits with mid-price LLMs: 72.5% against 74.5–76.0% for Kimi K3, MiniMax M3 and DeepSeek V4.1 Flash in one six-model study, and a median 11.6 F1 behind the best LLM per task in the largest pre-registered one\n🟢 Strong on binary and few-class work: 98.33% on 18,514 emails, and level with a dedicated reranker\n🟢 Best calibrated out of the box on familiar English tasks: median ECE 0.071 on the 13-subset public suite\n🟢 50 to a few hundred of your own labels and one fitted temperature fix most of its calibration error, and most of an open model's\n⚠️ Measured speed gains run from 0.5x to 12.1x and cost gains from 0.6x to 478x; TypeSafe's 193.6x and 444.6x have no named comparator\n⚠️ No calibration metric is published by TypeSafe, and the direction of its calibration error changes by domain\n⚠️ Swapping option names changed about a third of its answers in one unreplicated study\n⚠️ Plain Gemma 4 26B read by label probabilities (companion article) trails Jev by 2.1 points on the 3,880-record suite, is level on yes/no, 4.5 behind on multiple choice, and needs 50 labels to come within 0.01 of Jev's calibration\n❌ RLCD has no published method, and the model's size, base and training data are undisclosed\nScope: sources read on September 23, 2026, eight days after launch: 14 arXiv preprints, 104 GitHub repositories, 33 dev.to and Medium posts, and TypeSafe's own pages; the companion article's public-suite run finished on September 24 UTC. Every accuracy, calibration and timing figure here is a third party's measurement, marked in the full report as re-scored, checked or reported, except the Gemma figures in Compare and Contrast and the Summary, which are the author's own measurement in the companion article; that article's Jev figures come from Bespoke Labs, which sells a competing model. Every run that states a version used\njev-1.13.0\n. Sources were gathered and audited with AI assistance (Claude), and every figure was checked against its source, cited in the full report, except LangChain's, marked not re-checked there. The author is a Google Developer Expert and an AWS Community Builder; neither Google nor AWS funded, reviewed or saw this work. The author has no relationship with TypeSafe, Bespoke Labs or any replica reviewed.\nThe strategy for using primary sources to evaluate Jev was validated with an incremental step by step approach.\nReferences\nFull evidence review with every source:\nhttps://github.com/xbill9/gemma4-dev/blob/main/jev/reports/Jev%20independent%20evidence%20review.md\nCompanion measurement article:\nhttps://dev.to/gde/plain-gemma-4-26b-vs-jev-on-one-ec2-l4-21-points-behind-overall-level-on-yesno-45-behind-on-15k6\nTypeSafe launch post:\nhttps://typesafe.ai/blog/introducing-system-one-models-and-jev\nTypeSafe models and pricing:\nhttps://docs.typesafe.ai/models\nTypeSafe evals:\nhttps://evals.typesafe.ai/\nJev 1.13 known weaknesses:\nhttps://docs.typesafe.ai/model-jaggedness/jev-1.13\nIbrahim & Zaki, arXiv 2609.24574:\nhttps://arxiv.org/abs/2609.24574\nRafe & Das, arXiv 2609.24052:\nhttps://arxiv.org/abs/2609.24052\nLi et al., arXiv 2609.26550:\nhttps://arxiv.org/abs/2609.26550\nOption naming, arXiv 2609.26758:\nhttps://arxiv.org/abs/2609.26758\nJanardhan, jev-frontier-bench:\nhttps://github.com/manjunathshiva/jev-frontier-bench\nNimble public benchmarks:\nhttps://github.com/bespokelabsai/nimble/blob/main/docs/PUBLIC_BENCHMARKS.md\njev-phishing-bench:\nhttps://github.com/anisselbd/jev-phishing-bench\nLuce:\nhttps://github.com/scienthoon/luce\nOmarMujahid, jev-decision-bench:\nhttps://github.com/OmarMujahid/jev-decision-bench\nbitnovus, jev-spam-eval:\nhttps://github.com/bitnovus/jev-spam-eval\nSamuelSacco, jev-exploration:\nhttps://github.com/SamuelSacco/jev-exploration\nsanand0, BANKING77 pilot:\nhttps://sanand0.github.io/llmevals/jev/\nheiko-hotz, jev-evaluation:\nhttps://github.com/heiko-hotz/jev-evaluation\nywchiu, jev_benchmark:\nhttps://github.com/ywchiu/jev_benchmark\nikkun1222, Jev against a 310M encoder:\nhttps://dev.to/ikkun1222/jev-vs-a-310m-encoder-i-trained-myself-750-rows-three-tasks-two-different-winners-242e\nSimpleJev:\nhttps://github.com/featherless-ai/simple-jev\nvLLM PR #57250:\nhttps://github.com/vllm-project/vllm/pull/57250\nMatt Mastracci, Jev against DiffusionGemma:\nhttps://x.com/mmastrac/status/2100626193943052784\nGuo et al., On Calibration of Modern Neural Networks:\nhttps://arxiv.org/abs/1706.04599\nKadavath et al., Language Models (Mostly) Know What They Know:\nhttps://arxiv.org/abs/2207.05221\nBrown et al., GPT-3:\nhttps://arxiv.org/abs/2005.14165\nHendrycks et al., MMLU:\nhttps://arxiv.org/abs/2009.03300\nSchick & Schütze, verbalizers:\nhttps://arxiv.org/abs/2001.07676\nZhao et al., Calibrate Before Use:\nhttps://arxiv.org/abs/2102.09690\nGneiting & Raftery, Strictly Proper Scoring Rules, Prediction, and Estimation, Journal of the American Statistical Association 102 (2007)\nGeifman & El-Yaniv, Selective Classification:\nhttps://arxiv.org/abs/1705.08500\nChen et al., FrugalGPT:\nhttps://arxiv.org/abs/2305.05176\nGPT-4 technical report:\nhttps://arxiv.org/abs/2303.08774\nInstructGPT:\nhttps://arxiv.org/abs/2203.02155\nChristiano et al. 2017:\nhttps://arxiv.org/abs/1706.03741",
  "metadata": {
    "title": "Jev After Eight Days of Independent Tests: Level With Mid-Price LLMs, Behind the Frontier - DEV Community",
    "description": "What the independent measurements of TypeSafe's Jev found in its first eight days: arXiv preprints, GitHub evaluations and blog benchmarks, each traced to its primary source. Accuracy, calibration, speed, cost, failure modes, the prior art, the open alternatives, and what is still unmeasured. Tagged with ai, machinelearning, llm, gemma.",
    "keywords": "ai, machinelearning, llm, gemma, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/gde/jev-after-eight-days-of-independent-tests-level-with-mid-price-llms-behind-the-frontier-1kln",
    "og:title": "Jev After Eight Days of Independent Tests: Level With Mid-Price LLMs, Behind the Frontier",
    "og:description": "What the independent measurements of TypeSafe's Jev found in its first eight days: arXiv preprints, GitHub evaluations and blog benchmarks, each traced to its primary source. Accuracy, calibration, speed, cost, failure modes, the prior art, the open alternatives, and what is still unmeasured.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fxbill9%2Fgemma4-dev%2Fmain%2Fjev%2Fdevto-jev-review-banner.2360df33.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Jev After Eight Days of Independent Tests: Level With Mid-Price LLMs, Behind the Frontier",
    "twitter:description": "What the independent measurements of TypeSafe's Jev found in its first eight days: arXiv preprints, GitHub evaluations and blog benchmarks, each traced to its primary source. Accuracy, calibration, speed, cost, failure modes, the prior art, the open alternatives, and what is still unmeasured.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fxbill9%2Fgemma4-dev%2Fmain%2Fjev%2Fdevto-jev-review-banner.2360df33.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/xbill9/gemma4-dev/tree/main/jev",
      "label": "https://github.com/xbill9/gemma4-dev/tree/main/jev",
      "type": "a"
    },
    {
      "href": "https://github.com/xbill9/gemma4-dev/blob/main/jev/reports/Jev%20independent%20evidence%20review.md",
      "label": "https://github.com/xbill9/gemma4-dev/blob/main/jev/reports/Jev%20independent%20evidence%20review.md",
      "type": "a"
    },
    {
      "href": "https://dev.to/gde/plain-gemma-4-26b-vs-jev-on-one-ec2-l4-21-points-behind-overall-level-on-yesno-45-behind-on-15k6",
      "label": "https://dev.to/gde/plain-gemma-4-26b-vs-jev-on-one-ec2-l4-21-points-behind-overall-level-on-yesno-45-behind-on-15k6",
      "type": "a"
    },
    {
      "href": "https://typesafe.ai/blog/introducing-system-one-models-and-jev",
      "label": "https://typesafe.ai/blog/introducing-system-one-models-and-jev",
      "type": "a"
    },
    {
      "href": "https://docs.typesafe.ai/models",
      "label": "https://docs.typesafe.ai/models",
      "type": "a"
    },
    {
      "href": "https://evals.typesafe.ai/",
      "label": "https://evals.typesafe.ai/",
      "type": "a"
    },
    {
      "href": "https://docs.typesafe.ai/model-jaggedness/jev-1.13",
      "label": "https://docs.typesafe.ai/model-jaggedness/jev-1.13",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2609.24574",
      "label": "https://arxiv.org/abs/2609.24574",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2609.24052",
      "label": "https://arxiv.org/abs/2609.24052",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2609.26550",
      "label": "https://arxiv.org/abs/2609.26550",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2609.26758",
      "label": "https://arxiv.org/abs/2609.26758",
      "type": "a"
    },
    {
      "href": "https://github.com/manjunathshiva/jev-frontier-bench",
      "label": "https://github.com/manjunathshiva/jev-frontier-bench",
      "type": "a"
    },
    {
      "href": "https://github.com/bespokelabsai/nimble/blob/main/docs/PUBLIC_BENCHMARKS.md",
      "label": "https://github.com/bespokelabsai/nimble/blob/main/docs/PUBLIC_BENCHMARKS.md",
      "type": "a"
    },
    {
      "href": "https://github.com/anisselbd/jev-phishing-bench",
      "label": "https://github.com/anisselbd/jev-phishing-bench",
      "type": "a"
    },
    {
      "href": "https://github.com/scienthoon/luce",
      "label": "https://github.com/scienthoon/luce",
      "type": "a"
    },
    {
      "href": "https://github.com/OmarMujahid/jev-decision-bench",
      "label": "https://github.com/OmarMujahid/jev-decision-bench",
      "type": "a"
    },
    {
      "href": "https://github.com/bitnovus/jev-spam-eval",
      "label": "https://github.com/bitnovus/jev-spam-eval",
      "type": "a"
    },
    {
      "href": "https://github.com/SamuelSacco/jev-exploration",
      "label": "https://github.com/SamuelSacco/jev-exploration",
      "type": "a"
    },
    {
      "href": "https://sanand0.github.io/llmevals/jev/",
      "label": "https://sanand0.github.io/llmevals/jev/",
      "type": "a"
    },
    {
      "href": "https://github.com/heiko-hotz/jev-evaluation",
      "label": "https://github.com/heiko-hotz/jev-evaluation",
      "type": "a"
    },
    {
      "href": "https://github.com/ywchiu/jev_benchmark",
      "label": "https://github.com/ywchiu/jev_benchmark",
      "type": "a"
    },
    {
      "href": "https://dev.to/ikkun1222/jev-vs-a-310m-encoder-i-trained-myself-750-rows-three-tasks-two-different-winners-242e",
      "label": "https://dev.to/ikkun1222/jev-vs-a-310m-encoder-i-trained-myself-750-rows-three-tasks-two-different-winners-242e",
      "type": "a"
    },
    {
      "href": "https://github.com/featherless-ai/simple-jev",
      "label": "https://github.com/featherless-ai/simple-jev",
      "type": "a"
    },
    {
      "href": "https://github.com/vllm-project/vllm/pull/57250",
      "label": "https://github.com/vllm-project/vllm/pull/57250",
      "type": "a"
    },
    {
      "href": "https://x.com/mmastrac/status/2100626193943052784",
      "label": "https://x.com/mmastrac/status/2100626193943052784",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.04599",
      "label": "https://arxiv.org/abs/1706.04599",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2207.05221",
      "label": "https://arxiv.org/abs/2207.05221",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2005.14165",
      "label": "https://arxiv.org/abs/2005.14165",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.03300",
      "label": "https://arxiv.org/abs/2009.03300",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2001.07676",
      "label": "https://arxiv.org/abs/2001.07676",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2102.09690",
      "label": "https://arxiv.org/abs/2102.09690",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1705.08500",
      "label": "https://arxiv.org/abs/1705.08500",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2305.05176",
      "label": "https://arxiv.org/abs/2305.05176",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2303.08774",
      "label": "https://arxiv.org/abs/2303.08774",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2203.02155",
      "label": "https://arxiv.org/abs/2203.02155",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.03741",
      "label": "https://arxiv.org/abs/1706.03741",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:24Z",
  "source_url": "https://dev.to/gde/jev-after-eight-days-of-independent-tests-level-with-mid-price-llms-behind-the-frontier-1kln",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2609.24574",
      "bytes": 757232,
      "reused": true,
      "local_path": "papers/repair/d911ccf4ec114ebe5501.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2609.24052",
      "bytes": 1289080,
      "reused": true,
      "local_path": "papers/repair/e926d77aeb12f602d05d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2609.26550",
      "bytes": 2054435,
      "reused": true,
      "local_path": "papers/repair/4eaec50d6de8dbaa50a5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2609.26758",
      "bytes": 260769,
      "reused": true,
      "local_path": "papers/repair/b1bcc1c365f903e9276c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.04599",
      "bytes": 1349691,
      "reused": true,
      "local_path": "papers/repair/4f4f470ce76b55bd8bc5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2207.05221",
      "bytes": 2616253,
      "reused": true,
      "local_path": "papers/repair/e0a6a2885ecc52224784.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2005.14165",
      "bytes": 6768044,
      "reused": true,
      "local_path": "papers/repair/0c1629d8d07d676c2aae.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.03300",
      "bytes": 2074939,
      "reused": true,
      "local_path": "papers/repair/eabe88a3d618ccd5d1e4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2001.07676",
      "bytes": 486241,
      "reused": true,
      "local_path": "papers/repair/b0a84807f9d9b85ecc2d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2102.09690",
      "bytes": 695668,
      "reused": true,
      "local_path": "papers/repair/f904b24f8b1404d53b54.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1705.08500",
      "bytes": 614365,
      "reused": true,
      "local_path": "papers/repair/4797fb8a9af6b41043e3.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.05176",
      "bytes": 1171365,
      "reused": true,
      "local_path": "papers/repair/7364e996f1e4e16db047.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2303.08774",
      "bytes": 5245564,
      "reused": true,
      "local_path": "papers/repair/e00f74df20cd8064349a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2203.02155",
      "bytes": 1797405,
      "reused": true,
      "local_path": "papers/repair/4b8cf107df7118e0ffe8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.03741",
      "bytes": 3218596,
      "reused": true,
      "local_path": "papers/repair/06487c88cb70eb4ab099.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 26205,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/6afb6e675ab0/pages/source.html",
  "local_html": "attachments/6afb6e675ab0/pages/article.html",
  "body_base_path": "attachments/6afb6e675ab0/pages/",
  "body_markdown_path": "attachments/6afb6e675ab0/pages/article.md"
}