{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Sebastian Raschka / Ahead of AI",
  "title_listed": "Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)",
  "keywords": "LLM评估,基准,裁判模型",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://magazine.sebastianraschka.com/p/llm-evaluation-4-approaches",
  "publish_date": "2025-10",
  "corpus_stage": "评测/评估",
  "notes": "LLM评估四大方法",
  "id": "f61e677d50bc",
  "collected_at": "2026-10-01T20:32:56",
  "title": "Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)",
  "body_markdown": "How do we actually evaluate LLMs?\nIt’s a simple question, but one that tends to open up a much bigger discussion.\nWhen advising or collaborating on projects, one of the things I get asked most often is how to choose between different models and how to make sense of the evaluation results out there. (And, of course, how to measure progress when fine-tuning or developing our own.)\nSince this comes up so often, I thought it might be helpful to share a short overview of the main evaluation methods people use to compare LLMs. Of course, LLM evaluation is a very big topic that can’t be exhaustively covered in a single resource, but I think that having a clear mental map of these main approaches makes it much easier to interpret benchmarks, leaderboards, and papers.\nI originally planned to include these evaluation techniques in my upcoming book,\nBuild a Reasoning Model (From Scratch)\n, but they ended up being a bit outside the main scope. (The book itself focuses more on verifier-based evaluation.) So I figured that sharing this as a longer article with from-scratch code examples would be nice.\nIn\nBuild A Reasoning Model (From Scratch)\n, I am taking a hands-on approach to building a reasoning LLM from scratch.\nIf you liked “Build A Large Language Model (From Scratch)”, this book is written in a similar style in terms of building everything from scratch in pure PyTorch.\nReasoning is one of the most exciting and important recent advances in improving LLMs, but it’s also one of the easiest to misunderstand if you only hear the term reasoning and read about it in theory. So,\nin this book\n, I am taking a hands-on approach to building a reasoning LLM from scratch.\nThe book is currently in early-access with >100 pages already online, and I have just finished another 30 pages that are currently being added by the layout team. If you joined the early access program (a big\nthank you\nfor your support!), you should receive an email when those go live.\nPS: There’s a lot happening on the LLM research front right now. I’m still catching up on my growing list of bookmarked papers and plan to highlight some of the most interesting ones in the next article.\nBut now, let’s discuss the four main LLM evaluation methods along with their from-scratch code implementations to better understand their advantages and weaknesses.\nUnderstanding the main evaluation methods for LLMs\nThere are four common ways of evaluating trained LLMs in practice:\nmultiple choice\n,\nverifiers\n,\nleaderboards\n, and\nLLM judges\n, as shown in Figure 1 below. Research papers, marketing materials, technical reports, and model cards (a term for LLM-specific technical reports) often include results from two or more of these categories.\nFigure 1: An overview of the 4 different evaluations models covered in this article.\nFurthermore the four categories introduced here fall into two groups:\nbenchmark-based evaluation\nand\njudgment-based evaluation\n, as shown in the figure above.\n(There are also other measures, such as\ntraining loss,\nperplexity\n, and\nrewards\n, but they are usually used internally during model development.)\nThe following subsections provide brief overviews and examples of each of the four  methods.\nMethod 1: Evaluating answer-choice accuracy\nWe begin with a benchmark‑based method: multiple‑choice question answering.\nHistorically, one of the most widely used evaluation methods is multiple-choice benchmarks such as\nMMLU\n(short for Massive Multitask Language Understanding,\nhttps://huggingface.co/datasets/cais/mmlu\n). To illustrate this approach, figure 2 shows a representative task from the MMLU dataset.\nFigure 2: Evaluating an LLM on MMLU by comparing its multiple-choice prediction with the correct answer from the dataset.\nFigure 2 shows just a single example from the MMLU dataset. The complete MMLU dataset consists of 57 subjects (from high school math to biology) with about 16 thousand multiple-choice questions in total, and performance is measured in terms of accuracy (the fraction of correctly answered questions), for example 87.5% if 14,000 out of 16,000 questions are answered correctly.\nMultiple-choice benchmarks, such as MMLU, test an LLM’s knowledge recall in a straightforward, quantifiable way similar to standardized tests, many school exams, or theoretical driving tests.\nNote that figure 2 shows a simplified version of multiple-choice evaluation, where the model’s predicted answer letter is compared directly to the correct one. Two other popular methods exist that involve\nlog-probability scoring\n. I implemented them\nhere on GitHub\n. (As this builds on the concepts explained here, I recommended checking this out after completing this article.)\nThe following subsections illustrate how the MMLU scoring shown in figure 2 can be implemented in code.\n1.2 Loading the model\nFirst, before we can evaluate it on MMLU, we have to load the pre-trained model. Here, we are going to use a from-scratch implementation of Qwen3 0.6B in pure PyTorch, which requires only about 1.5 GB of RAM.\nNote that the Qwen3 model implementation details are not important here; we simply treat it as an LLM we want to evaluate. However, if you are curious, a from-scratch implementation walkthrough can be found in my previous\nUnderstanding and Implementing Qwen3 From Scratch\narticle, and the source code is also available\nhere on GitHub\n.\nInstead of copy & pasting the many lines of Qwen3 source code, we import it from my\nreasoning_from_scratch\nPython library, which can be installed via\npip install reasoning_from_scratch\nor\nuv add reasoning_from_scratch\nCode block 1: Loading a pre-trained model\nfrom pathlib import Path\nimport torch\nfrom reasoning_from_scratch.ch02 import get_device\nfrom reasoning_from_scratch.qwen3 import (\n    download_qwen3_small, Qwen3Tokenizer,\n    Qwen3Model, QWEN_CONFIG_06_B\n)\n\ndevice = get_device()\n\n# Set matmul precision to \"high\" to \n# enable Tensor Cores on compatible GPUs\ntorch.set_float32_matmul_precision(\"high\")\n\n# Uncomment the following line \n# if you encounter device compatibility issues\n# device = \"cpu\"\n\n# Use the base model by default\nWHICH_MODEL = \"base\"\n\nif WHICH_MODEL == \"base\":\n    download_qwen3_small(\n        kind=\"base\", tokenizer_only=False, out_dir=\"qwen3\"\n    )\n    tokenizer_path = Path(\"qwen3\") / \"tokenizer-base.json\"\n    model_path = Path(\"qwen3\") / \"qwen3-0.6B-base.pth\"\n    tokenizer = Qwen3Tokenizer(tokenizer_file_path=tokenizer_path)\n\nelif WHICH_MODEL == \"reasoning\":\n    download_qwen3_small(\n        kind=\"reasoning\", tokenizer_only=False, out_dir=\"qwen3\"\n    )\n    tokenizer_path = Path(\"qwen3\") / \"tokenizer-reasoning.json\"\n    model_path = Path(\"qwen3\") / \"qwen3-0.6B-reasoning.pth\"\n    tokenizer = Qwen3Tokenizer(\n        tokenizer_file_path=tokenizer_path,\n        apply_chat_template=True,\n        add_generation_prompt=True,\n        add_thinking=True,\n    )\n\nelse:\n    raise ValueError(f\"Invalid choice: WHICH_MODEL={WHICH_MODEL}\")\n\nmodel = Qwen3Model(QWEN_CONFIG_06_B)\nmodel.load_state_dict(torch.load(model_path))\nmodel.to(device)\n\n# Optionally enable model compilation for potential performance gains\nUSE_COMPILE = False\nif USE_COMPILE:\n    torch._dynamo.config.allow_unspec_int_on_nn_module = True\n    model = torch.compile(model)\n1.3 Checking the generated answer letter\nIn this section, we implement the simplest and perhaps most intuitive MMLU scoring method, which relies on checking whether a generated multiple-choice answer letter matches the correct answer. This is similar to what was illustrated earlier in Figure 2, which is shown below again for convenience.\nFigure 3: Evaluating an LLM on MMLU by comparing its multiple-choice prediction with the correct answer from the dataset.\nFor this, we will work with an example from the MMLU dataset:\nexample = {\n    \"question\": (\n        \"How many ways are there to put 4 distinguishable\"\n        \" balls into 2 indistinguishable boxes?\"\n    ),\n    “choices”: [\"7\", \"11\", \"16\", \"8\"],\n    “answer”: \"D\",\n}\nNext, we define a function to format the LLM prompts.\nCode block 2: Loading a pre-trained model\ndef format_prompt(example):\n    return (\n        f\"{example['question']}\\n\"\n        f\"A. {example['choices'][0]}\\n\"\n        f\"B. {example['choices'][1]}\\n\"\n        f\"C. {example['choices'][2]}\\n\"\n        f\"D. {example['choices'][3]}\\n\"\n        \"Answer: \"\n    )\n# Trailing space in \"Answer: \" encourages a single-letter next token\nLet’s execute the function on the MMLU example to get an idea of what the formatted LLM input looks like:\nprompt = format_prompt(example)\nprint(prompt)\nThe output is:\nHow many ways are there to put 4 distinguishable balls into 2\nindistinguishable boxes?\nHow many ways are there to put 4 distinguishable balls into 2\nindistinguishable boxes?\nA. 7\nB. 11\nC. 16\nD. 8\nAnswer:\nThe model prompt, as shown above, provides the model with a list of the different answer choices and ends with an\n“Answer: “\ntext that encourages the model to generate the correct answer.\nWhile it is not strictly necessary, it can sometimes also be helpful to provide additional questions along with the correct answers as input, so that the model can observe how it is expected to solve the task. (For example, cases where 5 examples are provided are also known as 5-shot MMLU.) However, for current generations of LLMs, where even the base models are quite capable, this is not required.\nLoading different MMLU samples\nYou can load examples from the MMLU dataset directly via the datasets library (which can be installed\nvia pip install datasets\nor\nuv add datasets\n):\nfrom datasets import load_dataset\n\nconfigs = get_dataset_config_names(\"cais/mmlu\")\ndataset = load_dataset(\"cais/mmlu\", \"high_school_mathematics\")\n# Inspect the first example from the test set:\nexample = dataset[\"test\"][0]\nprint(example)\nAbove, we used the\n“high_school_mathematics”\nsubset; to get a list of the other subsets, use the following code:\nfrom datasets import get_dataset_config_names\n\nsubsets = get_dataset_config_names(\"cais/mmlu\")\nprint(subsets)\nNext, we tokenize the prompt and wrap it in a PyTorch tensor object as input to the LLM:\nprompt_ids = tokenizer.encode(prompt)\nprompt_fmt = torch.tensor(prompt_ids, device=device)\n# Add batch dimension:\nprompt_fmt = prompt_fmt.unsqueeze(0)\nThen, with all that setup out of the way, we define the main scoring function below, which generates a few tokens (here, 8 tokens by default) and extracts the first instance of letter A/B/C/D that the model prints.\nCode block 3: Extracting the generated letter\nfrom reasoning_from_scratch.ch02_ex import (\n    generate_text_basic_stream_cache\n)\n\ndef predict_choice(\n    model, tokenizer, prompt_fmt, max_new_tokens=8\n):\n    pred = None\n    for t in generate_text_basic_stream_cache(\n        model=model,\n        token_ids=prompt_fmt,\n        max_new_tokens=max_new_tokens,\n        eos_token_id=tokenizer.eos_token_id,\n    ):\n        answer = tokenizer.decode(t.squeeze(0).tolist())\n        for letter in answer:\n            letter = letter.upper()\n            # stop as soon as a letter appears\n            if letter in \"ABCD\":\n                pred = letter\n                break\n        if pred:\n            break\n    return pred\nWe can then check the generated letter using the function from the code block above as follows:\npred1 = predict_choice(model, tokenizer, prompt_fmt)\nprint(\n    f\"Generated letter: {pred1}\\n\"\n    f\"Correct? {pred1 == example['answer']}\"\n)\nThe result is:\nGenerated letter: C\nCorrect? False\nAs we can see, the generated answer is incorrect (\nFalse\n) in this case.\nThis was just one of the 270 examples from the\nhigh_school_mathematics\nsubset in MMLU. The screenshot (Figure 4) below show’s the performance of the base model and reasoning variant when executed on the complete subset. The code for this is available\nhere on GitHub\n.\nFigure 4: Base and reasoning model performance on the MMLU\nhigh_school_mathematics\nsubset\nAssuming the questions have an equal answer probability, a random guesser (with uniform probability choosing A, B, C, or D) is expected to achieve 25% probability. So the both the base and reasoning model are not very good.\nMultiple-choice answer formats\nNote that this section implemented a simplified version of multiple-choice evaluation for illustration purposes, where the model’s predicted answer letter is compared directly to the correct one. In practice, more widely used variations exist, such as log-probability scoring, where we measure how likely the model considers each candidate answer rather than just checking the final letter choice. (We discuss probability-based scoring in chapter 4.) For reasoning models, evaluation can also involve assessing the likelihood of generating the correct answer when it is provided as input.\nFigure 5: Other MMLU scoring methods are described and shared on\nGitHub here\nHowever, regardless of which MMLU scoring variant we use, the evaluation still amounts to checking whether the model selects from the predefined answer options.\nA limitation of multiple‑choice benchmarks like MMLU is that they only measure an LLM’s ability to select from predefined options and thus is not very useful for evaluating reasoning capabilities\nbesides checking if and how much knowledge the model has forgotten compared to the base model. It does not capture free-form writing ability or real-world utility.\nStill, multiple-choice benchmarks remain simple and useful diagnostics: for example, a high MMLU score doesn’t necessarily mean the model is strong in practical use, but a low score can highlight potential knowledge gaps.\nMethod 2: Using verifiers to check answers\nRelated to multiple-choice question answering discussed in the previous section, verification-based approaches quantify the LLMs capabilities via an accuracy metric. However, in contrast to multiple-choice benchmarks, verification methods allow LLMs to provide a free-form answer. We then extract the relevant answer portion and use a so-called verifier to compare the answer portion to the correct answer provided in the dataset, as illustrated in Figure 6 below.\nFigure 6: Evaluating an LLM with a verification-based method in free-form question answering. The model generates a free-form answer (which may include multiple steps) and a final boxed answer, which is extracted and compared against the correct answer from the dataset.\nWhen we compare the extracted answer with the provided answer, as shown in figure above, we can employ external tools, such as code interpreters or calculator-like tools/software.\nThe downside is that this method can only be applied to domains that can be easily (and ideally deterministically) verified, such as math and code. Also, this approach can introduce additional complexity and dependencies, and it may shift part of the evaluation burden from the model itself to the external tool.\nHowever, because it allows us to generate an unlimited number of math problem variations programmatically and benefits from step-by-step reasoning, it has become a cornerstone of reasoning model evaluation and development.\nI wrote a comprehensive 35-page on this topic in my “Build a Reasoning Model (From Scratch)” book, so I am skipping the code implementation here. (I submitted the chapter last week. If you have the early access version, you’ll receive an email when it goes live and will be able to read it then. In the meantime, you can find the step-by-step code\nhere on GitHub\n.)\nFigure 7: Excerpt from the verification-based evaluation approach available\nhere on GitHub\nMethod 3: Comparing models using preferences and leaderboards\nSo far, we have covered two methods that offer easily quantifiable metrics such as model accuracy. However, none of the aforementioned methods evaluate LLMs in a more holistic way, including judging the style of the responses. In this section, as illustrated in Figure 8 below, we discuss a judgment-based method, namely, LLM leaderboards.\nFigure 8: A mental model of the topics covered in this book with a focus on the judgment- and benchmark-based evaluation methods covered in this appendix. Having already covered benchmark-based approaches (multiple choice, verifiers) in the previous section, we now introduce judgment-based approaches to measure LLM performance, with this subsection focusing on leaderboards.\nThe leaderboard method described here is a judgment-based approach where models are ranked not by accuracy values or other fixed benchmark scores but by user (or other LLM) preferences on their outputs.\nA popular leaderboard is\nLM Arena\n(formerly\nChatbot Arena\n), where users compare responses from two user-selected or anonymous models and vote for the one they prefer, as shown in Figure 9.\nFigure 9: Example of a judgment-based leaderboard interface (LM Arena). Two LLMs are given the same prompt, their responses are shown side by side, and users vote for the preferred answer.\nThese preference votes, which are collected as shown in the figure above, are then aggregated across all users into a leaderboard that ranks different models by user preference. A current snapshot of the LM Arena leaderboard (accessed on October 3, 2025) is shown below in Figure 10.\nFigure 10: Screenshot of the LM Arena leaderboard that shows the current leading LLMs based on user preferences on text tasks\nIn the remainder of this section, we will implement a simple example of a leaderboard.\nTo create a concrete example, consider users prompting different LLMs in a setup similar to Figure 9. The list below represents pairwise votes where the first model is the winner:\nvotes = [\n    (\"GPT-5\", \"Claude-3\"),\n    (\"GPT-5\", \"Llama-4\"),\n    (\"Claude-3\", \"Llama-3\"),\n    (\"Llama-4\", \"Llama-3\"),\n    (\"Claude-3\", \"Llama-3\"),\n    (\"GPT-5\", \"Llama-3\"),\n]\nIn the list above, each tuple in the votes list represents a pairwise preference between two models, written as\n(winner, loser)\n. So,\n(“GPT-5”, “Claude-3”)\nmeans that a user preferred GPT-5 over a Claude-3 model answer.\nIn the remainder of this section, we will turn the\nvotes\nlist into a leaderboard. For this, we will use the popular\nElo rating system\n, which was originally developed for ranking chess players.\nBefore we look at the concrete code implementation, in short, it works as follows. Each model starts with a baseline score. Then, after each comparison and the preference vote, the model’s rating is updated. (In Elo, the update magnitude depends on how surprising the outcome is.)\nSpecifically, if a user prefers a current model over a highly ranked model, the current model will get a relatively large ranking update and rank higher in the leaderboard. Vice versa, if it wins against a low-ranked opponent, the update is smaller. (And if the current model loses, it is updated in a similar fashion, but with ranking points getting subtracted instead of added.)\nThe code to turn these pairwise rankings into a leaderboard is shown in the code block below.\nCode block 4: Constructing a leaderboard\ndef elo_ratings(vote_pairs, k_factor=32,\n                initial_rating=1000):\n    # Initialize all models with the same base rating\n    ratings = {\n        model: initial_rating\n        for pair in vote_pairs\n        for model in pair\n    }\n\n    # Update ratings after each match\n    for winner, loser in vote_pairs:\n\n        # Expected score for the current winner\n        expected_winner = 1.0 / (\n            1.0 + 10 ** (\n                (ratings[loser] - ratings[winner])\n                / 400.0\n            )\n        )\n\n        # k_factor determines sensitivity of updates\n        ratings[winner] = (\n            ratings[winner]\n            + k_factor * (1 - expected_winner)\n        )\n        ratings[loser] = (\n            ratings[loser]\n            + k_factor * (0 - (1 - expected_winner))\n        )\n\n    return ratings\nThe\nelo_ratings\nfunction defined above takes the votes as input and turns it into a leaderboard, as follows:\nratings = elo_ratings(votes, k_factor=32, initial_rating=1000)\nfor model in sorted(ratings, key=ratings.get, reverse=True):\n    print(f\"{model:8s} : {ratings[model]:.1f}\")\nThis results in the following leaderboard ranking, where the higher the score, the better:\nGPT-5 : 1043.7\nClaude-3 : 1015.2\nLlama-4 : 1000.7\nLlama-3 : 940.4\nSo, how does this work? For each pair, we compute the expected score of the winner using the following formula:\nexpected_winner = 1 / (1 + 10 ** ((rating_loser - rating_winner) / 400))\nThis value\nexpected_winner\nis the model’s predicted chance to win in a no-draw setting based on the current ratings. It determines how large the rating update is.\nFirst, each model starts at\ninitial_rating = 1000\n. If the two ratings (winner and loser) are equal, we have\nexpected_winner = 0.5\n, which indicates an even match. In this case, the updates are:\nrating_winner + k_factor * (1 - 0.5) = rating_winner + 16\n\nrating_loser + k_factor * (0 - (1 - 0.5)) = rating_loser - 16\nNow, if a heavy favorite (a model with a high rating) wins, we have\nexpected_winner ≈ 1\n. The favorite gains only a small amount and the loser loses only a little:\nrating_winner + 32 * (1 - 0.99) = rating_winner + 0.32\n\nrating_loser + 32 * (0 - (1 - 0.99)) = rating_loser - 0.32\nHowever, if an underdog (a model with a low rating) wins, we have\nexpected_winner ≈ 0\n, and the winner gets almost the full\nk_factor\npoints while the loser loses about the same magnitude:\nrating_winner + 32 * (1 - 0.01) = rating_winner + 31.68\n\nrating_loser + 32 * (0 - (1 - 0.01)) = rating_loser - 31.68\nOrder matters\nThe Elo approach updates ratings after each match (model comparisons), so later results build on ratings that have already been updated. This means the same set of outcomes, when presented in a different order, can end with slightly different final scores. This effect is usually mild, but it can happen especially when an upset happens early versus late.\nTo reduce this order effect, we can shuffle the votes pairs and run the\nelo_ratings\nfunction multiple times and average the ratings.\nLeaderboard approaches such as the one described above provide a more dynamic view of model quality than static benchmark scores. However, the results can be influenced by user demographics, prompt selection, and voting biases. Benchmarks and leaderboards can also be gamed, and users may select responses based on style rather than correctness. Finally, compared to automated benchmark harnesses, leaderboards do not provide instant feedback on newly developed variants, which makes them harder to use during active model development.\nOther ranking methods\nThe LM Arena originally used the Elo method described in this section but recently transitioned to a statistical approach based on the Bradley–Terry model. The main advantage of the Bradley-Terry model is that, being statistically grounded, it allows the construction of confidence intervals to express uncertainty in the rankings. Also, in contrast to the Elo ratings, the Bradley-Terry model estimates all ratings jointly using a statistical fit over the entire dataset, which makes it immune to order effects.\nTo keep the reported scores in a familiar range, the Bradley-Terry model is fitted to produce values comparable to Elo. Even though the leaderboard no longer officially uses Elo ratings, the term “Elo” remains widely used by LLM researchers and practitioners when comparing models. A code example showing the Elo rating is available\nhere on GitHub\n.\nFigure 11: A comparison of Elo and Bradley-Terry rankings; the source code is available\nhere on GitHub\n.\nMethod 4: Judging responses with other LLMs\nIn the early days, LLMs were evaluated using statistical and heuristics-based methods, including a measure called\nBLEU\n, which is a crude measure of how well generated text matches reference text. The problem with such metrics is that they require exact word matches and don’t account for synonyms, word changes, and so on.\nOne solution to this problem, if we want to judge the written answer text as a whole, is to use relative rankings and leaderboard-based approaches as discussed in the previous section. However, a downside of leaderboards is the subjective nature of the preference-based comparisons as it involves human feedback (as well as the challenges that are associated with collecting this feedback).\nA related method is to use another LLM with a pre-defined grading\nrubric\n(i.e., an evaluation guide) to compare an LLM’s response to a reference response and judge the response quality based on a pre-defined rubric, as illustrated in Figure 12.\nFigure F12: Example of an LLM-judge evaluation. The model to be evaluated generates an answer, which is then scored by a separate judge LLM according to a rubric and a provided reference answer.\nIn practice, the judge-based approach shown in Figure 12 works well when the judge LLM is strong. Common setups use leading proprietary LLMs via an API (e.g., the GPT-5 API), though specialized judge models also exist. (E.g., one of the many examples is\nPhudge\n; ultimately, most of these specialized models are just smaller models fine-tuned to have similar scoring behavior as proprietary GPT models.)\nOne of the reasons why judges work so well is also that evaluating an answer is often easier than generating one.\nTo implement a judge-based model evaluation as shown in Figue 12 programmatically in Python, we could either load one of the larger Qwen3 models in PyTorch and prompt it with a grading rubric and the model answer we want to evaluate.\nAlternatively, we can use other LLMs through an API, for example the ChatGPT or Ollama API.\nAs we already know how to load Qwen3 models in PyTorch, to make it more interesting, in the remainder of the section, we will implement the judge-based evaluation shown in Figure 12 using the Ollama API in Python.\nSpecifically, we will use the 20-billion parameter gpt-oss open-weight model by OpenAI as it offers a good balance between capabilities and efficiency. For more information about gpt-oss, please see my\nFrom GPT-2 to gpt-oss: Analyzing the Architectural Advances\narticle:\nFrom GPT-2 to gpt-oss: Analyzing the Architectural Advances\nSebastian Raschka, PhD\n·\nAugust 9, 2025\nRead full story\n4.1 Implementing a LLM-as-a-judge approach in Ollama\nOllama\nis an efficient open-source application for running LLMs on a laptop. It serves as a wrapper around the open-source\nllama.cpp\nlibrary, which implements LLMs in pure C/C++ to maximize efficiency. However, note that Ollama is only a tool for generating text using LLMs (inference) and does not support training or fine-tuning LLMs.\nTo execute the following code, please install Ollama by visiting the official website at\nhttps://ollama.com\nand follow the provided instructions for your operating system:\nFor macOS and Windows users: Open the downloaded Ollama application. If prompted to install command-line usage, select “yes.”\nFor Linux users: Use the installation command available on the Ollama website.\nBefore implementing the model evaluation code, let’s first download the gpt-oss model and verify that Ollama is functioning correctly by using it from the command line terminal.\nExecute the following command on the command line (not in a Python session) to try out the 20 billion parameter gpt-oss model:\nollama run gpt-oss:20b\nThe first time you execute this command, the 20 billion parameter gpt-oss model, which takes up 14 GB of storage space, will be automatically downloaded. The output looks as follows:\n$ ollama run gpt-oss:20b\npulling manifest \npulling b112e727c6f1: 100% ▕██████████████████████▏  13 GB                         \npulling fa6710a93d78: 100% ▕██████████████████████▏ 7.2 KB                         \npulling f60356777647: 100% ▕██████████████████████▏  11 KB                         \npulling d8ba2f9a17b3: 100% ▕██████████████████████▏   18 B                         \npulling 55c108d8e936: 100% ▕██████████████████████▏  489 B                         \nverifying sha256 digest \nwriting manifest \nremoving unused layers \nsuccess\nAlternative Ollama models\nNote that the gpt-oss:20b in the ollama run gpt-oss:20b command refers to the 20 billion parameter gpt-oss model. Using Ollama with the gpt-oss:20b model requires approximately 13 GB of RAM. If your machine does not have sufficient RAM, you can try using a smaller model, such as the 4 billion parameter qwen3:4b model via ollama run qwen3:4b, which only requires around 4 GB of RAM.\nFor more powerful computers, you can also use the larger 120-billion parameter gpt-oss model by replacing gpt-oss:20b with gpt-oss:120b. However, keep in mind that this model requires significantly more computational resources.\nOnce the model download is complete, we are presented with a command-line interface that allows us to interact with the model. For example, try asking the model, “What is 1+2?”:\n>>> What is 1+2?\n\nThinking...\n\nUser asks: “What is 1+2?” This is simple: answer 3. Provide explanation? Possibly ask for simple\n\narithmetic. Provide answer: 3.\n\n...done thinking.\n\n1 + 2 = **3**\nYou can end this ollama run gpt-oss:20b session using the input\n/bye\n.\nYou can end this ollama run gpt-oss:20b session using the input /bye.\nIn the remainder of this section, we will use the ollama API. This approach requires that Ollama is running in the background. There are three different options to achieve this:\n1. Run the\nollama serve\ncommand in the terminal (recommended). This runs the Ollama backend as a server, usually on\nhttp://localhost:11434\n. Note that it doesn’t load a model until it’s called through the API (later in this section).\n2. Run the\nollama run gpt-oss:20b\ncommand similar to earlier, but keep it open and don’t exit the session via\n/bye\n. As discussed earlier, this opens a minimal convenience wrapper around a local Ollama server. Behind the scenes, it uses the same server API as ollama serve.\n3. Ollama desktop app. Opening the desktop app runs the same backend automatically and provides a graphical interface on top of it as shown in Figure 12 earlier.\nFigure 13: Two different options to keep the Ollama server (/application) running so we can use it via the Ollama API in Python.\nOllama server IP address\nOllama runs locally on our machine by starting a local server-like process. When running ollama serve in the terminal, as described above, you may encounter an error message saying\nError: listen tcp 127.0.0.1:11434: bind: address already in use\n.\nIf that’s the case, try use the command\nOLLAMA_HOST=127.0.0.1:11435 ollama serve\n(and if this address is also in use, try to increment the numbers by one until you find an address not in use.)\nThe following code verifies that the Ollama session is running properly before we use Ollama to evaluate the test set responses generated in the previous section:\nCode block 5: Checking if Ollama is running\nimport psutil\n\ndef check_if_running(process_name):\n    running = False\n    for proc in psutil.process_iter([\"name\"]):\n        if process_name in proc.info[\"name\"]:\n            running = True\n            break\n    return running\n\nollama_running = check_if_running(\"ollama\")\n\nif not ollama_running:\n    raise RuntimeError(\n        \"Ollama not running. \"\n        \"Launch ollama before proceeding.\"\n    )\nprint(\"Ollama running:\", check_if_running(\"ollama\"))\nEnsure that the output from executing the previous code displays Ollama running:\nTrue\n. If it shows\nFalse\n, please verify that the\nollama serve\ncommand or the Ollama application is actively running (see Figure 13).\nIn the remainder of this article, we will interact with the local gpt-oss model, running on our machine, through the Ollama REST API using Python. The following\nquery_model\nfunction demonstrates how to use the API:\nCode block 6: Querying a local Ollama model\nimport json\nimport urllib.request\n\n\ndef query_model(\n    prompt,\n    model=\"gpt-oss:20b\",\n    # If you used \n    # OLLAMA_HOST=127.0.0.1:11435 ollama serve\n    # update the address below\n    url=\"http://localhost:11434/api/chat\"\n):\n    # Create the data payload as a dictionary:\n    data = {\n        \"model\": model,\n        \"messages\": [\n            {\"role\": \"user\", \"content\": prompt}\n        ],\n        # Settings required for deterministic responses:\n        \"options\": {\n            \"seed\": 123,\n            \"temperature\": 0,\n            \"num_ctx\": 2048\n        }\n    }\n\n    # Convert the dictionary to JSON and encode it to bytes\n    payload = json.dumps(data).encode(\"utf-8\")\n\n    # Create a POST request and add headers\n    request = urllib.request.Request(  \n        url,\n        data=payload,\n        method=\"POST\"\n    )\n    request.add_header(\"Content-Type\", \"application/json\")\n\n    response_data = \"\"\n\n    # Send the request and capture the streaming response\n    with urllib.request.urlopen(request) as response:\n        while True:\n            line = response.readline().decode(\"utf-8\")\n            if not line:\n                break\n            # Parse each line into JSON\n            response_json = json.loads(line)\n            response_data += response_json[\"message\"][\"content\"]\n\n    return response_data\nHere’s an example of how to use the\nquery_model\nfunction that we just implemented:\nollama_model = \"gpt-oss:20b\"\nresult = query_model(\"What is 1+2?\", ollama_model)\nprint(result)\nThe resulting response is “3”. (It differs from what we’d get if we ran Ollama run or the Ollama application due to different default settings.)\nUsing the\nquery_model\nfunction, we can evaluate the responses generated by our model with a prompt that includes a grading rubric asking the gpt-oss model to rate our target model’s responses on a scale from 1 to 5 based on a correct answer as a reference.\nThe prompt we use for this is shown below:\nCode block 7: Setting up the prompt template including grading rubric\ndef rubric_prompt(instruction, reference_answer, model_answer):\n    rubric = (\n        \"You are a fair judge assistant. You will be \"\n        \"given an instruction, a reference answer, and \"\n        \"a candidate answer to evaluate, according to \"\n        \"the following rubric:\\n\\n\"\n        \"1: The response fails to address the \"\n        \"instruction, providing irrelevant, incorrect, \"\n        \"or excessively verbose content.\\n\"\n        \"2: The response partially addresses the \"\n        \"instruction but contains major errors, \"\n        \"omissions, or irrelevant details.\\n\"\n        \"3: The response addresses the instruction to \"\n        \"some degree but is incomplete, partially \"\n        \"correct, or unclear in places.\\n\"\n        \"4: The response mostly adheres to the \"\n        \"instruction, with only minor errors, \"\n        \"omissions, or lack of clarity.\\n\"\n        \"5: The response fully adheres to the \"\n        \"instruction, providing a clear, accurate, and \"\n        \"relevant answer in a concise and efficient \"\n        \"manner.\\n\\n\"\n        \"Now here is the instruction, the reference \"\n        \"answer, and the response.\\n\"\n    )\n\n    prompt = (\n        f\"{rubric}\\n\"\n        f\"Instruction:\\n{instruction}\\n\\n\"\n        f\"Reference Answer:\\n{reference_answer}\\n\\n\"\n        f\"Answer:\\n{model_answer}\\n\\n\"\n        f\"Evaluation: \"\n    )\n    return prompt\nThe\nmodel_answer\nin the\nrubric_prompt\nis intended to represent the response produced by our own model in practice. For illustration purposes, we hardcode a plausible model answer here rather than generating it dynamically. (However, feel free to use the Qwen3 model we loaded at the beginning of this article to generate a real\nmodel_answer\n).\nNext, let’s generate the rendered prompt for the Ollama model:\nrendered_prompt = rubric_prompt(\n    instruction=(\n        \"If all birds can fly, and a penguin is a bird, \"\n        \"can a penguin fly?\"\n    ),\n    reference_answer=(\n        \"Yes, according to the premise that all birds can fly, \"\n        \"a penguin can fly.\"\n    ),\n    model_answer=(\n        \"Yes – under those premises a penguin would be able to fly.\"\n    )\n)\nprint(rendered_prompt)\nThe output is as follows:\nYou are a fair judge assistant. You will be given an instruction, a\nreference answer, and a candidate answer to evaluate, according to the\nfollowing rubric:\n\n1: The response fails to address the instruction, providing irrelevant,\nincorrect, or excessively verbose content.\n2: The response partially addresses the instruction but contains major\nerrors, omissions, or irrelevant details.\n3: The response addresses the instruction to some degree but is\nincomplete, partially correct, or unclear in places.\n4: The response mostly adheres to the instruction, with only minor\nerrors, omissions, or lack of clarity.\n5: The response fully adheres to the instruction, providing a clear,\naccurate, and relevant answer in a concise and efficient manner.\n\nNow here is the instruction, the reference answer, and the response.\n\nInstruction:\nIf all birds can fly, and a penguin is a bird, can a penguin fly?\n\nReference Answer:\nYes, according to the premise that all birds can fly, a penguin can\nfly.\n\nAnswer:\nYes – under those premises a penguin would be able to fly.\n\nEvaluation:\nEnding the prompt in\n“Evaluation: “\nincentivizes the model to generate the answer. Let’s see how the gpt-oss:20b model judges the response:\nresult = query_model(rendered_prompt, ollama_model)\nprint(result)\nThe response is as follows:\n**Score: 5**\n\nThe candidate answer directly addresses the question, correctly applies the given premises, and concisely states that a penguin would be able to fly. It is accurate, relevant, and clear.\nAs we can see, the answer receives the highest score, which is reasonable, as it is indeed correct. While this was a simple example stepping through the process manually, we could take this idea further and implement a for-loop that iteratively queries the model (for example, the Qwen3 model we loaded earlier) with questions from an evaluation dataset and evaluate it via gpt-oss and calculate the average score. You can find an implementation of such a script where we evaluate the Qwen3 model on the MATH-500 dataset\nhere on GitHub\n.\nFigure 14: A comparison of the Qwen3 0.6 base and reasoning variants on the first 10 examples in MATH-500 evaluated by gpt-oss:20b as a judge. You can find the code\nhere on GitHub\n.\nScoring intermediate reasoning steps with process reward models\nRelated to symbolic verifiers and LLM judges, there is a class of learned models called\nprocess reward models\n(PRMs). Like judges, PRMs can evaluate reasoning traces beyond just the final answer, but unlike general judges, they focus specifically on the intermediate steps of reasoning. And unlike verifiers, which check correctness symbolically and usually only at the outcome level, PRMs provide step-by-step reward signals during training in reinforcement learning. We can categorize PRMs as “step-level judges,” which are predominantly developed for training, not pure evaluation. (In practice, PRMs are difficult to train reliably at scale. For example, DeepSeek R1 did not adopt PRMs and instead combined verifiers for the reasoning training.)\nJudge-based evaluations offer advantages over preference-based leaderboards, including scalability and consistency, as they do not rely on large pools of human voters. (Technically, it is possible to outsource the preference-based rating behind leaderboards to LLM judges as well). However, LLM judges also share similar weaknesses with human voters: results can be biased by model preferences, prompt design, and answer style. Also, there is a strong dependency on the choice of judge model and rubric, and they lack the reproducibility of fixed benchmarks.\nConclusion\nIn this article, we covered four different evaluation approaches: multiple choice, verifiers, leaderboards, and LLM judges.\nI know this was a long article, but I hope you found it useful for getting an overview of how LLMs are evaluated. A from-scratch approach like this can be verbose, but it is a great way to understand how these methods work under the hood, which in turn helps us identify weaknesses and areas for improvement.\nThat being said, you are probably wondering, “What is the best way to evaluate an LLM?” Unfortunately, there is no single best method since, as we have seen, each comes with different trade-offs. In short:\nMultiple-choice\n(+) Relatively quick and cheap to run at scale\n(+) Standardized and reproducible across papers (or model cards)\n(-) Measures basic knowledge recall\n(-) Does not reflect how LLMs are used in the real world\nVerifiers\n(+) Standardized, objective grading for domains with ground truth\n(+) Allows free-form answers (with some constraints on final answer formatting)\n(+) Can also score intermediate steps if using process verifiers or process reward models\n(-) Requires verifiable domains (for example, math or code), and building good verifiers can be tricky\n(-) Outcome-only verifiers evaluate only the final answer, not reasoning quality\nArena-style leaderboards (human pairwise preference)\n(+) Directly answers “Which model do people prefer?” on real prompts\n(+) Allows free-form answers and implicitly accounts for style, helpfulness, and safety\n(-) Expensive and time-intensive for humans\n(-) Does not measure correctness, only preference\n(-) Nonstationary populations can affect stability\nLLM-as-a-judge\n(+) Scalable across many tasks\n(+) Allows free-form answers\n(-) Dependent on the judge’s capability (ensembles can make this more robust)\n(-) Depends on rubric choice\nWhile I am usually not a big fan of radar plots, one can be helpful here to visualize these different evaluation areas, as shown below.\nFigure 15: A radar chart showing conceptually that we ideally want to pay attention to different areas when evaluating an LLM to identify its strengths and weaknesses.\nFor instance, a strong multiple-choice rating suggests that the model has solid general knowledge. Combine that with a strong verifier score, and the model is likely also answering technical questions correctly. However, if the model performs poorly on LLM-as-a-judge and leaderboard evaluations, it may struggle to write or articulate responses effectively and could benefit from some RLHF.\nSo, the best evaluation combines multiple areas. But ideally it also uses data that directly aligns with your goals or business problems. For example, suppose you are implementing an LLM to assist with legal or law-related tasks. It makes sense to run the model on standard benchmarks like MMLU as a quick sanity check, but ultimately you will want to tailor the evaluations to your target domain, such as law. You can find public benchmarks online that serve as good starting points, but in the end, you will want to test with your own proprietary data. Only then can you be reasonably confident that the model has not already seen the test data during training.\nIn any case, model evaluation is a very big and important topic. I hope this article was useful in explaining how the main approaches work, and that you took away a few useful insights for the next time you look at model evaluations or run them yourself.\nAs always,\nHappy tinkering!\nThis magazine is a personal passion project, and your support helps keep it alive.\nIf you’d like to support my work, please consider my\nBuild a Large Language Model (From Scratch)\nbook or its follow-up,\nBuild a Reasoning Model (From Scratch)\n. (I’m confident you’ll get a lot out of these; they explain how LLMs work in depth you won’t find elsewhere.)\nThanks for reading, and for helping support independent research!\nBuild a Large Language Model (From Scratch) is now available on\nAmazon\n. Build a Reasoning Model (From Scratch) is in\nEarly Access at Manning\n.\nIf you read the book and have a few minutes to spare, I’d really appreciate a\nbrief review\n. It helps us authors a lot!\nYour support means a great deal! Thank you!",
  "metadata": {
    "title": "Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)",
    "description": "Multiple-Choice Benchmarks, Verifiers, Leaderboards, and LLM Judges with Code Examples",
    "keywords": null,
    "author": "Sebastian Raschka, PhD",
    "og:type": "article",
    "og:title": "Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)",
    "og:description": "Multiple-Choice Benchmarks, Verifiers, Leaderboards, and LLM Judges with Code Examples",
    "og:image": "https://substackcdn.com/image/fetch/$s_!c7Za!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1748fa24-e946-47fb-bf1b-e488d08547fd_1764x1244.png",
    "og:url": "https://magazine.sebastianraschka.com/p/llm-evaluation-4-approaches",
    "twitter:title": "Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)",
    "twitter:description": "Multiple-Choice Benchmarks, Verifiers, Leaderboards, and LLM Judges with Code Examples",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!mEAB!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fsebastianraschka.substack.com%2Fapi%2Fv1%2Fpost_preview%2F175225406%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2025-10-10T18:59:36.605Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/f61e677d50bc/code/body-000.txt",
      "chars": 34,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-001.txt",
      "chars": 29,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-002.txt",
      "chars": 1613,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-003.txt",
      "chars": 204,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-004.txt",
      "chars": 329,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-005.txt",
      "chars": 45,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-006.txt",
      "chars": 117,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-007.txt",
      "chars": 236,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-008.txt",
      "chars": 109,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-009.txt",
      "chars": 150,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-010.txt",
      "chars": 669,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-011.txt",
      "chars": 141,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-012.txt",
      "chars": 34,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-013.txt",
      "chars": 176,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-014.txt",
      "chars": 829,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-015.txt",
      "chars": 171,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-016.txt",
      "chars": 65,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-017.txt",
      "chars": 72,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-018.txt",
      "chars": 120,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-019.txt",
      "chars": 114,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-020.txt",
      "chars": 116,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-021.txt",
      "chars": 22,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-022.txt",
      "chars": 537,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-023.txt",
      "chars": 193,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-024.txt",
      "chars": 453,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-025.txt",
      "chars": 1323,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-026.txt",
      "chars": 93,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-027.txt",
      "chars": 1369,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-028.txt",
      "chars": 387,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-029.txt",
      "chars": 1066,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-030.txt",
      "chars": 65,
      "language": "text"
    },
    {
      "file": "attachments/f61e677d50bc/code/body-031.txt",
      "chars": 200,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Q_QP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb34a8123-ef28-48bb-8ea2-28404b08d013_2461x997.png",
      "bytes": 105273,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-af3e6c67e4a55efdc1ef.png",
      "src": "https://substackcdn.com/image/fetch/$s_!Q_QP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb34a8123-ef28-48bb-8ea2-28404b08d013_2461x997.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!nwaB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc26764a9-6a26-4467-bb03-74b6cd1ed72b_1050x363.png",
      "bytes": 70073,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-3eef2f01aa55c10175cc.png",
      "src": "https://substackcdn.com/image/fetch/$s_!nwaB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc26764a9-6a26-4467-bb03-74b6cd1ed72b_1050x363.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!WmmA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8d5f7998-21be-4144-bfc2-57b2e0a4b1c4_1040x608.png",
      "bytes": 92860,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-5f07a80abdd63d8e1c3f.png",
      "src": "https://substackcdn.com/image/fetch/$s_!WmmA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8d5f7998-21be-4144-bfc2-57b2e0a4b1c4_1040x608.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!HSuI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F098958b6-4a5b-4070-a350-8f318abcede2_1815x1143.png",
      "bytes": 132845,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-736b67ccf74251c92a7b.png",
      "src": "https://substackcdn.com/image/fetch/$s_!HSuI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F098958b6-4a5b-4070-a350-8f318abcede2_1815x1143.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!MM--!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5fb712c-10f1-4240-b3e6-2cbf7ccfc356_1389x857.png",
      "bytes": 178347,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-af386ce01654f9da5f76.png",
      "src": "https://substackcdn.com/image/fetch/$s_!MM--!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5fb712c-10f1-4240-b3e6-2cbf7ccfc356_1389x857.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!RL2q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0304413b-aa78-4488-a157-841d6399d3e6_975x986.png",
      "bytes": 159122,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-675b7dcec51ee7cdb842.png",
      "src": "https://substackcdn.com/image/fetch/$s_!RL2q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0304413b-aa78-4488-a157-841d6399d3e6_975x986.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!P2NG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74860b8a-d12a-4781-8362-d5eb06feb389_1406x656.png",
      "bytes": 119271,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-e872fbda8810038f427a.png",
      "src": "https://substackcdn.com/image/fetch/$s_!P2NG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74860b8a-d12a-4781-8362-d5eb06feb389_1406x656.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!nm_T!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c6779c-5eb0-4cfd-88fd-b1b68f44b227_1114x457.png",
      "bytes": 81407,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-96296436f4caad5d760a.png",
      "src": "https://substackcdn.com/image/fetch/$s_!nm_T!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c6779c-5eb0-4cfd-88fd-b1b68f44b227_1114x457.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!m7Un!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1a1816e-790d-421c-a364-48efe66ae622_1600x1220.png",
      "bytes": 191139,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-d50fa9407a1beb2b3c8f.png",
      "src": "https://substackcdn.com/image/fetch/$s_!m7Un!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1a1816e-790d-421c-a364-48efe66ae622_1600x1220.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!KhbK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2eb7522f-34c1-4d0a-8f54-d5276a77e740_1115x818.png",
      "bytes": 70019,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-d5c9ec14c50692228329.png",
      "src": "https://substackcdn.com/image/fetch/$s_!KhbK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2eb7522f-34c1-4d0a-8f54-d5276a77e740_1115x818.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!hCOf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57300005-93c2-4984-b099-475e4c7d6aac_1301x688.png",
      "bytes": 145317,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-eed4d8a9a5062b47db7b.png",
      "src": "https://substackcdn.com/image/fetch/$s_!hCOf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57300005-93c2-4984-b099-475e4c7d6aac_1301x688.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!o8kH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b646f80-a08d-40f4-ae65-eb7b850c7ccd_1503x1337.png",
      "bytes": 304929,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-d45fcc6a03dc8f5a6b66.png",
      "src": "https://substackcdn.com/image/fetch/$s_!o8kH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b646f80-a08d-40f4-ae65-eb7b850c7ccd_1503x1337.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!kftt!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F529c4cc7-161c-4d7c-b186-06e68c771776_1564x926.png",
      "bytes": 11992,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-879f6e705e3570293b3a.png",
      "src": "https://substackcdn.com/image/fetch/$s_!kftt!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F529c4cc7-161c-4d7c-b186-06e68c771776_1564x926.png",
      "alt": "From GPT-2 to gpt-oss: Analyzing the Architectural Advances",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!NeHY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F475023aa-2070-4533-9776-fba77b711052_911x1313.png",
      "bytes": 251185,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-83591306f213153d0eaa.png",
      "src": "https://substackcdn.com/image/fetch/$s_!NeHY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F475023aa-2070-4533-9776-fba77b711052_911x1313.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!e5fC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04869674-1018-4da5-b287-100920e21b9b_1332x864.png",
      "bytes": 164616,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-e59ac6ad2e0dc6578640.png",
      "src": "https://substackcdn.com/image/fetch/$s_!e5fC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04869674-1018-4da5-b287-100920e21b9b_1332x864.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!vxlv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faacd21b7-bf77-4cdc-bdb4-efe254a7e1b1_1586x1317.png",
      "bytes": 88995,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-d0da361495d1b3870081.png",
      "src": "https://substackcdn.com/image/fetch/$s_!vxlv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faacd21b7-bf77-4cdc-bdb4-efe254a7e1b1_1586x1317.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!RCl_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27a118a0-5da6-4486-b1f0-3743754d0a77_8106x4044.webp",
      "bytes": 104982,
      "reused": false,
      "local_path": "attachments/f61e677d50bc/images/body-fddd7415077ca40fb7bf.webp",
      "src": "https://substackcdn.com/image/fetch/$s_!RCl_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27a118a0-5da6-4486-b1f0-3743754d0a77_8106x4044.webp",
      "alt": "Build a Large Language Model (From Scratch)",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://mng.bz/Nwr7",
      "label": "Build a Reasoning Model (From Scratch)",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!Q_QP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb34a8123-ef28-48bb-8ea2-28404b08d013_2461x997.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/cais/mmlu",
      "label": "https://huggingface.co/datasets/cais/mmlu",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!WmmA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8d5f7998-21be-4144-bfc2-57b2e0a4b1c4_1040x608.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/reasoning-from-scratch/tree/main/chF/02_mmlu",
      "label": "here on GitHub",
      "type": "a"
    },
    {
      "href": "https://magazine.sebastianraschka.com/p/qwen3-from-scratch",
      "label": "Understanding and Implementing Qwen3 From Scratch",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/reasoning-from-scratch/blob/main/reasoning_from_scratch/qwen3.py",
      "label": "here on GitHub",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/reasoning-from-scratch/blob/main/reasoning_from_scratch",
      "label": "reasoning_from_scratch",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/reasoning-from-scratch/blob/main/chF/02_mmlu/1_letter_matching.py",
      "label": "here on GitHub",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!HSuI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F098958b6-4a5b-4070-a350-8f318abcede2_1815x1143.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!MM--!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe5fb712c-10f1-4240-b3e6-2cbf7ccfc356_1389x857.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!RL2q!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0304413b-aa78-4488-a157-841d6399d3e6_975x986.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/reasoning-from-scratch/blob/main/ch03/01_main-chapter-code/ch03_main.ipynb",
      "label": "here on GitHub",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!P2NG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74860b8a-d12a-4781-8362-d5eb06feb389_1406x656.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!nm_T!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36c6779c-5eb0-4cfd-88fd-b1b68f44b227_1114x457.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://lmarena.ai/",
      "label": "LM Arena",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!m7Un!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1a1816e-790d-421c-a364-48efe66ae622_1600x1220.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!KhbK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2eb7522f-34c1-4d0a-8f54-d5276a77e740_1115x818.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Elo_rating_system",
      "label": "Elo rating system",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/reasoning-from-scratch/tree/main/chF/03_leaderboards",
      "label": "here on GitHub",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!hCOf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F57300005-93c2-4984-b099-475e4c7d6aac_1301x688.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!o8kH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b646f80-a08d-40f4-ae65-eb7b850c7ccd_1503x1337.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2405.08029",
      "label": "Phudge",
      "type": "a"
    },
    {
      "href": "https://magazine.sebastianraschka.com/p/from-gpt-2-to-gpt-oss-analyzing-the",
      "label": "From GPT-2 to gpt-oss: Analyzing the Architectural Advances Sebastian Raschka, PhD · August 9, 2025 Read full story",
      "type": "a"
    },
    {
      "href": "https://substack.com/profile/27393275-sebastian-raschka-phd",
      "label": "Sebastian Raschka, PhD",
      "type": "a"
    },
    {
      "href": "https://ollama.com/",
      "label": "Ollama",
      "type": "a"
    },
    {
      "href": "https://github.com/ggerganov/llama.cpp",
      "label": "llama.cpp",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!NeHY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F475023aa-2070-4533-9776-fba77b711052_911x1313.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/reasoning-from-scratch/tree/main/chF/04_llm-judge",
      "label": "here on GitHub",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!e5fC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04869674-1018-4da5-b287-100920e21b9b_1332x864.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!vxlv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faacd21b7-bf77-4cdc-bdb4-efe254a7e1b1_1586x1317.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://amzn.to/4fqvn0D",
      "label": "Build a Large Language Model (From Scratch)",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!RCl_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27a118a0-5da6-4486-b1f0-3743754d0a77_8106x4044.webp",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Build-Large-Language-Model-Scratch/dp/1633437167",
      "label": "brief review",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:25Z",
  "source_url": "https://magazine.sebastianraschka.com/p/llm-evaluation-4-approaches",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2405.08029",
      "bytes": 1024763,
      "reused": false,
      "local_path": "papers/repair/90fbcef60b9d6e082a9f.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 43659,
  "expected_body_images": 17,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/f61e677d50bc/pages/source.html",
  "local_html": "attachments/f61e677d50bc/pages/article.html",
  "body_base_path": "attachments/f61e677d50bc/pages/",
  "body_markdown_path": "attachments/f61e677d50bc/pages/article.md"
}