{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Chip Huyen",
  "title_listed": "Predictive Human Preference: From Model Ranking to Model Routing",
  "keywords": "模型路由,偏好预测,LLM",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://huyenchip.com/2024/02/28/predictive-human-preference.html",
  "publish_date": "2024-02",
  "corpus_stage": "推理部署",
  "notes": "人类偏好预测与模型路由",
  "id": "ddf42d3cde93",
  "collected_at": "2026-10-02T14:14:18",
  "title": "Predictive Human Preference: From Model Ranking to Model Routing",
  "body_markdown": "A challenge of building AI applications is choosing which model to use. What if we don’t have to? What if we can predict the best model for any prompt? Predictive human preference aims to predict which model users might prefer for a specific query.\nHuman preference has emerged to be both the Northstar and a powerful tool for AI model development. Human preference guides post-training techniques including\nRLHF\nand\nDPO\n. Human preference is also used to rank AI models, as used by LMSYS’s\nChatbot Arena\n.\nChatbot Arena aims to determine which model is generally preferred. I wanted to see if it’s possible to predict which model is preferred\nfor each query\n.\nOne use case of predictive human preference is model routing. For example, if we know in advance that for a prompt, users will prefer Claude Instant’s response over GPT-4, and Claude Instant is cheaper/faster than GPT-4, we can route this prompt to Claude Instant. Model routing has the potential to increase response quality while reducing costs and latency.\nAnother use case of predictive human preference is interpretability. Mapping out a model’s performance on different prompts can help us understand this model’s strengths and weaknesses. See section\nExperiment results\nfor examples.\nHere’s what predictive human preference for different model pairs looks like for the prompt “\nWhat’s the best way to cluster text embeddings?\n”. The predictions were generated by my toy preference predictor. The bright yellow color for the (GPT-4, GPT-3.5-Turbo) cell means that my predictor thinks GPT-4’s response is very likely to be preferred to that of GPT-3.5-Turbo’s for this prompt.\nThis post first discusses the correctness of Chatbot Arena, which will then be used as a baseline to evaluate the correctness of preference predictions. It then discusses how to build a preference predictor and the initial results.\nRanking Models Using Human Preference\nUsing preferential signals (comparisons) to rank models has grown in popularity in the last few years. Other than powering LMSYS’s\nChatbot Arena\n, it’s also used by many model providers (\nAnthropic\n, Gemini, ChatGPT, etc.) to evaluate their models in production.\nSide note\n: Friends who have deployed this in production told me that most users don’t read both options and just randomly vote for one. This introduces a lot of noise. However, the signals from the small percentage of users who vote correctly can sometimes be sufficient to help determine which model is preferred, as long as there’s minimal bias in the random voting.\nHow Preferential Ranking Works\nPreferential ranking works in two steps:\nCollect comparison data about user preference.\nCompute a model ranking from these comparisons.\nFor each request, two or more models are selected to respond. An evaluator, which can be human or AI, picks the winner. The evaluator shouldn’t know which models are being judged. Each comparison is called a\nmatch\n. This process results in a series of comparisons.\nMatch ID\nPrompt\nModel A\nModel B\nWinner\n1\n…\nModel 1\nModel 2\nModel 1\n2\n…\nModel 3\nModel 1\nModel 1\n3\n…\nModel 1\nModel 4\nModel 4\n...\n...\n...\n...\n...\nFrom these comparisons, we need to compute the rankings of all models. The two most common ranking algorithms are\nElo\n(from chess) and\nTrueSkill\n(from video games).\nWhile Chatbot Arena refers to their model scores “Elo scores”, they actually don’t use Elo. In December 2023, they switched to\nBradley-Terry\nbut scaled the resulting scores to make them look Elo-like (see their\nnotebook\n).\nGiven a history of match outcomes, the Bradley-Terry algorithm finds the model scores that maximize the likelihood of these match outcomes, turning model scoring into a maximum likelihood estimation problem. The input, for each training example, is the models that participate in the match. The output is the outcome of the match. Assuming there’s no draw, the outcome of a match is either 0 (a wins) or 1 (b wins).\nCorrectness of Chatbot Arena Ranking\nGiven the same match outcomes, different ranking algorithms can produce different rankings. For example, the ranking computed by Elo might differ from the ranking computed by Bradley-Terry. How do we know that a ranking is correct?\nAt its core, model ranking is a predictive problem. We compute a ranking from historical match outcomes and use it to predict future match outcomes. The quality of a ranking is determined by how accurately it can predict future match outcomes.\nLet’s say we have a match between model A and model B. If model A has a higher score, meaning that the ranking algorithm predicts that A wins. If users indeed prefer the higher-ranking model, the ranking algorithm makes a correct prediction.\nEval data\nTo compute the accuracy of Chatbot Arena ranking, I used their data published in July 2023, which consists of\n33K crowd-sourced comparisons\nfor matches among 20 models. I used this smaller dataset instead of\ntheir Jan 2024 dataset\nbecause this smaller dataset contains the prompt used for each match, which I need for predictive human preference. Benchmarking on this dataset allows me to compare my model with the Bradley-Terry algorithm later on.\nHere’s an example from their July 2023 dataset.\nprompt\nmodel_a\nmodel_b\nwinner\nmodel_a's response\nmodel_b's response\nwho was the last monarch of uk\nkoala-13b\nvicuna-13b\nmodel_a\nThe last monarch of the United Kingdom was Queen Elizabeth II, who reigned from 1952 to 2020.\nThe current monarch of the United Kingdom is Queen Elizabeth II. She has been the monarch since 1952, and is the longest-reigning monarch in British history.\nFor reference, the Bradley-Terry (BT) scores of the top 7 models in this dataset are as follows.\nGPT-4: 1189\nClaude-v1: 1150\nClaude-instant-v1: 1110\nGPT-3.5-Turbo: 1104\nWizardLM-13B: 1058\nVicuna-13b: 1040\nGuanaco-33b: 1031\nTo create a test set, I randomly select 10% of the data (3300 examples). Each match has three possible outcomes: model_a wins, model_b wins, or tie. This can still be framed as a binary classification problem if we treat a tied match as two matches: one in which model_a wins and one in which model_b wins.\nResults\nI found that for all non-tie matches in my test set, the model with the higher Bradley-Terry score is preferred 74.1% of the time. This means that if we always predict the higher-ranked model as the winner for a match, we’d have an accuracy of 74.1%.\nTest data\nOutput classes\n# samples\nBT's accuracy\nAll matches\nmodel_a wins\nmodel_b wins\ntie\n3,300\n53.33%\nNon-tie matches\nmodel_a wins\nmodel_b wins\n2,367\n74.1%\nNon-tie matches involving GPT-4\nmodel_a wins\nmodel_b wins\n355\n85.1% (always pick GPT-4 as winner)\nBack in July 2023, GPT-4 was considered the strongest model by a long shot (this was before Gemini, Mistral, Claude-v2). Did users always prefer GPT-4 to all other models? They didn’t. In 355 non-tie matches involving GPT-4, GPT-4 wins 85.1%.\nThis means that even though GPT-4 is the best model overall, there are prompts for which other models can outperform GPT-4. If we can figure out which prompts these are, and which models work best for them, we can route these prompts to the best-performing models, improving the response quality.\nPredicting Human Preference For Each Prompt\nIf a ranking algorithm is about figuring out which model is better overall, predictive human preference is about figuring out which model is better for each prompt. If we know in advance that for a particular prompt, GPT-3.5 works just as well as GPT-4, and GPT-3.5 is cheaper, we can route that prompt to GPT-3.5 instead. Or if we know that Mistral-7B works just as well as GPT-4 and Mistral-7B is faster, we can route our query to Mistral-7B instead.\nModel routing can also help with budget planning. Say, you only have enough budget to serve 50% of queries on the strongest model, and the rest to a weaker model, you want to make sure that you send to the weaker model only the queries that you’re confident it can do well on.\nExperiment setup\nI treat predictive human preference as a binary classification task. Given a match between 2 models, predict which one wins. If the probability of model_a winning is around 0.5, it can be considered a tie. If a Bradley-Terry model takes only\n(model_a, model_b)\nas the input, a preference predictor takes\n(prompt, model_a, model_b)\nas the input.\nThe architecture of my preference predictor looks like this. The model encoder and preference predictor are neural networks that can be trained independently or together. I used DistilBERT as my prompt encoder.\nTo train my model, I used 90% of LMSYS’s July 2023 dataset. I found that the predictor performed better using only non-tie matches (as opposed to using both tie and non-tie matches). I randomly flipped the order of models in a match 50% of the time.\nTo evaluate my model, I used 10% of this data. This is the same test data used to evaluate the correctness of Chatbot Arena’s ranking above.\nSplit\nAll matches\nNon-tie matches\nTrain\n29,700\n20,927\nTest\n3,300\n2,367\nNote\n: I should’ve made a separate validation set for hyperparameter tuning. However, given that I didn’t have a lot of data and this is only a proof of concept, I didn’t do it. (I’m also lazy.) The matches are among 20 models, corresponding to 190 model pairs. 20,927 comparisons mean that, on average, there are only 110 comparisons per model pair.\nExperiment results\nI evaluated my preference predictor under two settings:\nUsing only\nmodel_a\nand\nmodel_b\nas the input. This is to see whether this predictor, using only model names, can make better predictions about match outcomes than Chatbot Arena scores.\nUsing\n(prompt, model_a, model_b)\nas the input. This is to see whether including prompts helps improve match outcome prediction.\nI found that for all non-tie matches, my preference predictor can predict the match outcome accurately 75% of the time if not using prompts, and 76.2% of the time if using prompts. This suggests that human preference for models does change depending on the prompt. While the improvement doesn’t seem much, a 2.1% improvement can be significant at scale.\nEval data\n# eval samples\nChatbot Arena\nPreference predictor\n(without prompts)\nPreference predictor\n(with prompts)\nNon-tie matches\n2,367\n74.1%\n75%\n76.2%\nNon-tie matches involving GPT-4\n355\n85.1%\n86.2%\n87%\nKeep in mind that this predictor was trained with a small amount of crowd-sourced (e.g. noisy) data. The prompts crowdsourced are also simple. Among\n33K prompts\n, 180 (0.55%) of them are “hello” and “hi”. These simple prompts are insufficient to distinguish strong models from weak ones. I suspect that with more/better data, the performance of this predictor can significantly improve.\nDomain-specific and query-specific leaderboards\nRecall that 20 models correspond to 190 model pairs. To visualize how the predictor captures human preference, for each evaluation prompt, I generated 190 different inputs, one for each model pair.\nI then visualized the 190 predictions for 190 model pairs in a 20 x 20 grid, as shown below for the prompt “Derive the elastic wave equation.” I only included 9 models in the plot to make it readable. The diagonal values refer to comparing a model to itself, so the predicted preference should be 0.5.\nGiven the predicted preference for all model pairs for a prompt, I used a Bradley-Terry model (the same ranking algorithm that LMSYS uses) to create a leaderboard for this prompt. I used the same scaling that LMSYS uses to make the scores look Elo-like. Here’s the ranking of the 9 models shown above for the query “Derive the elastic wave equation.”\nThis also means that with this preference predictor, we can create a leaderboard for any arbitrary subset of data. We can have a leaderboard specific to any domain.\nModel ranking for the prompt\n\"Derive the elastic wave equation.\"\ngpt-4\n1214\nclaude-v1\n1162\ngpt-3.5-turbo\n1104\nclaude-instant-v1\n1110\nguanaco-33b\n1023\nvicuna-13b\n1007\nvicuna-7b\n985\nRWKV-4-Raven-14B\n970\ngpt4all-13b-snoozy\n915\nDespite being a toy predictor, the model seems to be able to capture different models’ performance patterns. One pattern is that for simple prompts, weak models can do (nearly) as well as strong models. For more challenging prompts, however, users are much more likely to prefer stronger models. Here’s a visualization of predicted human preference for an easy prompt (“hello, how are you?”) and a challenging prompt (“Explain why Planc length …”).\nHere are the model rankings for these two prompts. The score spread for the simple prompt is much less than the score spread for the challenging prompt. The models that are ranked differently for these two prompts are highlighted in red.\nThe predictor is also the most confident that GPT-4 will be preferred for queries in Russian and queries that involve code writing. For example, the average predicted win rate for the following Russian query of GPT-4 against all other models is 91.55%. Notice that for this query, while claude-v1 is predicted to do well on this query, claude-instant-v1 is predicted to do poorly.\nConclusion\nMy primitive experiment suggests that predictive human preference is feasible using a surprisingly small amount of data. There are many potential use cases for predictive human preference – model routing and interpretability are just two of them.\nPredictive human reference is the first and the most important step in model routing (the other key step is routing strategy). With more and more models being developed, each with different capabilities and a cost structure, model routing has clear economic values.\nI’m aware of four groups (two in stealth) that are working on model routing. One startup is Martian, which announced its\n$9M seed round\n. LMSYS is also working on model routing, which I think is a natural progression from their work in comparative evaluation.\nWhile my experiment used human-annotated comparisons, LMSYS folks told me that due to the noisiness of crowd-sourced annotations and the costs of expert annotations, they’ve found that using GPT-4 to compare two responses works better. Depending on the complexity of the queries, generating 10,000 comparisons using GPT-4 would cost only $200 - 500, making this very affordable for companies that want to test it out.\nThis is the most fun side project I’ve worked on in a while, so I’d love to talk more about it. For those interested, I’ll be hosting a casual 30-minute discussion on predictive human preference on Tuesday, Mar 5, 9.30am PST. Join our\nDiscord\nor email me if you want an invite!\nAcknowledgment\nThanks\nLuke Metz\nfor helping me with the experiments and coercing me into using JAX. While JAX is super cool and makes a lot of things easy, it also caused some of the weirdest bugs I’ve ever seen. I’m glad I used it though. Thanks\nHan-chung Lee\nfor feedback on the plots.",
  "metadata": {
    "title": "Predictive Human Preference: From Model Ranking to Model Routing",
    "description": "A challenge of building AI applications is choosing which model to use. What if we don’t have to? What if we can predict the best model for any prompt? Predi...",
    "keywords": null,
    "author": "Chip Huyen",
    "og:title": "Predictive Human Preference: From Model Ranking to Model Routing",
    "og:locale": "en_US",
    "og:description": "A challenge of building AI applications is choosing which model to use. What if we don’t have to? What if we can predict the best model for any prompt? Predictive human preference aims to predict which model users might prefer for a specific query.",
    "og:url": "https://huyenchip.com/2024/02/28/predictive-human-preference.html",
    "og:site_name": "Chip Huyen",
    "og:image": "https://huyenchip.com/assets/pics/predictive-preference/8-easy-hard-prompts.png",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "article:published_time": "2024-02-28T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/predictive-preference/1-cluster.png",
      "bytes": 378934,
      "reused": false,
      "local_path": "attachments/ddf42d3cde93/images/body-0594e4d8c80cdc4a16ab.png",
      "src": "https://huyenchip.com/assets/pics/predictive-preference/1-cluster.png",
      "alt": "Predictive human preference for all LLM model pairs",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/predictive-preference/2-chatgpt.png",
      "bytes": 233013,
      "reused": false,
      "local_path": "attachments/ddf42d3cde93/images/body-3083cd621566f85735cb.png",
      "src": "https://huyenchip.com/assets/pics/predictive-preference/2-chatgpt.png",
      "alt": "Predictive human preference for all LLM model pairs",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/predictive-preference/3-bradley-terry.png",
      "bytes": 86913,
      "reused": false,
      "local_path": "attachments/ddf42d3cde93/images/body-16daca72c2f4f096b27b.png",
      "src": "https://huyenchip.com/assets/pics/predictive-preference/3-bradley-terry.png",
      "alt": "Predictive human preference for all LLM model pairs",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/predictive-preference/4-preference-predictor.png",
      "bytes": 56374,
      "reused": false,
      "local_path": "attachments/ddf42d3cde93/images/body-7a69351719a26222b361.png",
      "src": "https://huyenchip.com/assets/pics/predictive-preference/4-preference-predictor.png",
      "alt": "Predictive human preference for all LLM model pairs",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/predictive-preference/5-predictive-preference-architecture.png",
      "bytes": 111521,
      "reused": false,
      "local_path": "attachments/ddf42d3cde93/images/body-8e7c56c9075e7680c981.png",
      "src": "https://huyenchip.com/assets/pics/predictive-preference/5-predictive-preference-architecture.png",
      "alt": "Predictive human preference for all LLM model pairs",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/predictive-preference/6-model-pairs.png",
      "bytes": 76164,
      "reused": false,
      "local_path": "attachments/ddf42d3cde93/images/body-16c6ec17f7b2f78e8ddc.png",
      "src": "https://huyenchip.com/assets/pics/predictive-preference/6-model-pairs.png",
      "alt": "Predictive human preference for all LLM model pairs",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/predictive-preference/7-derive.png",
      "bytes": 370546,
      "reused": false,
      "local_path": "attachments/ddf42d3cde93/images/body-68d4a082d8a9d3cb16da.png",
      "src": "https://huyenchip.com/assets/pics/predictive-preference/7-derive.png",
      "alt": "Predictive human preference for all LLM model pairs",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/predictive-preference/8-easy-hard-prompts.png",
      "bytes": 346409,
      "reused": false,
      "local_path": "attachments/ddf42d3cde93/images/body-d0947282f5f5550a0bfd.png",
      "src": "https://huyenchip.com/assets/pics/predictive-preference/8-easy-hard-prompts.png",
      "alt": "Predictive human preference for all LLM model pairs",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/predictive-preference/10-ranking.png",
      "bytes": 162584,
      "reused": false,
      "local_path": "attachments/ddf42d3cde93/images/body-de2991d4e1dc0394c45d.png",
      "src": "https://huyenchip.com/assets/pics/predictive-preference/10-ranking.png",
      "alt": "Predictive human preference for all LLM model pairs",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/predictive-preference/9-russian.png",
      "bytes": 378116,
      "reused": false,
      "local_path": "attachments/ddf42d3cde93/images/body-0c7f3a8de596d20c5c31.png",
      "src": "https://huyenchip.com/assets/pics/predictive-preference/9-russian.png",
      "alt": "Predictive human preference for all LLM model pairs",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://huyenchip.com/2023/05/02/rlhf.html",
      "label": "RLHF",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2305.18290",
      "label": "DPO",
      "type": "a"
    },
    {
      "href": "https://arena.lmsys.org/",
      "label": "Chatbot Arena",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2112.00861",
      "label": "Anthropic",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Elo_rating_system",
      "label": "Elo",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/TrueSkill",
      "label": "TrueSkill",
      "type": "a"
    },
    {
      "href": "https://lmsys.org/blog/2023-12-07-leaderboard/",
      "label": "Bradley-Terry",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/drive/1KdwokPjirkTmpO_P1WByFNFiqxWQquwH",
      "label": "notebook",
      "type": "a"
    },
    {
      "href": "https://drive.google.com/file/d/1ZXiBRtADf9HZ8eEarIFTy-qrPNDUP_H0/view",
      "label": "33K crowd-sourced comparisons",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/lmsys/chatbot_arena_conversations",
      "label": "33K prompts",
      "type": "a"
    },
    {
      "href": "https://techcrunch.com/2023/11/15/martians-tool-automatically-switches-between-llms-to-reduce-costs/",
      "label": "$9M seed round",
      "type": "a"
    },
    {
      "href": "https://discord.gg/Bgxhua5XVR",
      "label": "Discord",
      "type": "a"
    },
    {
      "href": "https://twitter.com/luke_metz",
      "label": "Luke Metz",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/hanchunglee/",
      "label": "Han-chung Lee",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:50Z",
  "source_url": "https://huyenchip.com/2024/02/28/predictive-human-preference.html",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.18290",
      "bytes": 1299212,
      "reused": true,
      "local_path": "papers/repair/ba63f83d413059cb8722.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2112.00861",
      "bytes": 2518122,
      "reused": false,
      "local_path": "papers/repair/198d6974de0bd55c65c4.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 14767,
  "expected_body_images": 10,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/ddf42d3cde93/pages/source.html",
  "local_html": "attachments/ddf42d3cde93/pages/article.html",
  "body_base_path": "attachments/ddf42d3cde93/pages/",
  "body_markdown_path": "attachments/ddf42d3cde93/pages/article.md"
}