{ "source_file": "0930", "source_platform": "个人博客", "author": "Eugene Yan", "title_listed": "How to Measure and Mitigate Position Bias", "keywords": "位置偏差,推荐,偏差", "domain": "机器学习", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://eugeneyan.com/writing/position-bias/", "publish_date": "2022-01", "corpus_stage": "推理部署", "notes": "推荐系统位置偏差测量与缓解", "id": "23d2a15d6f7a", "collected_at": "2026-10-02T14:17:12", "title": "How to Measure and Mitigate Position Bias", "body_markdown": "eugeneyan\nStart Here\nWriting\nSpeaking\nPrototyping\nAbout\nHow to Measure and Mitigate Position Bias\n[\nrecsys\n]\n· 7 min read\nPosition bias happens when higher positioned items are more likely to be seen and thus clicked\nregardless of their actual relevance\n. This leads to lesser engagement on lower ranked items. On Google Search, users click on the first position\n10x more than the tenth position\n. Similarly, we scan through Netflix’s recommendation carousels from left to right.\nPosition bias happens when we process items from top-to-bottom or left-to-right.\nPosition bias can happen for several reasons. Perhaps we trust the algorithm and just blindly click on top results. Or perhaps we’re less likely to view lower-ranked items (i.e., presentation bias), especially if they’re on the second page of Google Search’s results. Maybe after we click the first relevant result or play the first relevant movie, we ignore the rest of the items.\nBecause of position bias, it’s difficult to tell if users are engaging on recommendations because they’re truly relevant, or because the recommended item happens to rank highly. Training our models on biased historical data perpetuates the bias via a self-reinforcing feedback loop. This can lead to suboptimal outcomes where items that are more relevant but are shown in lower positions continue to get lower engagement and thus don’t improve their rank.\nHere are some notes on various ways to measure position bias. Some isolate the effect of position bias better while others are more feasible on logged data.\nMeasuring position bias via randomness\nThe straightforward way is to\nshuffle (top) results uniformly at random\n(\nRandN\nor RandTopN\n). This controls for item relevance by having the same item appear in multiple positions. Thus, any difference in customer engagement can be cleanly attributed to position bias. The downside is that this adversely impacts the customer experience and is usually too costly or out of the question.\nShuffling the top 4 items uniformly at random.\nAnother approach is to\nexploit the inherent randomness\nin item positions. We see this in historical data when we have multiple rankers in production or when the production ranker is updated frequently. This lets us to\nestimate position bias without intrusive interventions\nlike RandTopN. Nonetheless, unless we’re operating on a large scale with multiple rankers updating frequently, it can be hard to get such data.\nA weaker form of inherent randomness occurs when multiple widgets recommend the same item but in different positions. In the example below, the Popular on Netflix widget recommends Battleship in position 3 and Shooter in position 4 while the Trending widget recommends Battleship in position 1 and Shooter in position 6. In this approach, the context (i.e., user, item, or query) is constant. Nonetheless, other noise may seep in such as widget position or how the widget is presented (e.g., title, explanation).\nInherent randomness in how Battleship and Shooter are ranked for two widgets.\nIf we don’t have multiple widgets in production, we can consider recommendations that are ranked in different positions across different contexts. For example, in item-to-item recommendations, the same recommended item may be shown in different positions for different context items. However, as the context isn’t constant, this approach can be noisy because the same recommended item or search result has different relevance based on the context item/query.\nDespite the limitations of exploiting inherent randomness, the big advantage is that there’s no customer impact as we are measuring position bias via logged data.\nAnother approach is to\ninfer position bias via expectation maximization\n. One way is to\nmodel clicks via a position bias model\n. In the model, an item is clicked only if it is examined and relevant; examination depends only on the position while relevance depends only on the context and item. Google demonstrated this approach’s effectiveness on search logs from email and file storage services where there was only a single click for most queries. To do this, they represented queries and items via features instead of IDs.\nA middle ground is to\nadd a bit of randomness\ninto item positions. For example,\nFair\nPairs\nswaps items at positions\nk\nand\nk+1\nwhile\nRandPair\nswaps items at positions\n1\nand\nk\n. Similar to\nRandN\n, this keeps item relevance constant so we can measure the difference in customer engagement due to position bias. One downside is that RandPair can degrade customer experience especially if\nk\nis large.\nSwapping pairs of items via FairPair and RandPair.\nWe can also draw inspiration from\nserendipity in recommendations\nand add a random value to item positions. For example, we can add values randomly sampled from a normal distribution (e.g.,\nnp.random.normal\n) to each item’s position and re-sort them on the sum. Depending on the standard deviation, each item has the chance to gain +/-\nx\npositions.\nHowever, adding randomness to an item’s position doesn’t account for the underlying scores of the item recommendations. For example, a position 1 item that has 10x the raw score of a position 2 item is treated the same as a position 1 item that has 1.1x the score of a position 2 item.\nTo address this, we can add randomness to the raw scores instead. One way is\nBoltzmann exploration\nwhere we (i) normalize/softmax the raw scores into probabilities and then (ii) sample the recommendations (based on these probabilities) as we populate our widget starting from position 1. This gives each recommended item a chance to gain a higher position weighted by recommendation probability (which we can also use in\ncounterfactual evaluation\n). Unlike\nRandTopN\n, these approaches anchor each item to its original position before adding randomness, possibly reducing the negative customer impact.\nMitigating position bias\nIf we’re in the early days of building our recommender system or prioritize exploration over exploitation, adding some randomness can be a decent way to mitigate position bias while collecting click data. Because multiple items can appear in the same position (e.g., position 1), we can log which item performed better and train our models accordingly.\nIf adding randomness is not an option, we can use the measured/learned position bias to debias logged data. For example, the previous Google paper used inferred position bias to train models optimized on\ninverse propensity weighted precision\n.\nAlternatively, we can account for position bias by including positional features in our models. These positional features help the model learn how position affects reward. Then, during serving, we can set all items to have positional feature = 1 to negate the impact of position. More in Google’s\nRules of Machine Learning\n.\nConclusion\nDepending on the user experience (UX), the impact of position bias can vary. If we’re a large platform where customers organically view items via search, recommendations, category pages, etc. and we run several algorithms in production, the position bias from an individual algorithm gets washed out by other algorithms.\nOn the other hand, if we’re launching a new platform or have a UX where the bulk of item discovery comes from a single interface (e.g., restaurant search on food delivery apps, social media feeds), IMHO it’s a good idea to measure and mitigate position bias early.\nDo you know of other ways to measure and mitigate position bias? Would love to hear them in the comments below!\nReferences\nMinimally Invasive Randomization for Collecting Unbiased Preferences from Clickthrough Logs\nAn Experimental Comparison of Click Position Bias Models\nUnbiased Learning-to-Rank with Biased Feedback\nLearning to Rank with Selection Bias in Personal Search\nEstimating Position Bias without Intrusive Interventions\nPosition Bias Estimation for Unbiased Learning to Rank in Personal Search\nBoltzmann Exploration Done Right\nA Survey of Serendipity in Recommender Systems\nThanks to Yang Xinyi for reading drafts of this.\nIf you found this useful, please cite this write-up as:\nYan, Ziyou. (Apr 2022). How to Measure and Mitigate Position Bias. eugeneyan.com.\n https://eugeneyan.com/writing/position-bias/.\nor\n@article{yan2022position,\n title = {How to Measure and Mitigate Position Bias},\n author = {Yan, Ziyou},\n journal = {eugeneyan.com},\n year = {2022},\n month = {Apr},\n url = {https://eugeneyan.com/writing/position-bias/}\n}\nShare on:\nBrowse related tags:\n[\nrecsys\n]\nor\nSearch\n« Counterfactual Evaluation for Recommendation Systems\nBandits for Recommender Systems »\nJoin\n11,800+\nreaders getting updates on machine learning, RecSys, LLMs, and engineering.\nGet email updates", "metadata": { "title": "How to Measure and Mitigate Position Bias", "description": "Introducing randomness and/or learning from inherent randomness to mitigate position bias.", "keywords": null, "author": "Eugene Yan", "og:site_name": "eugeneyan.com", "og:title": "How to Measure and Mitigate Position Bias", "og:type": "article", "og:description": "Introducing randomness and/or learning from inherent randomness to mitigate position bias.", "og:url": "https://eugeneyan.com/writing/position-bias/", "og:image": "https://eugeneyan.com/assets/og_image/position-bias.jpg", "twitter:card": "summary_large_image", "twitter:domain": "eugeneyan.com", "twitter:title": "How to Measure and Mitigate Position Bias", "twitter:description": "Introducing randomness and/or learning from inherent randomness to mitigate position bias.", "twitter:image": "https://eugeneyan.com/assets/og_image/position-bias.jpg", "article:published_time": "2022-04-17T00:00:00+00:00", "article:author": "https://eugeneyan.com/about/", "article:section": "posts", "article:tag": "recsys" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/23d2a15d6f7a/code/body-000.txt", "chars": 237, "language": "text" } ], "images": [ { "status": "ok", "url": "https://eugeneyan.com/assets/icon-search.svg", "bytes": 425, "reused": false, "local_path": "attachments/23d2a15d6f7a/images/body-feeb4bd87b8704a903b2.svg", "src": "https://eugeneyan.com/assets/icon-search.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/position-bias.webp", "bytes": 64208, "reused": false, "local_path": "attachments/23d2a15d6f7a/images/body-3d0b658e5515ddd3b05e.webp", "src": "https://eugeneyan.com/assets/position-bias.webp", "alt": "Position bias happens when we process items from top-to-bottom or left-to-right.", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/randtopn.webp", "bytes": 12450, "reused": false, "local_path": "attachments/23d2a15d6f7a/images/body-1fef8c1c827f97922ace.webp", "src": "https://eugeneyan.com/assets/randtopn.webp", "alt": "Shuffling the top 4 items uniformly at random.", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/inherent-randomness.webp", "bytes": 45082, "reused": false, "local_path": "attachments/23d2a15d6f7a/images/body-40d8782492b83627cc9f.webp", "src": "https://eugeneyan.com/assets/inherent-randomness.webp", "alt": "Inherent randomness in how Battleship and Shooter are ranked for two widgets.", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/randpair.webp", "bytes": 15502, "reused": false, "local_path": "attachments/23d2a15d6f7a/images/body-4a7348524d79815f6226.webp", "src": "https://eugeneyan.com/assets/randpair.webp", "alt": "Swapping pairs of items via FairPair and RandPair.", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/icon-twitter.svg", "bytes": 892, "reused": false, "local_path": "attachments/23d2a15d6f7a/images/body-0083a7581b07e5f40e7b.svg", "src": "https://eugeneyan.com/assets/icon-twitter.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/icon-linkedin.svg", "bytes": 559, "reused": false, "local_path": "attachments/23d2a15d6f7a/images/body-f7a16c3e46e5838cec4d.svg", "src": "https://eugeneyan.com/assets/icon-linkedin.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/bluesky.svg", "bytes": 745, "reused": false, "local_path": "attachments/23d2a15d6f7a/images/body-be754e8ecdaf6de94f55.svg", "src": "https://eugeneyan.com/assets/bluesky.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/icon-facebook.svg", "bytes": 1123, "reused": false, "local_path": "attachments/23d2a15d6f7a/images/body-5d55b59733923cf14223.svg", "src": "https://eugeneyan.com/assets/icon-facebook.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/icon-mail.svg", "bytes": 841, "reused": false, "local_path": "attachments/23d2a15d6f7a/images/body-b179147617228dfafb9f.svg", "src": "https://eugeneyan.com/assets/icon-mail.svg", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://eugeneyan.com/", "label": "eugeneyan", "type": "a" }, { "href": "https://eugeneyan.com/start-here/", "label": "Start Here", "type": "a" }, { "href": "https://eugeneyan.com/writing/", "label": "Writing", "type": "a" }, { "href": "https://eugeneyan.com/speaking/", "label": "Speaking", "type": "a" }, { "href": "https://eugeneyan.com/prototyping/", "label": "Prototyping", "type": "a" }, { "href": "https://eugeneyan.com/about/", "label": "About", "type": "a" }, { "href": "https://eugeneyan.com/search/", "label": "", "type": "a" }, { "href": "https://eugeneyan.com/tag/recsys/", "label": "recsys", "type": "a" }, { "href": "https://www.sistrix.com/blog/why-almost-everything-you-knew-about-google-ctr-is-no-longer-valid/", "label": "10x more than the tenth position", "type": "a" }, { "href": "https://dl.acm.org/doi/10.1145/2911451.2911537", "label": "RandN", "type": "a" }, { "href": "https://arxiv.org/abs/1812.05161", "label": "estimate position bias without intrusive interventions", "type": "a" }, { "href": "https://dl.acm.org/doi/10.1145/3159652.3159732", "label": "model clicks via a position bias model", "type": "a" }, { "href": "https://arxiv.org/abs/cs/0605037", "label": "Fair", "type": "a" }, { "href": "https://dl.acm.org/doi/abs/10.1145/1341531.1341545", "label": "Pairs", "type": "a" }, { "href": "https://arxiv.org/abs/1608.04468", "label": "RandPair", "type": "a" }, { "href": "https://www.sciencedirect.com/science/article/abs/pii/S0950705116302763", "label": "serendipity in recommendations", "type": "a" }, { "href": "https://arxiv.org/abs/1705.10257", "label": "Boltzmann exploration", "type": "a" }, { "href": "https://eugeneyan.com/writing/counterfactual-evaluation/", "label": "counterfactual evaluation", "type": "a" }, { "href": "https://developers.google.com/machine-learning/guides/rules-of-ml", "label": "Rules of Machine Learning", "type": "a" }, { "href": "https://eugeneyan.com/writing/bandits/", "label": "Bandits for Recommender Systems »", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:52:24Z", "source_url": "https://eugeneyan.com/writing/position-bias/", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/1812.05161", "bytes": 251564, "reused": false, "local_path": "papers/repair/177817b7f3309aacf5d0.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/cs/0605037", "bytes": 111769, "reused": false, "local_path": "papers/repair/3fd566ec6bf8ed76a921.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1608.04468", "bytes": 354834, "reused": false, "local_path": "papers/repair/e4855d81b4413d5258eb.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1705.10257", "bytes": 236359, "reused": false, "local_path": "papers/repair/db392f6f35ab97b81256.pdf" } ], "actual_video_files": [], "body_chars": 8727, "expected_body_images": 10, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/23d2a15d6f7a/pages/source.html", "local_html": "attachments/23d2a15d6f7a/pages/article.html", "body_base_path": "attachments/23d2a15d6f7a/pages/", "body_markdown_path": "attachments/23d2a15d6f7a/pages/article.md" }