{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Eugene Yan",
  "title_listed": "How to Write Data Labeling/Annotation Guidelines",
  "keywords": "数据标注,指南,质量",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://eugeneyan.com/writing/labeling-guidelines/",
  "publish_date": "2023-02",
  "corpus_stage": "数据处理",
  "notes": "数据标注指南编写方法",
  "id": "968339fd76ca",
  "collected_at": "2026-10-02T14:16:58",
  "title": "How to Write Data Labeling/Annotation Guidelines",
  "body_markdown": "eugeneyan\nStart Here\nWriting\nSpeaking\nPrototyping\nAbout\nHow to Write Data Labeling/Annotation Guidelines\n[\nmachinelearning\nmechanism\nsurvey\n]\n· 6 min read\nWriting data labeling/annotation guidelines is hard. For the handful of times I’ve had to do it, I didn’t know where to start or how to improve. You would think that we’re\njust\nwriting instructions for a task but nooo, turns out there’s a lot more thinking involved.\nI know future me will have to write annotation guidelines again. Thus, present me would like to share a few things he’s learned. I’ll frame it around questions that I think a good guideline should answer (the principles of\nWhy, What, How\nare applicable). We’ll also see examples from guidelines by\nGoogle\nand\nBing Search\n. Addressing these questions should help us write good guidelines that ensure consistent and accurate data labels.\nWhy is the task important?\nWhat is the task?\nWhat do the terms in the task mean?\nHow should annotators decide?\nHow should the task be performed?\nWhy is the task important?\nA good guideline should explain\nwhy\nthe task is important.\nThis will motivate annotators to put in the effort it deserves. Whether the task is labeling images, annotating sentiment in text, or rating search relevance, tell them why it matters. Explain how the task’s outputs will be used to improve your product or machine learning models. Include screenshots of the UX and explain how the task helps power it.\nThe examples below are from the\nfirst pages of Google’s and Bing’s guidelines\n. They explain the task and how its output will be used to improve search results.\nGoogle's explanation on the Why\nBing's explanation on the Why\nWhat is the task?\nNext, explain the task so annotators understand exactly what they need to do.\nThis may involve helping annotators put themselves in the shoes of users, and possibly breaking the task into smaller pieces.\nGoogle’s guide has an entire section devoted to\nunderstanding the needs of a user\n. It discusses how people use search to perform simple and complex tasks, as well as the various user intents.\nThough it may be obvious, it's helpful to spell out the needs of a search user\nPeople using search have various intents such as to know, do, find a website, or visit in person\nBoth guidelines also\nexplain the labeling options\nearly in the respective sections.\nExplaining the labeling options for page quality\nExplaining the labeling options for needs met\nExplaining the labeling options for location quality\nWhat do the terms in the task mean?\nA good guideline should have clear definitions of the terms used.\nOne key challenge when writing annotation guidelines is ensuring that everyone is using the same terminology. This is especially important in technical fields with domain-specific vocabulary, or when developing new user experiences and coining new terms. This goes a long way towards improving the consistency and accuracy of labeled data.\nFor the seemingly basic task of rating search relevance, Google’s guidelines includes definitions for “query”, “locale”, and even “results”.\nDefining the various terms that show up in a search relevance task\nSimilarly, Bing defines the fields that show up in Bing search results.\nDefining the various fields in a Bing local page\nHow should annotators decide?\nHere’s the meat of the guidelines:\nExplaining how to decide on the labels\n. It may include broader considerations and context around the task, step-by-step instructions, and challenging examples and explanations.\nAs part of their guidelines on rating page quality, Google includes several aspects of a page that a reviewer should consider, such as page purpose, website type and reputation, and ads on the page.\nVarious factors to consider when defining page quality\nTo help annotators understand what they’re rating (i.e., search queries), Google includes several examples of queries and their intents. They also distinguish between “know”, “do”, and “website” queries.\nExplain the various types of queries a user could make\nBing includes a decision tree for rating match quality, and a step-by-step for rating location quality. Having annotators follow the same decision-making process can lead to more consistent and accurate results, and higher inter-rater agreement.\nA decision tree to assess match quality based on query intent and semantic scope\nA step-by-step to rate location quality given a query intent\nTo help annotators decide better and increase precision, we can provide examples of tasks with their associated output and explanations. This helps to calibrate annotators so we get higher inter-rater reliability. Google’s guidelines have a ton of these examples.\nVarious needs met ratings and their explanations\nHow should the task be performed?\nDon’t forget to\ninclude instructions on how to access and use the annotation tools and platforms, as well as the logistics of the task.\nFor example, the guide for page quality rating asks annotators\nnot to spend too much time on each rating\n, how to decide between two ratings, and how to decide when the criteria doesn’t cover it (they should use their judgment).\nHow much time to spend on each task and how to break ties\nIt’s also important to clarify what annotators should do if they can’t perform the task. Rather than forcing them to make a possibly incorrect judgment, we can provide an “Unsure” label or let them release the task.\nWhat annotators should do if they have difficulty with a task\n• • •\nImproving the quality of labels collected is not only about writing good labeling guidelines. We can also simplify the task to use binary responses instead of a numeric scale or multiple choice options. This can increase precision and throughput.\nIn addition, try to make labeling criteria as objective as possible. For example, “Is this adult content?” can be made more objective by rephrasing it as “Is this nudity?” Nonetheless, it’s okay if the task is somewhat subjective—that’s why we have human reviewers. As a comparison, Google’s guidelines ask annotators to “use your judgment” 19 times!\nTo evaluate your task and labeling guidelines, consider measuring how often reviewers agree with each other (i.e.,\ninter-rater reliability\n).\nCohen’s kappa\nis a commonly used metric. If Cohen’s kappa stays consistent or increases as you iterate on your task definition and labeling guidelines, you’re on the right track.\nIf you found this useful, please cite this write-up as:\nYan, Ziyou. (Mar 2023). How to Write Data Labeling/Annotation Guidelines. eugeneyan.com.\n        https://eugeneyan.com/writing/labeling-guidelines/.\nor\n@article{yan2023labeling,\n  title   = {How to Write Data Labeling/Annotation Guidelines},\n  author  = {Yan, Ziyou},\n  journal = {eugeneyan.com},\n  year    = {2023},\n  month   = {Mar},\n  url     = {https://eugeneyan.com/writing/labeling-guidelines/}\n}\nShare on:\nBrowse related tags:\n[\nmachinelearning\nmechanism\nsurvey\n]\nor\nSearch\n« Content Moderation & Fraud Detection - Patterns in Industry\nLLM-powered Biographies »\nJoin\n11,800+\nreaders getting updates on machine learning, RecSys, LLMs, and engineering.\nGet email updates",
  "metadata": {
    "title": "How to Write Data Labeling/Annotation Guidelines",
    "description": "Writing good instructions to achieve high precision and throughput.",
    "keywords": null,
    "author": "Eugene Yan",
    "og:site_name": "eugeneyan.com",
    "og:title": "How to Write Data Labeling/Annotation Guidelines",
    "og:type": "article",
    "og:description": "Writing good instructions to achieve high precision and throughput.",
    "og:url": "https://eugeneyan.com/writing/labeling-guidelines/",
    "og:image": "https://eugeneyan.com/assets/og_image/checkbox.jpg",
    "twitter:card": "summary_large_image",
    "twitter:domain": "eugeneyan.com",
    "twitter:title": "How to Write Data Labeling/Annotation Guidelines",
    "twitter:description": "Writing good instructions to achieve high precision and throughput.",
    "twitter:image": "https://eugeneyan.com/assets/og_image/checkbox.jpg",
    "article:published_time": "2023-03-12T00:00:00+00:00",
    "article:author": "https://eugeneyan.com/about/",
    "article:section": "posts",
    "article:tag": "survey"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/968339fd76ca/code/body-000.txt",
      "chars": 250,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-search.svg",
      "bytes": 425,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-feeb4bd87b8704a903b2.svg",
      "src": "https://eugeneyan.com/assets/icon-search.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-why.webp",
      "bytes": 46148,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-116bc0ab95e90eb04551.webp",
      "src": "https://eugeneyan.com/assets/google-why.webp",
      "alt": "Google's explanation on the Why",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/bing-why.webp",
      "bytes": 12488,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-0ebd5804c1d305c9b9d4.webp",
      "src": "https://eugeneyan.com/assets/bing-why.webp",
      "alt": "Bing's explanation on the Why",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-user-needs.webp",
      "bytes": 45290,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-baee56fa4db77c040d07.webp",
      "src": "https://eugeneyan.com/assets/google-user-needs.webp",
      "alt": "Though it may be obvious, it's helpful to spell out the needs of a search user",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-user-intent.webp",
      "bytes": 25396,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-2a4983bc22bb1a382cad.webp",
      "src": "https://eugeneyan.com/assets/google-user-intent.webp",
      "alt": "People using search have various intents such as to know, do, find a website, or visit in person",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-page-quality.webp",
      "bytes": 69908,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-1ebd7bb3c95ee0a163c3.webp",
      "src": "https://eugeneyan.com/assets/google-page-quality.webp",
      "alt": "Explaining the labeling options for page quality",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-needs-met.webp",
      "bytes": 49380,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-524d243c6671f904ed39.webp",
      "src": "https://eugeneyan.com/assets/google-needs-met.webp",
      "alt": "Explaining the labeling options for needs met",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/bing-location-quality.webp",
      "bytes": 29386,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-87e36c6f019129dfc4bc.webp",
      "src": "https://eugeneyan.com/assets/bing-location-quality.webp",
      "alt": "Explaining the labeling options for location quality",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-definitions.webp",
      "bytes": 94040,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-b4d9d152b7893c61e208.webp",
      "src": "https://eugeneyan.com/assets/google-definitions.webp",
      "alt": "Defining the various terms that show up in a search relevance task",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-definitions-2.webp",
      "bytes": 19294,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-5f098fd07ed8c9b93bb0.webp",
      "src": "https://eugeneyan.com/assets/google-definitions-2.webp",
      "alt": "Defining the various terms that show up in a search relevance task",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/bing-definitions.webp",
      "bytes": 35118,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-e999a47dfc1379ec6ada.webp",
      "src": "https://eugeneyan.com/assets/bing-definitions.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-pq-considerations.webp",
      "bytes": 56388,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-598c75808699ad4d426b.webp",
      "src": "https://eugeneyan.com/assets/google-pq-considerations.webp",
      "alt": "Various factors to consider when defining page quality",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-pq-considerations2.webp",
      "bytes": 87288,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-3d0ee6d4d2a7bbca68fd.webp",
      "src": "https://eugeneyan.com/assets/google-pq-considerations2.webp",
      "alt": "Various factors to consider when defining page quality",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-know.webp",
      "bytes": 78720,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-24e73eeae654c1240168.webp",
      "src": "https://eugeneyan.com/assets/google-know.webp",
      "alt": "Explain the various types of queries a user could make",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-do.webp",
      "bytes": 77042,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-a2366dd3d83189c41032.webp",
      "src": "https://eugeneyan.com/assets/google-do.webp",
      "alt": "Explain the various types of queries a user could make",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/bing-match-quality.webp",
      "bytes": 52878,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-6eba24b491b5cac66e39.webp",
      "src": "https://eugeneyan.com/assets/bing-match-quality.webp",
      "alt": "A decision tree to assess match quality based on query intent and semantic scope",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/bing-location-quality-steps.webp",
      "bytes": 64216,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-e22a8bb657a410507a6c.webp",
      "src": "https://eugeneyan.com/assets/bing-location-quality-steps.webp",
      "alt": "A step-by-step to rate location quality given a query intent",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-fully-meets.webp",
      "bytes": 30018,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-cb776590514c778a5657.webp",
      "src": "https://eugeneyan.com/assets/google-fully-meets.webp",
      "alt": "Various needs met ratings and their explanations",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-highly-meets.webp",
      "bytes": 48998,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-8be2015a6e270c9ee6de.webp",
      "src": "https://eugeneyan.com/assets/google-highly-meets.webp",
      "alt": "Various needs met ratings and their explanations",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-moderately-meets.webp",
      "bytes": 51430,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-cc2c46729eb6b541082a.webp",
      "src": "https://eugeneyan.com/assets/google-moderately-meets.webp",
      "alt": "Various needs met ratings and their explanations",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-slightly-meets.webp",
      "bytes": 42960,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-63a7f2f03bded2331e4d.webp",
      "src": "https://eugeneyan.com/assets/google-slightly-meets.webp",
      "alt": "Various needs met ratings and their explanations",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-fails-to-meet.webp",
      "bytes": 51516,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-8ddf596226096209e26c.webp",
      "src": "https://eugeneyan.com/assets/google-fails-to-meet.webp",
      "alt": "Various needs met ratings and their explanations",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-how-to-perform-task.webp",
      "bytes": 80578,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-c3ad91dbbbf3520789fb.webp",
      "src": "https://eugeneyan.com/assets/google-how-to-perform-task.webp",
      "alt": "How much time to spend on each task and how to break ties",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/google-unsure.webp",
      "bytes": 44538,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-503d896012c80534b4fd.webp",
      "src": "https://eugeneyan.com/assets/google-unsure.webp",
      "alt": "What annotators should do if they have difficulty with a task",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-twitter.svg",
      "bytes": 892,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-0083a7581b07e5f40e7b.svg",
      "src": "https://eugeneyan.com/assets/icon-twitter.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-linkedin.svg",
      "bytes": 559,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-f7a16c3e46e5838cec4d.svg",
      "src": "https://eugeneyan.com/assets/icon-linkedin.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/bluesky.svg",
      "bytes": 745,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-be754e8ecdaf6de94f55.svg",
      "src": "https://eugeneyan.com/assets/bluesky.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-facebook.svg",
      "bytes": 1123,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-5d55b59733923cf14223.svg",
      "src": "https://eugeneyan.com/assets/icon-facebook.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-mail.svg",
      "bytes": 841,
      "reused": false,
      "local_path": "attachments/968339fd76ca/images/body-b179147617228dfafb9f.svg",
      "src": "https://eugeneyan.com/assets/icon-mail.svg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://eugeneyan.com/",
      "label": "eugeneyan",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/start-here/",
      "label": "Start Here",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/",
      "label": "Writing",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/speaking/",
      "label": "Speaking",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/prototyping/",
      "label": "Prototyping",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/about/",
      "label": "About",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/search/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/machinelearning/",
      "label": "machinelearning",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/mechanism/",
      "label": "mechanism",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/survey/",
      "label": "survey",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/writing-docs-why-what-how/",
      "label": "Why, What, How",
      "type": "a"
    },
    {
      "href": "https://guidelines.raterhub.com/searchqualityevaluatorguidelines.pdf",
      "label": "Google",
      "type": "a"
    },
    {
      "href": "https://knowledgebase.oneforma.com/wp-content/uploads/2019/09/LIMQ-Guidelines-11.2018.pdf",
      "label": "Bing Search",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Inter-rater_reliability",
      "label": "inter-rater reliability",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/content-moderation/",
      "label": "« Content Moderation & Fraud Detection - Patterns in Industry",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/llm-bio/",
      "label": "LLM-powered Biographies »",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:19Z",
  "source_url": "https://eugeneyan.com/writing/labeling-guidelines/",
  "papers": [
    {
      "status": "ok",
      "url": "https://guidelines.raterhub.com/searchqualityevaluatorguidelines.pdf",
      "bytes": 8969290,
      "reused": false,
      "local_path": "papers/repair/bfda21c9205fc7a35adb.pdf"
    },
    {
      "status": "download_error",
      "url": "https://knowledgebase.oneforma.com/wp-content/uploads/2019/09/LIMQ-Guidelines-11.2018.pdf",
      "reason": "ConnectionError: HTTPSConnectionPool(host='knowledgebase.oneforma.com', port=443): Max retries exceeded with url: /wp-content/uploads/2019/09/LIMQ-Guidelines-11.2018.pdf (Caused by NameResolutionEr",
      "retried_at": "2026-10-20T22:24:42Z"
    }
  ],
  "actual_video_files": [],
  "body_chars": 7106,
  "expected_body_images": 29,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "download_error",
  "raw_html": "attachments/968339fd76ca/pages/source.html",
  "local_html": "attachments/968339fd76ca/pages/article.html",
  "body_base_path": "attachments/968339fd76ca/pages/",
  "body_markdown_path": "attachments/968339fd76ca/pages/article.md"
}