{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Hamel Husain / Parlance Labs",
  "title_listed": "Tools for curating LLM Data",
  "keywords": "数据清洗,LLM,数据处理",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://hamel.dev/notes/llm/finetuning/data_cleaning.html",
  "publish_date": "2023-11",
  "corpus_stage": "数据处理",
  "notes": "LLM数据清洗工具",
  "id": "6b80a4ada92b",
  "collected_at": "2026-10-02T14:29:03",
  "title": "Curating LLM data – Hamel’s Blog",
  "body_markdown": "Curating LLM data\nA review of tools\nPublished\nJuly 20, 2026\nTLDR\nI think many people should build their own data annotation/curation tools for LLMs. The benefits far outweigh costs in many situations, especially when using general-purpose front-end frameworks. It’s too critical of a task to outsource without careful consideration. Furthermore, you don’t want to get constrained by the limitations of a vendor’s tool early on.\nI recommend using\nShiny For Python\nfor reasons\ndiscussed here\n. I wouldn’t recommend\nStreamlit\nfor reasons\ndiscussed here\n.\nBackground\nOne pattern I noticed is that great AI researchers are willing to manually inspect lots of data. And more than that,\nthey build infrastructure that allows them to manually inspect data quickly.\nThough not glamorous, manually examining data gives valuable intuitions about the problem. The canonical example here is Andrej Karpathy doing the ImageNet 2000-way classification task himself.\n–\nJason Wei, AI Researcher at OpenAI\nI couldn’t agree with Jason more. I don’t think people look at their data enough. Building your own tools so you can quickly sort through and curate your data is one of the highest-impact activities you can do when working with LLMs. Looking at and curating your own data is critical for both evaluation and fine-tuning.\nThings I tried\nAt the outset, I tried to avoid building something myself. I tried the following vendors who provide tools for data curation/review:\nVendors\nWarning\nThese tools are at varying levels of maturity. I interacted with the developers on all these products, and they were super responsive, kind and aware of these limitations. I expect that these tools will improve significantly over time.\nSpacy Prodigy\n: This was my favorite “pre-packaged” tool/framework. They have the cleanest UI. However, I found it a bit difficult to quickly hack it for my specific needs. They have excellent features for lots of different NLP tasks. In the end, I ended up drawing inspiration from their UI and building my own tool.\nArgilla\n: This platform has lots of functionality, however the LLM functionality fell short for me. I couldn’t do simple things like sorting, filtering, and labeling. Their LLM vs non-LLM functionality has very different APIs, which makes things quite fragmented at the moment. I think it could have potential once it matures.\nLilac\n: I found that this was more of a dataset viewer rather than something that allowed me to label data and curate it. So it didn’t really fit my needs. The user interface did not seem that hackable/extendable.\nOne thing that became clear to me while trying these vendors is the importance of being able to hack these tools to fit your specific needs. Every company you work with will have an idiosyncratic tech stack and tools that you might want to integrate into this data annotation tool. This led me to build my own tools using general-purpose frameworks.\nGeneral Purpose Frameworks\nPython has really great front-end frameworks that are easy to use like\nGradio\nor\nPanel\nand\nStreamlit\n.\nThere is a new kid on the block,\nShiny For Python\n, was my favorite after evaluating all of them.\nReasons I liked Shiny the most:\nNative integration with\nQuarto\n.\nA powerful reactive model that is snappy.\nA small API that is easy to learn and keep in your head.\nAmazing\nWASM support\n, for example I have embedded a version of the app in this blog post!\nI found that Shiny apps always required much less code and were easier to understand than the other frameworks.",
  "metadata": {
    "title": "Curating LLM data – Hamel’s Blog",
    "description": "A review of tools",
    "keywords": null,
    "author": null,
    "og:type": "article",
    "og:title": "Curating LLM data – Hamel’s Blog",
    "og:description": "A review of tools",
    "og:image": "https://user-images.githubusercontent.com/1483922/208359430-f55d7503-3a98-4875-a35c-16314c9439d0.png",
    "og:site_name": "Hamel’s Blog",
    "og:locale": "en_US",
    "twitter:title": "Curating LLM data – Hamel’s Blog",
    "twitter:description": "A review of tools",
    "twitter:image": "https://user-images.githubusercontent.com/1483922/208359430-f55d7503-3a98-4875-a35c-16314c9439d0.png",
    "twitter:creator": "@HamelHusain",
    "twitter:site": "@HamelHusain",
    "twitter:card": "summary_large_image"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.googletagmanager.com/ns.html?id=GTM-PKGWQMKL",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://shiny.posit.co/py/",
      "label": "Shiny For Python",
      "type": "a"
    },
    {
      "href": "https://streamlit.io/",
      "label": "Streamlit",
      "type": "a"
    },
    {
      "href": "https://shiny.posit.co/py/docs/comp-streamlit.html",
      "label": "discussed here",
      "type": "a"
    },
    {
      "href": "https://x.com/_jasonwei/status/1708921475829481683?s=20",
      "label": "Jason Wei, AI Researcher at OpenAI",
      "type": "a"
    },
    {
      "href": "https://prodi.gy/",
      "label": "Spacy Prodigy",
      "type": "a"
    },
    {
      "href": "https://argilla.io/",
      "label": "Argilla",
      "type": "a"
    },
    {
      "href": "https://lilacml.com/",
      "label": "Lilac",
      "type": "a"
    },
    {
      "href": "https://www.gradio.app/",
      "label": "Gradio",
      "type": "a"
    },
    {
      "href": "https://panel.holoviz.org/",
      "label": "Panel",
      "type": "a"
    },
    {
      "href": "https://quarto.org/",
      "label": "Quarto",
      "type": "a"
    },
    {
      "href": "https://shiny.posit.co/py/get-started/shinylive.html",
      "label": "WASM support",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:44Z",
  "source_url": "https://hamel.dev/notes/llm/finetuning/data_cleaning.html",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 3512,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/6b80a4ada92b/pages/source.html",
  "local_html": "attachments/6b80a4ada92b/pages/article.html",
  "body_base_path": "attachments/6b80a4ada92b/pages/",
  "body_markdown_path": "attachments/6b80a4ada92b/pages/article.md"
}