{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Tae Kim",
  "title_listed": "Where I got labeled training data for Korean-Japanese-Chinese entity resolution",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/hannune/where-i-got-labeled-training-data-for-korean-japanese-chinese-entity-resolution-1j5d",
  "publish_date": "2026-09-23",
  "corpus_stage": "",
  "notes": "",
  "id": "22d7fe99e164",
  "collected_at": "2026-10-02T14:03:44",
  "title": "Where I got labeled training data for Korean-Japanese-Chinese entity resolution - DEV Community",
  "body_markdown": "Around pair 400 of the manual annotation batch, I got to a Korean holding company and its Japanese parent. The Japanese database had the company listed as a joint venture, the Korean filing described it as a wholly-owned subsidiary. Same entity, but the ownership description was genuinely different between sources. I marked it as a match since the names clearly referred to the same company, but I wrote a note that I still don't know what to do with.\nThat was about a month in. I ended up reviewing around 2,000 pairs by hand before I had enough coverage to feel reasonably confident in the KR-JP and KR-ZH models.\nThe reason it got to manual annotation: I couldn't find a public labeled dataset for Korean-Japanese-Chinese cross-lingual corporate name matching. For English-only ER, there are open sources. For this specific domain, I didn't find anything usable.\nThe Korean-English piece came together faster than I expected. Korean financial disclosure filings (DART) require companies to report both the Korean legal name and the English name. That turned out to be a clean source of verified KR-EN pairs. The company certified both names in a regulatory document, so I didn't have to guess whether they referred to the same entity.\ndef\nfetch_dart_name_pairs\n(\ncompany_code\n:\nstr\n)\n->\ndict\n:\nurl\n=\n\"\nhttps://opendart.fss.or.kr/api/company.json\n\"\nparams\n=\n{\n\"\ncrtfc_key\n\"\n:\nDART_API_KEY\n,\n\"\ncorp_code\n\"\n:\ncompany_code\n}\nresp\n=\nrequests\n.\nget\n(\nurl\n,\nparams\n=\nparams\n)\ndata\n=\nresp\n.\njson\n()\nreturn\n{\n\"\nko\n\"\n:\ndata\n.\nget\n(\n\"\ncorp_name\n\"\n),\n\"\nen\n\"\n:\ndata\n.\nget\n(\n\"\ncorp_name_eng\n\"\n),\n}\nEnter fullscreen mode\nExit fullscreen mode\nScraping recent DART filings got me a few thousand KR-EN positive pairs. Combined with negatives generated by sampling companies from different industry sectors, the KR-EN Splink model got decent coverage early.\nThe KR-JP and KR-ZH pairs didn't have an equivalent source. I went looking for companies with cross-listed securities that would appear in databases across multiple languages. Found maybe 800-900 verified triplets that way. Not many, but enough to get an initial model to a usable state for the core coverage.\nThe 2,000 manual annotations were the gap-filler. Mostly companies in the near-threshold band of early model runs, which sounds principled but in practice meant sitting with a lot of cases where I wasn't sure. Japanese holding company naming conventions have edge cases I'd never thought about before. A company could be listed under a regional holding name in Japanese filings but appear under the global brand in Korean corporate registries. Not wrong, just a different layer of the corporate structure.\nThe KR-EN model is the one I trust most. The KR-JP and KR-ZH models have held up for the core coverage but I've seen them struggle on cases that weren't well represented in the training set. I add pairs when I find errors in production, but it's slow.\nI build\ner-api\n, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data. More at\nhannune.ai\n.",
  "metadata": {
    "title": "Where I got labeled training data for Korean-Japanese-Chinese entity resolution - DEV Community",
    "description": "Around pair 400 of the manual annotation batch, I got to a Korean holding company and its Japanese... Tagged with entityresolution, dataengineering, python, nlp.",
    "keywords": "entityresolution, dataengineering, python, nlp, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/hannune/where-i-got-labeled-training-data-for-korean-japanese-chinese-entity-resolution-1j5d",
    "og:title": "Where I got labeled training data for Korean-Japanese-Chinese entity resolution",
    "og:description": "Around pair 400 of the manual annotation batch, I got to a Korean holding company and its Japanese...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fimages.unsplash.com%2Fphoto-1595507290691-53f7ac0179c6%3Fixid%3DM3w5NjA5Njd8MHwxfGFsbHx8fHx8fHx8fDE3OTAxMzMwNjh8%26ixlib%3Drb-4.1.0%26w%3D1200%26h%3D630%26fit%3Dcrop",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@HannuneTech",
    "twitter:title": "Where I got labeled training data for Korean-Japanese-Chinese entity resolution",
    "twitter:description": "Around pair 400 of the manual annotation batch, I got to a Korean holding company and its Japanese...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fimages.unsplash.com%2Fphoto-1595507290691-53f7ac0179c6%3Fixid%3DM3w5NjA5Njd8MHwxfGFsbHx8fHx8fHx8fDE3OTAxMzMwNjh8%26ixlib%3Drb-4.1.0%26w%3D1200%26h%3D630%26fit%3Dcrop"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/22d7fe99e164/code/body-000.txt",
      "chars": 342,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://er-api.hannune.ai/",
      "label": "er-api",
      "type": "a"
    },
    {
      "href": "https://hannune.ai/",
      "label": "hannune.ai",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:36Z",
  "source_url": "https://dev.to/hannune/where-i-got-labeled-training-data-for-korean-japanese-chinese-entity-resolution-1j5d",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 3049,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/22d7fe99e164/pages/source.html",
  "local_html": "attachments/22d7fe99e164/pages/article.html",
  "body_base_path": "attachments/22d7fe99e164/pages/",
  "body_markdown_path": "attachments/22d7fe99e164/pages/article.md"
}