{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Tae Kim",
  "title_listed": "The normalization step that breaks before your ER model does",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/hannune/the-normalization-step-that-breaks-before-your-er-model-does-38ia",
  "publish_date": "2026-09-25",
  "corpus_stage": "",
  "notes": "",
  "id": "2ef638dd7160",
  "collected_at": "2026-10-02T14:03:43",
  "title": "The normalization step that breaks before your ER model does - DEV Community",
  "body_markdown": "I spent three weeks tuning near-threshold match scores before realizing the pairs I was most worried about weren't getting bad scores. They weren't generating candidate pairs at all.\nThe thing I thought would fix it didn't. My first guess was blocking key design. I'd read enough about the invisibility of blocking errors that I figured the issue had to be there. I checked the blocking key function, checked the bucket sizes, didn't see anything obviously wrong. The pairs I was looking for were just absent.\nThen I added a simple audit: take 30 pairs that should match, print the blocking keys side by side. First run was Japanese companies from a securities exchange I'd added a few weeks before. Out of 30 pairs, 11 weren't generating matching keys.\nThe exchange uses full-width ASCII characters in company names.\nＡＢＣ株式会社\nuses full-width Latin letters (U+FF21 and so on), not regular ASCII. I hadn't known this was a thing.\nＡ\nand\nA\nare different code points, so the normalized name hashes to a different key.\nimport\nunicodedata\nname\n=\nunicodedata\n.\nnormalize\n(\n\"\nNFKC\n\"\n,\nname\n)\nEnter fullscreen mode\nExit fullscreen mode\nNFKC handles this. I ran the audit again, got 30 of 30. Then I ran the same audit on Korean data from a corporate registry and found a different problem.\nKorean company names have the entity type in multiple forms:\n주식회사 ABC\n,\nABC(주)\n, sometimes just\nABC㈜\nwhere\n㈜\nis a single precomposed Unicode character at U+3378 in the CJK Compatibility block. My stripping code handled\n주식회사\nand\n(주)\n. It didn't handle\n㈜\nbecause that's not a sequence of three characters; it's one codepoint that looks like a parenthetical when rendered.\nKOREAN_LEGAL_FORMS\n=\n[\nr\n\"\n주식회사\n\"\n,\nr\n\"\n유한회사\n\"\n,\nr\n\"\n\\(주\\)\n\"\n,\nr\n\"\n㈜\n\"\n,\n# U+3378, precomposed, won't match the spelled-out pattern\nr\n\"\n\\(유\\)\n\"\n,\nr\n\"\n㈔\n\"\n,\n# U+3354\n]\nEnter fullscreen mode\nExit fullscreen mode\nJapanese has\n㈱\n(U+3231) and\n㈲\n(U+3232) the same way. I found those in the same audit session.\nThe ordering thing tripped me up a bit. My thought was to run NFKC first and that would expand the precomposed symbols so the stripping patterns would match them. It doesn't work that way. NFKC doesn't decompose those particular codepoints; they pass through intact. So the precomposed forms have to be in the stripping list explicitly. Running NFKC before stripping is still right because it handles the full-width ASCII, but it doesn't help with the precomposed legal symbols.\nThere's still a category I don't have a clean answer for. Some Japanese company names use older traditional character forms that predate postwar simplification.\n國\nand\n国\nrefer to the same character. NFKC handles some variant pairs but not all. I've been adding them manually to a lookup table when they turn up in audits. That's obviously not comprehensive. At some point I should probably run a dedicated audit against a list of the known variant pairs, but I haven't done it, partly because I'm not sure how to get that list and partly because the other normalization issues got fixed first and this one fell off the queue.\nThe reason these are hard to catch is that they don't show up in any metric unless you build the blocking recall audit yourself.\nI build\ner-api\n, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data. More at\nhannune.ai\n.",
  "metadata": {
    "title": "The normalization step that breaks before your ER model does - DEV Community",
    "description": "Three weeks tuning ER threshold scores before realizing pairs were failing before scoring. A debugging story about Unicode normalization for East Asian corporate names. Tagged with entityresolution, dataengineering, python, nlp.",
    "keywords": "entityresolution, dataengineering, python, nlp, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/hannune/the-normalization-step-that-breaks-before-your-er-model-does-38ia",
    "og:title": "The normalization step that breaks before your ER model does",
    "og:description": "Three weeks tuning ER threshold scores before realizing pairs were failing before scoring. A debugging story about Unicode normalization for East Asian corporate names.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fimages.unsplash.com%2Fphoto-1596299786121-66852dfb4b61%3Fixid%3DM3w5NjA5Njd8MHwxfGFsbHx8fHx8fHx8fDE3OTAzMDU4MzB8%26ixlib%3Drb-4.1.0%26w%3D1200%26h%3D630%26fit%3Dcrop",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@HannuneTech",
    "twitter:title": "The normalization step that breaks before your ER model does",
    "twitter:description": "Three weeks tuning ER threshold scores before realizing pairs were failing before scoring. A debugging story about Unicode normalization for East Asian corporate names.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fimages.unsplash.com%2Fphoto-1596299786121-66852dfb4b61%3Fixid%3DM3w5NjA5Njd8MHwxfGFsbHx8fHx8fHx8fDE3OTAzMDU4MzB8%26ixlib%3Drb-4.1.0%26w%3D1200%26h%3D630%26fit%3Dcrop"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/2ef638dd7160/code/body-000.txt",
      "chars": 62,
      "language": "text"
    },
    {
      "file": "attachments/2ef638dd7160/code/body-001.txt",
      "chars": 171,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://er-api.hannune.ai/",
      "label": "er-api",
      "type": "a"
    },
    {
      "href": "https://hannune.ai/",
      "label": "hannune.ai",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:36Z",
  "source_url": "https://dev.to/hannune/the-normalization-step-that-breaks-before-your-er-model-does-38ia",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 3324,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/2ef638dd7160/pages/source.html",
  "local_html": "attachments/2ef638dd7160/pages/article.html",
  "body_base_path": "attachments/2ef638dd7160/pages/",
  "body_markdown_path": "attachments/2ef638dd7160/pages/article.md"
}