{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Tae Kim",
  "title_listed": "Blocking Key Design for Entity Resolution: Why Name Normalization Fails East Asian Corporate Data",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/hannune/blocking-key-design-for-entity-resolution-why-name-normalization-fails-east-asian-corporate-data-914",
  "publish_date": "2026-08-12",
  "corpus_stage": "",
  "notes": "",
  "id": "38234ed04661",
  "collected_at": "2026-10-02T14:03:55",
  "title": "Blocking Key Design for Entity Resolution: Why Name Normalization Fails East Asian Corporate Data - DEV Community",
  "body_markdown": "Entity resolution at scale starts with a problem most introductions skip: you can't compare every pair of records. A dataset with 500,000 entities has 125 billion possible pairs. At one millisecond per comparison, exhaustive comparison takes 1,400 days.\nBlocking reduces this to a tractable set. You partition records into blocks using one or more keys, and only compare records that share a key. The constraint: your blocking decision determines which true matches you'll never see. A match that falls into different blocks is a miss by design.\nBuilding the entity resolution layer for\n2asy.ai\n— a corporate intelligence knowledge graph for East Asian markets — blocking was where the approach broke down first, in a way that was hard to diagnose.\nThe Initial Approach: Name Normalization\nThe first blocking key was a normalized company name: strip punctuation, lowercase, remove common suffixes.\nimport\nre\nCOMMON_SUFFIXES\n=\n{\n\"\nco., ltd.\n\"\n,\n\"\nco.,ltd.\n\"\n,\n\"\nco. ltd.\n\"\n,\n\"\nltd.\n\"\n,\n\"\ninc.\n\"\n,\n\"\ncorp.\n\"\n,\n\"\ncorporation\n\"\n,\n\"\nco.\n\"\n,\n\"\nllc\n\"\n,\n\"\nplc\n\"\n,\n\"\ngmbh\n\"\n,\n\"\n주식회사\n\"\n,\n\"\n(주)\n\"\n,\n\"\n(사)\n\"\n,\n\"\n유한회사\n\"\n,\n\"\n株式会社\n\"\n,\n\"\n有限会社\n\"\n,\n\"\n合同会社\n\"\n,\n}\ndef\nnormalize_name\n(\nname\n:\nstr\n)\n->\nstr\n:\nname\n=\nname\n.\nlower\n().\nstrip\n()\nname\n=\nre\n.\nsub\n(\nr\n'\n[^ws]\n'\n,\n''\n,\nname\n)\nname\n=\nre\n.\nsub\n(\nr\n'\ns+\n'\n,\n'\n'\n,\nname\n)\nfor\nsuffix\nin\nsorted\n(\nCOMMON_SUFFIXES\n,\nkey\n=\nlen\n,\nreverse\n=\nTrue\n):\nif\nname\n.\nendswith\n(\nsuffix\n):\nname\n=\nname\n[:\n-\nlen\n(\nsuffix\n)].\nstrip\n()\nreturn\nname\nEnter fullscreen mode\nExit fullscreen mode\nTwo records sharing a normalized name go into the same block. Simple, fast.\nWhere It Failed\nThe failure mode was Korean and Japanese company names. A company registered as \"Samsung Electronics Co., Ltd.\" in a Korean regulatory filing might appear as \"Samsung Electronics\" in a Japanese document and \"삼성전자\" in a Korean tax filing.\nAfter normalization, those three forms produce different blocking keys:\nnormalize_name\n(\n\"\nSamsung Electronics Co., Ltd.\n\"\n)\n# → \"samsung electronics\"\nnormalize_name\n(\n\"\nSamsung Electronics\n\"\n)\n# → \"samsung electronics\"  ← same block\nnormalize_name\n(\n\"\n삼성전자\n\"\n)\n# → \"삼성전자\"  ← different block!\nEnter fullscreen mode\nExit fullscreen mode\nThe Korean script name falls into a separate block. It's never compared. The correct merge never happens.\nThe underlying issue: a blocking key derived from a single attribute assumes that field is stable across sources. It isn't. In East Asian corporate data, the company name field is the least stable field you have. It varies by language, script, legal suffix convention, transliteration standard, and filing jurisdiction.\nComposite Blocking with Multiple Key Strategies\nThe approach that worked was generating multiple candidate keys per record and forming a candidate pair if they share\nany\nkey.\nfrom\ntyping\nimport\nIterator\ndef\ngenerate_blocking_keys\n(\nrecord\n:\ndict\n)\n->\nIterator\n[\ntuple\n[\nstr\n,\nstr\n]]:\n\"\"\"\nYields (key_type, key_value) pairs for a company record.\n\"\"\"\nname\n=\nrecord\n.\nget\n(\n\"\nname\n\"\n,\n\"\"\n)\nnormalized\n=\nnormalize_name\n(\nname\n)\nif\nnormalized\n:\nyield\n(\n\"\nname_norm\n\"\n,\nnormalized\n)\nentity_type\n=\nrecord\n.\nget\n(\n\"\nentity_type\n\"\n,\n\"\"\n)\njurisdiction\n=\nrecord\n.\nget\n(\n\"\njurisdiction_code\n\"\n,\n\"\"\n)\nif\nentity_type\nand\njurisdiction\n:\nyield\n(\n\"\ntype_jurisdiction\n\"\n,\nf\n\"\n{\nentity_type\n}\n_\n{\njurisdiction\n}\n\"\n)\nromanized\n=\nromanize\n(\nname\n)\nif\nromanized\n:\nphonetic\n=\nsoundex\n(\nromanized\n)\nyield\n(\n\"\nphonetic\n\"\n,\nphonetic\n)\nreg_num\n=\nrecord\n.\nget\n(\n\"\nregistration_number\n\"\n,\n\"\"\n)\nif\nreg_num\n:\nyield\n(\n\"\nreg_prefix\n\"\n,\nreg_num\n[:\n6\n])\ndef\nbuild_candidate_pairs\n(\nrecords\n:\nlist\n[\ndict\n])\n->\nset\n[\ntuple\n[\nint\n,\nint\n]]:\n\"\"\"\nReturn all candidate pairs sharing at least one blocking key.\n\"\"\"\nkey_to_records\n:\ndict\n[\ntuple\n,\nlist\n[\nint\n]]\n=\n{}\nfor\nidx\n,\nrecord\nin\nenumerate\n(\nrecords\n):\nfor\nkey_type\n,\nkey_value\nin\ngenerate_blocking_keys\n(\nrecord\n):\nbucket\n=\n(\nkey_type\n,\nkey_value\n)\nkey_to_records\n.\nsetdefault\n(\nbucket\n,\n[]).\nappend\n(\nidx\n)\ncandidates\n:\nset\n[\ntuple\n[\nint\n,\nint\n]]\n=\nset\n()\nfor\nindices\nin\nkey_to_records\n.\nvalues\n():\nfor\ni\nin\nrange\n(\nlen\n(\nindices\n)):\nfor\nj\nin\nrange\n(\ni\n+\n1\n,\nlen\n(\nindices\n)):\na\n,\nb\n=\nsorted\n([\nindices\n[\ni\n],\nindices\n[\nj\n]])\ncandidates\n.\nadd\n((\na\n,\nb\n))\nreturn\ncandidates\nEnter fullscreen mode\nExit fullscreen mode\nA pair of records is a candidate if they share any key. This creates overlap — some pairs get compared more than once — but it eliminates the category of missed merges caused by name form variation.\nMeasuring the Impact\nAdding the phonetic key increased candidate pairs by about 40% while recovering merges we'd been systematically missing.\ndef\nevaluate_blocking\n(\nrecords\n,\nground_truth_pairs\n,\nblocking_fn\n):\ncandidates\n=\nblocking_fn\n(\nrecords\n)\ntrue_matches\n=\nset\n(\nground_truth_pairs\n)\nfound\n=\ncandidates\n&\ntrue_matches\nmissed\n=\ntrue_matches\n-\ncandidates\nrecall\n=\nlen\n(\nfound\n)\n/\nlen\n(\ntrue_matches\n)\nif\ntrue_matches\nelse\n0.0\nreduction_ratio\n=\n1.0\n-\nlen\n(\ncandidates\n)\n/\n(\nlen\n(\nrecords\n)\n*\n(\nlen\n(\nrecords\n)\n-\n1\n)\n/\n2\n)\nreturn\n{\n\"\nrecall\n\"\n:\nrecall\n,\n\"\ncandidate_pairs\n\"\n:\nlen\n(\ncandidates\n),\n\"\nreduction_ratio\n\"\n:\nreduction_ratio\n,\n\"\nmissed_true_matches\n\"\n:\nlen\n(\nmissed\n),\n}\n# Name-only blocking:   recall 0.73, candidate_pairs 42_000, missed 12_500\n# Composite blocking:   recall 0.94, candidate_pairs 59_000, missed 2_800\nEnter fullscreen mode\nExit fullscreen mode\nIn a corporate ownership graph, the cost of a false negative is high. Two nodes for the same company means ownership relationships assigned to one aren't visible when querying through the other. The silent failure is worse than the computational overhead.\nBlocking Key Design Doesn't End at Ship\nBlocking key design isn't a one-time decision. As source variety increases, keys that worked for the initial source set start missing merges from new sources.\nThe monitoring approach: sample records that weren't merged, check manually whether they should have been, and track false negative rate over time. When it spikes, it usually means a new source has naming conventions that none of the current keys handle.\ndef\nsample_unmerged_for_review\n(\nrecords\n:\nlist\n[\ndict\n],\nmerged_ids\n:\nset\n[\nint\n],\nsample_size\n:\nint\n=\n50\n,\n)\n->\nlist\n[\ndict\n]:\nunmerged\n=\n[\nr\nfor\ni\n,\nr\nin\nenumerate\n(\nrecords\n)\nif\ni\nnot\nin\nmerged_ids\n]\nby_jurisdiction\n:\ndict\n[\nstr\n,\nlist\n[\ndict\n]]\n=\n{}\nfor\nr\nin\nunmerged\n:\njur\n=\nr\n.\nget\n(\n\"\njurisdiction_code\n\"\n,\n\"\nUNK\n\"\n)\nby_jurisdiction\n.\nsetdefault\n(\njur\n,\n[]).\nappend\n(\nr\n)\nflagged\n=\n[]\nfor\njur\n,\nrecs\nin\nby_jurisdiction\n.\nitems\n():\ntotal_in_jur\n=\nsum\n(\n1\nfor\nr\nin\nrecords\nif\nr\n.\nget\n(\n\"\njurisdiction_code\n\"\n)\n==\njur\n)\nmerge_rate\n=\n1.0\n-\nlen\n(\nrecs\n)\n/\ntotal_in_jur\nif\nmerge_rate\n<\n0.3\nand\ntotal_in_jur\n>\n20\n:\nflagged\n.\nextend\n(\nrecs\n[:\n10\n])\nreturn\nflagged\n[:\nsample_size\n]\nEnter fullscreen mode\nExit fullscreen mode\nThe goal isn't zero false negatives — it's knowing which false negative rate your blocking tradeoffs are producing, and catching when that rate degrades without explanation.\nI work on entity resolution and corporate knowledge graphs at\ner-api.hannune.ai\nand\n2asy.ai\n. If you're designing blocking keys for multilingual or multi-script data, curious what strategies you've tried.",
  "metadata": {
    "title": "Blocking Key Design for Entity Resolution: Why Name Normalization Fails East Asian Corporate Data - DEV Community",
    "description": "Entity resolution at scale starts with a problem most introductions skip: you can't compare every... Tagged with python, dataengineering, graphs, machinelearning.",
    "keywords": "python, dataengineering, graphs, machinelearning, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/hannune/blocking-key-design-for-entity-resolution-why-name-normalization-fails-east-asian-corporate-data-914",
    "og:title": "Blocking Key Design for Entity Resolution: Why Name Normalization Fails East Asian Corporate Data",
    "og:description": "Entity resolution at scale starts with a problem most introductions skip: you can't compare every...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fimages.unsplash.com%2Fphoto-1659353684737-fde5734a2348%3Fw%3D1200%26h%3D630%26fit%3Dcrop",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@HannuneTech",
    "twitter:title": "Blocking Key Design for Entity Resolution: Why Name Normalization Fails East Asian Corporate Data",
    "twitter:description": "Entity resolution at scale starts with a problem most introductions skip: you can't compare every...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fimages.unsplash.com%2Fphoto-1659353684737-fde5734a2348%3Fw%3D1200%26h%3D630%26fit%3Dcrop"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/38234ed04661/code/body-000.txt",
      "chars": 517,
      "language": "text"
    },
    {
      "file": "attachments/38234ed04661/code/body-001.txt",
      "chars": 244,
      "language": "text"
    },
    {
      "file": "attachments/38234ed04661/code/body-002.txt",
      "chars": 1450,
      "language": "text"
    },
    {
      "file": "attachments/38234ed04661/code/body-003.txt",
      "chars": 705,
      "language": "text"
    },
    {
      "file": "attachments/38234ed04661/code/body-004.txt",
      "chars": 696,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://2asy.ai/",
      "label": "2asy.ai",
      "type": "a"
    },
    {
      "href": "https://er-api.hannune.ai/",
      "label": "er-api.hannune.ai",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:39Z",
  "source_url": "https://dev.to/hannune/blocking-key-design-for-entity-resolution-why-name-normalization-fails-east-asian-corporate-data-914",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 7116,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/38234ed04661/pages/source.html",
  "local_html": "attachments/38234ed04661/pages/article.html",
  "body_base_path": "attachments/38234ed04661/pages/",
  "body_markdown_path": "attachments/38234ed04661/pages/article.md"
}