{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Tae Kim",
  "title_listed": "What Splink actually runs on DuckDB when it scores entity pairs",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/hannune/what-splink-actually-runs-on-duckdb-when-it-scores-entity-pairs-1e3o",
  "publish_date": "2026-09-22",
  "corpus_stage": "",
  "notes": "",
  "id": "5b422b9140a8",
  "collected_at": "2026-10-02T14:03:44",
  "title": "What Splink actually runs on DuckDB when it scores entity pairs - DEV Community",
  "body_markdown": "When I added Splink to the ER API, the registry was at 44,798 entities. Before touching any configuration knobs, I looked at the SQL it was generating. I expected opaque query strings. What came out was readable enough that I could follow the logic directly.\nThe blocking step generates something like this:\nSELECT\nl\n.\nentity_id\nAS\nentity_id_l\n,\nr\n.\nentity_id\nAS\nentity_id_r\n,\nl\n.\nname_normalized\nAS\nname_l\n,\nr\n.\nname_normalized\nAS\nname_r\nFROM\nentities\nAS\nl\nINNER\nJOIN\nentities\nAS\nr\nON\nLEFT\n(\nl\n.\nname_normalized\n,\n4\n)\n=\nLEFT\n(\nr\n.\nname_normalized\n,\n4\n)\nWHERE\nl\n.\nentity_id\n<\nr\n.\nentity_id\nEnter fullscreen mode\nExit fullscreen mode\nLEFT(name_normalized, 4)\nis the blocking key. Entities that don't share those first four characters don't get compared at all. At 44,798 entities this brought candidate pairs from roughly 2 billion down to about 400,000.\nThe cross-join on the full entity table was what was slow. I caught this by looking at the DuckDB query plan while the blocking step ran — it was sitting there doing a lot more work than I wanted. Adding an index on the blocking key column fixed it immediately. Not complicated once I knew where to look, but I wouldn't have looked there without reading what was actually running.\nWhat the EM pass is doing\nFor each candidate pair, Splink builds comparison features before the EM loop:\nSELECT\nentity_id_l\n,\nentity_id_r\n,\nCASE\nWHEN\nname_l\n=\nname_r\nTHEN\n1\nELSE\n0\nEND\nAS\nexact_match\n,\njaro_winkler_similarity\n(\nname_l\n,\nname_r\n)\nAS\njaro_winkler_sim\n,\nCASE\nWHEN\ncountry_l\n=\ncountry_r\nTHEN\n1\nELSE\n0\nEND\nAS\nsame_country\nFROM\ncandidate_pairs\nEnter fullscreen mode\nExit fullscreen mode\nThe EM loop ran fast once I had the candidate table. DuckDB handles this in-process, so there's no serialization overhead between iterations. On my machine, 400,000 pairs, the whole convergence took a few seconds. That part of the pipeline wasn't a problem.\nWhat I spent a disproportionate amount of time on was preprocessing. Name normalization for Korean, Japanese, and Chinese inputs has a lot of edge cases, and the blocking key construction amplifies whatever noise gets through. Japanese company name suffixes alone took an afternoon to handle consistently — 株式会社 shows up at the front in some sources and the back in others, abbreviated in a few more. Once I had a normalization that wasn't silently introducing variance into the blocking key, I'd spent more hours there than on everything inside Splink's own configuration combined.\nNone of that is a Splink problem. The library doesn't know what language your inputs are in. But if you're running entity resolution over multilingual corporate names, the normalization you build before Splink touches the data will probably be where you spend most of your time. The EM loop will be fine.\nI build\ner-api\n, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data. More at\nhannune.ai\n.",
  "metadata": {
    "title": "What Splink actually runs on DuckDB when it scores entity pairs - DEV Community",
    "description": "When I added Splink to the ER API, the registry was at 44,798 entities. Before touching any... Tagged with entityresolution, duckdb, dataengineering, python.",
    "keywords": "entityresolution, duckdb, dataengineering, python, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/hannune/what-splink-actually-runs-on-duckdb-when-it-scores-entity-pairs-1e3o",
    "og:title": "What Splink actually runs on DuckDB when it scores entity pairs",
    "og:description": "When I added Splink to the ER API, the registry was at 44,798 entities. Before touching any...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgm13xpdmgfh6a5zzz51f.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@HannuneTech",
    "twitter:title": "What Splink actually runs on DuckDB when it scores entity pairs",
    "twitter:description": "When I added Splink to the ER API, the registry was at 44,798 entities. Before touching any...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgm13xpdmgfh6a5zzz51f.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/5b422b9140a8/code/body-000.txt",
      "chars": 275,
      "language": "text"
    },
    {
      "file": "attachments/5b422b9140a8/code/body-001.txt",
      "chars": 257,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://er-api.hannune.ai/",
      "label": "er-api",
      "type": "a"
    },
    {
      "href": "https://hannune.ai/",
      "label": "hannune.ai",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:36Z",
  "source_url": "https://dev.to/hannune/what-splink-actually-runs-on-duckdb-when-it-scores-entity-pairs-1e3o",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 2915,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/5b422b9140a8/pages/source.html",
  "local_html": "attachments/5b422b9140a8/pages/article.html",
  "body_base_path": "attachments/5b422b9140a8/pages/",
  "body_markdown_path": "attachments/5b422b9140a8/pages/article.md"
}