{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Tae Kim",
  "title_listed": "Cross-Lingual Entity Resolution: Why Your Knowledge Graph Has Four Samsungs",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/hannune/cross-lingual-entity-resolution-why-your-knowledge-graph-has-four-samsungs-2h0b",
  "publish_date": "2026-07-19",
  "corpus_stage": "",
  "notes": "",
  "id": "e56034a24075",
  "collected_at": "2026-10-02T14:04:05",
  "title": "Cross-Lingual Entity Resolution: Why Your Knowledge Graph Has Four Samsungs - DEV Community",
  "body_markdown": "Cross-lingual entity resolution is where knowledge graphs quietly break down.\nI ran into this building\ner-api\n, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data.\nThe problem is deceptively simple: Samsung Electronics appears as four different strings across source documents.\nKorean: 삼성전자\nEnglish: Samsung Electronics\nJapanese: サムスン電子\nChinese: 三星电子\nEach string hashes to a different node in the graph. A query for Samsung Electronics misses three-quarters of the data unless you happen to use the exact string form the graph was built with. One real company becomes four phantom entities.\nWhy String Matching Fails\nThe instinct is to use fuzzy string matching — Levenshtein distance, cosine similarity on character n-grams. This works for typos and abbreviations within the same script. It fails completely across scripts.\n\"Samsung Electronics\" and \"サムスン電子\" share no character sequences. No string distance metric will recognize them as the same entity without preprocessing.\nStep 1: Transliterate to a Canonical Form\nThe first step is converting all strings to a common script before comparison.\nimport\njaconv\n# Japanese: katakana/hiragana to romaji\nfrom\nhanja\nimport\ntranslate\nas\nhanja_to_hangul\nfrom\nhangul_romanize\nimport\nTransliterate\nfrom\nhangul_romanize.rule\nimport\nRevisedRomanization\ntransliterator\n=\nTransliterate\n(\nRevisedRomanization\n())\ndef\ncanonical_form\n(\ntext\n:\nstr\n,\nsource_lang\n:\nstr\n)\n->\nstr\n:\n\"\"\"\nConvert a name to a canonical Latin-ish form for comparison.\n\"\"\"\nif\nsource_lang\n==\n\"\nja\n\"\n:\ntext\n=\njaconv\n.\nkata2hepburn\n(\njaconv\n.\nhira2kata\n(\ntext\n))\nelif\nsource_lang\n==\n\"\nzh\n\"\n:\ntext\n=\nhanja_to_hangul\n(\ntext\n)\nsource_lang\n=\n\"\nko\n\"\nif\nsource_lang\n==\n\"\nko\n\"\n:\ntext\n=\ntransliterator\n.\ntranslit\n(\ntext\n)\nreturn\ntext\n.\nlower\n().\nstrip\n()\nEnter fullscreen mode\nExit fullscreen mode\nAfter transliteration:\n삼성전자 → \"samsung jeonja\"\nSamsung Electronics → \"samsung electronics\"\nサムスン電子 → \"samsung denshi\"\n三星电子 → \"삼성전자\" → \"samsung jeonja\"\nNot identical, but close enough that a similarity metric can catch them. Transliteration alone collapses a large fraction of duplicates before any similarity computation runs.\nStep 2: Blocking Keys\nComparing all n-squared candidate pairs across four languages at scale is not viable. For a corpus of 100k entities across four languages, that is 10 billion pairs.\nBlocking reduces candidates to a manageable set by only comparing pairs that share a blocking key — a coarse-grained signature that likely-duplicate entities share.\nimport\nre\ndef\nblocking_key\n(\nname\n:\nstr\n,\nlang\n:\nstr\n)\n->\nstr\n:\n\"\"\"\nGenerate a blocking key for grouping candidate pairs.\n    Strips legal suffixes, normalizes, takes first 4 chars.\n\"\"\"\ncanonical\n=\ncanonical_form\n(\nname\n,\nlang\n)\nlegal_suffixes\n=\n[\nr\n\"\n\\b(co|corp|inc|ltd|llc|plc|gmbh|ag|sa|bv)\\b\\.?\n\"\n,\nr\n\"\n\\b(주식회사|kabushiki kaisha|kk|株式会社|有限公司)\\b\n\"\n,\n]\nfor\npattern\nin\nlegal_suffixes\n:\ncanonical\n=\nre\n.\nsub\n(\npattern\n,\n\"\"\n,\ncanonical\n,\nflags\n=\nre\n.\nIGNORECASE\n)\ncanonical\n=\nre\n.\nsub\n(\nr\n\"\n[^a-z0-9]\n\"\n,\n\"\"\n,\ncanonical\n)\nreturn\ncanonical\n[:\n4\n]\nEnter fullscreen mode\nExit fullscreen mode\nAll entities that share the same blocking key enter the same candidate pool for pairwise comparison. Entities in different blocks never get compared.\nStep 3: Per-Language-Pair Similarity Thresholds\nOnce you have candidate pairs within a block, you need a similarity score. Splink's EM estimator works well here — it learns match probabilities from the data without requiring fully labeled examples.\nBut the threshold for what counts as a match is not the same across language pairs.\nKorean-English pairs tolerate more abbreviation variance. \"Samsung C&T\" vs. \"Samsung C&T Corporation\" — same entity, different abbreviation patterns. A Korean-English threshold of 0.75 captures this without over-merging.\nJapanese-Chinese pairs are tighter. The transliteration paths for Japanese and Chinese overlap less cleanly than Korean-Latin paths. A threshold of 0.85+ is safer here.\nLANG_PAIR_THRESHOLDS\n=\n{\n(\n\"\nko\n\"\n,\n\"\nen\n\"\n):\n0.75\n,\n(\n\"\nko\n\"\n,\n\"\nja\n\"\n):\n0.80\n,\n(\n\"\nko\n\"\n,\n\"\nzh\n\"\n):\n0.82\n,\n(\n\"\nen\n\"\n,\n\"\nja\n\"\n):\n0.78\n,\n(\n\"\nen\n\"\n,\n\"\nzh\n\"\n):\n0.80\n,\n(\n\"\nja\n\"\n,\n\"\nzh\n\"\n):\n0.85\n,\n}\ndef\nthreshold_for_pair\n(\nlang_a\n:\nstr\n,\nlang_b\n:\nstr\n)\n->\nfloat\n:\nkey\n=\ntuple\n(\nsorted\n([\nlang_a\n,\nlang_b\n]))\nreturn\nLANG_PAIR_THRESHOLDS\n.\nget\n(\nkey\n,\n0.80\n)\nEnter fullscreen mode\nExit fullscreen mode\nThe Hardest Part: Labeled Training Data\nSplink's EM estimator can run unsupervised, but performance is better with labeled examples.\nFor Korean-English, there is labeled data available from Korean financial filings — companies are required to report both the Korean and English names in disclosure documents. A financial data scraper gives you positive pairs cheaply.\nFor Korean-Chinese and Japanese-Chinese, there is almost no open labeled data. What actually worked:\nGenerating negative pairs\nby sampling clearly distinct companies from different industry sectors — any two companies with no overlapping industry codes and no name similarity are almost certainly different entities.\nBootstrapping positive pairs\nfrom a subset of companies with known cross-lingual identifiers (Dun & Bradstreet DUNS numbers, Bloomberg tickers) where the same company appears in all four language corpora.\nManual annotation\nfor the ambiguous middle tier — about 2,000 pairs reviewed by a bilingual Korean-Chinese speaker.\nIt is not elegant. But the EM estimator is sensitive enough to training data quality that this effort pays back in precision.\nThe Result: The Graph Shrinks\nAfter running cross-lingual ER on the initial corpus:\nNode count decreased by approximately 18%. Four phantom nodes collapsing to one real node, repeated across the corpus.\nGraph RAG retrieval precision for cross-lingual queries improved significantly. Queries for a company in English now correctly retrieve documents from Korean, Japanese, and Chinese sources.\nFalse positive rate stayed low because blocking plus per-language-pair thresholds prevented over-merging.\nThe graph shrinks after you apply this. Retrieval quality improves because queries no longer split across phantom duplicates.\nBuilding er-api, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data. More at\nhannune.ai\n.\nCover:\nAlina Grubnyak\non\nUnsplash",
  "metadata": {
    "title": "Cross-Lingual Entity Resolution: Why Your Knowledge Graph Has Four Samsungs - DEV Community",
    "description": "Cross-lingual entity resolution is where knowledge graphs quietly break down.  I ran into this... Tagged with entityresolution, knowledgegraph, nlp, python.",
    "keywords": "entityresolution, knowledgegraph, nlp, python, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/hannune/cross-lingual-entity-resolution-why-your-knowledge-graph-has-four-samsungs-2h0b",
    "og:title": "Cross-Lingual Entity Resolution: Why Your Knowledge Graph Has Four Samsungs",
    "og:description": "Cross-lingual entity resolution is where knowledge graphs quietly break down.  I ran into this...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fimages.unsplash.com%2Fphoto-1544197150-b99a580bb7a8%3Fw%3D1200%26h%3D630%26fit%3Dcrop",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@HannuneTech",
    "twitter:title": "Cross-Lingual Entity Resolution: Why Your Knowledge Graph Has Four Samsungs",
    "twitter:description": "Cross-lingual entity resolution is where knowledge graphs quietly break down.  I ran into this...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fimages.unsplash.com%2Fphoto-1544197150-b99a580bb7a8%3Fw%3D1200%26h%3D630%26fit%3Dcrop"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/e56034a24075/code/body-000.txt",
      "chars": 668,
      "language": "text"
    },
    {
      "file": "attachments/e56034a24075/code/body-001.txt",
      "chars": 562,
      "language": "text"
    },
    {
      "file": "attachments/e56034a24075/code/body-002.txt",
      "chars": 320,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://er-api.hannune.ai/",
      "label": "er-api",
      "type": "a"
    },
    {
      "href": "https://hannune.ai/",
      "label": "hannune.ai",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/@alinnnaaaa",
      "label": "Alina Grubnyak",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/",
      "label": "Unsplash",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:42Z",
  "source_url": "https://dev.to/hannune/cross-lingual-entity-resolution-why-your-knowledge-graph-has-four-samsungs-2h0b",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 6309,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/e56034a24075/pages/source.html",
  "local_html": "attachments/e56034a24075/pages/article.html",
  "body_base_path": "attachments/e56034a24075/pages/",
  "body_markdown_path": "attachments/e56034a24075/pages/article.md"
}