Around pair 400 of the manual annotation batch, I got to a Korean holding company and its Japanese parent. The Japanese database had the company listed as a joint venture, the Korean filing described it as a wholly-owned subsidiary. Same entity, but the ownership description was genuinely different between sources. I marked it as a match since the names clearly referred to the same company, but I wrote a note that I still don't know what to do with.
That was about a month in. I ended up reviewing around 2,000 pairs by hand before I had enough coverage to feel reasonably confident in the KR-JP and KR-ZH models.
The reason it got to manual annotation: I couldn't find a public labeled dataset for Korean-Japanese-Chinese cross-lingual corporate name matching. For English-only ER, there are open sources. For this specific domain, I didn't find anything usable.
The Korean-English piece came together faster than I expected. Korean financial disclosure filings (DART) require companies to report both the Korean legal name and the English name. That turned out to be a clean source of verified KR-EN pairs. The company certified both names in a regulatory document, so I didn't have to guess whether they referred to the same entity.
def
fetch_dart_name_pairs
(
company_code
:
str
)
->
dict
:
url
=
"
https://opendart.fss.or.kr/api/company.json
"
params
=
{
"
crtfc_key
"
:
DART_API_KEY
,
"
corp_code
"
:
company_code
}
resp
=
requests
.
get
(
url
,
params
=
params
)
data
=
resp
.
json
()
return
{
"
ko
"
:
data
.
get
(
"
corp_name
"
),
"
en
"
:
data
.
get
(
"
corp_name_eng
"
),
}
Enter fullscreen mode
Exit fullscreen mode
Scraping recent DART filings got me a few thousand KR-EN positive pairs. Combined with negatives generated by sampling companies from different industry sectors, the KR-EN Splink model got decent coverage early.
The KR-JP and KR-ZH pairs didn't have an equivalent source. I went looking for companies with cross-listed securities that would appear in databases across multiple languages. Found maybe 800-900 verified triplets that way. Not many, but enough to get an initial model to a usable state for the core coverage.
The 2,000 manual annotations were the gap-filler. Mostly companies in the near-threshold band of early model runs, which sounds principled but in practice meant sitting with a lot of cases where I wasn't sure. Japanese holding company naming conventions have edge cases I'd never thought about before. A company could be listed under a regional holding name in Japanese filings but appear under the global brand in Korean corporate registries. Not wrong, just a different layer of the corporate structure.
The KR-EN model is the one I trust most. The KR-JP and KR-ZH models have held up for the core coverage but I've seen them struggle on cases that weren't well represented in the training set. I add pairs when I find errors in production, but it's slow.
I build
er-api
, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data. More at
hannune.ai
.