{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Tae Kim",
  "title_listed": "Incremental Graph Updates for Corporate Knowledge Graphs: Three Problems Batch Pipelines Can't Solve",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/hannune/incremental-graph-updates-for-corporate-knowledge-graphs-three-problems-batch-pipelines-cant-solve-2phm",
  "publish_date": "2026-08-04",
  "corpus_stage": "",
  "notes": "",
  "id": "c13601d5f823",
  "collected_at": "2026-10-02T14:03:57",
  "title": "Incremental Graph Updates for Corporate Knowledge Graphs: Three Problems Batch Pipelines Can't Solve - DEV Community",
  "body_markdown": "Corporate knowledge graphs break in production the moment you treat them as a batch artifact.\nI ran into this building\n2asy.ai\n, a Graph-RAG system for East Asian corporate intelligence. The initial architecture was standard: ingest a document set, run NER and entity resolution across the batch, write the graph. Clean, deterministic, fast to prototype.\nThen the first update cycle hit.\nThe Three Problems That Show Up in Production\nProblem 1: Entity Resolution at the Boundary\nThe batch pipeline handled entity resolution using blocking keys across the entire document corpus. With incremental updates, you only have the incoming batch. The blocking keys no longer reach the existing graph nodes.\nIn practice this means: a new trade document arrives with \"Samsung Elec. Co., Ltd.\" The existing graph has \"Samsung Electronics Co., Ltd.\" as a canonical node with 847 existing edges. Without live-graph entity resolution, you insert a second node and start building a disconnected duplicate.\nHere is the resolution-at-boundary pattern that works:\nfrom\ndataclasses\nimport\ndataclass\nfrom\ntyping\nimport\nOptional\nimport\nnumpy\nas\nnp\n@dataclass\nclass\nEntityCandidate\n:\nraw_name\n:\nstr\nnormalized_name\n:\nstr\nlanguage\n:\nstr\ndoc_id\n:\nstr\nembedding\n:\nOptional\n[\nnp\n.\nndarray\n]\n=\nNone\ndef\nresolve_against_live_graph\n(\ncandidate\n:\nEntityCandidate\n,\ngraph_client\n,\n# neo4j.GraphDatabase or similar\nembed_fn\n,\nthreshold\n:\nfloat\n=\n0.85\n,\n)\n->\nOptional\n[\nstr\n]:\n\"\"\"\nRun entity resolution against existing graph nodes before inserting.\n    Returns the existing node ID if a match is found, else None.\n\"\"\"\n# Step 1: Blocking — retrieve candidate neighbors by name prefix\nblocking_query\n=\n\"\"\"\nMATCH (e:Entity)\n    WHERE e.blocking_key STARTS WITH $prefix\n    RETURN e.id AS id, e.canonical_name AS name, e.name_embedding AS emb\n    LIMIT 50\n\"\"\"\nprefix\n=\ncandidate\n.\nnormalized_name\n[:\n4\n].\nupper\n()\nneighbors\n=\ngraph_client\n.\nrun\n(\nblocking_query\n,\nprefix\n=\nprefix\n).\ndata\n()\nif\nnot\nneighbors\n:\nreturn\nNone\n# Step 2: Embedding similarity against blocked candidates\nif\ncandidate\n.\nembedding\nis\nNone\n:\ncandidate\n.\nembedding\n=\nembed_fn\n(\ncandidate\n.\nnormalized_name\n)\nbest_score\n=\n0.0\nbest_id\n=\nNone\nfor\nrow\nin\nneighbors\n:\nif\nrow\n[\n\"\nemb\n\"\n]\nis\nNone\n:\ncontinue\nexisting_emb\n=\nnp\n.\narray\n(\nrow\n[\n\"\nemb\n\"\n])\nscore\n=\nfloat\n(\nnp\n.\ndot\n(\ncandidate\n.\nembedding\n,\nexisting_emb\n)\n/\n(\nnp\n.\nlinalg\n.\nnorm\n(\ncandidate\n.\nembedding\n)\n*\nnp\n.\nlinalg\n.\nnorm\n(\nexisting_emb\n)\n+\n1e-9\n))\nif\nscore\n>\nbest_score\n:\nbest_score\n=\nscore\nbest_id\n=\nrow\n[\n\"\nid\n\"\n]\nreturn\nbest_id\nif\nbest_score\n>=\nthreshold\nelse\nNone\nEnter fullscreen mode\nExit fullscreen mode\nIf\nresolve_against_live_graph\nreturns a node ID, you merge onto that node. If it returns None, you insert as new. The full entity resolution run happens at the boundary, not after insertion.\nProblem 2: Supersession Without Time-Bounded Facts\nA company relocates its headquarters. The new filing contains the correct address. The old record is already in the graph. Naive append gives you both — which means every Graph-RAG query touching that entity gets conflicting facts. The LLM has to guess which one is current.\nThe fix is to model each property value as a time-bounded assertion rather than a mutable attribute:\nfrom\ndatetime\nimport\ndate\nfrom\nneo4j\nimport\nGraphDatabase\ndef\nupsert_fact_with_supersession\n(\ndriver\n:\nGraphDatabase\n,\nentity_id\n:\nstr\n,\nproperty_name\n:\nstr\n,\nnew_value\n:\nstr\n,\nvalid_from\n:\ndate\n,\nsource_doc_id\n:\nstr\n,\n)\n->\nNone\n:\n\"\"\"\nWrite a new fact, terminating any currently-open assertion for the same property.\n    Old assertions are retained for audit; only valid_to is set.\n\"\"\"\nwith\ndriver\n.\nsession\n()\nas\nsession\n:\nsession\n.\nrun\n(\n\"\"\"\n// Terminate the previous open assertion for this property\n            MATCH (e:Entity {id: $entity_id})-[:HAS_FACT]->(f:Fact)\n            WHERE f.property_name = $property_name\n              AND f.valid_to IS NULL\n            SET f.valid_to = date($valid_from) - duration(\n'\nP1D\n'\n)\n\"\"\"\n,\nentity_id\n=\nentity_id\n,\nproperty_name\n=\nproperty_name\n,\nvalid_from\n=\nstr\n(\nvalid_from\n))\nsession\n.\nrun\n(\n\"\"\"\n// Insert the new assertion with open valid_to\n            MATCH (e:Entity {id: $entity_id})\n            CREATE (e)-[:HAS_FACT]->(f:Fact {\n                property_name: $property_name,\n                value:          $new_value,\n                valid_from:     date($valid_from),\n                valid_to:       null,\n                source_doc_id:  $source_doc_id\n            })\n\"\"\"\n,\nentity_id\n=\nentity_id\n,\nproperty_name\n=\nproperty_name\n,\nnew_value\n=\nnew_value\n,\nvalid_from\n=\nstr\n(\nvalid_from\n),\nsource_doc_id\n=\nsource_doc_id\n)\nEnter fullscreen mode\nExit fullscreen mode\nQueries then specify a point-in-time context:\n// Get headquarters address as of 2024-03-01\nMATCH\n(\ne:\nEntity\n{\nid:\n$entity_id\n})\n-\n[\n:HAS_FACT\n]\n->\n(\nf:\nFact\n)\nWHERE\nf.property_name\n=\n'headquarters_address'\nAND\nf.valid_from\n<=\ndate\n(\n'2024-03-01'\n)\nAND\n(\nf.valid_to\nIS\nNULL\nOR\nf.valid_to\n>=\ndate\n(\n'2024-03-01'\n))\nRETURN\nf.value\nAS\naddress\nEnter fullscreen mode\nExit fullscreen mode\nThis is the pattern that makes Graph-RAG\nas-of\nqueries reliable. Without it, you are relying on insertion order, which is neither guaranteed nor queryable.\nProblem 3: Cascade Propagation After Node Merge\nWhen two entity nodes merge (because\nresolve_against_live_graph\nfound a match), the incoming node's edges need to be transferred to the surviving node. If your update pipeline does not model this explicitly, you end up with a partially disconnected graph where relationships point to deprecated node IDs.\ndef\nmerge_nodes_and_transfer_edges\n(\ndriver\n:\nGraphDatabase\n,\nsurviving_id\n:\nstr\n,\ndeprecated_id\n:\nstr\n,\n)\n->\nint\n:\n\"\"\"\nTransfer all edges from deprecated node to surviving node,\n    then mark deprecated node as merged.\n    Returns the number of edges transferred.\n\"\"\"\nwith\ndriver\n.\nsession\n()\nas\nsession\n:\n# Transfer outgoing edges\nresult\n=\nsession\n.\nrun\n(\n\"\"\"\nMATCH (deprecated:Entity {id: $deprecated_id})-[r]->(target)\n            WHERE NOT target.id = $surviving_id\n            MATCH (surviving:Entity {id: $surviving_id})\n            CALL apoc.refactor.from(r, surviving)\n            YIELD input, output\n            RETURN count(*) AS transferred\n\"\"\"\n,\ndeprecated_id\n=\ndeprecated_id\n,\nsurviving_id\n=\nsurviving_id\n)\noutgoing\n=\nresult\n.\nsingle\n()[\n\"\ntransferred\n\"\n]\n# Transfer incoming edges\nresult\n=\nsession\n.\nrun\n(\n\"\"\"\nMATCH (source)-[r]->(deprecated:Entity {id: $deprecated_id})\n            WHERE NOT source.id = $surviving_id\n            MATCH (surviving:Entity {id: $surviving_id})\n            CALL apoc.refactor.to(r, surviving)\n            YIELD input, output\n            RETURN count(*) AS transferred\n\"\"\"\n,\ndeprecated_id\n=\ndeprecated_id\n,\nsurviving_id\n=\nsurviving_id\n)\nincoming\n=\nresult\n.\nsingle\n()[\n\"\ntransferred\n\"\n]\n# Mark deprecated node\nsession\n.\nrun\n(\n\"\"\"\nMATCH (e:Entity {id: $deprecated_id})\n            SET e.merged_into = $surviving_id,\n                e.merged_at   = datetime()\n\"\"\"\n,\ndeprecated_id\n=\ndeprecated_id\n,\nsurviving_id\n=\nsurviving_id\n)\nreturn\noutgoing\n+\nincoming\nEnter fullscreen mode\nExit fullscreen mode\nThe\nmerged_into\nproperty lets you build a forwarding index. Any query that arrives with a deprecated node ID can be redirected to the surviving node without a full graph scan.\nTracking Dirty Neighbors\nAfter a merge, the direct neighbors of the deprecated node may have consistency issues — edges that now point through the merged node incorrectly, or facts that reference the deprecated entity in their source context.\nThe lightweight fix is a dirty-neighbor queue:\ndef\nflag_dirty_neighbors\n(\ndriver\n:\nGraphDatabase\n,\nnode_id\n:\nstr\n,\nreason\n:\nstr\n,\n)\n->\nlist\n[\nstr\n]:\n\"\"\"\nFlag the direct neighbors of a node for consistency recheck\n    in the next ingestion cycle.\n\"\"\"\nwith\ndriver\n.\nsession\n()\nas\nsession\n:\nresult\n=\nsession\n.\nrun\n(\n\"\"\"\nMATCH (n:Entity {id: $node_id})-[*1]-(neighbor:Entity)\n            WHERE NOT neighbor.id = $node_id\n            SET neighbor.dirty = true,\n                neighbor.dirty_reason = $reason,\n                neighbor.dirty_at = datetime()\n            RETURN collect(DISTINCT neighbor.id) AS neighbor_ids\n\"\"\"\n,\nnode_id\n=\nnode_id\n,\nreason\n=\nreason\n)\nreturn\nresult\n.\nsingle\n()[\n\"\nneighbor_ids\n\"\n]\nEnter fullscreen mode\nExit fullscreen mode\nAt the start of each ingestion cycle, you pull all dirty neighbors and run a consistency check before processing new documents. This handles cascade propagation without a full graph traversal on every update.\nThe Pattern Under the Pattern\nBatch rebuild works in a prototype. Production knowledge graphs need an explicit update model from the start:\nEntity resolution at the boundary\n: run resolution against the live graph before insertion, not just within the incoming batch\nTime-bounded facts\n: model property values as assertions with validity windows, not mutable attributes\nTracked propagation\n: flag dirty neighbors on merge so cascades are handled in the next cycle, not discovered by users\nAdding these after the fact means rebuilding the schema and reprocessing historical documents. The three problems above are not edge cases — they are the production steady state for any graph that receives continuous updates.\nI work on entity resolution and agentic infrastructure at\ner-api.hannune.ai\n. The Graph-RAG system this is based on runs against East Asian corporate registries where transliteration variance and entity deduplication are the hard problems.",
  "metadata": {
    "title": "Incremental Graph Updates for Corporate Knowledge Graphs: Three Problems Batch Pipelines Can't Solve - DEV Community",
    "description": "Corporate knowledge graphs break in production the moment you treat them as a batch artifact.  I ran... Tagged with graphdatabases, python, llm, dataengineering.",
    "keywords": "graphdatabases, python, llm, dataengineering, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/hannune/incremental-graph-updates-for-corporate-knowledge-graphs-three-problems-batch-pipelines-cant-solve-2phm",
    "og:title": "Incremental Graph Updates for Corporate Knowledge Graphs: Three Problems Batch Pipelines Can't Solve",
    "og:description": "Corporate knowledge graphs break in production the moment you treat them as a batch artifact.  I ran...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fimages.unsplash.com%2Fphoto-1644088379091-d574269d422f%3Fw%3D1200%26h%3D630%26fit%3Dcrop",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@HannuneTech",
    "twitter:title": "Incremental Graph Updates for Corporate Knowledge Graphs: Three Problems Batch Pipelines Can't Solve",
    "twitter:description": "Corporate knowledge graphs break in production the moment you treat them as a batch artifact.  I ran...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fimages.unsplash.com%2Fphoto-1644088379091-d574269d422f%3Fw%3D1200%26h%3D630%26fit%3Dcrop"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/c13601d5f823/code/body-000.txt",
      "chars": 1633,
      "language": "text"
    },
    {
      "file": "attachments/c13601d5f823/code/body-001.txt",
      "chars": 1452,
      "language": "text"
    },
    {
      "file": "attachments/c13601d5f823/code/body-002.txt",
      "chars": 278,
      "language": "text"
    },
    {
      "file": "attachments/c13601d5f823/code/body-003.txt",
      "chars": 1597,
      "language": "text"
    },
    {
      "file": "attachments/c13601d5f823/code/body-004.txt",
      "chars": 692,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://2asy.ai/",
      "label": "2asy.ai",
      "type": "a"
    },
    {
      "href": "https://er-api.hannune.ai/",
      "label": "er-api.hannune.ai",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:40Z",
  "source_url": "https://dev.to/hannune/incremental-graph-updates-for-corporate-knowledge-graphs-three-problems-batch-pipelines-cant-solve-2phm",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 9408,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/c13601d5f823/pages/source.html",
  "local_html": "attachments/c13601d5f823/pages/article.html",
  "body_base_path": "attachments/c13601d5f823/pages/",
  "body_markdown_path": "attachments/c13601d5f823/pages/article.md"
}