{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Jaival Suthar",
  "title_listed": "Knowledge Hub: Building a Multi-Source RAG System for Real-World Knowledge",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://jaivalsuthar.hashnode.dev/knowledge-hub-multi-source-rag-system",
  "publish_date": "2026-09-21",
  "corpus_stage": "",
  "notes": "",
  "id": "75199ac0a168",
  "collected_at": "2026-10-02T14:12:15",
  "title": "Knowledge Hub: Multi-Source RAG System",
  "body_markdown": "Knowledge Hub: Building a Multi-Source RAG System for Real-World Knowledge\nUpdated\nSeptember 21, 2026\n•\n26\nmin read\n•\nView as Markdown\nJ\nJaival Suthar\nI write technical articles that analyze real world engineering problems through the lens of production systems.\nMy work focuses on architectural tradeoffs, UI layer system design, performance optimization, and reliability patterns derived from shipping and maintaining scalable applications.\nPart of series\nBuilding AI Systems from Scratch\nMost RAG systems are easy to demonstrate.\nPut a document into a vector database, retrieve a few chunks, pass them to an LLM, and generate an answer.\nThe difficult part starts when the system has to answer a more uncomfortable question:\nWhat happens when the knowledge is no longer one document?\nThat was the problem I wanted to explore in M2 of my personal AI engineering project. In M1, I built Knowledge Vault, a controlled retrieval system around a single PDF. That constraint was intentional. It gave me a clean environment to study extraction, chunking, embeddings, retrieval, reranking, evaluation, and failure analysis.\nThe experiments taught me something important:\nRetrieval quality is not just about finding semantically similar text. It is about finding evidence that is actually useful for answering the question.\nM2 took that idea into a much less controlled environment.\nInstead of one PDF, the system needed to work across PDFs, Markdown, source code, GitHub repositories, and technical articles.\nThat changed almost everything.\nThe problem was no longer simply:\n“Can I retrieve the right passage?”\nIt became:\n“Can I preserve the identity, structure, and provenance of knowledge as it moves through heterogeneous sources, and can I measure how different retrieval strategies behave against that knowledge?”\nThat became\nKnowledge Hub\n.\n🔗 Github Repository:\nhttps://github.com/Jaival-Suthar/knowledge-hub\nM1 solved a controlled problem. It created a bigger one.\nKnowledge Vault had a useful property: the corpus was predictable. There was one source. The source had a document structure. The chunks represented passages from that document. Dense retrieval was enough to establish a strong baseline. But real engineering knowledge doesn’t look like that.\nA personal knowledge base might contain:\nPDFs\nMarkdown notes\nsource code\nGit repositories\ntechnical articles\nconfiguration files\ntests\ndocumentation\nproject metadata\nA function inside a TypeScript file is not conceptually the same thing as a paragraph in a PDF. A GitHub repository is not a document. A technical article is not necessarily stored locally. A code identifier can be highly important even when its semantic embedding is not particularly close to the query.\nSo I defined the M2 problem around three requirements:\nSource diversity —\ningest heterogeneous knowledge without creating source-specific retrieval systems.\nRetrieval diversity\n— compare semantic and lexical retrieval instead of assuming one method is sufficient.\nEvidence traceability\n— preserve enough provenance to understand where every retrieved chunk came from.\nThe architecture therefore had to separate ingestion from retrieval.\nThe central design decision: one canonical representation\nThe most important architectural decision in M2 was not Qdrant, BM25, RRF, Tree-sitter, or FastAPI. It was the canonical data model.\nI wanted source-specific ingestion to disappear once data entered the retrieval system.\nThe core representation became:\nDocument\n   ↓\nChunk\n   ↓\nProvenance\nEvery ingestion adapter could do whatever was necessary to understand its source. But once the content became canonical\nDocument\nand\nChunk\nobjects, retrieval should not care whether the original source was:\nPDF\nMarkdown\nTypeScript\nGitHub\nHashnode\nThis gave the system a clean boundary:\nThat separation ended up being one of the most important parts of the project. Without it, every retrieval strategy would eventually become coupled to every source type.\nThe ingestion problem was harder than it looked\nSupporting another source is not simply adding another parser.\nEach source has a different definition of structure.\nThat became very obvious during M2.\nMarkdown\nMarkdown has semantic structure that plain text extraction would throw away. Headings, lists, tables, blockquotes, and code blocks all carry meaning. The Markdown ingestion path therefore became heading-aware and token-aware while preserving important structural boundaries.\nThe goal wasn’t to create the smallest possible chunks. It was to create chunks that remained meaningful when retrieved independently.\nCode is not prose\nThis was one of the biggest differences from M1. For a PDF, a paragraph is a reasonable unit of meaning. For source code, arbitrary token windows are often a poor abstraction. A function should ideally remain identifiable as a function. A class should retain its class context. An interface should not be separated from the information that defines it.\nThat led to AST-aware code ingestion using\nTree-sitter\n.\nThe parser extracts semantic symbols such as:\nclasses\nmethods\nfunctions\ninterfaces\ntypes\nenums\nstructs\nnamespaces\nThese symbols become the basis for semantic chunks.\nThe distinction matters because the retrieval system should be able to answer a query such as:\n“Where is authentication implemented?”\nwith a meaningful implementation unit rather than an arbitrary slice of source code.\nThe ingestion pipeline was validated against a real code corpus.\nFor the PerfEngine validation:\n1,928 ZIP members were inspected\n252 supported files were identified\n252 files were successfully parsed\n730 AST symbols were extracted\n730 semantic chunks were produced\n252 canonical documents were created\nThat validation was important because it tested the ingestion contract against a non-trivial repository rather than a toy example.\nZIP ingestion also became a security boundary\nOnce repositories and project archives entered the system, ingestion was no longer just a parsing problem.\nIt became an input validation problem.\nThe ZIP ingestion path therefore explicitly handled cases such as:\npath traversal\nabsolute paths\nsymlinks\nencrypted archives\nextraction limits\nThis is an example of something that doesn’t appear in a typical RAG architecture diagram.\nBut ingestion is an attack surface.\nIf the system accepts arbitrary archives, extraction behavior needs to be treated as part of the engineering design rather than as a convenience function.\nGitHub is not a document\nA GitHub repository introduced another layer of complexity.\nA repository has:\na remote identity\na ref\na commit\na directory structure\nmultiple files\nmultiple languages\ngenerated files\ntests\nconfiguration\ndocumentation\nThe ingestion pipeline therefore uses shallow clones, explicit refs, commit validation, bounded timeouts, isolated temporary workspaces, and cleanup.\nMore importantly, provenance records the repository context.\nA retrieved code chunk should not merely say:\n\"function authenticate()...\"\nIt should be possible to understand where that code came from.\nFor the RippleTalk repository, the validation run processed 38 files and produced 25 semantic chunks.\nThe retrieval layer could then treat those chunks exactly like any other canonical chunks.\nArticles introduced a different kind of source boundary\nTechnical articles are closer to Markdown than source code, but they introduce an acquisition problem. The source may live remotely.\nIts content may change. Its identity is not simply a local filesystem path.\nFor Hashnode articles, the ingestion path therefore separates acquisition from parsing and chunking.\nThe existing acquisition layer retrieves the article’s public Markdown representation. That representation then passes through the existing article parser and article-aware chunker before being converted into the canonical Knowledge Hub representation. The canonical layer preserves the original chunk content, deterministic identity, source URI, heading context, ordering, and provenance.\nThe important design principle remained the same:\nSource-specific complexity should stop at the ingestion boundary.\nAfter ingestion, retrieval should operate on the same canonical representation.\nM0 and M2 have deliberately different responsibilities\nOne architectural clarification became particularly important while building M2.\nThe local LLM from M0 is\nnot the retrieval engine\n.\nKnowledge Hub’s dense retrieval uses an embedding model:\nBAAI/bge-small-en-v1.5\nand stores vectors in Qdrant.\nBM25 provides the lexical retrieval path. The Qwen3 model from M0 belongs to the inference/generation side of the larger system.\nThat separation matters because it allows retrieval experiments to be measured independently of generation. If retrieval quality changes, I want to know that the retrieval system changed.\nI don’t want improvements or regressions to be hidden behind a different generated answer.\nThis was one of the principles carried forward from M1:\nMeasure retrieval separately from inference.\nKnowledge Hub therefore does not require the local LLM to perform dense retrieval.\nDense retrieval became the baseline again\nSince dense retrieval was already established in M1, I kept it as the semantic baseline. The embedding model converts chunks and queries into vector representations. Qdrant performs similarity search.\nThe retrieval contract returns ranked canonical chunks rather than source-specific objects.\nBut this time, I did not want to stop at establishing a baseline. I wanted to understand what different retrieval strategies contributed.\nThat led to a controlled experiment comparing:\nDense retrieval\nBM25 retrieval\nDense + BM25 with Reciprocal Rank Fusion\nThe initial benchmark deliberately used the controlled PDF corpus.\nThis was important because I wanted to isolate retrieval behavior before introducing the additional variability of the complete multi-source knowledge base.\nThe evaluation contained:\n35 total questions\n30 answerable questions\n5 unanswerable questions\n70 explicit gold/acceptable evidence records\nThe 30 answerable queries were used for the Dense/BM25/Hybrid retrieval comparison.\nThe blind spot: exact terminology matters\nConsider a query containing:\nHybridRetriever\nrrf_k\ncandidate_k\ncontent_role\nThese are not ordinary natural-language concepts.\nThey are identifiers.\nA semantic model may understand their general meaning, but exact lexical matching can be extremely valuable.\nThis led to the next experiment:\nWhat does BM25 contribute when dense retrieval already exists?\nBM25 became the second retrieval signal\nThe BM25 implementation indexes chunk content together with provenance metadata.\nThe tokenizer was intentionally designed to preserve useful identifiers while also handling forms such as camelCase.\nThat matters for technical corpus.\nA query containing:\nCrossEncoder reranking\nshould be able to match the exact terminology present in source code or technical documentation.\nThe benchmark showed a clear tradeoff.\nBM25\nDense retrieval remained stronger on this benchmark, particularly in the top-ranked results. BM25, however, was substantially faster and still recovered a meaningful amount of relevant evidence at larger candidate depths.\nThe important observation was not simply that one retriever performed better than the other.\nThe more useful question was:\n“Do they retrieve different useful evidence?”\nBefore building hybrid retrieval, I measured complementarity\nI compared the candidate sets produced by Dense and BM25 over the 30 answerable benchmark queries.\nThe results were:\nDense only       5\nBM25 only        2\nBoth            19\nNeither          4\nThe average candidate intersection was approximately:\n4.77 / 20\nwith a mean Jaccard similarity of:\n0.1393\nMore importantly:\nBM25 rescued Dense: 2 queries\nDense rescued BM25: 5 queries\nThe two retrievers were not producing identical candidate sets.\nThey were seeing the corpus differently.\nThat gave me a concrete reason to combine them.\nHybrid retrieval therefore became an experimental hypothesis rather than an architectural assumption:\nIf the two retrievers expose different evidence, can rank fusion recover useful evidence that either method would miss on its own?\nRRF: combining different retrieval signals\nFor the hybrid system I used\nReciprocal Rank Fusion\n.\nThe important property of RRF is that it doesn’t require the raw scores from different retrieval systems to be directly comparable.\nDense similarity scores and BM25 scores have different meanings and scales.\nInstead, RRF operates on rank positions.\nConceptually:\nDense ranking ──┐\n                ├──> RRF ──> unified ranking\nBM25 ranking ───┘\nThe implementation uses 1-based ranks and merges results by canonical\nchunk_id\n.\nThe same candidate depth is passed to both retrievers.\nThis gives the experiment a controlled structure:\nQuery\n  ├── Dense → top K candidates\n  └── BM25  → top K candidates\n             ↓\n            RRF\n             ↓\n        unified candidates\nThen the downstream evidence pipeline can operate on the fused result.\nThe hybrid result was more interesting than simply picking a winner\nThe controlled PDF benchmark produced these results:\nThe result was not simply that Hybrid retrieval was “better.”\nThe more precise observation was that\nHybrid retrieval achieved the highest Recall@20 on this benchmark, while Dense retrieval retained stronger early-ranking metrics.\nSpecifically:\nRecall@20\nDense    80.00%\nBM25     70.00%\nHybrid   86.67%\nHybrid therefore recovered relevant evidence within the top 20 for more queries than either individual retriever.\nAt the same time:\nMRR\nDense    0.5322\nHybrid   0.4615\nnDCG@5\nDense    0.4768\nHybrid   0.3930\nSo the additional candidate coverage from RRF did not translate into stronger top-ranked retrieval on this benchmark.\nLatency also increased because the Hybrid retriever executes both retrieval paths:\nDense    23.06 ms\nBM25      0.73 ms\nHybrid   32.64 ms\nThis made the result more interesting than a simple winner-takes-all comparison.\nThe experiment provided evidence that combining the two retrieval signals could increase deeper candidate coverage, while the ranking metrics showed that the fused ranking behaved differently from Dense retrieval at the top of the list.\nThat distinction is important.\nIf I looked only at Recall@20, Hybrid appeared to provide a clear coverage gain.\nIf I also looked at Recall@1, MRR, and nDCG@5, the result became more nuanced.\nThis is exactly why retrieval evaluation needs multiple metrics rather than a single “accuracy” number.\nRRF sensitivity: was the result dependent on one constant?\nOnce Hybrid retrieval showed higher Recall@20, I wanted to know whether that behavior depended heavily on the RRF constant.\nInstead of treating\nrrf_k = 60\nas a magic value, I evaluated:\nk = 10\nk = 20\nk = 40\nk = 60\nk = 100\nk = 200\nThe sensitivity matrix was:\nThe interesting part was the stability.\nAcross a 20× range of RRF constants:\nRecall@1  = 36.67%\nRecall@10 = 76.67%\nRecall@20 = 86.67%\nremained unchanged.\nThe variation was concentrated in the top five positions.\nSo I did not interpret\nk=40\nas a universal optimum.\nThe narrower conclusion was:\nOn this benchmark, the RRF constant had little effect on candidate coverage. Its measurable effect was concentrated in the ordering of the highest-ranked results.\nThat was useful because it showed that the observed Hybrid coverage was not dependent on finding one “magic” RRF constant.\nRetrieval depth changed the interpretation\nThe hybrid experiment reinforced an idea from M1:\nCandidate retrieval and final ranking are different problems.\nA system can successfully retrieve the correct evidence into its candidate pool without placing that evidence near the top.\nThat means:\nCandidate Recall\n        ≠\nFinal Ranking Quality\nThis distinction becomes especially important once multiple retrieval systems are fused.\nHybrid retrieval can improve the breadth of the candidate pool while still requiring another mechanism to determine which evidence should appear first.\nThat is where structural filtering and reranking enter.\nRetrieval needed to understand source structure\nM1 had already shown that semantic relevance does not automatically imply evidentiary usefulness.\nM2 made that problem broader.\nA knowledge base can contain:\nnavigation\nmetadata\nimplementation\ndocumentation\ntests\nconfiguration\nreference material\nactual evidence\nA table of contents can be semantically related to a question.\nA test can mention an API without explaining how it works.\nA configuration file can contain the exact keyword from a query while still being poor evidence for the user’s question.\nSo M2 introduced canonical content roles:\nnavigation\nmetadata\ndocumentation\nimplementation\nevidence\nreference\ntest\nconfiguration\nThese roles became part of the retrieval metadata rather than being thrown away during ingestion.\nStructural eligibility was deliberately applied after RRF\nThe final retrieval pipeline became:\nDense Retrieval ──┐\n                  ├──> RRF\nBM25 Retrieval ───┘\n          ↓\nStructural Eligibility\n          ↓\nMetadata Filtering\n          ↓\nCross-Encoder Reranking\n          ↓\nFinal Evidence\nThe ordering is intentional.\nI didn’t want one retriever to make a source-level decision independently and then prevent another retriever from contributing.\nBoth retrieval signals first contribute candidates. Then the system applies source-aware controls. Unknown or missing structural roles are preserved rather than aggressively discarded. That makes the filtering policy conservative.\nMetadata filtering added explicit control\nThe next requirement was more operational.\nSometimes semantic relevance is not enough.\nIf I know that I want evidence from:\nsource_type = github\nlanguage = typescript\nrepository = RippleTalk\nthe retrieval system should be able to express that constraint directly.\nM2 therefore supports metadata filtering across dimensions such as:\nsource_type\nproject\nlanguage\nrepository\npath\ncontent_role\nThe semantics are:\nAND across dimensions\nOR within a dimension\nFor example:\nlanguage = typescript\nAND\nrepository = RippleTalk\nwhile multiple accepted repositories can be expressed as an OR within the repository dimension.\nThis is different from relevance. It is\nretrieval control\n. That distinction became increasingly important as the corpus grew.\nMore sophisticated ranking did not automatically mean better ranking\nM1 already taught me to be skeptical of the assumption that adding a reranker automatically improves retrieval.\nM2 reinforced it.\nThe pipeline supports an optional BGE cross-encoder reranker.\nThe reranker scores query/chunk pairs directly and reorders the candidate set.\nConceptually:\nretrieved candidates\n        ↓\n(query, chunk)\n        ↓\ncross-encoder\n        ↓\nreranker score\n        ↓\nnew ranking\nIt sounds like an obvious improvement. The benchmark did not support that assumption. In the evaluated configuration, reranking reduced the measured ranking metrics while adding substantial latency.\nThat made the decision straightforward:\nThe reranker remains optional rather than becoming a mandatory stage.\nThis is an important engineering outcome.\nA component does not deserve a permanent place in the architecture simply because it is more sophisticated.\nIf a component increases complexity and latency without improving the measured objective, the correct response is to treat it as an experiment rather than an unquestioned dependency.\nThe controlled PDF experiment was only the first stage\nThe single-PDF benchmark was useful for controlled comparison. But M2 was ultimately designed for heterogeneous sources. So I created a combined Knowledge Hub corpus and a separate multi-source gold-evidence dataset.\nThe multi-source dataset contained\n21 answerable questions\nspanning articles, GitHub/code, and project sources, with explicit gold evidence attached to the questions. The dataset was established before the final source-aware retrieval experimentation.\nThis changed the experiment.\nThe question was no longer:\n“Which retriever performs best on one controlled document?”\nIt became:\n“Does the retrieval architecture continue to behave coherently once different source types share the same knowledge space?”\nAfter the multi-source gold questions were in place, I implemented the remaining retrieval stages and evaluated them progressively as an ablation rather than assuming that every additional stage would improve the system.\nThe multi-source retrieval ablation\nThe final experiment evaluated six configurations over the same\n21 answerable questions\n:\nThe complete E1–E6 ablation was based on the 21-question multi-source benchmark.\nThe results were more interesting than a simple “more stages means better retrieval”.\nE3 — Hybrid + RRF\nreached\n100% Recall@5\n, with MRR\n0.7143\nand nDCG@5\n0.6980\n. This provided empirical evidence that Dense and BM25 were complementary on this heterogeneous corpus.\nE4 — Structural eligibility\ndid not change the measured ranking metrics on this benchmark. Recall@5, MRR, and nDCG@5 remained identical to E3, while latency increased only from\n27.47 ms to 28.32 ms\n. That means the benchmark did not demonstrate a retrieval-quality improvement from structural filtering, even though the stage provides deterministic structural quality control.\nE5 — Cross-encoder reranking\nproduced the most surprising result. Recall@5 fell from\n100% to 95.24%\n, MRR from\n0.7143 to 0.6742\n, and nDCG@5 from\n0.6980 to 0.6642\n, while latency increased from\n28.32 ms to 5532.58 ms\n. The reranker therefore remained an optional component rather than becoming a mandatory stage.\nE6 — Source-aware metadata filtering\npreserved Recall@5 at\n95.24%\nand Recall@10/20 at\n100%\n, while slightly changing MRR from\n0.6742 to 0.6766\nand nDCG@5 from\n0.6642 to 0.6663\n. Its larger architectural value is explicit control over retrieval scope by provenance, rather than a large recall improvement on this benchmark.\nThe important conclusion is therefore not that every stage improved retrieval.\nIt is:\nDifferent retrieval stages have different roles, and empirical evaluation is necessary to determine whether each stage actually contributes to retrieval quality.\nThe two evaluation stages therefore answered different questions:\nControlled PDF benchmark\n        ↓\nIsolate retrieval behavior\n        ↓\nDense / BM25 / Hybrid\n        ↓\nRRF-k sensitivity\n        ↓\nUnderstand ranking vs coverage\nMulti-source gold-evidence benchmark\n        ↓\nEvaluate heterogeneous sources\n        ↓\nHybrid retrieval\n        ↓\nStructural eligibility\n        ↓\nCross-encoder reranking\n        ↓\nSource-aware metadata filtering\n        ↓\nMeasure the contribution and cost of each stage\nKeeping these experiments separate prevented me from treating one benchmark as representative of the entire system.\nThe retrieval contract became more important than the retrievers\nOne architectural lesson from M2 was that retrieval implementations should not own the entire data model. Dense retrieval knows about vectors. BM25 knows about lexical terms. RRF knows about ranks. The reranker knows about query-document pair scores. None of them should need to understand the full ingestion architecture.\nThat is why the shared retrieval contract is built around the canonical\nChunk\nand its provenance. The retrieval layer can therefore evolve independently.\nToday:\nDense\nBM25\nHybrid\ncan operate over the same representation.\nThe important part is not the number of retrievers.\nIt is that they agree on what a retrieved piece of evidence is.\nEvaluation became part of the architecture\nOne of the biggest changes from M1 to M2 was the evaluation harness.\nThe controlled PDF benchmark contains:\n35 total questions\n30 answerable questions\n5 unanswerable questions\n70 explicit gold/acceptable evidence records\nThe evaluation measures:\nRecall@1\nRecall@5\nRecall@10\nRecall@20\nMRR\nnDCG@5\nlatency\nThe goal was not to produce a single leaderboard number.\nThe goal was to make retrieval changes measurable.\nA retrieval experiment should answer:\nWhat changed?\nWhy did it change?\nDid candidate coverage improve?\nDid ranking improve?\nWhat did it cost?\nThat mindset became one of the most useful parts of the project.\nThe API became the system boundary\nOnce the ingestion and retrieval layers were stable, I needed a clean way to expose the system. That became the FastAPI layer.\nThe API exposes:\nGET  /health\nPOST /search\nPOST /retrieve\nPOST /ingest\nGET  /documents\nGET  /sources\nThe important design choice was to keep the API thin.\nIt does not contain a second retrieval implementation.\nIt does not duplicate ingestion logic.\nIt does not recreate the metadata model.\nIt calls into the existing canonical pipeline.\nThe architecture therefore becomes:\n┌───────────────┐\n                  │    FastAPI    │\n                  └───────┬───────┘\n                          │\n                ┌─────────▼─────────┐\n                │ Canonical Retrieval│\n                │      Pipeline      │\n                └─────────┬─────────┘\n                          │\n        ┌─────────────────┼─────────────────┐\n        │                 │                 │\n      Dense              BM25            Filters\n        │                 │                 │\n        └─────────────────┼─────────────────┘\n                          │\n                        Qdrant\nSwagger/OpenAPI makes the subsystem directly inspectable without requiring a separate client.\nThat matters because the project is no longer just an experiment running through scripts.\nIt now has a stable interface.\nPersistence also changes the architecture\nQdrant is not treated simply as a temporary vector store. The API startup path hydrates the corpus view from persisted data.\nThat gives the service a useful property:\nThe API does not need to reconstruct the entire corpus from scratch every time it starts. The canonical documents and chunks remain persisted, while the service reconstructs the operational view it needs.\nThis is another example of separating:\nsource acquisition\nfrom:\nruntime retrieval\nWhat M2 actually completed\nAt the end of M2, Knowledge Hub can:\ningest PDFs\ningest Markdown\ningest source code\nparse code using Tree-sitter\ningest GitHub repositories\ningest Hashnode articles\npreserve canonical provenance\npersist canonical chunks in Qdrant\nperform dense retrieval\nperform BM25 retrieval\ncombine retrieval signals with RRF\napply structural eligibility\napply metadata filters\noptionally rerank candidates\nevaluate retrieval with a reproducible benchmark\nexpose the system through FastAPI\nexpose OpenAPI/Swagger for interactive inspection\nThe final validation suite reached:\n324 passed\n6 skipped\nThe system was also validated through real corpus ingestion, GitHub ingestion, Hashnode ingestion, PerfEngine parsing, Uvicorn/API startup, Swagger/OpenAPI access, PDF retrieval, Ruff, compile checks, and repository diff validation.\nThat was the point where I considered M2 complete.\nWhat M2 intentionally does not solve\nThere is one boundary I want to make explicit.\nKnowledge Hub does not automatically synchronize changing sources.\nIf a GitHub repository changes, the system does not yet automatically detect the new commit and incrementally update only the affected documents.\nIf an article changes, the system does not automatically determine which chunks became stale.\nIf a source disappears, deletion propagation is not automatic.\nFor the completed M2 system, ingestion is explicit.\nThat is intentional scope control.\nFuture work\nKnowledge Hub does not yet include the following capabilities. They are planned as productionization work and are not part of the completed M2 implementation. They fall into two areas.\nIncremental indexing\nToday, ingestion is explicit. Making the system maintain itself as sources change would require tracking identity and version at every level:\ncontent_hash\ndocument identity\nchunk identity\nsource version\nEach document and chunk would be classified by change state:\nNEW\nMODIFIED\nUNCHANGED\nDELETED\nFor GitHub, this would mean tracking the repository, branch/ref, commit SHA, and file path, so that a new commit updates only the affected documents.\nFor articles, it would mean tracking article identity, content hash, and update/version information, so that stale chunks can be detected and replaced.\nDeletion would also propagate, so that a source that disappears no longer leaves evidence behind in the index.\nRetrieval Inspector\nThe second area is observability. Rather than presenting only the final answer, the Retrieval Inspector would expose the retrieval pipeline as a sequence of stages:\nQUERY\n  ↓\nDense Results\n  ↓\nBM25 Results\n  ↓\nRRF Results\n  ↓\nStructural Filtering\n  ↓\nMetadata Filtering\n  ↓\nReranking\n  ↓\nFinal Evidence\nAt each stage, it would show what the candidate set looked like and what changed, with enough information to answer a question like:\nWhy did this chunk rank here?\nTogether, these two additions would move Knowledge Hub from a system that can be measured at a point in time to one that stays current and can be explained continuously.\nThe biggest lessons from M2\n**Multi-source retrieval is an ingestion problem before it is a retrieval problem\n**It is tempting to think that adding more sources simply means adding more documents to the vector database.\nIt doesn’t.\nDifferent sources have different structures, identities, acquisition mechanisms, and security considerations.\nIf those differences are not handled before retrieval, the retrieval layer becomes polluted with source-specific assumptions.\nThe canonical Document → Chunk → Provenance model was therefore more important than any individual retriever.\n**Dense retrieval is powerful, but semantic similarity is not the whole search problem\n**\nDense retrieval performed strongly on the benchmark. But the complementarity analysis showed that BM25 and Dense did not retrieve the same candidates. This was particularly relevant for technical terminology and identifiers. The lesson isn’t that BM25 replaces embeddings.\nIt is that different retrieval signals encode different notions of relevance.\nHybrid retrieval changed coverage more than ranking\nI did not want to add BM25 simply because “hybrid search is what production systems do.” The candidate overlap gave me an empirical reason to test it. Then the benchmark showed where the hybrid system actually helped:\nRecall@20:\nDense   80.00%\nHybrid  86.67%\nwhile top-ranked metrics did not improve.\nThe RRF sensitivity experiment added another layer.\nAcross\nk=10\nthrough\nk=200\n, Recall@10 and Recall@20 remained unchanged. The variation was concentrated in the top-five ordering.\nSo the more accurate conclusion is:\nOn this benchmark, hybrid retrieval improved deeper candidate coverage more than it improved ranking quality.\nThat is a much more useful result than calling hybrid retrieval universally better.\nMore ranking sophistication can make a system worse\nThe reranker was technically capable. That did not make it automatically useful. On the evaluated configuration, it reduced measured ranking metrics while increasing latency.\nThat is a reminder that architecture should follow measured objectives. Not the other way around.\nProvenance is not metadata decoration\nOnce multiple sources enter the system, provenance becomes part of retrieval correctness.\nKnowing that a chunk came from:\nrepository\n→ commit\n→ file\n→ symbol\nor:\narticle\n→ section\n→ chunk\nchanges how confidently the evidence can be interpreted.\nWithout provenance, retrieval can tell you\nwhat\nmatched. With provenance, it can also tell you\nwhere it came from\n.\nRetrieval quality needs multiple measurements\nRecall@20 answered one question.\nMRR answered another.\nnDCG@5 answered another.\nLatency answered a completely different one.\nThe hybrid experiment made this obvious. One metric suggested a gain.\nOther metrics showed that the gain did not translate into better top-ranked evidence.\nThe RRF sensitivity experiment showed that even a parameter sweep can affect ranking without changing deeper candidate coverage. That is why I don’t want to reduce retrieval evaluation to a single number.\nWhat changed from M1 to M2?\nM1 was about understanding retrieval failure inside a controlled document.\nM2 was about making the retrieval system survive heterogeneity.\nThe progression looks roughly like this:\nM1\nSingle PDF\n   ↓\nDense Retrieval\n   ↓\nReranking\n   ↓\nStructural Failure Analysis\n   ↓\nPassage-Level Evaluation\nM2 expanded the problem:\nMultiple Sources\n   ↓\nCanonical Ingestion\n   ↓\nSource-Aware Provenance\n   ↓\nDense + BM25\n   ↓\nRRF Hybrid Retrieval\n   ↓\nRRF-k Sensitivity\n   ↓\nStructural Eligibility\n   ↓\nMetadata Filtering\n   ↓\nOptional Reranking\n   ↓\nReproducible Evaluation\n   ↓\nFastAPI Boundary\nThe interesting part is that M2 did not simply add features.\nIt changed the questions I could ask about the system.\nThe real outcome of M2\nWhen I started M2, I thought the primary problem was:\n“How do I make RAG work across more sources?”\nBy the end, that no longer felt like the right question.\nThe harder problem was:\nHow do I preserve evidence identity across heterogeneous sources while keeping retrieval strategies interchangeable, measurable, and inspectable?\nThat led to several design decisions that I would not have made from a typical RAG tutorial:\nsource-specific ingestion with source-agnostic retrieval\ncanonical chunks with explicit provenance\nAST-aware code chunking\nlexical retrieval alongside dense retrieval\nempirical candidate complementarity analysis\nRRF rather than score normalization\nRRF sensitivity evaluation\nstructural eligibility after candidate fusion\nmetadata filtering as explicit retrieval control\noptional rather than mandatory reranking\npassage-level evaluation\nretrieval/inference separation\na thin API boundary over the actual system\nThe biggest lesson is probably the same one that emerged from M1:\nThe difficult part of RAG is not making the pipeline produce an answer.\nThe difficult part is being able to explain why the system retrieved what it retrieved, measure where it fails, change one component at a time, and know whether the change actually improved the objective.\nM1 taught me how to investigate retrieval failure inside one document.\nM2 forced me to do that across an entire knowledge base.\nAnd that is a much more interesting engineering problem.\n#\nrag\n#\nai\n#\ninformation-retrieval\n#\nvector-search\n#\npython\n#\nqdrant\n#\nsystem-design",
  "metadata": {
    "title": "Knowledge Hub: Multi-Source RAG System",
    "description": "Explore Knowledge Hub, a multi-source RAG system for PDFs, code, GitHub, and articles with hybrid retrieval, evaluation, provenance, and FastAPI.",
    "keywords": null,
    "author": null,
    "og:title": "Knowledge Hub: Multi-Source RAG System",
    "og:description": "Explore Knowledge Hub, a multi-source RAG system for PDFs, code, GitHub, and articles with hybrid retrieval, evaluation, provenance, and FastAPI.",
    "og:url": "https://medium.com/@jaivalsuthar/knowledge-hub-multi-source-rag-system-9977a52a1c05",
    "og:site_name": "Jaival Suthar | Engineering Portfolio",
    "og:image": "https://cdn.hashnode.com/uploads/og-images/6972fc71207454ddbf7903dd/6566a354-f967-4f38-bd4d-e29519a96c86.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:creator": "https://x.com/Jaival_Suthar",
    "twitter:title": "Knowledge Hub: Multi-Source RAG System",
    "twitter:description": "Explore Knowledge Hub, a multi-source RAG system for PDFs, code, GitHub, and articles with hybrid retrieval, evaluation, provenance, and FastAPI.",
    "twitter:image": "https://cdn.hashnode.com/uploads/og-images/6972fc71207454ddbf7903dd/6566a354-f967-4f38-bd4d-e29519a96c86.png",
    "article:published_time": "2026-09-21T16:42:44.185Z",
    "article:modified_time": "2026-09-21T16:42:44.185Z",
    "article:author": "https://jaivalsuthar.hashnode.dev/@JaivalSuthar"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/75199ac0a168/code/body-000.txt",
      "chars": 36,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-001.txt",
      "chars": 40,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-002.txt",
      "chars": 29,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-003.txt",
      "chars": 23,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-004.txt",
      "chars": 47,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-005.txt",
      "chars": 23,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-006.txt",
      "chars": 76,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-007.txt",
      "chars": 10,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-008.txt",
      "chars": 7,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-009.txt",
      "chars": 60,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-010.txt",
      "chars": 81,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-011.txt",
      "chars": 141,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-012.txt",
      "chars": 58,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-013.txt",
      "chars": 36,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-014.txt",
      "chars": 39,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-015.txt",
      "chars": 54,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-016.txt",
      "chars": 44,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-017.txt",
      "chars": 57,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-018.txt",
      "chars": 49,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-019.txt",
      "chars": 87,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-020.txt",
      "chars": 196,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-021.txt",
      "chars": 67,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-022.txt",
      "chars": 44,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-023.txt",
      "chars": 50,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-024.txt",
      "chars": 117,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-025.txt",
      "chars": 163,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-026.txt",
      "chars": 272,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-027.txt",
      "chars": 18,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-028.txt",
      "chars": 104,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-029.txt",
      "chars": 84,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-030.txt",
      "chars": 610,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-031.txt",
      "chars": 19,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-032.txt",
      "chars": 18,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-033.txt",
      "chars": 21,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-034.txt",
      "chars": 61,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-035.txt",
      "chars": 31,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-036.txt",
      "chars": 138,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-037.txt",
      "chars": 30,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-038.txt",
      "chars": 41,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-039.txt",
      "chars": 36,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-040.txt",
      "chars": 26,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-041.txt",
      "chars": 113,
      "language": "text"
    },
    {
      "file": "attachments/75199ac0a168/code/body-042.txt",
      "chars": 265,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/edd6b0a2-bd87-4c65-8a60-0f1ba14164c5.png",
      "bytes": 1293155,
      "reused": true,
      "local_path": "attachments/75199ac0a168/images/000.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/edd6b0a2-bd87-4c65-8a60-0f1ba14164c5.png",
      "alt": "Knowledge Hub: Building a Multi-Source RAG System for Real-World Knowledge",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/21b16bea-b125-4900-9b1e-61da5de9f8e7.png",
      "bytes": 101152,
      "reused": true,
      "local_path": "attachments/75199ac0a168/images/001.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/21b16bea-b125-4900-9b1e-61da5de9f8e7.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/723d1601-5aeb-4208-85ee-a074d5f41262.png",
      "bytes": 221858,
      "reused": true,
      "local_path": "attachments/75199ac0a168/images/002.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/723d1601-5aeb-4208-85ee-a074d5f41262.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/c46baff4-24d1-45d4-a09b-1247fc543621.png",
      "bytes": 262497,
      "reused": true,
      "local_path": "attachments/75199ac0a168/images/003.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/c46baff4-24d1-45d4-a09b-1247fc543621.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/09dc5b2e-0231-45ec-a4f8-795a27f8dfb4.png",
      "bytes": 150255,
      "reused": true,
      "local_path": "attachments/75199ac0a168/images/004.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/09dc5b2e-0231-45ec-a4f8-795a27f8dfb4.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/2ddd15f8-5121-4abb-a322-34a51f91b8f7.png",
      "bytes": 407917,
      "reused": true,
      "local_path": "attachments/75199ac0a168/images/005.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/2ddd15f8-5121-4abb-a322-34a51f91b8f7.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/80ac5297-718d-42a5-a93e-94c131e85f4a.png",
      "bytes": 116535,
      "reused": true,
      "local_path": "attachments/75199ac0a168/images/006.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6972fc71207454ddbf7903dd/80ac5297-718d-42a5-a93e-94c131e85f4a.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://jaivalsuthar.hashnode.dev/knowledge-hub-multi-source-rag-system.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@JaivalSuthar",
      "label": "J",
      "type": "a"
    },
    {
      "href": "https://x.com/Jaival_Suthar",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/jaival-suthar-57195b257/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/series/building-ai-systems-from-scratch",
      "label": "Part of series Building AI Systems from Scratch",
      "type": "a"
    },
    {
      "href": "https://github.com/Jaival-Suthar/knowledge-hub",
      "label": "https://github.com/Jaival-Suthar/knowledge-hub",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/rag",
      "label": "# rag",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/ai",
      "label": "# ai",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/information-retrieval",
      "label": "# information-retrieval",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/vector-search",
      "label": "# vector-search",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/python",
      "label": "# python",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/qdrant",
      "label": "# qdrant",
      "type": "a"
    },
    {
      "href": "https://jaivalsuthar.hashnode.dev/tag/system-design",
      "label": "# system-design",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:10:22",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:03Z",
  "source_url": "https://jaivalsuthar.hashnode.dev/knowledge-hub-multi-source-rag-system",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 34326,
  "expected_body_images": 7,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/75199ac0a168/pages/source.html",
  "local_html": "attachments/75199ac0a168/pages/article.html",
  "body_base_path": "attachments/75199ac0a168/pages/",
  "body_markdown_path": "attachments/75199ac0a168/pages/article.md"
}