{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Vicki Boykis",
  "title_listed": "Querying 3 billion vectors",
  "keywords": "向量检索,embedding,性能",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://vickiboykis.com/2026/02/21/querying-3-billion-vectors/",
  "publish_date": "2026-02",
  "corpus_stage": "数据处理",
  "notes": "30亿向量查询",
  "id": "c9e852cb114a",
  "collected_at": "2026-10-02T15:22:57",
  "title": "Querying 3 billion vectors | ✰Vicki Boykis✰",
  "body_markdown": "Querying 3 billion vectors\nFeb 21 2026\nRecently, I got nerd-sniped by this exchange between\nJeff Dean\nand someone trying to query 3 billion vectors.\nI was curious to see if I could implement the\noptimal map-reduce\nsolution he alludes to in his reply.\nA vector is a list/array of floating point numbers of\nn\ndimensions, where\nn\nis the length of the list. The reason you might perform vector search is to find words or items that are semantically similar to each other, a common pattern in search, recommendations, and generative retrieval applications\nlike Cursor\nwhich heavily leverage\nembeddings.\nI started by writing an extremely naive implementation which made the following assumptions:\nwe have 3 billion searchable (document) vectors and ~1k query vectors (a number I made up)\nBoth of the vector sets are stored on disk in\n.npy\nformat (simple format for\nstoring numpy arrays\nWe’d like to compare each of the query vectors against the larger pool of document vectors and return the resulting similarity (dot product) for each of the vector combinations.\n3k total reference vectors (to see if we could intially run this amount before scaling)\nThe vectors are of dimensionality (n) 768, a common dimensionality for many models that allow for\nsimilarity-based embedding queries\nimport\nnumpy\nas\nnp\nfrom\nloguru\nimport\nlogger\nimport\ntime\nimport\nos\n# start with 3_000 vectors to keep things small\ntotal_vectors_num\n=\n3_000_000_000\nquery_vectors_num\n=\n1_000\ndef\ngenerate_random_vectors\n(num_vectors:int)\n->\nnp\n.\narray:\nlogger\n.\ninfo(\nf\n\"Generating\n{\nnum_vectors\n}\nvectors...\"\n)\nrng\n=\nnp\n.\nrandom\n.\ndefault_rng()\nvectors\n=\nrng\n.\nrandom((num_vectors,\n768\n))\nreturn\nvectors\ndef\nget_dot_products\n(vectors_file:np\n.\narray, query_vectors:np\n.\narray)\n->\nlist[np\n.\narray]:\ntotal_products_computed\n=\n0\ndot_products\n=\n[]\nfor\nv\nin\nvectors_file:\nfor\nqv\nin\nquery_vectors:\ndot_product\n=\nv\n@\nqv\ndot_products\n.\nappend(dot_product)\ntotal_products_computed\n+=\n1\nif\ntotal_products_computed\n%\n100000\n==\n0\n:\nlogger\n.\ninfo(\nf\n\"Total vectors processed:\n{\ntotal_products_computed\n}\n\"\n)\nreturn\ndot_products\n# Generate initial vectors and query vectors and write to disk\ndoc_vectors\n=\ngenerate_random_vectors(total_vectors_num)\nquery_vectors\n=\ngenerate_random_vectors(query_vectors_num)\nnp\n.\nsave(\n'vectors.npy'\n, doc_vectors)\n# Load vectors from disk\nlogger\n.\ninfo(\n\"Loading file from disk...\"\n)\nvectors_file\n=\nnp\n.\nload(\n'vectors.npy'\n)\nstart_time\n=\ntime\n.\ntime()\nlogger\n.\ninfo(\n\"Getting dot products...\"\n)\nresults\n=\nget_dot_products(vectors_file, query_vectors)\nend_time\n=\ntime\n.\ntime()\nlogger\n.\ninfo(\nf\n\"Execution time:\n{\nend_time\n-\nstart_time\n:\n.4f\n}\nseconds\"\n)\nlogger\n.\ninfo(\nf\n\"Number of dot products computed:\n{\nlen(results)\n}\n\"\n)\nThis, predictably, didn’t do so great, even on my M2 Macbook, even at 3,000 vectors, one million times less than 3 billion embeddings, taking 2 seconds.\nResults:\n2025-12-13 17:53:25.675 | INFO     | __main__:generate_random_vectors:9 - Generating\n3000\nvectors...\n2025-12-13 17:53:25.691 | INFO     | __main__:generate_random_vectors:9 - Generating\n1000\nvectors...\n2025-12-13 17:53:25.698 | INFO     | __main__:<module>:39 - Loading file from disk...\n2025-12-13 17:53:25.700 | INFO     | __main__:<module>:43 - Getting dot products...\n2025-12-13 17:53:27.688 | INFO     | __main__:get_dot_products:24 - Total vectors processed:3000000\n2025-12-13 17:53:27.688 | INFO     | __main__:<module>:47 - Execution time: 1.9877 seconds\n2025-12-13 17:53:27.688 | INFO     | __main__:<module>:48 - Number of dot products computed:\n3000000\nSo I\nvectorized the numpy operation\n, which made things much faster.\nimport\nnumpy\nas\nnp\nfrom\nloguru\nimport\nlogger\nimport\ntime\ntotal_vectors_num\n=\n3_000\nquery_vectors_num\n=\n1_000\ndef\ngenerate_random_vectors\n(num_vectors:int)\n->\nnp\n.\narray:\nlogger\n.\ninfo(\nf\n\"Generating\n{\nnum_vectors\n}\nvectors...\"\n)\nrng\n=\nnp\n.\nrandom\n.\ndefault_rng()\nvectors\n=\nrng\n.\nrandom((num_vectors,\n768\n))\nreturn\nvectors\ndef\nget_dot_products_vectorized\n(vectors_file:np\n.\narray, query_vectors:np\n.\narray):\ndot_products\n=\nvectors_file\n@\nquery_vectors\n.\nT\nreturn\ndot_products\n.\nflatten()\n# collapse into single dim\n# Generate initial vectors and query vectors and write to disk\nram_vectors\n=\ngenerate_random_vectors(total_vectors_num)\nquery_vectors\n=\ngenerate_random_vectors(query_vectors_num)\nnp\n.\nsave(\n'vectors.npy'\n, ram_vectors)\n# Load vectors from disk\nlogger\n.\ninfo(\n\"Loading file from disk...\"\n)\nvectors_file\n=\nnp\n.\nload(\n'vectors.npy'\n)\nstart_time\n=\ntime\n.\ntime()\nlogger\n.\ninfo(\n\"Getting dot products...\"\n)\nresults\n=\nget_dot_products_vectorized(vectors_file, query_vectors)\nend_time\n=\ntime\n.\ntime()\nlogger\n.\ninfo(\nf\n\"Execution time:\n{\nend_time\n-\nstart_time\n:\n.4f\n}\nseconds\"\n)\nlogger\n.\ninfo(\nf\n\"Number of dot products computed:\n{\nlen(results)\n}\n\"\n)\nAt .017 seconds, this was a big improvement!\nResults:\n2025-12-13 17:52:52.810 | INFO     | __main__:generate_random_vectors:9 - Generating\n3000\nvectors...\n2025-12-13 17:52:52.831 | INFO     | __main__:generate_random_vectors:9 - Generating\n1000\nvectors...\n2025-12-13 17:52:52.874 | INFO     | __main__:<module>:39 - Loading file from disk...\n2025-12-13 17:52:52.876 | INFO     | __main__:<module>:43 - Getting dot products...\n2025-12-13 17:52:52.887 | INFO     | __main__:<module>:47 - Execution time: 0.0107 seconds\n2025-12-13 17:52:52.887 | INFO     | __main__:<module>:48 - Number of dot products computed:\n3000000\nI tried a 3 million sample size with this improvement. This took 12 seconds.\n2025-12-13 19:39:43.830 | INFO     | __main__:generate_random_vectors:12 - Generating 3000000 vectors...\n2025-12-13 19:39:57.509 | INFO     | __main__:generate_random_vectors:12 - Generating 1000 vectors...\n2025-12-13 19:39:58.978 | INFO     | __main__:<module>:57 - Loading file from disk...\n2025-12-13 19:40:00.131 | INFO     | __main__:<module>:61 - Getting dot products...\n2025-12-13 19:40:12.984 | INFO     | __main__:<module>:65 - Execution time: 12.8491 seconds\n2025-12-13 19:40:12.992 | INFO     | __main__:<module>:66 - Number of dot products computed: 3000000000\nWe could also reduce even further by converting the data to float32:\ndoc_vectors\n=\ngenerate_random_vectors(total_vectors_num)\n.\nastype(np\n.\nfloat32)\nquery_vectors\n=\ngenerate_random_vectors(query_vectors_num)\n.\nastype(np\n.\nfloat32)\n2025-12-13 18:13:52.152 | INFO     | __main__:generate_random_vectors:10 - Generating 3000 vectors...\n2025-12-13 18:13:52.168 | INFO     | __main__:generate_random_vectors:10 - Generating 1000 vectors...\n2025-12-13 18:13:52.176 | INFO     | __main__:<module>:55 - Loading file from disk...\n2025-12-13 18:13:52.178 | INFO     | __main__:<module>:59 - Getting dot products...\n2025-12-13 18:13:52.182 | INFO     | __main__:<module>:63 - Execution time: 0.0045 seconds\n2025-12-13 18:13:52.182 | INFO     | __main__:<module>:64 - Number of dot products computed: 3000000\nWith these small improvements, we’ve already sped up inference to ~13 seconds for 3 million vectors, which means for 3 billion, it would take 1000x longer, or ~3216 minutes.\n|approach    | query_vectors | doc_vectors   | time     |\n|----------- |---------------|---------------|----------|\n| Naive      | 1,000         | 3,000         | 1.9877s  |\n| Vectorized | 1,000         | 3,000         | 0.0107s  |\n| Vectorized | 1,000         | 3,000,000     | 12.8491s |\n| Np.Float32 | 1,000         | 3,0000        | 0.0045s  |\nWhen we start to run it to test, however, we run into a different problem: OOM. Why? The amount of memory needed to process 3 billion objects, each as\nfloat32\nobject that’s 4 bytes in size, would be 8 million GB.\nvectors\n=\nrng\n.\nrandom((\n1\n,\n768\n))\n.\nastype(np\n.\nfloat32)\nprint(vectors\n.\nnbytes)\n3072\nprint(vectors\n.\nitemsize)\n4\nbytes_per_float32\n=\n4\nmemory_gb\n=\n(\n3000000000\n*\n1000\n*\n768\n*\nbytes_per_float32)\n/\n(\n1024\n**\n3\n)\n8583068.84765625\n=\n8.6\nTB\nIn order to improve this, we would need to do some heavy lifting of the kind Jeff Dean prescribed. First, we could to change the code to use generators and batch the comparison operations. We could write every n operations to disk, either directly or through memory mapping. Or, we could use system-level optimized code calls - we could rewrite the code in Rust or C, or use a library like\nSimSIMD\nexplicitly made for similarity comparisons between vectors at scale.\nBefore I started on any further optimizations, upon further inspection, there were some things about the problem that I realized weren’t clear to me: 3 billion vector embeddings queried a few thousand times could mean:\nI have a single query vector, and I query all 3 billion vectors once, get the dot product, and get all results\nI have a single query vector, I query all 3 billion vectors once, get the dot product, and return top-k results, which is easier because we can do ANN search\nIn this case, do I need to return the two initial vectors also? Or just the result?\nDo I need to re-rank the results by similarity in any way?\nI have 1,000 query vectors, and I query all 3 billion vectors once, and get the dot product of all results\nAre these vectors already in-memory when we intially start working with them or will they always be on-disk? Are we reading them one at a time, or streaming them?\nWhat kind of machine are we assuming: Are we running this locally? What are the specs of the machine? Are we assuming the vectors come to us in a specific, optimized format?\nDo we have GPUs and are we allowed to use them?\nHow\nbig are our embeddings?\n- this is extremely important and could significantly impact our representation, input vector size and output results\nAre we assuming we can compress their representation at all, i.e. is compressiong from float64 to float32 tolerable wrt to accuracy?\nHow much time do we have to generate this one-off project? Are we sure it’s really a one-off?\nAll of these dictate the additional time and resources spent on the solution. What I realized is the same thing I’ve seen so many of these problems over the years, that the technical solution is no longer the hardest one to achieve: the hardest one is nailing down the requirements.\n#embeddings\n#python\n#search\n#engineering",
  "metadata": {
    "title": "Querying 3 billion vectors | ✰Vicki Boykis✰",
    "description": "Requirements are hard",
    "keywords": "embeddings,python,search,engineering,",
    "author": null,
    "og:type": "article",
    "og:url": "https://vickiboykis.com/2026/02/21/querying-3-billion-vectors/",
    "og:title": "Querying 3 billion vectors",
    "og:description": "Requirements are hard",
    "og:image": "https://vickiboykis.com/images/logo.png",
    "twitter:card": "summary",
    "twitter:url": "https://vickiboykis.com/2026/02/21/querying-3-billion-vectors/",
    "twitter:title": "Querying 3 billion vectors",
    "twitter:description": "Requirements are hard",
    "twitter:image": "https://vickiboykis.com/images/logo.png",
    "twitter:site": "@vboykis",
    "article:published_time": "2026-02-21T00:00:00+00:00",
    "article:modified_time": "2026-02-21T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/c9e852cb114a/code/body-000.txt",
      "chars": 1476,
      "language": "text"
    },
    {
      "file": "attachments/c9e852cb114a/code/body-001.txt",
      "chars": 664,
      "language": "text"
    },
    {
      "file": "attachments/c9e852cb114a/code/body-002.txt",
      "chars": 1126,
      "language": "text"
    },
    {
      "file": "attachments/c9e852cb114a/code/body-003.txt",
      "chars": 564,
      "language": "text"
    },
    {
      "file": "attachments/c9e852cb114a/code/body-004.txt",
      "chars": 573,
      "language": "text"
    },
    {
      "file": "attachments/c9e852cb114a/code/body-005.txt",
      "chars": 154,
      "language": "text"
    },
    {
      "file": "attachments/c9e852cb114a/code/body-006.txt",
      "chars": 566,
      "language": "text"
    },
    {
      "file": "attachments/c9e852cb114a/code/body-007.txt",
      "chars": 348,
      "language": "text"
    },
    {
      "file": "attachments/c9e852cb114a/code/body-008.txt",
      "chars": 223,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://gist.github.com/user-attachments/assets/ecca4afd-81bf-45a4-9043-ad7da174d93a",
      "bytes": 482864,
      "reused": true,
      "local_path": "attachments/c9e852cb114a/images/000.jpg",
      "src": "https://gist.github.com/user-attachments/assets/ecca4afd-81bf-45a4-9043-ad7da174d93a",
      "alt": "image",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/LRitzdorf/TheJeffDeanFacts",
      "label": "Jeff Dean",
      "type": "a"
    },
    {
      "href": "https://vickiboykis.com/2021/06/06/the-humble-hash-aggregate/",
      "label": "optimal map-reduce",
      "type": "a"
    },
    {
      "href": "https://read.engineerscodex.com/p/how-cursor-indexes-codebases-fast",
      "label": "like Cursor",
      "type": "a"
    },
    {
      "href": "https://vickiboykis.com/what_are_embeddings/",
      "label": "embeddings.",
      "type": "a"
    },
    {
      "href": "https://numpy.org/doc/stable/reference/generated/numpy.lib.format.html",
      "label": "storing numpy arrays",
      "type": "a"
    },
    {
      "href": "https://pythonspeed.com/articles/vectorization-python/",
      "label": "vectorized the numpy operation",
      "type": "a"
    },
    {
      "href": "https://github.com/ashvardanian/SimSIMD",
      "label": "SimSIMD",
      "type": "a"
    },
    {
      "href": "https://vickiboykis.com/2025/09/01/how-big-are-our-embeddings-now-and-why/",
      "label": "big are our embeddings?",
      "type": "a"
    },
    {
      "href": "https://vickiboykis.com/tags/embeddings/",
      "label": "#embeddings",
      "type": "a"
    },
    {
      "href": "https://vickiboykis.com/tags/python/",
      "label": "#python",
      "type": "a"
    },
    {
      "href": "https://vickiboykis.com/tags/search/",
      "label": "#search",
      "type": "a"
    },
    {
      "href": "https://vickiboykis.com/tags/engineering/",
      "label": "#engineering",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:15Z",
  "source_url": "https://vickiboykis.com/2026/02/21/querying-3-billion-vectors/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 10085,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/c9e852cb114a/pages/source.html",
  "local_html": "attachments/c9e852cb114a/pages/article.html",
  "body_base_path": "attachments/c9e852cb114a/pages/",
  "body_markdown_path": "attachments/c9e852cb114a/pages/article.md"
}