{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Muhammad Fahad Bashir",
  "title_listed": "5.Vector Stores: Efficient Storage and Retrieval for Embeddings",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://mfahadbashir.hashnode.dev/5vector-stores-efficient-storage-and-retrieval-for-embeddings",
  "publish_date": "2024-09-12",
  "corpus_stage": "",
  "notes": "",
  "id": "e97cd2ba5c93",
  "collected_at": "2026-10-02T14:10:58",
  "title": "5.Vector Stores: Efficient Storage and Retrieval for Embeddings",
  "body_markdown": "5.Vector Stores: Efficient Storage and Retrieval for Embeddings\nPublished\nSeptember 12, 2024\n•\n7\nmin read\n•\nView as Markdown\nM\nMuhammad Fahad Bashir\nMachine Learning, Artificial Intelligence\nPart of series\nImplementing RAG systems from Scratch in-depth\nIn this continuation of our series of Retrieval-Augmented Generation (RAG), we will learn about the\nfinal step of the ingestion pipeline—vector stores\n. Previously, we covered embeddings in detail, from understanding what they are to implementing them with examples. Now, let’s move forward and explore vector stores: what they are, why they’re important, and how they operate.\nWhat is a Vector Store?\nAs we know, embeddings are numerical representations of data, typically stored in vector form. To access and retrieve relevant data using these vectors (a process known as retrieval), we need a dedicated space to store them—a vector store.\nImage source\nBut why do we need vector stores when we have traditional databases?\nTraditional databases are designed to store and retrieve exact matches, like finding the name of a song from a database when you know the exact title. These systems are not built to handle vector embeddings, which represent high-dimensional data used for similarity searches rather than exact matches. This is where vector stores come in.For understanding more in-depth about vector databse and traditional database\ncheck out source\nWhy Use Vector Stores?\nVector stores are specialized databases designed to store and query high-dimensional data like embeddings. They are essential for two key reasons:\nHigh-speed similarity search\n– Vector stores allow us to quickly find data that is similar, rather than exactly the same, which is crucial for applications like recommendation systems and semantic search.\nScalability and integration\n– They are built to handle vast amounts of data efficiently, making them suitable for use cases where scalability is a priority.\nBenefits of Vector Stores\nUsing vector stores offers several advantages:\nEfficient data storage\n– They can store complex, high-dimensional embeddings.\nFast retrieval\n– They are optimized for rapid similarity searches.\nSemantic search\n– They can perform searches based on the meaning of the data rather than relying on exact matches. This allows for more nuanced and relevant search results.\nFor example, by storing embeddings in a vector store, we can perform efficient similarity searches to retrieve the most relevant documents or chunks of text based on a query. It’s like having a search engine that understands the meaning behind the words, not just the words themselves.\nUse Cases of Vector Stores\nVector databases are not limited to recommendation systems. They are widely used in:\nRAG (Semantic Search)\n– For retrieving the most relevant information in tasks such as Q&A or document retrieval.\nAnomaly Detection\n– For identifying outliers in large datasets.\nBioinformatics -\nGenetic sequence storage , Protein structure querying\nHow Do Vector Stores Work?\nThere are two main techniques used in vector stores for performing similarity searches:\n1. Cosine Similarity\nCosine similarity measures the cosine of the angle between two vectors in a multi-dimensional space. In vector databases, cosine similarity is used for comparing high-dimensional embeddings to find how similar one vector (e.g., a query) is to others stored in the database.\nImagine vectors as arrows pointing in different directions in space. Cosine similarity measures how close these arrows are to each other. If two arrows point in the same direction, they are considered similar, regardless of their length.\nFor instance, let's say you're searching for a movie recommendation. Cosine similarity would find movies with themes, genres, or actors similar to the one you're searching for. Even if the exact movie isn't found, it can recommend one that's \"close enough\" in meaning or content.\n2. Dot Product\nThink of the dot product as measuring how much one list of numbers (vector A) overlaps with another list (vector B). If they overlap a lot, the result will be a large number, meaning they’re very similar.\nFor example, let’s say you're trying to find a song similar to one you're listening to. If the songs share many similar attributes (like tempo, artist, or genre), the dot product between their \"vectors\" will be high, meaning they’re very similar. The more attributes they share, the closer the match.\nTypes of Vector Stores\nA vector store takes care of storing embedded data and performing vector search for you. Langchain provides support for different vector stores. Some of the names are FAISS, Chroma, Pinecone, and Qdrant. each vector has its features.\nHere is the link to view the vector store\ns\nThey are further divided into two main categories.\nLocally Managed\n– In this case, you are responsible for all the steps: creating the store, saving the data, and managing it. Examples include\nChroma and FAISS.\nFully Managed / Cloud\n– Here, a third-party provider manages the store for you, typically accessed via an API. This service usually incurs costs but can be free for limited usage. Examples include\nPinecone and Qdrant\n.\nPractical Example\nNow, let’s take a look at practical examples of both locally managed and fully managed vector stores in action.\nLocally Managed\nwe will be using chromadb for this example which is mainly locally managed.You have to initialize\nChroma\nas the vector store to store and retrieve the embeddings.Here is the code.\n!pip install chromadb\n# installing package\nfrom\nlangchain.vectorstores\nimport\nChroma\n# Initialize Chroma vector store\nchroma_store = Chroma(embedding_function=embed_model, persist_directory=\n\"chroma_db\"\n)\n# Add documents to Chroma\nchroma_store.add_texts([doc.page_content\nfor\ndoc\nin\nsplit])\n# Save the Chroma vector store\nchroma_store.persist()\nNote: the\nembedding_function\npassed to Chroma should be an instance of an embedding model, not a list of embeddings.\nFully Managed / Cloud\nFor this example we will be using the online service\nQdrant\n.[\nThe leading open source vector database and similarity search engine]\nStep 1: Sign Up for Qdrant Cloud\nVisit Qdrant Cloud and sign up for an account.\nCreate a new project (you have to create a cluster and generate an API key).\nStep 2: Install required library\npip install qdrant-client langchain\nStep 3: Import module & Set up qdrant client in code\nfrom\nqdrant_client\nimport\nQdrantClient\nfrom\nlangchain.vectorstores\nimport\nQdrant\nfrom\nqdrant_client.http\nimport\nmodels\nas\nrest\n\nurl =\n\"<---qdrant cloud cluster url here --->\"\n# e.g., \"https://sampledb-abc12345.qdrantcloud.com\"\napi_key =\n\"<---api key here--->\"\nqdrant_client = QdrantClient(\n    url=url,\n    api_key=api_key,\n)\nStep 4: Prepare Embeddings and Set Collection Name\nStep 5: Create a Collection in Qdrant Cloud\nYou need to ensure that the collection you're going to use is created in Qdrant Cloud. If the collection doesn’t exist, you can create it.\nembeddings = embedded\n# Use the embeddings you have generated earlier\ncollection_name =\n\"my_documents\"\n# You can name this according to your preference\nqdrant_client.recreate_collection(\n    collection_name=collection_name,\n    vectors_config=rest.VectorParams(\n        size=len(embeddings[\n0\n]),\n# Embedding vector size\ndistance=rest.Distance.COSINE\n# Choose the distance metric\n)\n)\nStep 6:upsert the document:\nNow, you’ll upsert the embeddings and corresponding documents into Qdrant.\n# Create a list of dictionary objects, each having the document and its embedding\ndocuments = [\n    {\n\"id\"\n: i,\n\"text\"\n: doc.page_content,\n\"embedding\"\n: embedding}\nfor\ni, (doc, embedding)\nin\nenumerate(zip(split, embeddings))\n]\n\nqdrant_client.upload_collection(\n    collection_name=collection_name,\n    vectors=[doc[\n\"embedding\"\n]\nfor\ndoc\nin\ndocuments],\n# The embedding vectors\npayload=[{\n\"page_content\"\n: doc[\n\"text\"\n]}\nfor\ndoc\nin\ndocuments],\n# Original text payload\nids=[doc[\n\"id\"\n]\nfor\ndoc\nin\ndocuments],\n# Unique ID for each document\nbatch_size=\n50\n# Adjust batch size as necessary\n)\n# LangChain Qdrant vector store\nvector_store = Qdrant(client=qdrant_client, collection_name=collection_name, embeddings=embed_model)\nFinally, you can perform a similarity search on your vector store. This will retrieve documents based on how similar they are to the query embedding.\nView the Google Colab Notebook for playing around with code for both examples:\nClick Here\nYou can also explore other different vector stores as well.\nConclusion\nIn this article, we explored the role of\nvector stores\nin the RAG ingestion pipeline. We discussed why traditional databases aren't suited for handling embeddings and how vector stores, with techniques like\ncosine similarity\nand\ndot product\n, enable efficient similarity searches.\nThrough practical examples of both\nlocally managed\n(Chroma) and\ncloud-based\n(Qdrant) vector stores, we demonstrated how they are essential for AI applications like recommendation systems and semantic search.\nBy mastering vector stores, you can significantly enhance your AI systems' ability to search and retrieve relevant information quickly and at scale.\n#\ntutorial\n#\nrag\n#\nchatbot\n#\nvector-database\n#\nvector-embeddings\n#\ndata-ingestion\n#\nqdrant\n#\nchromadb\n#\ngoogle-colab\n65\nviews",
  "metadata": {
    "title": "5.Vector Stores: Efficient Storage and Retrieval for Embeddings",
    "description": "Vector stores efficiently handle high-dimensional data embeddings for advanced similarity searches, scalability, and diverse applications",
    "keywords": null,
    "author": null,
    "og:title": "5.Vector Stores: Efficient Storage and Retrieval for Embeddings",
    "og:description": "Vector stores efficiently handle high-dimensional data embeddings for advanced similarity searches, scalability, and diverse applications",
    "og:url": "https://mfahadbashir.hashnode.dev/5vector-stores-efficient-storage-and-retrieval-for-embeddings",
    "og:site_name": "Muhammad Fahad Bashir",
    "og:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1726052576256/739822e4-4c43-48b4-b6dc-9e3cd4e69a74.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "5.Vector Stores: Efficient Storage and Retrieval for Embeddings",
    "twitter:description": "Vector stores efficiently handle high-dimensional data embeddings for advanced similarity searches, scalability, and diverse applications",
    "twitter:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1726052576256/739822e4-4c43-48b4-b6dc-9e3cd4e69a74.png",
    "article:published_time": "2024-09-12T02:17:37.357Z",
    "article:author": "https://mfahadbashir.hashnode.dev/@mfahadbashir"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/e97cd2ba5c93/code/body-000.txt",
      "chars": 44,
      "language": "text"
    },
    {
      "file": "attachments/e97cd2ba5c93/code/body-001.txt",
      "chars": 304,
      "language": "text"
    },
    {
      "file": "attachments/e97cd2ba5c93/code/body-002.txt",
      "chars": 36,
      "language": "text"
    },
    {
      "file": "attachments/e97cd2ba5c93/code/body-003.txt",
      "chars": 328,
      "language": "text"
    },
    {
      "file": "attachments/e97cd2ba5c93/code/body-004.txt",
      "chars": 398,
      "language": "text"
    },
    {
      "file": "attachments/e97cd2ba5c93/code/body-005.txt",
      "chars": 727,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/res/hashnode/image/upload/v1726052576256/739822e4-4c43-48b4-b6dc-9e3cd4e69a74.png",
      "bytes": 209217,
      "reused": true,
      "local_path": "attachments/e97cd2ba5c93/images/000.png",
      "src": "https://cdn.hashnode.com/res/hashnode/image/upload/v1726052576256/739822e4-4c43-48b4-b6dc-9e3cd4e69a74.png",
      "alt": "5.Vector Stores: Efficient Storage and Retrieval for Embeddings",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/res/hashnode/image/upload/v1726048375360/b1ecdddd-a92d-404c-b53c-5ea75bea91d7.jpeg",
      "bytes": 60793,
      "reused": true,
      "local_path": "attachments/e97cd2ba5c93/images/002.jpeg",
      "src": "https://cdn.hashnode.com/res/hashnode/image/upload/v1726048375360/b1ecdddd-a92d-404c-b53c-5ea75bea91d7.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/res/hashnode/image/upload/v1726053337283/5b782452-221b-4013-8baf-7c0c3382be45.jpeg",
      "bytes": 148278,
      "reused": true,
      "local_path": "attachments/e97cd2ba5c93/images/003.jpeg",
      "src": "https://cdn.hashnode.com/res/hashnode/image/upload/v1726053337283/5b782452-221b-4013-8baf-7c0c3382be45.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/res/hashnode/image/upload/v1726048555050/7e126a4b-a566-4d15-ac5f-e9f0e1f2422d.png",
      "bytes": 43504,
      "reused": true,
      "local_path": "attachments/e97cd2ba5c93/images/004.png",
      "src": "https://cdn.hashnode.com/res/hashnode/image/upload/v1726048555050/7e126a4b-a566-4d15-ac5f-e9f0e1f2422d.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.botpenguin.com/assets/website/Screenshot_2024_03_13_at_1_45_10_PM_8c008710ae.webp",
      "bytes": 15338,
      "reused": true,
      "local_path": "attachments/e97cd2ba5c93/images/005.webp",
      "src": "https://cdn.botpenguin.com/assets/website/Screenshot_2024_03_13_at_1_45_10_PM_8c008710ae.webp",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/res/hashnode/image/upload/v1726107070445/b2453102-c411-411b-b4e1-c1288cbfa4d2.png",
      "bytes": 35528,
      "reused": true,
      "local_path": "attachments/e97cd2ba5c93/images/006.png",
      "src": "https://cdn.hashnode.com/res/hashnode/image/upload/v1726107070445/b2453102-c411-411b-b4e1-c1288cbfa4d2.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=9VgpXcfJYvw",
      "src": "https://www.youtube.com/watch?v=9VgpXcfJYvw",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/e97cd2ba5c93/videos/repair-9VgpXcfJYvw.mp4",
      "bytes": 250324143,
      "valid": true,
      "height": 720,
      "width": 1280,
      "duration": "3740.801000",
      "reused": true,
      "reused_from": "xinxing-zimeiti-corpus/attachments/9ea31de51baa/videos/repair-9VgpXcfJYvw.mp4"
    }
  ],
  "links": [
    {
      "href": "https://mfahadbashir.hashnode.dev/5vector-stores-efficient-storage-and-retrieval-for-embeddings.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@mfahadbashir",
      "label": "M",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/muhammad-fahad-bashir-09568216a/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://mfahadbashir.hashnode.dev/series/rag-series",
      "label": "Part of series Implementing RAG systems from Scratch in-depth",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=9VgpXcfJYvw",
      "label": "Image source",
      "type": "a"
    },
    {
      "href": "https://www.newsletter.theaiengineering.com/p/understanding-vector-databases-a",
      "label": "check out source",
      "type": "a"
    },
    {
      "href": "https://python.langchain.com/v0.2/docs/integrations/vectorstores/",
      "label": "Here is the link to view the vector store",
      "type": "a"
    },
    {
      "href": "https://qdrant.tech/",
      "label": "Qdrant",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/drive/16wsUBGBdHoLTFRLvZebUObdHeSb80dn-?usp=sharing",
      "label": "Click Here",
      "type": "a"
    },
    {
      "href": "https://mfahadbashir.hashnode.dev/tag/tutorial",
      "label": "# tutorial",
      "type": "a"
    },
    {
      "href": "https://mfahadbashir.hashnode.dev/tag/rag",
      "label": "# rag",
      "type": "a"
    },
    {
      "href": "https://mfahadbashir.hashnode.dev/tag/chatbot",
      "label": "# chatbot",
      "type": "a"
    },
    {
      "href": "https://mfahadbashir.hashnode.dev/tag/vector-database",
      "label": "# vector-database",
      "type": "a"
    },
    {
      "href": "https://mfahadbashir.hashnode.dev/tag/vector-embeddings",
      "label": "# vector-embeddings",
      "type": "a"
    },
    {
      "href": "https://mfahadbashir.hashnode.dev/tag/data-ingestion",
      "label": "# data-ingestion",
      "type": "a"
    },
    {
      "href": "https://mfahadbashir.hashnode.dev/tag/qdrant",
      "label": "# qdrant",
      "type": "a"
    },
    {
      "href": "https://mfahadbashir.hashnode.dev/tag/chromadb",
      "label": "# chromadb",
      "type": "a"
    },
    {
      "href": "https://mfahadbashir.hashnode.dev/tag/google-colab",
      "label": "# google-colab",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:08:37",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:46Z",
  "source_url": "https://mfahadbashir.hashnode.dev/5vector-stores-efficient-storage-and-retrieval-for-embeddings",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 9222,
  "expected_body_images": 6,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/e97cd2ba5c93/pages/source.html",
  "local_html": "attachments/e97cd2ba5c93/pages/article.html",
  "body_base_path": "attachments/e97cd2ba5c93/pages/",
  "body_markdown_path": "attachments/e97cd2ba5c93/pages/article.md"
}