{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Shawhin Talebi",
  "title_listed": "Multimodal RAG: Process Any File Type with AI",
  "keywords": "RAG,多模态",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "部分免费/会员墙",
  "url": "https://towardsdatascience.com/multimodal-rag-process-any-file-type-with-ai-e6921342c903/",
  "publish_date": "2024-12",
  "corpus_stage": "预训练/SFT",
  "notes": "",
  "id": "206b52e8efb0",
  "collected_at": "2026-10-01T22:22:12",
  "title": "Multimodal RAG: Process Any File Type with AI | Towards Data Science",
  "body_markdown": "Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\n# Multimodal RAG: Process Any File Type with AI\nA beginner-friendly guide with example (Python) code\n[Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\nDecember 5, 202412 min read\nThis is the third article in a [larger series](https://shawhin.medium.com/list/multimodal-ai-fe9521d0e77a) on multimodal AI. In the previous posts, we discussed [multimodal LLMs](https://towardsdatascience.com/multimodal-models-llms-that-can-see-and-hear-5c6737c981d3) and [embedding models](https://towardsdatascience.com/multimodal-embeddings-an-introduction-5dc36975966f), respectively. In this article, we will combine these ideas to enable the development of multimodal RAG systems. I'll start by reviewing key concepts and then share example code for implementing such a system.\n![Image from Canva.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/1KUfCT0odhqkTsPzf1ljx6A.png)\nImage from Canva.\n* * *\nLanguage models like GPT, LLaMA, and Claude learn a tremendous amount of world knowledge via their pre-training. This makes them powerful tools for solving custom problems and answering complex questions.\nHowever, **there is knowledge that even the most advanced language models are ignorant of**. This includes proprietary information within organizations, events that occurred after a model's pre-training data collection, and specialized knowledge that is not prevalent on the internet.\nAlthough this ignorance limits a model's out-of-the-box capabilities, there is **a popular technique to overcome these limitations** : retrieval augmented generation (or RAG for short).\n## **What is RAG?**\n**RAG** is an approach for **improving a model's response quality by dynamically providing the relevant context** for a given prompt. Here's an example of when this might be helpful.\nSay, I forgot the name of a Python library a colleague mentioned in yesterday's meeting. This isn't something ChatGPT can help me with because it does not know the meeting's contents.\nHowever, RAG could help with this by taking my question (e.g. \"What was the name of that Python library that Rachel mentioned in yesterday's meeting?\"), automatically pulling the meeting transcript, then providing my original query and the transcript to an LLM.\n![Basic design of a RAG system. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/1STVyqpJkhoKZWYdR-2-xqA.png)\nBasic design of a RAG system. Image by author.\n## **Multimodal RAG**\nAlthough improving LLMs with RAG unlocks several practical use cases, there are some situations where relevant information exists in non-text formats, e.g., images, videos, charts, and tables. In such cases, we can go one step further and build **multimodal RAG systems** , **AI systems capable of processing text and non-text data**.\nMultimodal RAG enables more sophisticated inferences beyond what is conveyed by text alone. For example, it could analyze someone's facial expressions and speech tonality to give a richer context to a meeting's transcription.\n![Basic design of a Multimodal RAG system. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/1mqRCTYThFcZmGcVtw6s1cw.png)\nBasic design of a Multimodal RAG system. Image by author.\n## **3 Levels of MRAG**\nWhile there are several ways to implement a multimodal RAG (MRAG) system, here I will focus on three basic strategies at increasing levels of sophistication.\n  1. Translate modalities to text.\n  2. Text-only retrieval + MLLM\n  3. Multimodal retrieval + MLLM\n\n\nThe following discussion assumes **you already have a basic understanding of**[**RAG**](https://towardsdatascience.com/how-to-improve-llms-with-rag-abdc132f76ac)**and multimodal models**. The following articles discussed these topics: RAG, [Multimodal LLMs](https://towardsdatascience.com/multimodal-models-llms-that-can-see-and-hear-5c6737c981d3), and [Multimodal Embeddings](https://towardsdatascience.com/multimodal-embeddings-an-introduction-5dc36975966f).\n### **Level 1: Translate modalities to text**\nA simple way to make a RAG system multimodal is by **translating new modalities to text before storing them in the knowledge base**. This could be as simple as converting meeting recordings into text transcripts, using an existing multimodal LLM (MLLM) to generate image captions, or converting tables to a readable text format (e.g., .csv or .json).\n![Visual overview of Level 1 of MRAG. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/17QqhRIlnU7TQsCMnVDb6KA.png)\nVisual overview of Level 1 of MRAG. Image by author.\nThe key upside of this approach is that it **requires minimal changes to an existing RAG system**. Additionally, by explicitly generating text representations of non-text modalities, one has better control over the features of the data to extract. For instance, captions of analytical figures may include both a description and key insights.\nOf course, the downside of this strategy is that the **model's responses cannot directly use non-textual data** , which means that the translation from, say, image to text can create a critical information bottleneck.\n### **Level 2: Text-only retrieval + MLLM**\nAnother approach is to generate text representations of all items in the knowledge base, e.g., descriptions and meta-tags, for retrieval, but to **pass the original modality to a multimodal LLM (MLLM)**. For example, image metadata is used for the retrieval step, and the associated image is passed to a model for inference.\n![Visual overview of Level 2 of MRAG. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/1JoUZLYezY3q95zngSmDJIA.png)\nVisual overview of Level 2 of MRAG. Image by author.\nThis maintains many of the benefits of Level 1 while mitigating its limitations. Namely, text features of items in the knowledge base can be optimized for search, but the downstream model can use the full richness of each item's original modality.\nThe key difference with this approach is that it requires an **MLLM** , which is **an LLM capable of processing non-text data**. This unlocks more advanced reasoning capabilities, as demonstrated by models like GPT-4o or LLaMA 3.2 Vision.\n### **Level 3: Multimodal retrieval + MLLM**\nAlthough we could use keyword-based search in the retrieval processes for Level 1 and Level 2, it is a common practice to use so-called **vector search**. This consists of **generating vector representations (i.e., embeddings)** of items in the knowledge base and then **performing a search by computing similarity scores** between an input query and each item in the knowledge base.\nTraditionally, this requires that the query and knowledge base items are text-based. However, as we saw in the [previous article](https://towardsdatascience.com/multimodal-embeddings-an-introduction-5dc36975966f) of this series, there exist **multimodal embedding models** that **generate aligned vector representations of both text and non-text data**.\nTherefore, we can use multimodal embeddings to perform multimodal retrieval. This works the same way as text-based vector search, but now the embedding space co-locates similar concepts independent of its original modality. The results of such a retrieval strategy can then be passed directly to a MLLM.\n![Visual overview of Level 3 of MRAG. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/1YwMdXXTGBMj9QSjAwkojdA.png)\nVisual overview of Level 3 of MRAG. Image by author.\n## **Example Code: Multimodal Blog Question-Answering Assistant**\nWith a basic understanding of how Multimodal RAG works, let's see how we can build such a system. Here, I will create a question-answering assistant that can access the text and figures from the previous two blogs in this series.\nThe Python code for this example is freely available at the [GitHub repo](https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/3-multimodal-rag).\n* * *\n### Imports & Data Loading\nWe start by importing a few handy libraries and modules.\npython\n\n```\nimport jsonfrom transformers import CLIPProcessor, CLIPTextModelWithProjectionfrom torch import load, matmul, argsortfrom torch.nn.functional import softmax\n```\n\nNext, we'll import text and image chunks from the [Multimodal LLMs](https://towardsdatascience.com/multimodal-models-llms-that-can-see-and-hear-5c6737c981d3) and [Multimodal Embeddings](https://towardsdatascience.com/multimodal-embeddings-an-introduction-5dc36975966f) blog posts. These are saved in .json files, which can be loaded into Python as a list of dictionaries.\npython\n\n```\n# load text chunkswith open('data/text_content.json', 'r', encoding='utf-8') as f:        text_content_list = json.load(f)# load imageswith open('data/image_content.json', 'r', encoding='utf-8') as f:        image_content_list = json.load(f)\n```\n\nWhile I won't review the data preparation process here, the code I used is on the [GitHub repo](https://github.com/ShawhinT/YouTube-Blog/blob/main/multimodal-ai/3-multimodal-rag/1-data_prep.ipynb).\nWe will also load the multimodal embeddings (from CLIP) for each item in _text_content _list_ and _image_content _list_. These are saved as pytorch tensors.\npython\n\n```\n# load embeddingstext_embeddings = load('data/text_embeddings.pt', weights_only=True)image_embeddings = load('data/image_embeddings.pt', weights_only=True)print(text_embeddings.shape)print(image_embeddings.shape)# >> torch.Size([86, 512])# >> torch.Size([17, 512])\n```\n\nPrinting the shape of these tensors, we see they are represented via 512-dimensional embeddings. And we have 86 text chunks and 17 images.\n### Multimodal Search\nWith our knowledge base loaded, we can now define a query for vector search. This will consist of translating an input query into an embedding using CLIP. We do this similarly to the examples from the [previous post](https://towardsdatascience.com/multimodal-embeddings-an-introduction-5dc36975966f).\npython\n\n```\n# queryquery = \"What is CLIP's contrastive loss function?\"# embed query (4 steps)# 1) load modelmodel = CLIPTextModelWithProjection.from_pretrained(\"openai/clip-vit-base-patch16\")# 2) load data processorprocessor = CLIPProcessor.from_pretrained(\"openai/clip-vit-base-patch16\")# 3) pre-process textinputs = processor(text=[text], return_tensors=\"pt\", padding=True)# 4) compute embeddings with CLIPoutputs = model(**inputs)# extract embeddingquery_embed = outputs.text_embedsprint(query_embed.shape)# >> torch.Size([1, 512])\n```\n\nPrinting the shape, we see we have a single vector representing the query.\nTo perform a vector search over the knowledge base, we need to do the following.\n  1. Compute similarities between the query embedding and all the text and image embeddings.\n  2. Rescale the similarities to range from 0 to 1 via the softmax function.\n  3. Sort the scaled similarities and return the top k results.\n  4. Finally, filter the results to only keep items above a pre-defined similarity threshold.\n\n\nHere's what that looks like in code for the text chunks.\npython\n\n```\n# define k and simiarlity thresholdk = 5threshold = 0.05# multimodal search over articlestext_similarities = matmul(query_embed, text_embeddings.T)# rescale similarities via softmaxtemp=0.25text_scores = softmax(text_similarities/temp, dim=1)# return top k filtered text resultsisorted_scores = argsort(text_scores, descending=True)[0]sorted_scores = text_scores[0][isorted_scores]itop_k_filtered = [idx.item()                     for idx, score in zip(isorted_scores, sorted_scores)                     if score.item() >= threshold][:k]top_k = [text_content_list[i] for i in itop_k_filtered]print(top_k)\n```\n\ntext\n\n```\n# top k results[{'article_title': 'Multimodal Embeddings: An Introduction',  'section': 'Contrastive Learning',  'text': 'Two key aspects of CL contribute to its effectiveness'}]\n```\n\nAbove, we see the top text results. Notice we only have one item, even though _k_ =5. This is because the 2nd-5th items were below the 0.1 threshold.\nInterestingly, this item doesn't seem helpful to our initial query of _\"What is CLIP's contrastive loss function?\"_ This highlights **one of the key challenges of vector search** : _items similar to a given query may not necessarily help answer it_.\nOne way we can mitigate this issue is having less stringent restrictions on our search results by increasing _k_ and lowering the similarity _threshold_ , then hoping the LLM can work out what's helpful vs. not.\nTo do this, I'll first package the vector search steps into a Python function.\npython\n\n```\ndef similarity_search(query_embed, target_embeddings, content_list,                       k=5, threshold=0.05, temperature=0.5):    \"\"\"       Perform similarity search over embeddings and return top k results.    \"\"\"    # Calculate similarities    similarities = torch.matmul(query_embed, target_embeddings.T)    # Rescale similarities via softmax    scores = torch.nn.functional.softmax(similarities/temperature, dim=1)    # Get sorted indices and scores    sorted_indices = scores.argsort(descending=True)[0]    sorted_scores = scores[0][sorted_indices]    # Filter by threshold and get top k    filtered_indices = [        idx.item() for idx, score in zip(sorted_indices, sorted_scores)         if score.item() >= threshold    ][:k]    # Get corresponding content items and scores    top_results = [content_list[i] for i in filtered_indices]    result_scores = [scores[0][i].item() for i in filtered_indices]    return top_results, result_scores\n```\n\nThen, set more inclusive search parameters.\npython\n\n```\n# search over text chunkstext_results, text_scores = similarity_search(query_embed, text_embeddings,                     text_content_list, k=15, threshold=0.01, temperature=0.25)# search over imagesimage_results, image_scores = similarity_search(query_embed, image_embeddings,                     image_content_list, k=5, threshold=0.25, temperature=0.5)\n```\n\nThis results in 15 text results and 1 image result.\ntext\n\n```\n1 - Two key aspects of CL contribute to its effectiveness2 - To make a class prediction, we must extract the image logits and evaluate which class corresponds to the maximum.3 - Next, we can import a version of the clip model and its associated data processor. Note: the processor handles tokenizing input text and image preparation.4 - The basic idea behind using CLIP for 0-shot image classification is to pass an image into the model along with a set of possible class labels. Then, a classification can be made by evaluating which text input is most similar to the input image.5 - We can then match the best image to the input text by extracting the text logits and evaluating the image corresponding to the maximum.6 - The code for these examples is freely available on the GitHub repository.7 - We see that (again) the model nailed this simple example. But let's try some trickier examples.8 - Next, we'll preprocess the image/text inputs and pass them into the model.9 - Another practical application of models like CLIP is multimodal RAG, which consists of the automated retrieval of multimodal context to an LLM. In the next article of this series, we will see how this works under the hood and review a concrete example.10 - Another application of CLIP is essentially the inverse of Use Case 1. Rather than identifying which text label matches an input image, we can evaluate which image (in a set) best matches a text input (i.e. query)—in other words, performing a search over images.11 - This has sparked efforts toward expanding LLM functionality to include multiple modalities.12 - GPT-4o - Input: text, images, and audio. Output: text.FLUX - Input: text. Output: images.Suno - Input: text. Output: audio.13 - The standard approach to aligning disparate embedding spaces is contrastive learning (CL). A key intuition of CL is to represent different views of the same information similarly [5].14 - While the model is less confident about this prediction with a 54.64% probability, it correctly implies that the image is not a meme.15 - [8] Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities\n```\n\n![Image search result.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/1rq89PAcqQ_lHgkYhkf5T4g.png)\nImage search result.\n### Prompting MLLM\nAlthough most of these text item results do not seem helpful to our query, the image result is exactly what we're looking for. Nevertheless, given these search results, let's see how LLaMA 3.2 Vision responds to this query.\nWe first will structure the search results as well-formatted strings.\npython\n\n```\ntext_context = \"\"for text in text_results:    if text_results:        text_context = text_context + \"**Article title:** \"                                               + text['article_title'] + \"n\"        text_context = text_context + \"**Section:**  \"                                               + text['section'] + \"n\"        text_context = text_context + \"**Snippet:** \"                                               + text['text'] + \"nn\"\n```\n\npython\n\n```\nimage_context = \"\"for image in image_results:    if image_results:        image_context = image_context + \"**Article title:** \"                                           + image['article_title'] + \"n\"        image_context = image_context + \"**Section:**  \"                                           + image['section'] + \"n\"        image_context = image_context + \"**Image Path:**  \"                                           + image['image_path'] + \"n\"        image_context = image_context + \"**Image Caption:** \"                                           + image['caption'] + \"nn\"\n```\n\nNote the metadata that accompanies each text and image item. This will help the LLaMA better understand the context of the content.\nNext, we interleave the text and image results in a prompt.\npython\n\n```\n# construct prompt templateprompt = f\"\"\"Given the query \"{query}\" and the following relevant snippets:{text_context}{image_context}Please provide a concise and accurate answer to the query, incorporating relevant information from the provided snippets where possible.\"\"\"\n```\n\nThe final prompt is quite long, so I won't print it here. However, it is fully displayed in the [example notebook](https://github.com/ShawhinT/YouTube-Blog/blob/main/multimodal-ai/3-multimodal-rag/2-mrag_example.ipynb) on GitHub.\nFinally, we can use [ollama](https://ollama.com/) to pass this prompt to LLaMA 3.2 Vision.\npython\n\n```\nollama.pull('llama3.2-vision')response = ollama.chat(    model='llama3.2-vision',    messages=[{        'role': 'user',        'content': prompt,        'images': [image[\"image_path\"] for image in image_results]    }])print(response['message']['content'])\n```\n\ntext\n\n```\nThe image depicts a contrastive loss function for aligning text and image representations in multimodal models. The function is designed to minimize the difference between the similarity of positive pairs (text-image) and negative pairs (text-text or image-image). This loss function is commonly used in CLIP, which stands for Contrastive Language-Image Pre-training.**Key Components:***   **Positive Pairs:** Text-image pairs where the text describes an image.*   **Negative Pairs:** Text-text or image-image pairs that do not belong to the same class.*   **Contrastive Loss Function:** Calculates the difference between positive and negative pairs' similarities.**How it Works:**1.  **Text-Image Embeddings:** Generate embeddings for both text and images using a multimodal encoder (e.g., CLIP).2.  **Positive Pair Similarity:** Calculate the similarity score between each text-image pair.3.  **Negative Pair Similarity:** Calculate the similarity scores between all negative pairs.4.  **Contrastive Loss Calculation:** Compute the contrastive loss by minimizing the difference between positive and negative pairs' similarities.**Benefits:***   **Multimodal Alignment:** Aligns text and image representations for better understanding of visual content from text descriptions.*   **Improved Performance:** Enhances performance in downstream tasks like image classification, retrieval, and generation.\n```\n\nThe model correctly picks up that the image contains the information it needs and explains the general intuition of how it works. However, it **misunderstands the meaning of positive and negative pairs** , thinking that a negative pair corresponds to a pair of the same modality.\nWhile we went through the implementation details step-by-step, I packaged everything into a nice UI using Gradio in this [notebook](https://github.com/ShawhinT/YouTube-Blog/blob/main/multimodal-ai/3-multimodal-rag/3-mrag_UI.ipynb) on the GitHub repo.\n> [**YouTube-Blog/multimodal-ai/3-multimodal-rag at main · ShawhinT/YouTube-Blog**](https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/3-multimodal-rag)\n## Conclusion\nMultimodal RAG systems can synthesize knowledge stored in a variety of formats, expanding what's possible with AI. Here, we reviewed 3 simple strategies for developing such a system and then saw an example implementation of a multimodal blog QA assistant.\nAlthough the example worked well enough for this demonstration, there are clear limitations to the search process. A few techniques that may improve this include using a **reranker to refine similarity search** results and to improve search quality via **fine-tuned multimodal embeddings**.\nIf you want to see future posts on these topics, let me know in the comments :)\n**More on Multimodal models 👇**\n> [**Multimodal AI**](https://shawhin.medium.com/list/fe9521d0e77a)\n* * *\n**My website** : <https://www.shawhintalebi.com/>\n[1] [RAG](https://towardsdatascience.com/how-to-improve-llms-with-rag-abdc132f76ac)\n[2] [Multimodal LLMs](https://towardsdatascience.com/multimodal-models-llms-that-can-see-and-hear-5c6737c981d3)\n[3] [Multimodal Embeddings](https://towardsdatascience.com/multimodal-embeddings-an-introduction-5dc36975966f)\n* * *\n![](https://assets.insightmediagroup.io/media/1787657074748_utev3i.jpeg)\nWritten By\n## [Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[See all from Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[AI](https://towardsdatascience.com/tag/ai/)[Hands On Tutorials](https://towardsdatascience.com/tag/hands-on-tutorials/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[multimodal-rag](https://towardsdatascience.com/tag/multimodal-rag/)[Python](https://towardsdatascience.com/tag/python/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-rag-process-any-file-type-with-ai-e6921342c903%2F&title=Multimodal%20RAG%3A%20Process%20Any%20File%20Type%20with%20AI)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-rag-process-any-file-type-with-ai-e6921342c903%2F&title=Multimodal%20RAG%3A%20Process%20Any%20File%20Type%20with%20AI)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-rag-process-any-file-type-with-ai-e6921342c903%2F&text=Multimodal%20RAG%3A%20Process%20Any%20File%20Type%20with%20AI)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/0nEISi8WUO4lq7zj7-scaled.jpg)](https://towardsdatascience.com/unconventional-datetime-transformation-made-easy-using-pythons-pandas-library-aa9a8c65d885/)\n## [Unconventional Datetime Transformation Made Easy Using Python's Pandas Library](https://towardsdatascience.com/unconventional-datetime-transformation-made-easy-using-pythons-pandas-library-aa9a8c65d885/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nExplained with a real-life example\n[Jin Cui](https://towardsdatascience.com/author/jin-cui/)\nJuly 25, 20235 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/09/1GNDPa8qBVEs1LI6348OfMQ-scaled.jpeg)](https://towardsdatascience.com/data-scientists-think-like-biologists-b681a9795627/)\n## [Data Scientists: Think like Biologists!](https://towardsdatascience.com/data-scientists-think-like-biologists-b681a9795627/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nA Checklist for Biological Machine Learning\n[Joseph Bloom](https://towardsdatascience.com/author/joseph-bloom/)\nSeptember 18, 20208 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1jNLph0vO8g5H6WCfWBnsPg-scaled.jpeg)](https://towardsdatascience.com/how-to-extend-pandas-dataframes-with-custom-methods-to-supercharge-code-functionality-readability-f4ae983ad715/)\n## [How to Extend Pandas DataFrames with Custom Methods to Supercharge Code Functionality & Readability](https://towardsdatascience.com/how-to-extend-pandas-dataframes-with-custom-methods-to-supercharge-code-functionality-readability-f4ae983ad715/)\n[Coding](https://towardsdatascience.com/category/programming/coding/)\nA step-by-step guide to extending pandas DataFrames with custom methods including full examples for implementing probability and expectancy\n[Graham Harrison](https://towardsdatascience.com/author/grahamharrison-86487/)\nOctober 10, 20236 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/0kUAWW_qq5OSP877v-scaled.jpg)](https://towardsdatascience.com/python-6-coding-hygiene-tips-that-helped-me-got-promoted-c70b4380f1c7/)\n## [Python: 6 coding hygiene tips that helped me get promoted.](https://towardsdatascience.com/python-6-coding-hygiene-tips-that-helped-me-got-promoted-c70b4380f1c7/)\n[Programming](https://towardsdatascience.com/category/programming/)\nIn this article, my top 6 obvious but not always applied coding tips that I wish someone forced me to follow 3 years ago and that I will...\n[Sajid Lhessani](https://towardsdatascience.com/author/lhessani-sajid/)\nDecember 3, 20205 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/08/0Sz4zTjoQIjWY1OvO-scaled.jpg)](https://towardsdatascience.com/here-are-30-ways-that-will-make-you-a-pro-at-creating-numpy-arrays-932b77d9a1eb/)\n## [Here Are 30 Ways That Will Make You a Pro at Creating NumPy Arrays](https://towardsdatascience.com/here-are-30-ways-that-will-make-you-a-pro-at-creating-numpy-arrays-932b77d9a1eb/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA comprehensive guide to creating NumPy arrays\n[Avi Chawla](https://towardsdatascience.com/author/avi_chawla/)\nAugust 31, 20227 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/11/0EdW1YRwPSi4skGYV.png)](https://towardsdatascience.com/transformers-numeracy-pt-1-calculators-for-natural-language-e09a471c5ebe/)\n## [NLP & Numeracy (Pt 1) - Calculators for Natural Language](https://towardsdatascience.com/transformers-numeracy-pt-1-calculators-for-natural-language-e09a471c5ebe/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nWhat is numeracy in NLP and why it matters?\n[Misho Dungarov](https://towardsdatascience.com/author/mihail.dungarov/)\nNovember 22, 202013 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/06/1d_6Y2uTmnRbinHgzfiXVDg-scaled.jpeg)](https://towardsdatascience.com/flux-jl-on-mnist-variations-of-a-theme-c3cd7a949f8c/)\n## [Flux.jl on MNIST - Variations of a theme](https://towardsdatascience.com/flux-jl-on-mnist-variations-of-a-theme-c3cd7a949f8c/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nFlux.jl is a ML-stack offering lightweight components to create models and train them. Using the MNIST dataset we will see how easy it is...\n[Roland Schätzle](https://towardsdatascience.com/author/schaetzle.ka/)\nJune 23, 202212 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/01/0jvjJv2FfHKm_Ygti-scaled.jpg)](https://towardsdatascience.com/speaking-probes-self-interpreting-models-7a3dc6cb33d6/)\n## [Speaking Probes: Self-Interpreting Models?](https://towardsdatascience.com/speaking-probes-self-interpreting-models-7a3dc6cb33d6/)\n[NLP](https://towardsdatascience.com/category/artificial-intelligence/nlp/)\nCan language models aid in their interpretation?\n[Guy Dar](https://towardsdatascience.com/author/guydar/)\nJanuary 16, 202317 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/0lt-ZB2ZTQJqlYrYT.png)](https://towardsdatascience.com/how-to-make-real-time-machine-learning-work-for-user-journeys-5d41fa073005/)\n## [How to Make Real-Time Machine Learning Work for User Journeys](https://towardsdatascience.com/how-to-make-real-time-machine-learning-work-for-user-journeys-5d41fa073005/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nWhy combining in-session and historical behavior is the right approach for B2C organizations\n[Michael Firn](https://towardsdatascience.com/author/vidora/)\nAugust 25, 20217 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n",
  "metadata": {
    "title": "Multimodal RAG: Process Any File Type with AI | Towards Data Science",
    "description": "A beginner-friendly guide with example (Python) code",
    "keywords": null,
    "author": "Shaw Talebi",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Multimodal RAG: Process Any File Type with AI",
    "og:description": "A beginner-friendly guide with example (Python) code",
    "og:url": "https://towardsdatascience.com/multimodal-rag-process-any-file-type-with-ai-e6921342c903/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/1KUfCT0odhqkTsPzf1ljx6A.png",
    "og:image:width": "1280",
    "og:image:height": "720",
    "og:image:type": "image/png",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Shaw Talebi",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "12 min read",
    "article:published_time": "2024-12-05T20:19:55.000Z",
    "article:modified_time": "2024-12-05T20:19:55.000Z",
    "article:author": "https://towardsdatascience.com/author/shawhin/"
  },
  "paywall": {
    "listed_paid": true,
    "detected": false,
    "marker": "",
    "public_only": true,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/206b52e8efb0/code/block_000.txt",
      "chars": 157
    },
    {
      "language": "txt",
      "file": "attachments/206b52e8efb0/code/block_001.txt",
      "chars": 242
    },
    {
      "language": "txt",
      "file": "attachments/206b52e8efb0/code/block_002.txt",
      "chars": 265
    },
    {
      "language": "txt",
      "file": "attachments/206b52e8efb0/code/block_003.txt",
      "chars": 523
    },
    {
      "language": "txt",
      "file": "attachments/206b52e8efb0/code/block_004.txt",
      "chars": 605
    },
    {
      "language": "txt",
      "file": "attachments/206b52e8efb0/code/block_005.txt",
      "chars": 179
    },
    {
      "language": "txt",
      "file": "attachments/206b52e8efb0/code/block_006.txt",
      "chars": 949
    },
    {
      "language": "txt",
      "file": "attachments/206b52e8efb0/code/block_007.txt",
      "chars": 356
    },
    {
      "language": "txt",
      "file": "attachments/206b52e8efb0/code/block_008.txt",
      "chars": 2138
    },
    {
      "language": "txt",
      "file": "attachments/206b52e8efb0/code/block_009.txt",
      "chars": 443
    },
    {
      "language": "txt",
      "file": "attachments/206b52e8efb0/code/block_010.txt",
      "chars": 582
    },
    {
      "language": "txt",
      "file": "attachments/206b52e8efb0/code/block_011.txt",
      "chars": 271
    },
    {
      "language": "txt",
      "file": "attachments/206b52e8efb0/code/block_012.txt",
      "chars": 256
    },
    {
      "language": "txt",
      "file": "attachments/206b52e8efb0/code/block_013.txt",
      "chars": 1402
    }
  ],
  "images": [
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/1KUfCT0odhqkTsPzf1ljx6A.png",
      "alt": "Image from Canva.",
      "local_path": "attachments/206b52e8efb0/images/000.png",
      "bytes": 1298146,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/1STVyqpJkhoKZWYdR-2-xqA.png",
      "alt": "Basic design of a RAG system. Image by author.",
      "local_path": "attachments/206b52e8efb0/images/001.png",
      "bytes": 50595,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/1mqRCTYThFcZmGcVtw6s1cw.png",
      "alt": "Basic design of a Multimodal RAG system. Image by author.",
      "local_path": "attachments/206b52e8efb0/images/002.png",
      "bytes": 59543,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/17QqhRIlnU7TQsCMnVDb6KA.png",
      "alt": "Visual overview of Level 1 of MRAG. Image by author.",
      "local_path": "attachments/206b52e8efb0/images/003.png",
      "bytes": 35293,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/1JoUZLYezY3q95zngSmDJIA.png",
      "alt": "Visual overview of Level 2 of MRAG. Image by author.",
      "local_path": "attachments/206b52e8efb0/images/004.png",
      "bytes": 36658,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/1YwMdXXTGBMj9QSjAwkojdA.png",
      "alt": "Visual overview of Level 3 of MRAG. Image by author.",
      "local_path": "attachments/206b52e8efb0/images/005.png",
      "bytes": 36577,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/12/1rq89PAcqQ_lHgkYhkf5T4g.png",
      "alt": "Image search result.",
      "local_path": "attachments/206b52e8efb0/images/006.png",
      "bytes": 54384,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/09/1GNDPa8qBVEs1LI6348OfMQ-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/206b52e8efb0/images/007.jpeg",
      "bytes": 619422,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1jNLph0vO8g5H6WCfWBnsPg-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/206b52e8efb0/images/008.jpeg",
      "bytes": 267812,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/0kUAWW_qq5OSP877v-scaled.jpg",
      "alt": "",
      "local_path": "attachments/206b52e8efb0/images/009.jpg",
      "bytes": 262209,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/11/0EdW1YRwPSi4skGYV.png",
      "alt": "",
      "local_path": "attachments/206b52e8efb0/images/010.png",
      "bytes": 552457,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/06/1d_6Y2uTmnRbinHgzfiXVDg-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/206b52e8efb0/images/011.jpeg",
      "bytes": 369460,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/01/0jvjJv2FfHKm_Ygti-scaled.jpg",
      "alt": "",
      "local_path": "attachments/206b52e8efb0/images/012.jpg",
      "bytes": 520773,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/0lt-ZB2ZTQJqlYrYT.png",
      "alt": "",
      "local_path": "attachments/206b52e8efb0/images/013.png",
      "bytes": 1023877,
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "src": "https://www.youtube.com/c/TowardsDataScience",
      "quality": "unavailable",
      "local_path": "",
      "note": "无1080p且无720p，未交付更低清晰度"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/shawhin/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/multimodal-models-llms-that-can-see-and-hear-5c6737c981d3"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/multimodal-embeddings-an-introduction-5dc36975966f"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/how-to-improve-llms-with-rag-abdc132f76ac"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/ai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/hands-on-tutorials/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/multimodal-rag/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/python/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/unconventional-datetime-transformation-made-easy-using-pythons-pandas-library-aa9a8c65d885/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/data-science/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/jin-cui/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/data-scientists-think-like-biologists-b681a9795627/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://shawhin.medium.com/list/multimodal-ai-fe9521d0e77a"
    },
    {
      "type": "external",
      "href": "https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/3-multimodal-rag"
    },
    {
      "type": "external",
      "href": "https://github.com/ShawhinT/YouTube-Blog/blob/main/multimodal-ai/3-multimodal-rag/1-data_prep.ipynb"
    },
    {
      "type": "external",
      "href": "https://github.com/ShawhinT/YouTube-Blog/blob/main/multimodal-ai/3-multimodal-rag/2-mrag_example.ipynb"
    },
    {
      "type": "external",
      "href": "https://ollama.com/"
    },
    {
      "type": "external",
      "href": "https://github.com/ShawhinT/YouTube-Blog/blob/main/multimodal-ai/3-multimodal-rag/3-mrag_UI.ipynb"
    },
    {
      "type": "external",
      "href": "https://shawhin.medium.com/list/fe9521d0e77a"
    },
    {
      "type": "external",
      "href": "https://www.shawhintalebi.com/"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=Multimodal+RAG%3A+Process+Any+File+Type+with+AI&u=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-rag-process-any-file-type-with-ai-e6921342c903%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=Multimodal+RAG%3A+Process+Any+File+Type+with+AI&url=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-rag-process-any-file-type-with-ai-e6921342c903%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=Multimodal+RAG%3A+Process+Any+File+Type+with+AI&url=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-rag-process-any-file-type-with-ai-e6921342c903%2F"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/c/TowardsDataScience"
    },
    {
      "type": "external",
      "href": "https://www.threads.net/@towardsdatascience"
    },
    {
      "type": "external",
      "href": "https://bsky.app/profile/towardsdatascience.com"
    },
    {
      "type": "external",
      "href": "https://thenewstack.io"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh"
    },
    {
      "type": "external",
      "href": "https://insightmediagroup.io/creator-network"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T11:11:49Z",
  "source_url": "https://towardsdatascience.com/multimodal-rag-process-any-file-type-with-ai-e6921342c903/",
  "delivery_status": "excluded_paid",
  "discovery_status": "excluded_paid",
  "missing_reason": "按原约定排除付费/会员全文"
}