{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Shawhin Talebi",
  "title_listed": "Multimodal Embeddings: An Introduction",
  "keywords": "多模态,embedding",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "部分免费/会员墙",
  "url": "https://towardsdatascience.com/multimodal-embeddings-an-introduction-5dc36975966f/",
  "publish_date": "2024-11",
  "corpus_stage": "预训练/SFT",
  "notes": "",
  "id": "5c504a56c5b4",
  "collected_at": "2026-10-01T22:22:12",
  "title": "Multimodal Embeddings: An Introduction | Towards Data Science",
  "body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Data Science](https://towardsdatascience.com/category/data-science/)\n# Multimodal Embeddings: An Introduction\nMapping text and images into a common space\n[Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\nNovember 29, 20247 min read\nThis is the 2nd article in a [larger series](https://shawhin.medium.com/list/multimodal-ai-fe9521d0e77a) on multimodal AI. In the [previous post](https://towardsdatascience.com/multimodal-models-llms-that-can-see-and-hear-5c6737c981d3), we saw how to augment [large language models (LLMs)](https://shawhin.medium.com/list/large-language-models-llms-8e009ae3054c) to understand new data modalities (e.g., images, audio, video). One such approach relied on encoders that generate vector representations (i.e. embeddings) of non-text data. In this article, I will discuss _multimodal_ embeddings and share what they can do via two practical use cases.\n![Image from Canva.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1a6BF-kEeo8rd7OW2a3JYGA.png)\nImage from Canva.\n* * *\nAI research is traditionally split into distinct fields: NLP, computer vision (CV), robotics, human-computer interface (HCI), etc. However, countless practical tasks require the **integration of these different research areas** e.g. autonomous vehicles (CV + robotics), AI agents (NLP + CV + HCI), personalized learning (NLP + HCI), etc.\nAlthough these fields aim to solve different problems and work with different data types, they all share a fundamental process. Namely, **generating useful numerical representations of real-world phenomena**.\nHistorically, this was done by hand. This means that researchers and practitioners would use their (or other people's) expertise to explicitly transform data into a more helpful form. Today, however, _these can be derived another way_.\n## **Embeddings**\n**Embeddings** are **(useful) numerical representations of data learned implicitly through model training**. For example, through learning how to predict text, BERT learned representations of text, which are helpful for many NLP tasks [1]. Another example is the Vision Transformer (ViT), trained for image classification on Image Net, which can be repurposed for other applications [2].\nA key point here is that these learned embedding spaces will have some underlying structure so that **similar concepts are located close together**. As shown in the toy examples below.\n![Toy represetation of text and image embeddings, respectively. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1jpmC6Kx7DxVeikEr15vooA.png)\nToy represetation of text and image embeddings, respectively. Image by author.\nOne **key limitation** of the previously mentioned models is they are restricted to a single data modality, e.g., text or images. Preventing cross-modal applications like image captioning, content moderation, image search, and more. _But what if we could merge these two representations?_\n## **Multimodal Embeddings**\nAlthough text and images may look very different to us, in a neural network, these are **represented via the same mathematical object** , i.e., a vector. Therefore, in principle, text, images, or any other data modality can processed by a single model.\nThis fact underlies **multimodal embeddings** , which **represent multiple data modalities in the same vector space** such that similar concepts are co-located (independent of their original representations).\n![Toy representation of multimodal embedding space. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/15d3HBNjNIXLy0oMIvJjxWw.png)\nToy representation of multimodal embedding space. Image by author.\nFor example, CLIP encodes text and images into a shared embedding space [3]. A key insight from CLIP is that by aligning text and image representations, the **model is capable of 0-shot image classification on an arbitrary set of target classes** since any input text can be treated as a class label (we will see a concrete example of this later).\nHowever, this idea is not limited to text and images. Virtually any data modalities can be aligned in this way e.g., text-audio, audio-image, text-EEG, image-tabular, and text-video. Unlocking use cases such as video captioning, advanced OCR, audio transcription, video search, and EEG-to-text [4].\n## **Contrastive Learning**\nThe standard approach to aligning disparate embedding spaces is **contrastive learning (CL)**. A key intuition of CL is to **represent different views of the same** _**information**_**similarly** [5].\nThis consists of learning representations that **maximize the similarity between positive pairs** and **minimize the similarity of negative pairs**. In the case of an image-text model, a positive pair might be an image with an appropriate caption, while a negative pair would be an image with an irrelevant caption (as shown below).\n![Example positive and negative pairs used in contrastive training. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1AGHBVjzwjXapJSe4aUPrjg.png)\nExample positive and negative pairs used in contrastive training. Image by author.\n**Two key aspects** **of CL** contribute to its effectiveness\n  1. Since positive and negative pairs can be curated from the data's inherent structure (e.g., metadata from web images), CL training data **do not require manual labeling** , which unlocks larger-scale training and more powerful representations [3].\n  2. It simultaneously maximizes positive and minimizes negative pair similarity via a special loss function, as demonstrated by CLIP [3].\n\n\n![CLIP&#039;s contrastive loss for text-image representation alignment \\[3\\]. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/12X1aT8fzFsgbqn23zXmmAA.png)\nCLIP's contrastive loss for text-image representation alignment [3]. Image by author.\n##  **Example Code:** Using CLIP for 0-shot classification and image search\nWith a high-level understanding of how multimodal embeddings work, let's see two concrete examples of what they can do. Here, I will use the open-source [CLIP model](https://huggingface.co/openai/clip-vit-base-patch16) to perform two tasks: 0-shot image classification and image search.\nThe **code for these examples** is freely available on the [GitHub repository](https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/2-mm-embeddings).\n* * *\n### Use case 1: 0-shot Image Classification\nThe basic idea behind using CLIP for 0-shot image classification is to pass an image into the model along with a set of possible class labels. Then, a classification can be made by **evaluating which text input is most similar to the input image**.\nWe'll start by importing the [Hugging Face Transformers library](https://huggingface.co/docs/transformers/en/installation) so that the CLIP model can be downloaded locally. Additionally, the PIL library is used to load images in Python.\npython\n\n```\nfrom transformers import CLIPProcessor, CLIPModel\nfrom PIL import Image\n```\n\nNext, we can import a version of the clip model and its associated data processor. _Note: the processor handles tokenizing input text and image preparation._\nini\n\n```\n# import model\nmodel = CLIPModel.from_pretrained(\"openai/clip-vit-base-patch16\")\n\n# import processor (handles text tokenization and image preprocessing)\nprocessor = CLIPProcessor.from_pretrained(\"openai/clip-vit-base-patch16\") \n```\n\nWe load in the below image of a cat and create a list of two possible class labels: \"_a photo of a cat_ \" or \"_a photo of a dog_ \".\nini\n\n```\n# load image\nimage = Image.open(\"images/cat_cute.png\")\n\n# define text classes\ntext_classes = [\"a photo of a cat\", \"a photo of a dog\"]\n```\n\n![Input cat photo. Image from Canva.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1Nzo536sqahqm1Q24Ms2vmA.png)\nInput cat photo. Image from Canva.\nNext, we'll preprocess the image/text inputs and pass them into the model.\nini\n\n```\n# pass image and text classes to processor\ninputs = processor(text=text_classes, images=image, return_tensors=\"pt\", \n                                                    padding=True)\n\n# pass inputs to CLIP\noutputs = model(**inputs) # note: \"**\" unpacks dictionary items\n```\n\nTo make a class prediction, we must extract the image logits and evaluate which class corresponds to the maximum.\nmakefile\n\n```\n# image-text similarity score\nlogits_per_image = outputs.logits_per_image \n# convert scores to probs via softmax\nprobs = logits_per_image.softmax(dim=1) \n\n# print prediction\npredicted_class = text_classes[probs.argmax()]\nprint(predicted_class, \"| Probability = \", \n                       round(float(probs[0][probs.argmax()]),4))\n```\n\ntext\n\n```\n>> a photo of a cat | Probability =  0.9979\n```\n\nThe model nailed it with a 99.79% probability that it's a cat photo. However, this was a super easy one. Let's see what happens when we change the class labels to: \"_ugly cat_ \" and \"_cute cat_ \" for the same image.\ntext\n\n```\n>> cute cat | Probability =  0.9703\n```\n\nThe model easily identified that the image was indeed a cute cat. Let's do something more challenging like the labels: \"_cat meme_ \" or \"_not cat meme_ \".\ntext\n\n```\n>> not cat meme | Probability =  0.5464\n```\n\nWhile the model is less confident about this prediction with a 54.64% probability, it correctly implies that the image is not a meme.\n### Use case 2: Image Search\nAnother application of CLIP is essentially the inverse of Use Case 1. Rather than identifying which text label matches an input image, we can evaluate **which image (in a set) best matches a text input (i.e. query)** —in other words, performing a search over images.\nWe start by storing a set of images in a list. Here, I have three images of a cat, dog, and goat, respectively.\npython\n\n```\n# create list of images to search over\nimage_name_list = [\"images/cat_cute.png\", \"images/dog.png\", \"images/goat.png\"]\n\nimage_list = []\nfor image_name in image_name_list:\n    image_list.append(Image.open(image_name))\n```\n\nNext, we can define a query like \"_a cute dog_ \" and pass it and the images into CLIP.\npython\n\n```\n# define a query\nquery = \"a cute dog\"\n\n# pass images and query to CLIP\ninputs = processor(text=query, images=image_list, return_tensors=\"pt\", \n                                                  padding=True)\n```\n\nWe can then match the best image to the input text by extracting the text logits and evaluating the image corresponding to the maximum.\npython\n\n```\n# compute logits and probabilities\noutputs = model(**inputs)\nlogits_per_text = outputs.logits_per_text\nprobs = logits_per_text.softmax(dim=1)\n\n# print best match\nbest_match = image_list[probs.argmax()]\nprob_match = round(float(probs[0][probs.argmax()]),4)\n\nprint(\"Match probability: \",prob_match)\ndisplay(best_match)\n```\n\ntext\n\n```\n>> Match probability:  0.9817\n```\n\n![Best match for query &quot;a cute dog&quot;. Image from Canva.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/14wnqr5p_7N3QD5EkXIQeew.png)\nBest match for query \"a cute dog\". Image from Canva.\nWe see that (again) the model nailed this simple example. But let's try some trickier examples.\npython\n\n```\nquery = \"something cute but metal 🤘\"\n```\n\ntext\n\n```\n>> Match probability:  0.7715\n```\n\n![Best match for query &quot;something cute but metal 🤘&quot;. Image from Canva.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1tIY3_ONQQT_cracAPWm8NQ.png)\nBest match for query \"something cute but metal 🤘\". Image from Canva.\npython\n\n```\nquery = \"a good boy\"\n```\n\ntext\n\n```\n>> Match probability:  0.8248\n```\n\n![Best match for query &quot;a good boy&quot;. Image from Canva.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/14wnqr5p_7N3QD5EkXIQeew.png)\nBest match for query \"a good boy\". Image from Canva.\npython\n\n```\nquery = \"the best pet in the world\"\n```\n\ntext\n\n```\n>> Match probability:  0.5664\n```\n\n![Best match for query &quot;the best pet in the world&quot;. Image from Canva.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1Nzo536sqahqm1Q24Ms2vmA.png)\nBest match for query \"the best pet in the world\". Image from Canva.\nAlthough this last prediction is quite controversial, all the other matches were spot on! This is likely since images like these are ubiquitous on the internet and thus were seen many times in CLIP's pre-training.\n> [**YouTube-Blog/multimodal-ai/2-mm-embeddings at main · ShawhinT/YouTube-Blog**](https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/2-mm-embeddings)\n## What's Next?\nMultimodal embeddings unlock countless AI use cases that involve multiple data modalities. Here, we saw two such use cases, i.e., 0-shot image classification and image search using CLIP.\nAnother practical application of models like CLIP is multimodal RAG, which consists of the automated retrieval of multimodal context to an LLM. In the [next article](https://medium.com/towards-data-science/multimodal-rag-process-any-file-type-with-ai-e6921342c903) of this [series](https://shawhin.medium.com/list/multimodal-ai-fe9521d0e77a), we will see how this works under the hood and review a concrete example.\n**More on Multimodal models 👇**\n> [**Multimodal AI**](https://shawhin.medium.com/list/fe9521d0e77a)\n* * *\n**My website** : <https://www.shawhintalebi.com/>\n  * [1] [BERT](https://arxiv.org/abs/1810.04805)\n  * [2] [ViT](https://arxiv.org/abs/2010.11929)\n  * [3] [CLIP](https://arxiv.org/abs/2103.00020)\n  * [4] [Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)](https://arxiv.org/abs/2410.07507)\n  * [5] [A Simple Framework for Contrastive Learning of Visual Representations](https://arxiv.org/abs/2002.05709)\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657074748_utev3i.jpeg)\nWritten By\n## [Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[See all from Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[AI](https://towardsdatascience.com/tag/ai/)[Computer Vision](https://towardsdatascience.com/tag/computer-vision/)[Data Science](https://towardsdatascience.com/tag/data-science/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[multimodal-ai](https://towardsdatascience.com/tag/multimodal-ai/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-embeddings-an-introduction-5dc36975966f%2F&title=Multimodal%20Embeddings%3A%20An%20Introduction)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-embeddings-an-introduction-5dc36975966f%2F&title=Multimodal%20Embeddings%3A%20An%20Introduction)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-embeddings-an-introduction-5dc36975966f%2F&text=Multimodal%20Embeddings%3A%20An%20Introduction)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/09/03VwWo3pQYvBtUqKH-scaled.jpg)](https://towardsdatascience.com/many-hyperparameters-use-gin-fdf6741d282/)\n## [Many hyperparameters? Use Gin](https://towardsdatascience.com/many-hyperparameters-use-gin-fdf6741d282/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nHow to handle all the parameters\n[Pascal Janetzky](https://towardsdatascience.com/author/pascaljanetzky/)\nSeptember 18, 20213 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/0uOPnfFD7c89vMbGz-scaled.jpg)](https://towardsdatascience.com/beginner-explanation-for-data-transformation-9add3102f3bf/)\n## [Beginner Explanation for Data Transformation](https://towardsdatascience.com/beginner-explanation-for-data-transformation-9add3102f3bf/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\n[Cornellius Wijaya](https://towardsdatascience.com/author/cornelliusyudhawijaya/)\nMay 10, 20210 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/1JTEqCz-VU16nkkUwzyWp_w.png)](https://towardsdatascience.com/a-practical-summary-of-matplotlib-in-13-python-snippets-4d07f0011bdf/)\n## [A Practical Summary of Matplotlib in 13 Python Snippets](https://towardsdatascience.com/a-practical-summary-of-matplotlib-in-13-python-snippets-4d07f0011bdf/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nFundamental scripts for basic data visualization in Python\n[Lucas Soares](https://towardsdatascience.com/author/lucas-soares/)\nAugust 31, 20216 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/11/1_4D-6Vr-ILDSwXk9dwRGLg-scaled.jpeg)](https://towardsdatascience.com/count-occurrences-of-python-list-item-e87782d8954c/)\n## [How to Count The Occurrences of a List Item in Python](https://towardsdatascience.com/count-occurrences-of-python-list-item-e87782d8954c/)\n[Programming](https://towardsdatascience.com/category/programming/)\nUnderstanding how to compute the number of occurrences of a specific list element in Python\n[Giorgos Myrianthous](https://towardsdatascience.com/author/gmyrianthous/)\nNovember 5, 20213 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1JCKBay3gHyvWhWtTduv2Kg-scaled.jpeg)](https://towardsdatascience.com/the-intuition-behind-concordance-index-survival-analysis-3c961fc11ce8/)\n## [The Intuition behind Concordance Index - Survival Analysis](https://towardsdatascience.com/the-intuition-behind-concordance-index-survival-analysis-3c961fc11ce8/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nRanking accuracy versus absolute accuracy\n[Antonieta Mastrogiuseppe](https://towardsdatascience.com/author/fyi.oamd/)\nNovember 28, 202416 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/0jrob-mez4_AINggt.jpg)](https://towardsdatascience.com/create-a-fast-auto-documented-maintainable-and-easy-to-use-python-api-crud-routes-and-routing-7e8f35ebda46/)\n## [Create a fast auto-documented, maintainable and easy-to-use Python API - CRUD routes and routing...](https://towardsdatascience.com/create-a-fast-auto-documented-maintainable-and-easy-to-use-python-api-crud-routes-and-routing-7e8f35ebda46/)\n[Programming](https://towardsdatascience.com/category/programming/)\nLearn how to catch requests and obtain the information that they carry\n[Mike Huls](https://towardsdatascience.com/author/mikehuls/)\nOctober 5, 20217 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/08/1-Db8jSBEGrJApfCYBdoueg-scaled.jpeg)](https://towardsdatascience.com/turn-and-face-the-strange-a3a59fdb69e5/)\n## [Turn and Face the Strange](https://towardsdatascience.com/turn-and-face-the-strange-a3a59fdb69e5/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nHow to leverage anomaly detection methods to improve your supervised learning\n[Evie Fowler](https://towardsdatascience.com/author/efowler_86027/)\nAugust 11, 20238 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/Bernedoodle-pups_same-different.jpg)](https://towardsdatascience.com/emergent-object-binding-from-self-supervised-not-supervised-learning/)\n## [Do Labels Make AI Blind? Self-Supervision Solves the Age-Old Binding Problem](https://towardsdatascience.com/emergent-object-binding-from-self-supervised-not-supervised-learning/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA new NeurIPS 2025 paper shows how self-supervised learning imbues ViT with better image understanding than supervised training\n[Jonathan Williford](https://towardsdatascience.com/author/williford/)\nDecember 4, 202513 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/1Gm1KYS7PediO_bWtGKTIwA.jpeg)](https://towardsdatascience.com/generate-pencil-sketch-from-photo-in-python-7c56802d8acb/)\n## [Generate Pencil Sketch from Photo in Python](https://towardsdatascience.com/generate-pencil-sketch-from-photo-in-python-7c56802d8acb/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nImage manipulation using OpenCV tutorial with code\n[Abhijith Chandradas](https://towardsdatascience.com/author/abhijithchandradas/)\nJuly 29, 20213 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n",
  "metadata": {
    "title": "Multimodal Embeddings: An Introduction | Towards Data Science",
    "description": "Mapping text and images into a common space",
    "keywords": null,
    "author": "Shaw Talebi",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Multimodal Embeddings: An Introduction",
    "og:description": "Mapping text and images into a common space",
    "og:url": "https://towardsdatascience.com/multimodal-embeddings-an-introduction-5dc36975966f/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1a6BF-kEeo8rd7OW2a3JYGA.png",
    "og:image:width": "1280",
    "og:image:height": "720",
    "og:image:type": "image/png",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Shaw Talebi",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "7 min read",
    "article:published_time": "2024-11-29T15:02:12.000Z",
    "article:modified_time": "2024-11-29T15:02:12.000Z",
    "article:author": "https://towardsdatascience.com/author/shawhin/"
  },
  "paywall": {
    "listed_paid": true,
    "detected": false,
    "marker": "",
    "public_only": true,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_000.txt",
      "chars": 72
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_001.txt",
      "chars": 228
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_002.txt",
      "chars": 134
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_003.txt",
      "chars": 270
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_004.txt",
      "chars": 330
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_005.txt",
      "chars": 44
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_006.txt",
      "chars": 36
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_007.txt",
      "chars": 40
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_008.txt",
      "chars": 216
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_009.txt",
      "chars": 207
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_010.txt",
      "chars": 317
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_011.txt",
      "chars": 30
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_012.txt",
      "chars": 37
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_013.txt",
      "chars": 30
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_014.txt",
      "chars": 21
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_015.txt",
      "chars": 30
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_016.txt",
      "chars": 36
    },
    {
      "language": "txt",
      "file": "attachments/5c504a56c5b4/code/block_017.txt",
      "chars": 30
    }
  ],
  "images": [
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1a6BF-kEeo8rd7OW2a3JYGA.png",
      "alt": "Image from Canva.",
      "local_path": "attachments/5c504a56c5b4/images/000.png",
      "bytes": 279045,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1jpmC6Kx7DxVeikEr15vooA.png",
      "alt": "Toy represetation of text and image embeddings, respectively. Image by author.",
      "local_path": "attachments/5c504a56c5b4/images/001.png",
      "bytes": 141230,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/15d3HBNjNIXLy0oMIvJjxWw.png",
      "alt": "Toy representation of multimodal embedding space. Image by author.",
      "local_path": "attachments/5c504a56c5b4/images/002.png",
      "bytes": 130473,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1AGHBVjzwjXapJSe4aUPrjg.png",
      "alt": "Example positive and negative pairs used in contrastive training. Image by author.",
      "local_path": "attachments/5c504a56c5b4/images/003.png",
      "bytes": 196032,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/12X1aT8fzFsgbqn23zXmmAA.png",
      "alt": "CLIP&#039;s contrastive loss for text-image representation alignment [3]. Image by author.",
      "local_path": "attachments/5c504a56c5b4/images/004.png",
      "bytes": 50874,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1Nzo536sqahqm1Q24Ms2vmA.png",
      "alt": "Input cat photo. Image from Canva.",
      "local_path": "attachments/5c504a56c5b4/images/005.png",
      "bytes": 127383,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/14wnqr5p_7N3QD5EkXIQeew.png",
      "alt": "Best match for query &quot;a cute dog&quot;. Image from Canva.",
      "local_path": "attachments/5c504a56c5b4/images/006.png",
      "bytes": 179601,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1tIY3_ONQQT_cracAPWm8NQ.png",
      "alt": "Best match for query &quot;something cute but metal 🤘&quot;. Image from Canva.",
      "local_path": "attachments/5c504a56c5b4/images/007.png",
      "bytes": 146072,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/14wnqr5p_7N3QD5EkXIQeew.png",
      "alt": "Best match for query &quot;a good boy&quot;. Image from Canva.",
      "local_path": "attachments/5c504a56c5b4/images/008.png",
      "bytes": 179601,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1Nzo536sqahqm1Q24Ms2vmA.png",
      "alt": "Best match for query &quot;the best pet in the world&quot;. Image from Canva.",
      "local_path": "attachments/5c504a56c5b4/images/009.png",
      "bytes": 127383,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/09/03VwWo3pQYvBtUqKH-scaled.jpg",
      "alt": "",
      "local_path": "attachments/5c504a56c5b4/images/010.jpg",
      "bytes": 359019,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/0uOPnfFD7c89vMbGz-scaled.jpg",
      "alt": "",
      "local_path": "attachments/5c504a56c5b4/images/011.jpg",
      "bytes": 278255,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/1JTEqCz-VU16nkkUwzyWp_w.png",
      "alt": "",
      "local_path": "attachments/5c504a56c5b4/images/012.png",
      "bytes": 292530,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/11/1_4D-6Vr-ILDSwXk9dwRGLg-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/5c504a56c5b4/images/013.jpeg",
      "bytes": 377496,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1JCKBay3gHyvWhWtTduv2Kg-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/5c504a56c5b4/images/014.jpeg",
      "bytes": 774752,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/0jrob-mez4_AINggt.jpg",
      "alt": "",
      "local_path": "attachments/5c504a56c5b4/images/015.jpg",
      "bytes": 79383,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/08/1-Db8jSBEGrJApfCYBdoueg-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/5c504a56c5b4/images/016.jpeg",
      "bytes": 298660,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/Bernedoodle-pups_same-different.jpg",
      "alt": "",
      "local_path": "attachments/5c504a56c5b4/images/017.jpg",
      "bytes": 322727,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/1Gm1KYS7PediO_bWtGKTIwA.jpeg",
      "alt": "",
      "local_path": "attachments/5c504a56c5b4/images/018.jpeg",
      "bytes": 47079,
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "src": "https://www.youtube.com/c/TowardsDataScience",
      "quality": "unavailable",
      "local_path": "",
      "note": "无1080p且无720p，未交付更低清晰度"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/data-science/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/shawhin/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/multimodal-models-llms-that-can-see-and-hear-5c6737c981d3"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/ai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/computer-vision/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/data-science/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/multimodal-ai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/many-hyperparameters-use-gin-fdf6741d282/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/deep-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/pascaljanetzky/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/beginner-explanation-for-data-transformation-9add3102f3bf/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/cornelliusyudhawijaya/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/a-practical-summary-of-matplotlib-in-13-python-snippets-4d07f0011bdf/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://shawhin.medium.com/list/multimodal-ai-fe9521d0e77a"
    },
    {
      "type": "external",
      "href": "https://shawhin.medium.com/list/large-language-models-llms-8e009ae3054c"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/openai/clip-vit-base-patch16"
    },
    {
      "type": "external",
      "href": "https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/2-mm-embeddings"
    },
    {
      "type": "external",
      "href": "https://huggingface.co/docs/transformers/en/installation"
    },
    {
      "type": "external",
      "href": "https://medium.com/towards-data-science/multimodal-rag-process-any-file-type-with-ai-e6921342c903"
    },
    {
      "type": "external",
      "href": "https://shawhin.medium.com/list/fe9521d0e77a"
    },
    {
      "type": "external",
      "href": "https://www.shawhintalebi.com/"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/1810.04805"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2010.11929"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2103.00020"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2410.07507"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2002.05709"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=Multimodal+Embeddings%3A+An+Introduction&u=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-embeddings-an-introduction-5dc36975966f%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=Multimodal+Embeddings%3A+An+Introduction&url=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-embeddings-an-introduction-5dc36975966f%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=Multimodal+Embeddings%3A+An+Introduction&url=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-embeddings-an-introduction-5dc36975966f%2F"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/c/TowardsDataScience"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T11:11:49Z",
  "source_url": "https://towardsdatascience.com/multimodal-embeddings-an-introduction-5dc36975966f/",
  "delivery_status": "excluded_paid",
  "discovery_status": "excluded_paid",
  "missing_reason": "按原约定排除付费/会员全文"
}