{ "source_file": "0930", "source_platform": "Medium", "author": "Shawhin Talebi", "title_listed": "Multimodal Models - LLMs that can see and hear", "keywords": "多模态,LLM", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "部分免费/会员墙", "url": "https://towardsdatascience.com/multimodal-models-llms-that-can-see-and-hear-5c6737c981d3/", "publish_date": "2024-11", "corpus_stage": "预训练/SFT", "notes": "", "id": "c22b733ba8ff", "collected_at": "2026-10-01T22:22:12", "title": "Multimodal Models - LLMs that can see and hear | Towards Data Science", "body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\n# Multimodal Models - LLMs that can see and hear\nAn introduction with example Python code\n[Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\nNovember 19, 20249 min read\n# **Multimodal Models** - **LLMs That Can See and Hear**\nThis is the first post in a larger series on [Multimodal AI](https://shawhin.medium.com/list/multimodal-ai-fe9521d0e77a). A **Multimodal Model (MM)** is an **AI system capable of processing or generating multiple data modalities** (e.g., text, image, audio, video). In this article, I will discuss a particular type of MM that builds on top of a large language model (LLM). I'll start with a high-level overview of such models and then share example code for using LLaMA 3.2 Vision to perform various image-to-text tasks.\n![Photo by Sincerely Media on Unsplash](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/0YE-Q-OuWnrgrUrQw-scaled.jpg)\nPhoto by [Sincerely Media](https://unsplash.com/@sincerelymedia?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\n* * *\n[Large language models (LLMs)](https://towardsdatascience.com/a-practical-introduction-to-llms-65194dda1148) have marked a fundamental shift in AI research and development. However, despite their broader impacts, they are still **fundamentally limited**.\nNamely, LLMs can only process and generate text, making them _blind_ to other modalities such as images, video, audio, and more. This is a major limitation since **some tasks rely on non-text data,** e.g., analyzing engineering blueprints, reading body language or speech tonality, and interpreting plots and infographics.\nThis has sparked efforts toward expanding LLM functionality to include multiple modalities.\n## **What is a Multimodal Model?**\nA **Multimodal Model (MM)** is an **AI system that can process multiple data modalities as input or output (or both)** [1]. Below are a few examples.\n * **GPT-4o** - Input: text, images, and audio. Output: text.\n * **FLUX** - Input: text. Output: images.\n * **Suno** - Input: text. Output: audio.\n\n\n![Example mutlimodal models. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1yvfu8VAp1UgCw4SVvUe77Q.png)\nExample mutlimodal models. Image by author.\nWhile there are several ways to create models that can process multiple data modalities, a recent line of research seeks to use **LLMs as the core reasoning engine of a multimodal system** [2]. Such models are called multimodal large language models (or large multimodal models) [2][3].\nOne benefit of using existing LLM as a starting point for MMs is that they've **demonstrated a strong ability to acquire world knowledge through large-scale pre-training** , which can be leveraged to process concepts appearing in non-textual representations.\n## **3 Paths to Multimodality**\nHere, I will focus on multimodal models developed from an LLM. Three popular approaches are described below.\n 1. **LLM + Tools** : Augment LLMs with pre-built components\n 2. **LLM + Adapters** : Augment LLMs with multi-modal encoders or decoders, which are aligned via adapter fine-tuning\n 3. **Unified Models** : Expand LLM architecture to fuse modalities at pre-training\n\n\n## **Path 1: LLM + Tools**\nThe simplest way to make an LLM multimodal is by **adding external modules that can readily translate between text and an arbitrary modality**. For example, a transcription model (e.g. Whisper) can be connected to an LLM to translate input speech into text, or a text-to-image model can generate images based on LLM outputs.\nThe key benefit of such an approach is **simplicity**. Tools can quickly be assembled without any additional model training.\nThe downside, however, is that the quality of such a system may be limited. Just like when playing a game of telephone, messages mutate when passed from person to person. **Information may degrade going from one module to another via text descriptions only.**\n![An example of information degradation during message passing. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1Nwc-ZhRFKH17LWWmsNhbdA.png)\nAn example of information degradation during message passing. Image by author.\n## **Path 2: LLM + Adapters**\nOne way to mitigate the \"telephone problem\" is by optimizing the representations of new modalities to align with the LLM's internal concept space. For example, ensuring an image of a dog and the description of one _look_ similar to the LLM.\nThis is possible through the use of **adapters** , a relatively small set of **parameters that appropriately translate a dense vector representation for a downstream model** [2][4][5].\nAdapters can be trained using, for example, image-caption pairs, where the adapter learns to translate an image encoding into a representation compatible with the LLM [2][4][6]. One way to achieve this is via contrastive learning [2], which I will discuss more in the [next article](https://towardsdatascience.com/multimodal-embeddings-an-introduction-5dc36975966f) of this series.\n![A simple strategy for integrating images into an LLM via an image encoding adapter. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1pyqGh5Cbrk_EMlPYtrfrQw.png)\nA simple strategy for integrating images into an LLM via an image encoding adapter. Image by author.\nThe benefits of using adapters to augment LLMs include **better alignment between novel modality representations in a data-efficient way**. Since many pre-trained embedding, language, and diffusion models are available in today's AI landscape, one can readily fuse models based on their needs. Notable examples from the open-source community are LLaVA, LLaMA 3.2 Vision, Flamingo, MiniGPT4, Janus, Mini-Omni2, and IDEFICS [3][5][7][8].\nHowever, this data efficiency comes at a price. Just like how adapter-based fine-tuning approaches (e.g. [LoRA](https://towardsdatascience.com/fine-tuning-large-language-models-llms-23473d763b91#8e86)) can only nudge an LLM so far, the same holds in this context. Additionally, pasting various encoders and decoders to an LLM may result in overly complicated model architectures.\n## **Path 3: Unified Models**\nThe final way to make an LLM multimodal is by **incorporating multiple modalities at the pre-training stage**. This works by adding modality-specific tokenizers (rather than pre-trained encoder/decoder models) to the model architecture and expanding the embedding layer to accommodate new modalities [9].\nWhile this approach comes with significantly greater technical challenges and computational requirements, it enables the **seamless integration of multiple modalities into a shared concept space** , unlocking better reasoning capabilities and efficiencies [10].\nThe preeminent example of this unified approach is (presumably) GPT-4o, which processes text, image, and audio inputs to enable **expanded reasoning capabilities at faster inference times than previous versions of GPT-4**. Other models that follow this approach include Gemini, Emu3, BLIP, and Chameleon [9][10].\nTraining these models typically entails multi-step pre-training on a set of (multimodal) tasks, such as language modeling, text-image contrastive learning, text-to-video generation, and others [7][9][10].\n## **Example: Using** LLaMA 3.2 Vision for Image-based Tasks\nWith a basic understanding of how LLM-based multimodal models work under the hood, let's see what we can do with them. Here, I will use LLaMA 3.2 Vision to perform various image-to-text tasks.\nTo run this example, [download Ollama](https://ollama.com/download) and its [Python library](https://pypi.org/project/ollama/). This enables the model to run locally i.e. no need for external API calls.\nThe example code is freely available on [GitHub](https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/1-mm-llms).\n* * *\n### Importing model\nWe start by importing ollama.\npython\n\n```\nimport ollama\n```\n\nNext, we'll download the model locally. Here, we use LLaMA 3.2 Vision 11B.\npython\n\n```\nollama.pull('llama3.2-vision')\n```\n\n### Visual QA\nNow, we're ready to use the model! Here's how we can do basic visual question answering.\npython\n\n```\n# pass image and question to model\nresponse = ollama.chat(\n model='llama3.2-vision',\n messages=[{\n 'role': 'user',\n 'content': 'What is in this image?',\n 'images': ['images/shaw-sitting.jpeg']\n }]\n)\n\n# print response\nprint(response['message']['content'])\n```\n\nThe image is of me from a networking event (as shown below).\n![Image of me from networking event at Richardson IQ. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1lvX8Mut8SQ1vDhsaewLQ_g.jpeg)\nImage of me from networking event at Richardson IQ. Image by author.\nThe model's response is shown below. While it has trouble reading what's on my hat, it does a decent job inferring the context of the photo.\nplaintext\n\n```\nThis image shows a man sitting on a yellow ottoman with his hands clasped \ntogether. He is wearing a black polo shirt with a name tag that says \"Shaw\" \nand a black baseball cap with white text that reads, \"THE DATA ENREPRENEUR.\" \nThe background of the image appears to be an office or lounge area, with a \nlarge screen on the wall behind him displaying a presentation slide. There are \nalso several chairs and tables in the background, suggesting that this may be \na meeting room or common area for employees to gather and work.\n```\n\nIf you run this on your machine, you may run into a long wait time until the model generates a response. One thing we can do to make this less painful is to enable streaming.\npython\n\n```\n# create stream\nstream = ollama.chat(\n model='llama3.2-vision',\n messages=[{\n 'role': 'user',\n 'content': 'Can you write a caption for this image?',\n 'images': ['images/shaw-sitting.jpeg']\n }],\n stream=True,\n)\n\n# print chunks in stream as they become available\nfor chunk in stream:\n print(chunk['message']['content'], end='', flush=True)\n```\n\nInterestingly, we get a qualitatively different response when prompting the model in a slightly different way for the same image.\nplaintext\n\n```\nThis image features a man sitting on a yellow chair. He is wearing a black \npolo shirt with a blue name tag that says \"Shaw\", khaki pants, and a black \nbaseball cap with white text that reads \"THE DATA ENTHUSIAST\". The man has his \nhands clasped together in front of him and appears to be smiling.\n\nThe background of the image consists of a room with various pieces of \nfurniture. There is a green ottoman to the left of the yellow chair, and two \nblue chairs on the right side of the image. A brown table or desk sits behind \nthe man, along with a fireplace. The walls are painted teal blue and have a \nwooden accent wall featuring holes for hanging items.\n\nThe overall atmosphere suggests that this may be a modern office space or \nco-working area where people can come to work, relax, or socialize.\n```\n\n### Explaining Memes\nObjectively describing a scene is simpler than understanding and explaining humor. Let's see how the model explains the meme below.\n![Building with AI meme. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1IqUoZEX2CYOsX6oFIVeuIw.jpeg)\nBuilding with AI meme. Image by author.\npython\n\n```\n# ask model to explain meme\nstream = ollama.chat(\n model='llama3.2-vision',\n messages=[{\n 'role': 'user',\n 'content': 'Can you explain this meme to me?',\n 'images': ['images/ai-meme.jpeg']\n }],\n stream=True,\n)\n\n# print stream\nfor chunk in stream:\n print(chunk['message']['content'], end='', flush=True)\n```\n\nplaintext\n\n```\nThe meme depicts Patrick Star from SpongeBob SquarePants, surrounded by \nvarious AI tools and symbols. The caption reads \"Trying to build with AI \ntoday...\" The image humorously illustrates the challenges of using AI in \nbuilding projects, implying that it can be overwhelming and frustrating.\n```\n\nThe model does a good job here. It understands that the image is funny while also conveying the pain that people face.\n### OCR\nThe last use case is optical character recognition (OCR). This involves extracting text from images, which is valuable in a wide range of contexts. Here, I'll see if the model can translate a screenshot from my notes app to a markdown file.\n![Screenshot of 5 AI project ideas. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1PRSGngwjIVW01cLHK41lNg.jpeg)\nScreenshot of [5 AI project ideas](https://towardsdatascience.com/5-ai-projects-you-can-build-this-weekend-with-python-c57724e9c461). Image by author.\npython\n\n```\n# ask model to read screenshot and convert to markdown\nstream = ollama.chat(\n model='llama3.2-vision',\n messages=[{\n 'role': 'user',\n 'content': 'Can you transcribe the text from this screenshot in a \n markdown format?',\n 'images': ['images/5-ai-projects.jpeg']\n }],\n stream=True,\n)\n\n# read stream\nfor chunk in stream:\n print(chunk['message']['content'], end='', flush=True)\n```\n\ntext\n\n```\nHere is the transcription of the text in markdown format:\n\n5 AI Projects You Can Build This Weekend (with Python)\n\n1. **Resume Optimization (Beginner)**\n * Idea: build a tool that adapts your resume for a specific job description\n2. **YouTube Lecture Summarizer (Beginner)**\n * Idea: build a tool that takes YouTube video links and summarizes them\n3. **Automatically Organizing PDFs (Intermediate)**\n * Idea: build a tool to analyze the contents of each PDF and organize them \ninto folders based on topics\n4. **Multimodal Search (Intermediate)**\n * Idea: use multimodal embeddings to represent user queries, text knowledge, \nand images in a single space\n5. **Desktop QA (Advanced)**\n * Idea: connect a multimodal knowledge base to a multimodal model like \nLlama-3.2-11B-Vision\n\nNote that I've added some minor formatting changes to make the text more \nreadable in markdown format. Let me know if you have any further requests.\n```\n\nAgain, the model does a decent job out of the box. While it missed the header, it accurately captured the content and formatting of the project ideas.\n> [**YouTube-Blog/multimodal-ai/1-mm-llms at main · ShawhinT/YouTube-Blog**](https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/1-mm-llms)\n## What's next?\nMultimodal models are AI systems that can process multiple data modalities as inputs or outputs (or both). A recent trend for developing these systems involves adding modalities to large language models (LLMs).\nHowever, there are other types of multimodal models. In the [next article](https://towardsdatascience.com/multimodal-embeddings-an-introduction-5dc36975966f) of this series, I will discuss multimodal embedding models, which encode multiple data modalities (e.g. text and images) into a shared representation space.\n**More on Multimodal models 👇**\n> [**Multimodal AI**](https://shawhin.medium.com/list/fe9521d0e77a)\n* * *\n👉 **Get FREE access to every new story I write**[**(Learn More)**](https://shawhin.medium.com/subscribe)\n[1] [Multimodal Machine Learning: A Survey and Taxonomy](https://arxiv.org/abs/1705.09406)\n[2] [A Survey on Multimodal Large Language Models](https://arxiv.org/abs/2306.13549)\n[3] [Visual Instruction Tuning](https://arxiv.org/abs/2304.08485)\n[4] [GPT-4o System Card](https://arxiv.org/abs/2410.21276)\n[5] [Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](https://arxiv.org/abs/2410.13848)\n[6] [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/abs/2103.00020)\n[7] [Flamingo: a Visual Language Model for Few-Shot Learning](https://arxiv.org/abs/2204.14198)\n[8] [Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities](https://arxiv.org/abs/2410.11190)\n[9] [Emu3: Next-Token Prediction is All You Need](https://arxiv.org/abs/2409.18869)\n[10] [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://arxiv.org/abs/2405.09818)\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657074748_utev3i.jpeg)\nWritten By\n## [Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[See all from Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[AI](https://towardsdatascience.com/tag/ai/)[Getting Started](https://towardsdatascience.com/tag/getting-started/)[Llm](https://towardsdatascience.com/tag/llm/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[multimodal-ai](https://towardsdatascience.com/tag/multimodal-ai/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-models-llms-that-can-see-and-hear-5c6737c981d3%2F&title=Multimodal%20Models%20-%20LLMs%20that%20can%20see%20and%20hear)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-models-llms-that-can-see-and-hear-5c6737c981d3%2F&title=Multimodal%20Models%20-%20LLMs%20that%20can%20see%20and%20hear)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fmultimodal-models-llms-that-can-see-and-hear-5c6737c981d3%2F&text=Multimodal%20Models%20-%20LLMs%20that%20can%20see%20and%20hear)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1x6pKNbi-mznjKWFYrUDgfw.png)](https://towardsdatascience.com/visualizing-gradient-descent-parameters-in-torch-332a63d1e5c5/)\n## [Visualizing Gradient Descent Parameters in Torch](https://towardsdatascience.com/visualizing-gradient-descent-parameters-in-torch-332a63d1e5c5/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nPrying behind the interface to see the effects of SGD parameters on your model training\n[P.G. Baumstarck](https://towardsdatascience.com/author/thepgb/)\nFebruary 26, 20247 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/0PNm_tZzRC9kjH99m-scaled.jpg)](https://towardsdatascience.com/data-scientists-dont-work-at-startups-for-your-first-job-45274957c3d6/)\n## [Data Scientists, Don't Work at Startups For Your First Job](https://towardsdatascience.com/data-scientists-dont-work-at-startups-for-your-first-job-45274957c3d6/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nYou will learn a lot of things, but most of them are probably wrong\n[Rionaldi Chandraseta](https://towardsdatascience.com/author/chandraseta/)\nAugust 10, 20218 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/1OtZdV6HfCuHygPvMchJOzA.jpeg)](https://towardsdatascience.com/the-four-policy-classes-of-reinforcement-learning-38185daa6c8a/)\n## [The Four Policy Classes of Reinforcement Learning](https://towardsdatascience.com/the-four-policy-classes-of-reinforcement-learning-38185daa6c8a/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThoughts and Theory A comprehensive classification of solution strategies for reinforcement learning Policies in Reinforcement...\n[Wouter van Heeswijk, PhD](https://towardsdatascience.com/author/wvheeswijk/)\nMay 11, 20217 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/1K3dCVTr_0G3HX7tZc0cBxg-scaled.jpeg)](https://towardsdatascience.com/augmented-assignments-python-caa4990811a0/)\n## [Augmented Assignments in Python](https://towardsdatascience.com/augmented-assignments-python-caa4990811a0/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nUnderstanding how augmented assignment expressions work in Python and why you should be careful when using them with mutable objects\n[Giorgos Myrianthous](https://towardsdatascience.com/author/gmyrianthous/)\nDecember 2, 20214 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/12/0BQZFmE9s3RNm-2L4-scaled.jpg)](https://towardsdatascience.com/you-cant-spell-diffusion-without-u-60635f569579/)\n## [You can't spell Diffusion without U](https://towardsdatascience.com/you-cant-spell-diffusion-without-u-60635f569579/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nUnderstanding the core building blocks of Stable Diffusion\n[Sairam Sundaresan](https://towardsdatascience.com/author/artofsaience/)\nDecember 23, 20225 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/0sH_rtHpzfktmOeUh-scaled.jpg)](https://towardsdatascience.com/how-to-crack-python-errors-like-a-pro-221ac76409af/)\n## [How to Crack Python Errors Like a Pro](https://towardsdatascience.com/how-to-crack-python-errors-like-a-pro-221ac76409af/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nAs you crack manually an exception, the developer you become is more important than what you fix.\n[Houssem Ben Braiek](https://towardsdatascience.com/author/houssem.benbraiek/)\nDecember 31, 20217 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/10/07jM8tPkxgsI-Ke88-scaled.jpg)](https://towardsdatascience.com/embracing-uncertainty-the-power-of-fuzzy-logic-in-decision-making-73abb7c30ac4/)\n## [Embracing Uncertainty: The Power of Fuzzy Logic in Decision-Making](https://towardsdatascience.com/embracing-uncertainty-the-power-of-fuzzy-logic-in-decision-making-73abb7c30ac4/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nExploring how fuzzy logic enhances AI, systems thinking, and real-world applications\n[Niklas Lang](https://towardsdatascience.com/author/niklas_lang/)\nOctober 9, 202411 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/0QN3xznUDhgWeTihU-scaled.jpg)](https://towardsdatascience.com/focal-loss-a-better-alternative-for-cross-entropy-1d073d92d075/)\n## [Focal Loss : A better alternative for Cross-Entropy](https://towardsdatascience.com/focal-loss-a-better-alternative-for-cross-entropy-1d073d92d075/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nFocal loss is said to perform better than Cross-Entropy loss in many cases. But why Cross-Entropy loss fails, and how Focal loss addresses...\n[Roshan Nayak](https://towardsdatascience.com/author/roshannayak610/)\nApril 26, 20225 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/01/1jk_M2Cg05eN5Tt3XHByGhg.png)](https://towardsdatascience.com/support-vector-machine-mnist-digit-classification-with-python-including-my-hand-written-digits-83d6eca7004a/)\n## [Support Vector Machine: MNIST Digit Classification with Python; Including my Hand Written Digits](https://towardsdatascience.com/support-vector-machine-mnist-digit-classification-with-python-including-my-hand-written-digits-83d6eca7004a/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nUnderstanding SVM Series : Part 3\n[Saptashwa Bhattacharyya](https://towardsdatascience.com/author/saptashwa/)\nJanuary 21, 20199 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n", "metadata": { "title": "Multimodal Models - LLMs that can see and hear | Towards Data Science", "description": "An introduction with example Python code", "keywords": null, "author": "Shaw Talebi", "og:locale": "en_US", "og:type": "article", "og:title": "Multimodal Models - LLMs that can see and hear", "og:description": "An introduction with example Python code", "og:url": "https://towardsdatascience.com/multimodal-models-llms-that-can-see-and-hear-5c6737c981d3/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/0YE-Q-OuWnrgrUrQw-scaled.jpg", "og:image:width": "2560", "og:image:height": "1920", "og:image:type": "image/jpeg", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Shaw Talebi", "twitter:label2": "Est. reading time", "twitter:data2": "9 min read", "article:published_time": "2024-11-19T11:01:59.000Z", "article:modified_time": "2024-11-19T11:01:59.000Z", "article:author": "https://towardsdatascience.com/author/shawhin/" }, "paywall": { "listed_paid": true, "detected": false, "marker": "", "public_only": true, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "language": "txt", "file": "attachments/c22b733ba8ff/code/block_000.txt", "chars": 14 }, { "language": "txt", "file": "attachments/c22b733ba8ff/code/block_001.txt", "chars": 31 }, { "language": "txt", "file": "attachments/c22b733ba8ff/code/block_002.txt", "chars": 285 }, { "language": "txt", "file": "attachments/c22b733ba8ff/code/block_003.txt", "chars": 529 }, { "language": "txt", "file": "attachments/c22b733ba8ff/code/block_004.txt", "chars": 374 }, { "language": "txt", "file": "attachments/c22b733ba8ff/code/block_005.txt", "chars": 802 }, { "language": "txt", "file": "attachments/c22b733ba8ff/code/block_006.txt", "chars": 339 }, { "language": "txt", "file": "attachments/c22b733ba8ff/code/block_007.txt", "chars": 294 }, { "language": "txt", "file": "attachments/c22b733ba8ff/code/block_008.txt", "chars": 430 }, { "language": "txt", "file": "attachments/c22b733ba8ff/code/block_009.txt", "chars": 927 } ], "images": [ { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/0YE-Q-OuWnrgrUrQw-scaled.jpg", "alt": "Photo by Sincerely Media on Unsplash", "local_path": "attachments/c22b733ba8ff/images/000.jpg", "bytes": 159093, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1yvfu8VAp1UgCw4SVvUe77Q.png", "alt": "Example mutlimodal models. Image by author.", "local_path": "attachments/c22b733ba8ff/images/001.png", "bytes": 145440, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1Nwc-ZhRFKH17LWWmsNhbdA.png", "alt": "An example of information degradation during message passing. Image by author.", "local_path": "attachments/c22b733ba8ff/images/002.png", "bytes": 254327, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1pyqGh5Cbrk_EMlPYtrfrQw.png", "alt": "A simple strategy for integrating images into an LLM via an image encoding adapter. Image by author.", "local_path": "attachments/c22b733ba8ff/images/003.png", "bytes": 107827, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1lvX8Mut8SQ1vDhsaewLQ_g.jpeg", "alt": "Image of me from networking event at Richardson IQ. Image by author.", "local_path": "attachments/c22b733ba8ff/images/004.jpeg", "bytes": 158928, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1IqUoZEX2CYOsX6oFIVeuIw.jpeg", "alt": "Building with AI meme. Image by author.", "local_path": "attachments/c22b733ba8ff/images/005.jpeg", "bytes": 92624, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1PRSGngwjIVW01cLHK41lNg.jpeg", "alt": "Screenshot of 5 AI project ideas. Image by author.", "local_path": "attachments/c22b733ba8ff/images/006.jpeg", "bytes": 99780, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1x6pKNbi-mznjKWFYrUDgfw.png", "alt": "", "local_path": "attachments/c22b733ba8ff/images/007.png", "bytes": 944643, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/0PNm_tZzRC9kjH99m-scaled.jpg", "alt": "", "local_path": "attachments/c22b733ba8ff/images/008.jpg", "bytes": 563636, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/1OtZdV6HfCuHygPvMchJOzA.jpeg", "alt": "", "local_path": "attachments/c22b733ba8ff/images/009.jpeg", "bytes": 169333, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/1K3dCVTr_0G3HX7tZc0cBxg-scaled.jpeg", "alt": "", "local_path": "attachments/c22b733ba8ff/images/010.jpeg", "bytes": 566053, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/12/0BQZFmE9s3RNm-2L4-scaled.jpg", "alt": "", "local_path": "attachments/c22b733ba8ff/images/011.jpg", "bytes": 683840, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/0sH_rtHpzfktmOeUh-scaled.jpg", "alt": "", "local_path": "attachments/c22b733ba8ff/images/012.jpg", "bytes": 530256, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/10/07jM8tPkxgsI-Ke88-scaled.jpg", "alt": "", "local_path": "attachments/c22b733ba8ff/images/013.jpg", "bytes": 89511, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/0QN3xznUDhgWeTihU-scaled.jpg", "alt": "", "local_path": "attachments/c22b733ba8ff/images/014.jpg", "bytes": 824575, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2019/01/1jk_M2Cg05eN5Tt3XHByGhg.png", "alt": "", "local_path": "attachments/c22b733ba8ff/images/015.png", "bytes": 58824, "downloaded": true } ], "audio": [], "videos": [ { "src": "https://www.youtube.com/c/TowardsDataScience", "quality": "unavailable", "local_path": "", "note": "无1080p且无720p，未交付更低清晰度" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/shawhin/" }, { "type": "internal", "href": "https://towardsdatascience.com/a-practical-introduction-to-llms-65194dda1148" }, { "type": "internal", "href": "https://towardsdatascience.com/multimodal-embeddings-an-introduction-5dc36975966f" }, { "type": "internal", "href": "https://towardsdatascience.com/fine-tuning-large-language-models-llms-23473d763b91" }, { "type": "internal", "href": "https://towardsdatascience.com/5-ai-projects-you-can-build-this-weekend-with-python-c57724e9c461" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/ai/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/getting-started/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/llm/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/multimodal-ai/" }, { "type": "internal", "href": "https://towardsdatascience.com/visualizing-gradient-descent-parameters-in-torch-332a63d1e5c5/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/thepgb/" }, { "type": "internal", "href": "https://towardsdatascience.com/data-scientists-dont-work-at-startups-for-your-first-job-45274957c3d6/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://shawhin.medium.com/list/multimodal-ai-fe9521d0e77a" }, { "type": "external", "href": "https://unsplash.com/@sincerelymedia" }, { "type": "external", "href": "https://unsplash.com" }, { "type": "external", "href": "https://ollama.com/download" }, { "type": "external", "href": "https://pypi.org/project/ollama/" }, { "type": "external", "href": "https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/1-mm-llms" }, { "type": "external", "href": "https://shawhin.medium.com/list/fe9521d0e77a" }, { "type": "external", "href": "https://shawhin.medium.com/subscribe" }, { "type": "external", "href": "https://arxiv.org/abs/1705.09406" }, { "type": "external", "href": "https://arxiv.org/abs/2306.13549" }, { "type": "external", "href": "https://arxiv.org/abs/2304.08485" }, { "type": "external", "href": "https://arxiv.org/abs/2410.21276" }, { "type": "external", "href": "https://arxiv.org/abs/2410.13848" }, { "type": "external", "href": "https://arxiv.org/abs/2103.00020" }, { "type": "external", "href": "https://arxiv.org/abs/2204.14198" }, { "type": "external", "href": "https://arxiv.org/abs/2410.11190" }, { "type": "external", "href": "https://arxiv.org/abs/2409.18869" } ], "repair_version": 2, "repaired_at": "2026-10-14T11:11:49Z", "source_url": "https://towardsdatascience.com/multimodal-models-llms-that-can-see-and-hear-5c6737c981d3/", "delivery_status": "excluded_paid", "discovery_status": "excluded_paid", "missing_reason": "按原约定排除付费/会员全文" }