{ "source_file": "0930", "source_platform": "Medium", "author": "Avishek Biswas", "title_listed": "How Vision Language Models Are Trained from \"Scratch\"", "keywords": "", "domain": "图像处理", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://towardsdatascience.com/how-vision-language-models-are-trained-from-scratch/", "publish_date": "2026-03-13", "corpus_stage": "", "notes": "", "id": "ccb16b00665a", "collected_at": "2026-10-02T10:20:33", "title": "How Vision Language Models Are Trained from “Scratch” | Towards Data Science", "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\n# How Vision Language Models Are Trained from “Scratch”\nA deep dive into exactly how text-only language models are fine-tuned to *see* images\n[Avishek Biswas](https://towardsdatascience.com/author/neural.avb/)\nMarch 13, 202611 min read\n![](../images/000.jpg)\nRecently, I went on an adventure to transform a small text-only language model and gift it the power of vision. This article is to summarize all my learnings, and take a deeper look at the network architectures behind modern Vision Language Models.\n_The code is open-source, you can check out the GitHub link at the end of the article. There is also a 30-minute companion YouTube video that explains the whole article in a visually rich format._ \n \n_Also, unless otherwise mentioned, all images in this article are produced by the author._\n**Wait, are you really going to be \"training from scratch\"?**\nYes... I mean no... _it's a bit nuanced_.\n**Research labs in 2026 do not train multimodal models from \"scratch\" anymore.** It is simply too expensive to teach a model vision and (textual) language at the same time! It requires more data, compute, time, and money. Also, it often leads to poorer results.\nInstead, labs take existing pretrained text-only language models, and finetune it to give \"vision capabilities\". In theory (and practice), this is way more compute-efficient.\n![](../images/001.png)\nLet's talk about Vision Language Models!\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\n## The standard architecture\nAlthough it's less data-intensive, finetuning text-only LMs to _suddenly_ start seeing images _would_ surely open its own can of worms.\n 1. **how do we embed the image,** i.e. convert it into numerical representations that a neural network can understand?\n 2. how do we tune the image embeddings to be **compatible with text**?\n 3. how do we adjust the weights of the text model so that it retains it's previous world knowledge, but also **generate text from image embeddings**?\n\n\n![](../images/002.png)\nThe image embeddings coming out of the VIT and Q-Former passes through an MLP layer. Followed by a series of Decoder layer and trainable LORA adapters.\nThese modules are:\n 1. **The Image Backbone:** A model that converts raw images into embeddings.\n 2. **The Adapter Layer:** These are models that convert the image embeddings into a \"text-compatible\" embedding. This is the main challenging part - what architectures to use, what loss functions, etc.\n 3. **The Language Layer:** The language model we will train to input the adapted embeddings and generate text from it.\n\n\nLet's discuss them one by one.\n## 1. The Image Backbone\nThe goal of your image backbone is simple:\n**Input:** A raw 2D pixel map/image.\n**Output:** A sequence of vector embeddings representing the image\nIn general, we just use an off-the-shelf image model that has been pretrained with massive corpus of images, generally on self-supervised tasks.\nYou can use a Convolutional Neural Network (like a ResNet) to use as an image backbone. But modern state-of-the-art VLMs have almost entirely shifted to ViTs because they scale better with data and are more flexible for multimodal fusion.\n![](../images/003.png)\nVision Transformers input an image, extract patches out of it, and then pass them through bidirectional self-attention layers. This contextualizes each patch embedding with each other to form a contextual sequence of image embeddings.\n_**Do I train the image backbone or keep it frozen?**_\nIn most VLM research, there is a clear trend toward keeping backbones static (frozen) to save costs. Also, vision-language training generally needs paired image-text datasets. Since these datasets are always much smaller than the VIT's pretraining dataset, finetuning the backbone often leads to overfitting and degraded performance.\nBy keeping these weights frozen, we are basically transferring the ownership of vision-language learning to latter parts of the network (i.e. the adapter layer and the text backbone).\nIn my experiments, I used the [ViT-Base model](https://huggingface.co/google/vit-base-patch16-224). This model takes the image input, splits it into patches of 16x16 images and applies self-attention on them to generate an embedding sequence of 197 vectors. Each vector is 768 dimesnions long (the embedding size of the VIT).\n## 2. The Adapter Layer\nThis is where we are going to spend the majority of our time. We have converted images into embeddings already, but those embeddings are completely text-unaware.\nVision Transformers are pre-trained purely on image pixels. Not on their captions, or any local textual features. The role of the adapter is to ground the pixel-based-image-embeddings into a (often shorter sequence of) text-based-image-embeddings.\nThere are many ways to do this, like using CLIP models, but we are going to look at one of the more popular approaches — the Query Former of the Q-Former.\n### Q-Former\nAlright — so what is a Q-Former? A Q-Former or the Query-Former was introduced in the [BLIP-2 paper](https://arxiv.org/abs/2301.12597).\n![](../images/body-ba75190797c8121dca2f.png)\nFrom the BLIP-2 Paper\n**How do I train a Q-Former?**\nStandard Q-Formers can be trained using any multimodal image-text pair dataset. For example, you can use the Conceptual Captions dataset, which is a massive corpus of images and their corresponding captions. In my project, I took just 50,000 pairs to train the Q-Former.\nYou can train a Q-Former from scratch, but the BLIP-2 recommendation is to use an pretrained BERT model. So that's what we will do.\nAt a high level, here is our basic gameplan:\n * Train a multi-modal joint embedding space. A space where text and images \"know\" each other.\n * Basically, we will input pairs of image and captions — and embed both of them in the same joint space.\n * Images and incompatible captions will be mapped in separate places in this new embedding space, and compatible captions will be mapped close to each other.\n\n\n![](../images/body-c292dc9550c316c4e6dc.png)\nLayer 1: Self Attention\n![](../images/body-42384cc6ae53e8b1ef76.png)\nLayer 2: Self-Attention & Cross-Attention with the VIT Features\n \n**Setting up cross-attention layers**\nThere's a problem — BERT models are purely text models. They have no idea what an image is.\nSo, our objective is to first introduce new cross-attention layers to marry the vision embeddings coming out of the VIT and the text embeddings from BERT. Let's break down step-by-step how we convert BERT into a Q-Former:\n * Sample an image and text pair from the dataset\n * Pass the image through the frozen VIT model to convert the image into image embeddings, shaped [197, 768]\n * Initialize \"learnable query embeddings\". These are (say 32) vector embeddings that we will use to convert the image embedding sequence into a text-grounded token embedding sequence. Notice that 32 is much lower than the original VIT embedding sequence length (197).\n * We input the text caption embeddings and the query embeddings into the first BERT layer. The layer applies self-attention on these inputs. \n \nFor now, let's assume that the query tokens only attend among themselves and the text tokens among themselves (i.e. the query tokens and the text tokens do not _see_ each other). \n\n * In the 2nd layer of the BERT, something INTERESTING happens. The two sets of embeddings go through another self-attention layer like before. But this time, we also use a cross-attention layer to contextualize the query embeddings with the ViT image embeddings we calculated earlier. \n \nOf course, normal BERT does not have any cross-attention layers, so we introduce these multi-headed cross-attention layers ourselves. \n\n * Just like this, we alternate between a pure self-attention layer (where queries and text independently self-attend among themselves) followed by a cross-attention layer (where the query embeddings attend to the frozen VIT embeddings). \n\n * In the final layer, we choose a joint embedding training loss, like ITC (Image Text Contrastive Loss) or ITM (Image Text Matching Loss) or ITG (Image Text Generation Loss etc). More on this later.\n\n\n### What does cross-attention do?\nIt contextualizes the image content with the query embeddings. You can imagine each query is trying to match a specific embedding pattern against the 197 VIT embeddings. \nFor example, if a query has a high match with a single image vector, it will capture that feature very prominently. If the query matches with a combination of vectors, it will be an average of those embeddings, and so on. \nRemember we trained 32 of these query embeddings, so you are allowing the Q-Former to learn multiple different co-activations within the image embeddings. Due to the nature of training, these coactivations are encouraged to maximize alignment between image and text.\n> **As we train the Q-Former, both the initial query embeddings and the cross-attention weights will be optimized so we can extract relevant features from the VIT image tokens.**\nThe query former embeddings are not trying to capture every detail of these 197 embeddings — instead they are trying to learn how to combine them into a compact 32 token sequence.\nNote that after the Q-Former is trained, we won't actually use the text part of the Q-Former for anything. We will simply pass the query-embeddings through the Q-former and alternatively run self-attention and cross-attention only on them.\n### Loss functions for training Q-Formers\n[![AVB](../images/006.jpg)](https://x.com/neural_avb/status/2000961550094627086)\n[AVB](https://x.com/neural_avb/status/2000961550094627086)\n[@neural_avb](https://x.com/neural_avb/status/2000961550094627086)\n·[Follow](https://x.com/intent/follow?screen_name=neural_avb)\n[](https://x.com/neural_avb/status/2000961550094627086)\nThere is SO MUCH COOL SHIT you can do just by configuring the attention mask in different ways. So icydk, I will be training a small text-only LM to have vision capabilities. As a pretraining step, I need to first train a joint image-text embedding space which will be later used[Show more](https://x.com/neural_avb/status/2000961550094627086)\n[![Image](../images/005.jpg)](https://x.com/neural_avb/status/2000961550094627086)\n[![AVB](../images/006.jpg)](https://x.com/neural_avb/status/2000823976717901829)\nAVB\n@neural_avb\nAnother VLM update. I'm doing a 2 phase training process similar in LLava/BLIP etc. The eventual goal is to tune a text-only SLM like Qwen3-0.6B to have vision capabilities. The first step is to train a \"Q-Former network\". Finished writing one. Basically I initialized with a\n[![Image](../images/007.jpg)](https://x.com/neural_avb/status/2000823976717901829)\n[4:09 PM · Dec 16, 2025](https://x.com/neural_avb/status/2000961550094627086)[](https://help.x.com/en/x-for-websites-ads-info-and-privacy)\n[301](https://x.com/intent/like?tweet_id=2000961550094627086)[Reply](https://x.com/intent/tweet?in_reply_to=2000961550094627086)Copy link\n[Read 7 replies](https://x.com/neural_avb/status/2000961550094627086)\nHow the Q-Former model is trained is actually closely related to how we attend between query and text tokens throughout the layers.\n 1. **Image-Text-Contrastive Loss** (our setup) \nFor this task, we use a **unimodal self-attention mask**. Query tokens attend among each other, text token among each other. \n \nThe loss function can be any standard CLIP-like contrastive loss function. We will align the image and text in the same embedding space. Basically, we take the output of the queries and the output of the text encoder and compute their similarity. If an image and a caption belong together, we want their vectors to be as close as possible. \n \nThis forces the queries to extract a \"global\" visual representation that matches the general theme of the text without actually looking at the words yet. \n\n 2. **Image-Text Matching Loss** (ITM) \nThis uses a **bi-directional self-attention mask**. Here, every query token is allowed to see every text token, and every text token can see every query! \n \nFor the loss fucntion, we use a binary classification task where the model has to predict: \"Is this image and this text a match—Yes or No?\". Binary cross-entropy loss. \n \nBecause the modalities are fully mixed, the model can do fine-grained comparisons. The queries can look at specific objects in the image (via the cross-attention) and verify if they correspond to specific words in the text. This is much more detailed than the contrastive loss and ensures the 32 tokens are capturing localized details.\n 3. **Image-Text Generation Loss (ITG)** \nFinally, we have the generative task. For this, we use a **multimodal causal mask**. The queries can still see each other, but the text tokens are now treated like a sequence. Each text token can see all 32 query tokens - which act as a visual prefix. But they can only see the text tokens that came _before_ it. \n \nFor the loss function, we just train the model to predict the next token in the caption. By forcing the model to literally \"write\" the description based on the queries, we ensure that those 32 tokens contain every bit of visual information necessary for a language model to understand the scene.\n\n\nFor my project, I just used the simplest — ITC. For a small dataset like I was using, this was the easiest way! BLIP-2 recommends to use a mixture of all these training methods. The github repo shared at the end of the article provides the recipe to use any of the above attention schemes.\n![](../images/body-c57b23ef1749304be7c0.jpeg)\nA trained Q-Former model learns to match similar text and image pairs\nIn the next section, we will do the final step — training the VLM!\n## 3. The Language Layer\nNow comes the final step. We will use the VIT and the Q-Former to make a language model into a vision model. I picked one of the smallest instruction-tuned language models — the [SmolLM2-135M](http://HuggingFaceTB/SmolLM2-135M). Thankfully, this part is not as complicated as the Q-Former training.\nWe have the image embeddings (coming from the VIT and the Q-Former), and we have the text tokens (coming from the SmolLM tokenizer). Let's see some details.\n![](../images/body-4776025afe5609d58a34.png)\nFrom the BLIP-2 Paper\n * We sample an image and a caption from our dataset \n\n * We randomly pick from a list of simple system prompts, similar to: \"_You are a helpful assistant. Respond truthfully to the user_.\" \n \nWe also pick the user query from a list of prompts, for example: \"_What do you see in this image?_ \" \n \nWe tokenize the output captions sampled from the dataset as well. \n \nThese 3 things form the text tokens. We tokenize them all using the SmolLM2 tokenizer, but we are not going to insert it into the LLM just yet — we must process the image first. \n\n * We pass the image through the frozen VIT, then through the Q-Former (again, note that the text captions are not passed into the Q-Former, only the image pipeline is executed) \n\n * We introduce a small MLP layer that converts the Q-Former output into new embeddings that are of the same shape as the LLM's expected embedding size. As we train, this MLP layer will map the Q-Former embedding into the LLM embedding space. \n\n * Now that we have the text tokens sequence and the new image embeddings (VIT -> Q-Former -> MLP). We will pass the text tokens through the LLM's native embedding layer. We sandwich the text and image embeddings in the following sequence: \n \n \n...and forward pass it through the rest of the LLM. \n\n * Why that specific sequence? Since autoregressive LLMs use causal masking, we will essentially be training models to generate the output (caption) sequence given the entire prefix (system prompt, user prompt, and the image embeddings). \n\n * We add LoRA adapters (Low-Rank Adaptation Matrices) instead of training the entire LLM from scratch. By wrapping our LLM with LoRA, we freeze the original millions of parameters and only train tiny, low-rank matrices injected into the attention layers. This makes the model trainable on consumer hardware while keeping all that pre-existing intelligence intact. \n\n * And that’s it! We pass these stitched embeddings and labels into the LLM. The model attends to the text instruction _and_ the visual tokens simultaneously, and thanks to LoRA, it learns how to update its internal wiring to understand this new visual language. Only the Q-Former layers, the MLP layer, and the LORA adapters are trained. Everything else is kept frozen.\n\n\n![](../images/body-0bc2b91163814acecae6.png)\nSome results! You can find more results in the youtube video mentioned at the end of the article!\nAfter training this for just a few hours on a small subset of data, the trained VLM can now see the images and generate text about it. Machine Learning is so beautiful when it works.\n## In summary\nYou can find the full github repository here:\n\n \nAnd watch the youtube video here:\nPlease accept cookies to access this content\nLet's summarize all the modules in Vision Language pipelines. \n * A vision backbone (like the VIT) that takes image input and converts it into embeddings\n * An adapter layer (like the Q-Former) that grounds the image with text\n * An LLM that we train to consolidate the text and image embeddings to learn the language of vision\n\n\n**My Patreon:** \n\n**My YouTube channel:** \n\n**Follow me on Twitter:** \n\n**I am building Paper Breakdown, a place to study research papers** \n\n**Read my articles:** \n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-849620ef9680d77fc51e.jpeg)\nWritten By\n## [Avishek Biswas](https://towardsdatascience.com/author/neural.avb/)\n[See all from Avishek Biswas](https://towardsdatascience.com/author/neural.avb/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Large Multimodal Models](https://towardsdatascience.com/tag/large-multimodal-models/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Multimodal Learning](https://towardsdatascience.com/tag/multimodal-learning/)[Vision Language Model](https://towardsdatascience.com/tag/vision-language-model/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fhow-vision-language-models-are-trained-from-scratch%2F&title=How%20Vision%20Language%20Models%20Are%20Trained%20from%20%E2%80%9CScratch%E2%80%9D)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fhow-vision-language-models-are-trained-from-scratch%2F&title=How%20Vision%20Language%20Models%20Are%20Trained%20from%20%E2%80%9CScratch%E2%80%9D)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fhow-vision-language-models-are-trained-from-scratch%2F&text=How%20Vision%20Language%20Models%20Are%20Trained%20from%20%E2%80%9CScratch%E2%80%9D)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](../images/008.jpg)](https://towardsdatascience.com/summarising-best-practices-for-prompt-engineering-c5e86c483af4/)\n## [Summarising Best Practices for Prompt Engineering](https://towardsdatascience.com/summarising-best-practices-for-prompt-engineering-c5e86c483af4/)\n[ChatGPT](https://towardsdatascience.com/category/artificial-intelligence/chatgpt/)\nHow to build your own LLM-based application using OpenAI API\n[Dmytro Nikolaiev (Dimid)](https://towardsdatascience.com/author/andimid/)\nMay 28, 202314 min read\n * [![](../images/009.png)](https://towardsdatascience.com/to-make-an-impact-in-any-industry-domain-knowledge-is-critical-bdd93a304a1d/)\n## [To Make an Impact in Any Industry, Domain Knowledge Is Critical](https://towardsdatascience.com/to-make-an-impact-in-any-industry-domain-knowledge-is-critical-bdd93a304a1d/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nAbiodun Olaoye discusses his work on renewable energy\n[TDS Editors](https://towardsdatascience.com/author/towardsdatascience/)\nDecember 14, 20225 min read\n * [![](../images/010.png)](https://towardsdatascience.com/how-to-build-an-llm-application-360848c957db/)\n## [How to Build an LLM Application](https://towardsdatascience.com/how-to-build-an-llm-application-360848c957db/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nUsing Langchain and OpenAI to Build LLM Centred Apps\n[John Adeojo](https://towardsdatascience.com/author/johnadeojo/)\nJuly 21, 20236 min read\n * [![](../images/011.jpg)](https://towardsdatascience.com/conditional-random-field-tutorial-in-pytorch-ca0d04499463/)\n## [Conditional Random Field Tutorial in PyTorch 🔥](https://towardsdatascience.com/conditional-random-field-tutorial-in-pytorch-ca0d04499463/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nLet's say I have two identical dice, but one is fair and the other is...\n[Freddy Boulton](https://towardsdatascience.com/author/freddyalfonsoboulton/)\nMay 3, 20189 min read\n * [![](../images/012.webp)](https://towardsdatascience.com/gpt-6-astra-just-hit-openais-highest-cybersecurity-risk-level/)\n## [GPT-6 Astra Just Hit OpenAI's Highest Cybersecurity Risk Level](https://towardsdatascience.com/gpt-6-astra-just-hit-openais-highest-cybersecurity-risk-level/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nWhy this matters less for what Astra can do and more for what every other model hasn't been tested for\n[Benjamin Nweke](https://towardsdatascience.com/author/benjaminnweke/)\nSeptember 21, 20267 min read\n * [![](../images/013.png)](https://towardsdatascience.com/the-ultimate-guide-to-finding-outliers-in-your-time-series-data-part-1-1bf81e09ade4-2/)\n## [The Ultimate Guide to Finding Outliers in Your Time-Series Data (Part 1)](https://towardsdatascience.com/the-ultimate-guide-to-finding-outliers-in-your-time-series-data-part-1-1bf81e09ade4-2/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nEffective statistical methods and tools for outlier detection in time-series analysis\n[Sara Nobrega](https://towardsdatascience.com/author/saranobregafn/)\nMay 21, 202419 min read\n * [![](../images/014.png)](https://towardsdatascience.com/announcement-tensorflow-2-0-is-coming-7e17e8e333e3/)\n## [Announcement: TensorFlow 2.0 is coming!](https://towardsdatascience.com/announcement-tensorflow-2-0-is-coming-7e17e8e333e3/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nAnd TensorFlow 1.x is going to be...\n[Admond Lee](https://towardsdatascience.com/author/admond1994/)\nNovember 23, 20183 min read\n * [![](../images/015.gif)](https://towardsdatascience.com/empowering-llms-to-think-deeper-by-erasing-thoughts/)\n## [Empowering LLMs to Think Deeper by Erasing Thoughts](https://towardsdatascience.com/empowering-llms-to-think-deeper-by-erasing-thoughts/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nWe examine the fundamental limits of CoT-based test-time scaling and explore efficient methods for deep reasoning in LLMs.\n[Chenxiao Yang](https://towardsdatascience.com/author/chenxiao.yang/)\nMay 12, 20259 min read\n * [![](../images/016.png)](https://towardsdatascience.com/llms-weapons-of-mass-disinformation-4def0dc3dc7/)\n## [LLMs, new Weapons of Mass Disinformation?](https://towardsdatascience.com/llms-weapons-of-mass-disinformation-4def0dc3dc7/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nYou thought the 2016 Brexit and US presidential campaigns were bad? Think again.\n[Anthony Mensier](https://towardsdatascience.com/author/anthony.mensier/)\nMay 29, 202318 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500) Productivity Tips, Data Career Insights, and Other Recent Must-Reads ](https://towardsdatascience.com/productivity-tips-data-career-insights-and-other-recent-must-reads-5c47eb7a2fe0/ \"Productivity Tips, Data Career Insights, and Other Recent Must-Reads\")\n\n\n", "metadata": { "title": "How Vision Language Models Are Trained from “Scratch” | Towards Data Science", "description": "A deep dive into exactly how text-only language models are fine-tuned to *see* images", "keywords": null, "author": "Avishek Biswas", "og:locale": "en_US", "og:type": "article", "og:title": "How Vision Language Models Are Trained from “Scratch”", "og:description": "A deep dive into exactly how text-only language models are fine-tuned to *see* images", "og:url": "https://towardsdatascience.com/how-vision-language-models-are-trained-from-scratch/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/03/Article_Thumbnail.jpg", "og:image:width": "1280", "og:image:height": "720", "og:image:type": "image/jpeg", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Avishek Biswas", "twitter:label2": "Est. reading time", "twitter:data2": "11 min read", "article:published_time": "2026-03-13T11:30:09.000Z", "article:modified_time": "2026-03-13T11:30:09.000Z", "article:author": "https://towardsdatascience.com/author/neural.avb/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 202, "error": "" }, "code_blocks": [], "images": [ { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:05:35Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:05:35Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:05:35Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:05:35Z" } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/c/TowardsDataScience", "kind": "channel_reference", "status": "reference_only" }, { "url": "https://www.youtube.com/@avb_fj", "kind": "channel_reference", "status": "reference_only" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/large-language-models/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/neural.avb/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/neural-avb/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/deep-dives/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/large-multimodal-models/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/multimodal-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/vision-language-model/" }, { "type": "internal", "href": "https://towardsdatascience.com/summarising-best-practices-for-prompt-engineering-c5e86c483af4/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/chatgpt/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/andimid/" }, { "type": "internal", "href": "https://towardsdatascience.com/to-make-an-impact-in-any-industry-domain-knowledge-is-critical-bdd93a304a1d/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/towardsdatascience/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://roadmap.sh/ai-engineer" }, { "type": "external", "href": "https://huggingface.co/google/vit-base-patch16-224" }, { "type": "external", "href": "https://arxiv.org/abs/2301.12597" }, { "type": "external", "href": "https://x.com/neural_avb/status/2000961550094627086" }, { "type": "external", "href": "https://x.com/intent/follow?screen_name=neural_avb" }, { "type": "external", "href": "https://x.com/neural_avb/status/2000823976717901829" }, { "type": "external", "href": "https://help.x.com/en/x-for-websites-ads-info-and-privacy" }, { "type": "external", "href": "https://x.com/intent/like?tweet_id=2000961550094627086" }, { "type": "external", "href": "https://x.com/intent/tweet?in_reply_to=2000961550094627086" }, { "type": "external", "href": "http://huggingfacetb/SmolLM2-135M" }, { "type": "external", "href": "https://github.com/avbiswas/vlm" }, { "type": "external", "href": "https://www.patreon.com/NeuralBreakdownwithAVB" }, { "type": "external", "href": "https://www.youtube.com/@avb_fj" }, { "type": "external", "href": "https://x.com/neural_avb" }, { "type": "external", "href": "https://paperbreakdown.com" }, { "type": "external", "href": "https://www.facebook.com/sharer/sharer.php?title=How+Vision+Language+Models+Are+Trained+from+%E2%80%9CScratch%E2%80%9D&u=https%3A%2F%2Ftowardsdatascience.com%2Fhow-vision-language-models-are-trained-from-scratch%2F" }, { "type": "external", "href": "https://www.linkedin.com/shareArticle?mini=true&title=How+Vision+Language+Models+Are+Trained+from+%E2%80%9CScratch%E2%80%9D&url=https%3A%2F%2Ftowardsdatascience.com%2Fhow-vision-language-models-are-trained-from-scratch%2F" } ], "repair_version": 2, "repaired_at": "2026-10-20T17:32:33Z", "source_url": "https://towardsdatascience.com/how-vision-language-models-are-trained-from-scratch/", "delivery_status": "unverified", "discovery_status": "http_403", "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片", "repair_error": "", "actual_video_files": [], "expected_saved_body_images": 4, "image_discovery": "saved_markdown", "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\n# How Vision Language Models Are Trained from “Scratch”\nA deep dive into exactly how text-only language models are fine-tuned to *see* images\n[Avishek Biswas](https://towardsdatascience.com/author/neural.avb/)\nMarch 13, 202611 min read\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/03/Article_Thumbnail.jpg)\nRecently, I went on an adventure to transform a small text-only language model and gift it the power of vision. This article is to summarize all my learnings, and take a deeper look at the network architectures behind modern Vision Language Models.\n_The code is open-source, you can check out the GitHub link at the end of the article. There is also a 30-minute companion YouTube video that explains the whole article in a visually rich format._ \n \n_Also, unless otherwise mentioned, all images in this article are produced by the author._\n**Wait, are you really going to be \"training from scratch\"?**\nYes... I mean no... _it's a bit nuanced_.\n**Research labs in 2026 do not train multimodal models from \"scratch\" anymore.** It is simply too expensive to teach a model vision and (textual) language at the same time! It requires more data, compute, time, and money. Also, it often leads to poorer results.\nInstead, labs take existing pretrained text-only language models, and finetune it to give \"vision capabilities\". In theory (and practice), this is way more compute-efficient.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/03/Article_Thumbnail-1024x576.png)\nLet's talk about Vision Language Models!\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\n## The standard architecture\nAlthough it's less data-intensive, finetuning text-only LMs to _suddenly_ start seeing images _would_ surely open its own can of worms.\n 1. **how do we embed the image,** i.e. convert it into numerical representations that a neural network can understand?\n 2. how do we tune the image embeddings to be **compatible with text**?\n 3. how do we adjust the weights of the text model so that it retains it's previous world knowledge, but also **generate text from image embeddings**?\n\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/03/CleanShot-2026-03-11-at-03.10.50-1024x293.png)\nThe image embeddings coming out of the VIT and Q-Former passes through an MLP layer. Followed by a series of Decoder layer and trainable LORA adapters.\nThese modules are:\n 1. **The Image Backbone:** A model that converts raw images into embeddings.\n 2. **The Adapter Layer:** These are models that convert the image embeddings into a \"text-compatible\" embedding. This is the main challenging part - what architectures to use, what loss functions, etc.\n 3. **The Language Layer:** The language model we will train to input the adapted embeddings and generate text from it.\n\n\nLet's discuss them one by one.\n## 1. The Image Backbone\nThe goal of your image backbone is simple:\n**Input:** A raw 2D pixel map/image.\n**Output:** A sequence of vector embeddings representing the image\nIn general, we just use an off-the-shelf image model that has been pretrained with massive corpus of images, generally on self-supervised tasks.\nYou can use a Convolutional Neural Network (like a ResNet) to use as an image backbone. But modern state-of-the-art VLMs have almost entirely shifted to ViTs because they scale better with data and are more flexible for multimodal fusion.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/03/image-166-edited.png)\nVision Transformers input an image, extract patches out of it, and then pass them through bidirectional self-attention layers. This contextualizes each patch embedding with each other to form a contextual sequence of image embeddings.\n_**Do I train the image backbone or keep it frozen?**_\nIn most VLM research, there is a clear trend toward keeping backbones static (frozen) to save costs. Also, vision-language training generally needs paired image-text datasets. Since these datasets are always much smaller than the VIT's pretraining dataset, finetuning the backbone often leads to overfitting and degraded performance.\nBy keeping these weights frozen, we are basically transferring the ownership of vision-language learning to latter parts of the network (i.e. the adapter layer and the text backbone).\nIn my experiments, I used the [ViT-Base model](https://huggingface.co/google/vit-base-patch16-224). This model takes the image input, splits it into patches of 16x16 images and applies self-attention on them to generate an embedding sequence of 197 vectors. Each vector is 768 dimesnions long (the embedding size of the VIT).\n## 2. The Adapter Layer\nThis is where we are going to spend the majority of our time. We have converted images into embeddings already, but those embeddings are completely text-unaware.\nVision Transformers are pre-trained purely on image pixels. Not on their captions, or any local textual features. The role of the adapter is to ground the pixel-based-image-embeddings into a (often shorter sequence of) text-based-image-embeddings.\nThere are many ways to do this, like using CLIP models, but we are going to look at one of the more popular approaches — the Query Former of the Q-Former.\n### Q-Former\nAlright — so what is a Q-Former? A Q-Former or the Query-Former was introduced in the [BLIP-2 paper](https://arxiv.org/abs/2301.12597).\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/03/CleanShot-2026-03-11-at-03.19.47@2x-1024x819.png)\nFrom the BLIP-2 Paper\n**How do I train a Q-Former?**\nStandard Q-Formers can be trained using any multimodal image-text pair dataset. For example, you can use the Conceptual Captions dataset, which is a massive corpus of images and their corresponding captions. In my project, I took just 50,000 pairs to train the Q-Former.\nYou can train a Q-Former from scratch, but the BLIP-2 recommendation is to use an pretrained BERT model. So that's what we will do.\nAt a high level, here is our basic gameplan:\n * Train a multi-modal joint embedding space. A space where text and images \"know\" each other.\n * Basically, we will input pairs of image and captions — and embed both of them in the same joint space.\n * Images and incompatible captions will be mapped in separate places in this new embedding space, and compatible captions will be mapped close to each other.\n\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/03/CleanShot-2026-03-11-at-03.05.15-1-1024x825.png)\nLayer 1: Self Attention\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/03/CleanShot-2026-03-11-at-03.06.32-1-1024x695.png)\nLayer 2: Self-Attention & Cross-Attention with the VIT Features\n \n**Setting up cross-attention layers**\nThere's a problem — BERT models are purely text models. They have no idea what an image is.\nSo, our objective is to first introduce new cross-attention layers to marry the vision embeddings coming out of the VIT and the text embeddings from BERT. Let's break down step-by-step how we convert BERT into a Q-Former:\n * Sample an image and text pair from the dataset\n * Pass the image through the frozen VIT model to convert the image into image embeddings, shaped [197, 768]\n * Initialize \"learnable query embeddings\". These are (say 32) vector embeddings that we will use to convert the image embedding sequence into a text-grounded token embedding sequence. Notice that 32 is much lower than the original VIT embedding sequence length (197).\n * We input the text caption embeddings and the query embeddings into the first BERT layer. The layer applies self-attention on these inputs. \n \nFor now, let's assume that the query tokens only attend among themselves and the text tokens among themselves (i.e. the query tokens and the text tokens do not _see_ each other). \n\n * In the 2nd layer of the BERT, something INTERESTING happens. The two sets of embeddings go through another self-attention layer like before. But this time, we also use a cross-attention layer to contextualize the query embeddings with the ViT image embeddings we calculated earlier. \n \nOf course, normal BERT does not have any cross-attention layers, so we introduce these multi-headed cross-attention layers ourselves. \n\n * Just like this, we alternate between a pure self-attention layer (where queries and text independently self-attend among themselves) followed by a cross-attention layer (where the query embeddings attend to the frozen VIT embeddings). \n\n * In the final layer, we choose a joint embedding training loss, like ITC (Image Text Contrastive Loss) or ITM (Image Text Matching Loss) or ITG (Image Text Generation Loss etc). More on this later.\n\n\n### What does cross-attention do?\nIt contextualizes the image content with the query embeddings. You can imagine each query is trying to match a specific embedding pattern against the 197 VIT embeddings. \nFor example, if a query has a high match with a single image vector, it will capture that feature very prominently. If the query matches with a combination of vectors, it will be an average of those embeddings, and so on. \nRemember we trained 32 of these query embeddings, so you are allowing the Q-Former to learn multiple different co-activations within the image embeddings. Due to the nature of training, these coactivations are encouraged to maximize alignment between image and text.\n> **As we train the Q-Former, both the initial query embeddings and the cross-attention weights will be optimized so we can extract relevant features from the VIT image tokens.**\nThe query former embeddings are not trying to capture every detail of these 197 embeddings — instead they are trying to learn how to combine them into a compact 32 token sequence.\nNote that after the Q-Former is trained, we won't actually use the text part of the Q-Former for anything. We will simply pass the query-embeddings through the Q-former and alternatively run self-attention and cross-attention only on them.\n### Loss functions for training Q-Formers\n[![AVB](https://pbs.twimg.com/profile_images/2015375309147611136/WKvfQ-oV_normal.jpg)](https://x.com/neural_avb/status/2000961550094627086)\n[AVB](https://x.com/neural_avb/status/2000961550094627086)\n[@neural_avb](https://x.com/neural_avb/status/2000961550094627086)\n·[Follow](https://x.com/intent/follow?screen_name=neural_avb)\n[](https://x.com/neural_avb/status/2000961550094627086)\nThere is SO MUCH COOL SHIT you can do just by configuring the attention mask in different ways. So icydk, I will be training a small text-only LM to have vision capabilities. As a pretraining step, I need to first train a joint image-text embedding space which will be later used[Show more](https://x.com/neural_avb/status/2000961550094627086)\n[![Image](https://pbs.twimg.com/media/G8TXBn3b0AAPQEW?format=jpg&name=small)](https://x.com/neural_avb/status/2000961550094627086)\n[![AVB](https://pbs.twimg.com/profile_images/2015375309147611136/WKvfQ-oV_normal.jpg)](https://x.com/neural_avb/status/2000823976717901829)\nAVB\n@neural_avb\nAnother VLM update. I'm doing a 2 phase training process similar in LLava/BLIP etc. The eventual goal is to tune a text-only SLM like Qwen3-0.6B to have vision capabilities. The first step is to train a \"Q-Former network\". Finished writing one. Basically I initialized with a\n[![Image](https://pbs.twimg.com/media/G8RYlwYa4AQU6VW?format=jpg&name=small)](https://x.com/neural_avb/status/2000823976717901829)\n[4:09 PM · Dec 16, 2025](https://x.com/neural_avb/status/2000961550094627086)[](https://help.x.com/en/x-for-websites-ads-info-and-privacy)\n[301](https://x.com/intent/like?tweet_id=2000961550094627086)[Reply](https://x.com/intent/tweet?in_reply_to=2000961550094627086)Copy link\n[Read 7 replies](https://x.com/neural_avb/status/2000961550094627086)\nHow the Q-Former model is trained is actually closely related to how we attend between query and text tokens throughout the layers.\n 1. **Image-Text-Contrastive Loss** (our setup) \nFor this task, we use a **unimodal self-attention mask**. Query tokens attend among each other, text token among each other. \n \nThe loss function can be any standard CLIP-like contrastive loss function. We will align the image and text in the same embedding space. Basically, we take the output of the queries and the output of the text encoder and compute their similarity. If an image and a caption belong together, we want their vectors to be as close as possible. \n \nThis forces the queries to extract a \"global\" visual representation that matches the general theme of the text without actually looking at the words yet. \n\n 2. **Image-Text Matching Loss** (ITM) \nThis uses a **bi-directional self-attention mask**. Here, every query token is allowed to see every text token, and every text token can see every query! \n \nFor the loss fucntion, we use a binary classification task where the model has to predict: \"Is this image and this text a match—Yes or No?\". Binary cross-entropy loss. \n \nBecause the modalities are fully mixed, the model can do fine-grained comparisons. The queries can look at specific objects in the image (via the cross-attention) and verify if they correspond to specific words in the text. This is much more detailed than the contrastive loss and ensures the 32 tokens are capturing localized details.\n 3. **Image-Text Generation Loss (ITG)** \nFinally, we have the generative task. For this, we use a **multimodal causal mask**. The queries can still see each other, but the text tokens are now treated like a sequence. Each text token can see all 32 query tokens - which act as a visual prefix. But they can only see the text tokens that came _before_ it. \n \nFor the loss function, we just train the model to predict the next token in the caption. By forcing the model to literally \"write\" the description based on the queries, we ensure that those 32 tokens contain every bit of visual information necessary for a language model to understand the scene.\n\n\nFor my project, I just used the simplest — ITC. For a small dataset like I was using, this was the easiest way! BLIP-2 recommends to use a mixture of all these training methods. The github repo shared at the end of the article provides the recipe to use any of the above attention schemes.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/03/qformer-990x1024.jpeg)\nA trained Q-Former model learns to match similar text and image pairs\nIn the next section, we will do the final step — training the VLM!\n## 3. The Language Layer\nNow comes the final step. We will use the VIT and the Q-Former to make a language model into a vision model. I picked one of the smallest instruction-tuned language models — the [SmolLM2-135M](http://HuggingFaceTB/SmolLM2-135M). Thankfully, this part is not as complicated as the Q-Former training.\nWe have the image embeddings (coming from the VIT and the Q-Former), and we have the text tokens (coming from the SmolLM tokenizer). Let's see some details.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/03/CleanShot-2026-03-11-at-03.20.32@2x-1024x150.png)\nFrom the BLIP-2 Paper\n * We sample an image and a caption from our dataset \n\n * We randomly pick from a list of simple system prompts, similar to: \"_You are a helpful assistant. Respond truthfully to the user_.\" \n \nWe also pick the user query from a list of prompts, for example: \"_What do you see in this image?_ \" \n \nWe tokenize the output captions sampled from the dataset as well. \n \nThese 3 things form the text tokens. We tokenize them all using the SmolLM2 tokenizer, but we are not going to insert it into the LLM just yet — we must process the image first. \n\n * We pass the image through the frozen VIT, then through the Q-Former (again, note that the text captions are not passed into the Q-Former, only the image pipeline is executed) \n\n * We introduce a small MLP layer that converts the Q-Former output into new embeddings that are of the same shape as the LLM's expected embedding size. As we train, this MLP layer will map the Q-Former embedding into the LLM embedding space. \n\n * Now that we have the text tokens sequence and the new image embeddings (VIT -> Q-Former -> MLP). We will pass the text tokens through the LLM's native embedding layer. We sandwich the text and image embeddings in the following sequence: \n \n \n...and forward pass it through the rest of the LLM. \n\n * Why that specific sequence? Since autoregressive LLMs use causal masking, we will essentially be training models to generate the output (caption) sequence given the entire prefix (system prompt, user prompt, and the image embeddings). \n\n * We add LoRA adapters (Low-Rank Adaptation Matrices) instead of training the entire LLM from scratch. By wrapping our LLM with LoRA, we freeze the original millions of parameters and only train tiny, low-rank matrices injected into the attention layers. This makes the model trainable on consumer hardware while keeping all that pre-existing intelligence intact. \n\n * And that’s it! We pass these stitched embeddings and labels into the LLM. The model attends to the text instruction _and_ the visual tokens simultaneously, and thanks to LoRA, it learns how to update its internal wiring to understand this new visual language. Only the Q-Former layers, the MLP layer, and the LORA adapters are trained. Everything else is kept frozen.\n\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/03/CleanShot-2026-03-11-at-02.49.59-1024x568.png)\nSome results! You can find more results in the youtube video mentioned at the end of the article!\nAfter training this for just a few hours on a small subset of data, the trained VLM can now see the images and generate text about it. Machine Learning is so beautiful when it works.\n## In summary\nYou can find the full github repository here:\n\n \nAnd watch the youtube video here:\nPlease accept cookies to access this content\nLet's summarize all the modules in Vision Language pipelines. \n * A vision backbone (like the VIT) that takes image input and converts it into embeddings\n * An adapter layer (like the Q-Former) that grounds the image with text\n * An LLM that we train to consolidate the text and image embeddings to learn the language of vision\n\n\n**My Patreon:** \n\n**My YouTube channel:** \n\n**Follow me on Twitter:** \n\n**I am building Paper Breakdown, a place to study research papers** \n\n**Read my articles:** \n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657599061_2dgnrr.jpeg)\nWritten By\n## [Avishek Biswas](https://towardsdatascience.com/author/neural.avb/)\n[See all from Avishek Biswas](https://towardsdatascience.com/author/neural.avb/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Large Multimodal Models](https://towardsdatascience.com/tag/large-multimodal-models/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Multimodal Learning](https://towardsdatascience.com/tag/multimodal-learning/)[Vision Language Model](https://towardsdatascience.com/tag/vision-language-model/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fhow-vision-language-models-are-trained-from-scratch%2F&title=How%20Vision%20Language%20Models%20Are%20Trained%20from%20%E2%80%9CScratch%E2%80%9D)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fhow-vision-language-models-are-trained-from-scratch%2F&title=How%20Vision%20Language%20Models%20Are%20Trained%20from%20%E2%80%9CScratch%E2%80%9D)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fhow-vision-language-models-are-trained-from-scratch%2F&text=How%20Vision%20Language%20Models%20Are%20Trained%20from%20%E2%80%9CScratch%E2%80%9D)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/05/0jtQT-m5tMZJgAEWq.jpg)](https://towardsdatascience.com/summarising-best-practices-for-prompt-engineering-c5e86c483af4/)\n## [Summarising Best Practices for Prompt Engineering](https://towardsdatascience.com/summarising-best-practices-for-prompt-engineering-c5e86c483af4/)\n[ChatGPT](https://towardsdatascience.com/category/artificial-intelligence/chatgpt/)\nHow to build your own LLM-based application using OpenAI API\n[Dmytro Nikolaiev (Dimid)](https://towardsdatascience.com/author/andimid/)\nMay 28, 202314 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/12/1bJjL_RCnMZjrY6IcdpUfHg.png)](https://towardsdatascience.com/to-make-an-impact-in-any-industry-domain-knowledge-is-critical-bdd93a304a1d/)\n## [To Make an Impact in Any Industry, Domain Knowledge Is Critical](https://towardsdatascience.com/to-make-an-impact-in-any-industry-domain-knowledge-is-critical-bdd93a304a1d/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nAbiodun Olaoye discusses his work on renewable energy\n[TDS Editors](https://towardsdatascience.com/author/towardsdatascience/)\nDecember 14, 20225 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1oN3dnz84eKQ1OLUYDFC68A.png)](https://towardsdatascience.com/how-to-build-an-llm-application-360848c957db/)\n## [How to Build an LLM Application](https://towardsdatascience.com/how-to-build-an-llm-application-360848c957db/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nUsing Langchain and OpenAI to Build LLM Centred Apps\n[John Adeojo](https://towardsdatascience.com/author/johnadeojo/)\nJuly 21, 20236 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2018/05/0i-IyD2Yjd2fOBsdP.jpg)](https://towardsdatascience.com/conditional-random-field-tutorial-in-pytorch-ca0d04499463/)\n## [Conditional Random Field Tutorial in PyTorch 🔥](https://towardsdatascience.com/conditional-random-field-tutorial-in-pytorch-ca0d04499463/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nLet's say I have two identical dice, but one is fair and the other is...\n[Freddy Boulton](https://towardsdatascience.com/author/freddyalfonsoboulton/)\nMay 3, 20189 min read\n * [![](https://assets.insightmediagroup.io/media/1789666049105_b6e9pj.webp)](https://towardsdatascience.com/gpt-6-astra-just-hit-openais-highest-cybersecurity-risk-level/)\n## [GPT-6 Astra Just Hit OpenAI's Highest Cybersecurity Risk Level](https://towardsdatascience.com/gpt-6-astra-just-hit-openais-highest-cybersecurity-risk-level/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nWhy this matters less for what Astra can do and more for what every other model hasn't been tested for\n[Benjamin Nweke](https://towardsdatascience.com/author/benjaminnweke/)\nSeptember 21, 20267 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1-bJviPoC9q6taoVrvQ29OQ.png)](https://towardsdatascience.com/the-ultimate-guide-to-finding-outliers-in-your-time-series-data-part-1-1bf81e09ade4-2/)\n## [The Ultimate Guide to Finding Outliers in Your Time-Series Data (Part 1)](https://towardsdatascience.com/the-ultimate-guide-to-finding-outliers-in-your-time-series-data-part-1-1bf81e09ade4-2/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nEffective statistical methods and tools for outlier detection in time-series analysis\n[Sara Nobrega](https://towardsdatascience.com/author/saranobregafn/)\nMay 21, 202419 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2018/11/07EiPGNpgTSMHpoEj.png)](https://towardsdatascience.com/announcement-tensorflow-2-0-is-coming-7e17e8e333e3/)\n## [Announcement: TensorFlow 2.0 is coming!](https://towardsdatascience.com/announcement-tensorflow-2-0-is-coming-7e17e8e333e3/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nAnd TensorFlow 1.x is going to be...\n[Admond Lee](https://towardsdatascience.com/author/admond1994/)\nNovember 23, 20183 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/combined_animation.gif)](https://towardsdatascience.com/empowering-llms-to-think-deeper-by-erasing-thoughts/)\n## [Empowering LLMs to Think Deeper by Erasing Thoughts](https://towardsdatascience.com/empowering-llms-to-think-deeper-by-erasing-thoughts/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nWe examine the fundamental limits of CoT-based test-time scaling and explore efficient methods for deep reasoning in LLMs.\n[Chenxiao Yang](https://towardsdatascience.com/author/chenxiao.yang/)\nMay 12, 20259 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/05/1bB9wBT8Dgevbd_W-qcHcuw.png)](https://towardsdatascience.com/llms-weapons-of-mass-disinformation-4def0dc3dc7/)\n## [LLMs, new Weapons of Mass Disinformation?](https://towardsdatascience.com/llms-weapons-of-mass-disinformation-4def0dc3dc7/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nYou thought the 2016 Brexit and US presidential campaigns were bad? Think again.\n[Anthony Mensier](https://towardsdatascience.com/author/anthony.mensier/)\nMay 29, 202318 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500) Productivity Tips, Data Career Insights, and Other Recent Must-Reads ](https://towardsdatascience.com/productivity-tips-data-career-insights-and-other-recent-must-reads-5c47eb7a2fe0/ \"Productivity Tips, Data Career Insights, and Other Recent Must-Reads\")\n\n\n", "body_base_path": "attachments/ccb16b00665a/pages/", "body_markdown_path": "attachments/ccb16b00665a/pages/archived-body.md", "local_html": "attachments/ccb16b00665a/pages/archived-body.html", "body_source": "prior_capture" }