![Revisit consent button](../images/body-df29beacd7496220cc94.svg) We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg) We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data. * Cookie BCTempID * Duration 10 minutes * Description No description available. * Cookie __cf_bm * Duration 1 hour * Description This cookie, set by Cloudflare, is used to support Cloudflare Bot Management. * Cookie AWSALBCORS * Duration 7 days * Description Amazon Web Services set this cookie for load balancing. * Cookie _cfuvid * Duration session * Description Cloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services * Cookie li_gc * Duration 6 months * Description Linkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes. * Cookie __hssrc * Duration session * Description This cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session. * Cookie __hssc * Duration 1 hour * Description HubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie. * Cookie wpEmojiSettingsSupports * Duration session * Description WordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly. * Cookie BCSessionID * Duration 1 year 1 month 4 days * Description Blueconic sets this cookie as a unique identifier for the BlueConic profile. * Cookie _octo * Duration 1 year * Description No description available. * Cookie logged_in * Duration 1 year * Description No description available. * Cookie __Secure-YEC * Duration past * Description YouTube sets this cookie to stores the user's video player preferences using embedded YouTube video * Cookie __eoi * Duration 6 months * Description Description is currently not available. * Cookie AWSALBTGCORS * Duration 7 days * Description No description available. * Cookie login-status-p * Duration past * Description Description is currently not available. * Cookie AWSALBTG * Duration 7 days * Description No description available. * Cookie csrf_token * Duration session * Description No description available. * Cookie token_v2 * Duration 1 day * Description Description is currently not available. * Cookie D * Duration 1 year * Description Description is currently not available. * Cookie PHPSESSID * Duration session * Description This cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed. * Cookie VISITOR_PRIVACY_METADATA * Duration 6 months * Description YouTube sets this cookie to store the user's cookie consent state for the current domain. * Cookie cookietest * Duration session * Description The cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience. * Cookie __Host-airtable-session * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie __Host-airtable-session.sig * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie m * Duration 1 year 1 month 4 days * Description Stripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly. * Cookie BIGipServer* * Duration session * Description Marketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. * Cookie __cfruid * Duration session * Description Cloudflare sets this cookie to identify trusted web traffic. * Cookie _GRECAPTCHA * Duration 6 months * Description Google Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks. * Cookie __Secure-YNID * Duration 6 months * Description Google cookie used to protect user security and prevent fraud, especially during the login process. * Cookie cookieyes-consent * Duration 1 year * Description CookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors. Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features. * Cookie lidc * Duration 1 day * Description LinkedIn sets the lidc cookie to facilitate data center selection. * Cookie brw * Duration 1 year * Description No description available. * Cookie brwConsent * Duration 5 minutes * Description Description is currently not available. * Cookie WMF-Uniq * Duration 1 year * Description Description is currently not available. * Cookie loom_anon_comment * Duration 1 year * Description No description available. * Cookie loom_referral_video * Duration session * Description Description is currently not available. * Cookie VISITOR_INFO1_LIVE * Duration 6 months * Description A cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface. * Cookie yt-remote-connected-devices * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie ytidb::LAST_RESULT_ENTRY_KEY * Duration Never Expires * Description The cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future. * Cookie yt-remote-device-id * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie yt-remote-session-name * Duration session * Description The yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video. * Cookie yt-remote-fast-check-period * Duration session * Description The yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos. * Cookie yt-remote-session-app * Duration session * Description The yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player. * Cookie yt-remote-cast-available * Duration session * Description The yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player. * Cookie yt-remote-cast-installed * Duration session * Description The yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video. * Cookie cp_session * Duration 3 months * Description Codepen sets this cookie for Help systems found in the website. * Cookie loid * Duration 1 year 1 month 4 days * Description This cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform. Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc. * Cookie __hstc * Duration 6 months * Description Hubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session). * Cookie hubspotutk * Duration 6 months * Description HubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts. * Cookie _ga * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors. * Cookie _ga_* * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to store and count page views. * Cookie __Host-psifi.analyticsTrace * Duration 6 hours * Description Description is currently not available. * Cookie __Host-psifi.analyticsTraceV2 * Duration 6 hours * Description Description is currently not available. * Cookie _gh_sess * Duration session * Description GitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form. * Cookie YSC * Duration session * Description YSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages. * Cookie ajs_anonymous_id * Duration 1 year * Description This cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before. * Cookie vuid * Duration 1 year 1 month 4 days * Description Vimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors. * Cookie AWSALB * Duration 7 days * Description AWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target. * Cookie acq * Duration past * Description Description is currently not available. * Cookie acq.sig * Duration past * Description Description is currently not available. * Cookie ptc * Duration 2 years * Description No description available. Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns. * Cookie muc_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to collect user behaviour and interaction data to optimize the website. * Cookie guest_id_marketing * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie guest_id_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie personalization_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting. * Cookie guest_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences. * Cookie bcookie * Duration 1 year * Description LinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs. * Cookie __Secure-ROLLOUT_TOKEN * Duration 6 months * Description YouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment. * Cookie yt.innertube::nextId * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie yt.innertube::requests * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie session_tracker * Duration session * Description This cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website. * Cookie edgebucket * Duration session * Description Reddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content. * Cookie did * Duration 1 year * Description Arbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year. Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation [](https://towardsdatascience.com/) Publish AI, ML & data-science insights to a global community of data professionals. Sign in[Submit an Article](https://write.insightmediagroup.io) [Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/) * * * [Write For TDS](https://towardsdatascience.com/submissions/) [LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search [Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/) # How Vision Language Models Are Trained from “Scratch” A deep dive into exactly how text-only language models are fine-tuned to *see* images [Avishek Biswas](https://towardsdatascience.com/author/neural.avb/) March 13, 202611 min read ![](../images/000.jpg) Recently, I went on an adventure to transform a small text-only language model and gift it the power of vision. This article is to summarize all my learnings, and take a deeper look at the network architectures behind modern Vision Language Models. _The code is open-source, you can check out the GitHub link at the end of the article. There is also a 30-minute companion YouTube video that explains the whole article in a visually rich format._ _Also, unless otherwise mentioned, all images in this article are produced by the author._ **Wait, are you really going to be "training from scratch"?** Yes... I mean no... _it's a bit nuanced_. **Research labs in 2026 do not train multimodal models from "scratch" anymore.** It is simply too expensive to teach a model vision and (textual) language at the same time! It requires more data, compute, time, and money. Also, it often leads to poorer results. Instead, labs take existing pretrained text-only language models, and finetune it to give "vision capabilities". In theory (and practice), this is way more compute-efficient. ![](../images/001.png) Let's talk about Vision Language Models! Learn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap. ## The standard architecture Although it's less data-intensive, finetuning text-only LMs to _suddenly_ start seeing images _would_ surely open its own can of worms. 1. **how do we embed the image,** i.e. convert it into numerical representations that a neural network can understand? 2. how do we tune the image embeddings to be **compatible with text**? 3. how do we adjust the weights of the text model so that it retains it's previous world knowledge, but also **generate text from image embeddings**? ![](../images/002.png) The image embeddings coming out of the VIT and Q-Former passes through an MLP layer. Followed by a series of Decoder layer and trainable LORA adapters. These modules are: 1. **The Image Backbone:** A model that converts raw images into embeddings. 2. **The Adapter Layer:** These are models that convert the image embeddings into a "text-compatible" embedding. This is the main challenging part - what architectures to use, what loss functions, etc. 3. **The Language Layer:** The language model we will train to input the adapted embeddings and generate text from it. Let's discuss them one by one. ## 1. The Image Backbone The goal of your image backbone is simple: **Input:** A raw 2D pixel map/image. **Output:** A sequence of vector embeddings representing the image In general, we just use an off-the-shelf image model that has been pretrained with massive corpus of images, generally on self-supervised tasks. You can use a Convolutional Neural Network (like a ResNet) to use as an image backbone. But modern state-of-the-art VLMs have almost entirely shifted to ViTs because they scale better with data and are more flexible for multimodal fusion. ![](../images/003.png) Vision Transformers input an image, extract patches out of it, and then pass them through bidirectional self-attention layers. This contextualizes each patch embedding with each other to form a contextual sequence of image embeddings. _**Do I train the image backbone or keep it frozen?**_ In most VLM research, there is a clear trend toward keeping backbones static (frozen) to save costs. Also, vision-language training generally needs paired image-text datasets. Since these datasets are always much smaller than the VIT's pretraining dataset, finetuning the backbone often leads to overfitting and degraded performance. By keeping these weights frozen, we are basically transferring the ownership of vision-language learning to latter parts of the network (i.e. the adapter layer and the text backbone). In my experiments, I used the [ViT-Base model](https://huggingface.co/google/vit-base-patch16-224). This model takes the image input, splits it into patches of 16x16 images and applies self-attention on them to generate an embedding sequence of 197 vectors. Each vector is 768 dimesnions long (the embedding size of the VIT). ## 2. The Adapter Layer This is where we are going to spend the majority of our time. We have converted images into embeddings already, but those embeddings are completely text-unaware. Vision Transformers are pre-trained purely on image pixels. Not on their captions, or any local textual features. The role of the adapter is to ground the pixel-based-image-embeddings into a (often shorter sequence of) text-based-image-embeddings. There are many ways to do this, like using CLIP models, but we are going to look at one of the more popular approaches — the Query Former of the Q-Former. ### Q-Former Alright — so what is a Q-Former? A Q-Former or the Query-Former was introduced in the [BLIP-2 paper](https://arxiv.org/abs/2301.12597). ![](../images/body-ba75190797c8121dca2f.png) From the BLIP-2 Paper **How do I train a Q-Former?** Standard Q-Formers can be trained using any multimodal image-text pair dataset. For example, you can use the Conceptual Captions dataset, which is a massive corpus of images and their corresponding captions. In my project, I took just 50,000 pairs to train the Q-Former. You can train a Q-Former from scratch, but the BLIP-2 recommendation is to use an pretrained BERT model. So that's what we will do. At a high level, here is our basic gameplan: * Train a multi-modal joint embedding space. A space where text and images "know" each other. * Basically, we will input pairs of image and captions — and embed both of them in the same joint space. * Images and incompatible captions will be mapped in separate places in this new embedding space, and compatible captions will be mapped close to each other. ![](../images/body-c292dc9550c316c4e6dc.png) Layer 1: Self Attention ![](../images/body-42384cc6ae53e8b1ef76.png) Layer 2: Self-Attention & Cross-Attention with the VIT Features **Setting up cross-attention layers** There's a problem — BERT models are purely text models. They have no idea what an image is. So, our objective is to first introduce new cross-attention layers to marry the vision embeddings coming out of the VIT and the text embeddings from BERT. Let's break down step-by-step how we convert BERT into a Q-Former: * Sample an image and text pair from the dataset * Pass the image through the frozen VIT model to convert the image into image embeddings, shaped [197, 768] * Initialize "learnable query embeddings". These are (say 32) vector embeddings that we will use to convert the image embedding sequence into a text-grounded token embedding sequence. Notice that 32 is much lower than the original VIT embedding sequence length (197). * We input the text caption embeddings and the query embeddings into the first BERT layer. The layer applies self-attention on these inputs. For now, let's assume that the query tokens only attend among themselves and the text tokens among themselves (i.e. the query tokens and the text tokens do not _see_ each other). * In the 2nd layer of the BERT, something INTERESTING happens. The two sets of embeddings go through another self-attention layer like before. But this time, we also use a cross-attention layer to contextualize the query embeddings with the ViT image embeddings we calculated earlier. Of course, normal BERT does not have any cross-attention layers, so we introduce these multi-headed cross-attention layers ourselves. * Just like this, we alternate between a pure self-attention layer (where queries and text independently self-attend among themselves) followed by a cross-attention layer (where the query embeddings attend to the frozen VIT embeddings). * In the final layer, we choose a joint embedding training loss, like ITC (Image Text Contrastive Loss) or ITM (Image Text Matching Loss) or ITG (Image Text Generation Loss etc). More on this later. ### What does cross-attention do? It contextualizes the image content with the query embeddings. You can imagine each query is trying to match a specific embedding pattern against the 197 VIT embeddings. For example, if a query has a high match with a single image vector, it will capture that feature very prominently. If the query matches with a combination of vectors, it will be an average of those embeddings, and so on. Remember we trained 32 of these query embeddings, so you are allowing the Q-Former to learn multiple different co-activations within the image embeddings. Due to the nature of training, these coactivations are encouraged to maximize alignment between image and text. > **As we train the Q-Former, both the initial query embeddings and the cross-attention weights will be optimized so we can extract relevant features from the VIT image tokens.** The query former embeddings are not trying to capture every detail of these 197 embeddings — instead they are trying to learn how to combine them into a compact 32 token sequence. Note that after the Q-Former is trained, we won't actually use the text part of the Q-Former for anything. We will simply pass the query-embeddings through the Q-former and alternatively run self-attention and cross-attention only on them. ### Loss functions for training Q-Formers [![AVB](../images/006.jpg)](https://x.com/neural_avb/status/2000961550094627086) [AVB](https://x.com/neural_avb/status/2000961550094627086) [@neural_avb](https://x.com/neural_avb/status/2000961550094627086) ·[Follow](https://x.com/intent/follow?screen_name=neural_avb) [](https://x.com/neural_avb/status/2000961550094627086) There is SO MUCH COOL SHIT you can do just by configuring the attention mask in different ways. So icydk, I will be training a small text-only LM to have vision capabilities. As a pretraining step, I need to first train a joint image-text embedding space which will be later used[Show more](https://x.com/neural_avb/status/2000961550094627086) [![Image](../images/005.jpg)](https://x.com/neural_avb/status/2000961550094627086) [![AVB](../images/006.jpg)](https://x.com/neural_avb/status/2000823976717901829) AVB @neural_avb Another VLM update. I'm doing a 2 phase training process similar in LLava/BLIP etc. The eventual goal is to tune a text-only SLM like Qwen3-0.6B to have vision capabilities. The first step is to train a "Q-Former network". Finished writing one. Basically I initialized with a [![Image](../images/007.jpg)](https://x.com/neural_avb/status/2000823976717901829) [4:09 PM · Dec 16, 2025](https://x.com/neural_avb/status/2000961550094627086)[](https://help.x.com/en/x-for-websites-ads-info-and-privacy) [301](https://x.com/intent/like?tweet_id=2000961550094627086)[Reply](https://x.com/intent/tweet?in_reply_to=2000961550094627086)Copy link [Read 7 replies](https://x.com/neural_avb/status/2000961550094627086) How the Q-Former model is trained is actually closely related to how we attend between query and text tokens throughout the layers. 1. **Image-Text-Contrastive Loss** (our setup) For this task, we use a **unimodal self-attention mask**. Query tokens attend among each other, text token among each other. The loss function can be any standard CLIP-like contrastive loss function. We will align the image and text in the same embedding space. Basically, we take the output of the queries and the output of the text encoder and compute their similarity. If an image and a caption belong together, we want their vectors to be as close as possible. This forces the queries to extract a "global" visual representation that matches the general theme of the text without actually looking at the words yet. 2. **Image-Text Matching Loss** (ITM) This uses a **bi-directional self-attention mask**. Here, every query token is allowed to see every text token, and every text token can see every query! For the loss fucntion, we use a binary classification task where the model has to predict: "Is this image and this text a match—Yes or No?". Binary cross-entropy loss. Because the modalities are fully mixed, the model can do fine-grained comparisons. The queries can look at specific objects in the image (via the cross-attention) and verify if they correspond to specific words in the text. This is much more detailed than the contrastive loss and ensures the 32 tokens are capturing localized details. 3. **Image-Text Generation Loss (ITG)** Finally, we have the generative task. For this, we use a **multimodal causal mask**. The queries can still see each other, but the text tokens are now treated like a sequence. Each text token can see all 32 query tokens - which act as a visual prefix. But they can only see the text tokens that came _before_ it. For the loss function, we just train the model to predict the next token in the caption. By forcing the model to literally "write" the description based on the queries, we ensure that those 32 tokens contain every bit of visual information necessary for a language model to understand the scene. For my project, I just used the simplest — ITC. For a small dataset like I was using, this was the easiest way! BLIP-2 recommends to use a mixture of all these training methods. The github repo shared at the end of the article provides the recipe to use any of the above attention schemes. ![](../images/body-c57b23ef1749304be7c0.jpeg) A trained Q-Former model learns to match similar text and image pairs In the next section, we will do the final step — training the VLM! ## 3. The Language Layer Now comes the final step. We will use the VIT and the Q-Former to make a language model into a vision model. I picked one of the smallest instruction-tuned language models — the [SmolLM2-135M](http://HuggingFaceTB/SmolLM2-135M). Thankfully, this part is not as complicated as the Q-Former training. We have the image embeddings (coming from the VIT and the Q-Former), and we have the text tokens (coming from the SmolLM tokenizer). Let's see some details. ![](../images/body-4776025afe5609d58a34.png) From the BLIP-2 Paper * We sample an image and a caption from our dataset * We randomly pick from a list of simple system prompts, similar to: "_You are a helpful assistant. Respond truthfully to the user_." We also pick the user query from a list of prompts, for example: "_What do you see in this image?_ " We tokenize the output captions sampled from the dataset as well. These 3 things form the text tokens. We tokenize them all using the SmolLM2 tokenizer, but we are not going to insert it into the LLM just yet — we must process the image first. * We pass the image through the frozen VIT, then through the Q-Former (again, note that the text captions are not passed into the Q-Former, only the image pipeline is executed) * We introduce a small MLP layer that converts the Q-Former output into new embeddings that are of the same shape as the LLM's expected embedding size. As we train, this MLP layer will map the Q-Former embedding into the LLM embedding space. * Now that we have the text tokens sequence and the new image embeddings (VIT -> Q-Former -> MLP). We will pass the text tokens through the LLM's native embedding layer. We sandwich the text and image embeddings in the following sequence: ...and forward pass it through the rest of the LLM. * Why that specific sequence? Since autoregressive LLMs use causal masking, we will essentially be training models to generate the output (caption) sequence given the entire prefix (system prompt, user prompt, and the image embeddings). * We add LoRA adapters (Low-Rank Adaptation Matrices) instead of training the entire LLM from scratch. By wrapping our LLM with LoRA, we freeze the original millions of parameters and only train tiny, low-rank matrices injected into the attention layers. This makes the model trainable on consumer hardware while keeping all that pre-existing intelligence intact. * And that’s it! We pass these stitched embeddings and labels into the LLM. The model attends to the text instruction _and_ the visual tokens simultaneously, and thanks to LoRA, it learns how to update its internal wiring to understand this new visual language. Only the Q-Former layers, the MLP layer, and the LORA adapters are trained. Everything else is kept frozen. ![](../images/body-0bc2b91163814acecae6.png) Some results! You can find more results in the youtube video mentioned at the end of the article! After training this for just a few hours on a small subset of data, the trained VLM can now see the images and generate text about it. Machine Learning is so beautiful when it works. ## In summary You can find the full github repository here: And watch the youtube video here: Please accept cookies to access this content Let's summarize all the modules in Vision Language pipelines. * A vision backbone (like the VIT) that takes image input and converts it into embeddings * An adapter layer (like the Q-Former) that grounds the image with text * An LLM that we train to consolidate the text and image embeddings to learn the language of vision **My Patreon:** **My YouTube channel:** **Follow me on Twitter:** **I am building Paper Breakdown, a place to study research papers** **Read my articles:** ## Register Now to Continue Reading ### Unlock Towards Data Science with a Free Account All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account * * * ![](../images/body-849620ef9680d77fc51e.jpeg) Written By ## [Avishek Biswas](https://towardsdatascience.com/author/neural.avb/) [See all from Avishek Biswas](https://towardsdatascience.com/author/neural.avb/) [Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Large Multimodal Models](https://towardsdatascience.com/tag/large-multimodal-models/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Multimodal Learning](https://towardsdatascience.com/tag/multimodal-learning/)[Vision Language Model](https://towardsdatascience.com/tag/vision-language-model/) Share This Article * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fhow-vision-language-models-are-trained-from-scratch%2F&title=How%20Vision%20Language%20Models%20Are%20Trained%20from%20%E2%80%9CScratch%E2%80%9D) * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fhow-vision-language-models-are-trained-from-scratch%2F&title=How%20Vision%20Language%20Models%20Are%20Trained%20from%20%E2%80%9CScratch%E2%80%9D) * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fhow-vision-language-models-are-trained-from-scratch%2F&text=How%20Vision%20Language%20Models%20Are%20Trained%20from%20%E2%80%9CScratch%E2%80%9D) Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. [Write for TDS](https://towardsdatascience.com/submissions/) ## Related Articles * [![](../images/008.jpg)](https://towardsdatascience.com/summarising-best-practices-for-prompt-engineering-c5e86c483af4/) ## [Summarising Best Practices for Prompt Engineering](https://towardsdatascience.com/summarising-best-practices-for-prompt-engineering-c5e86c483af4/) [ChatGPT](https://towardsdatascience.com/category/artificial-intelligence/chatgpt/) How to build your own LLM-based application using OpenAI API [Dmytro Nikolaiev (Dimid)](https://towardsdatascience.com/author/andimid/) May 28, 202314 min read * [![](../images/009.png)](https://towardsdatascience.com/to-make-an-impact-in-any-industry-domain-knowledge-is-critical-bdd93a304a1d/) ## [To Make an Impact in Any Industry, Domain Knowledge Is Critical](https://towardsdatascience.com/to-make-an-impact-in-any-industry-domain-knowledge-is-critical-bdd93a304a1d/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Abiodun Olaoye discusses his work on renewable energy [TDS Editors](https://towardsdatascience.com/author/towardsdatascience/) December 14, 20225 min read * [![](../images/010.png)](https://towardsdatascience.com/how-to-build-an-llm-application-360848c957db/) ## [How to Build an LLM Application](https://towardsdatascience.com/how-to-build-an-llm-application-360848c957db/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) Using Langchain and OpenAI to Build LLM Centred Apps [John Adeojo](https://towardsdatascience.com/author/johnadeojo/) July 21, 20236 min read * [![](../images/011.jpg)](https://towardsdatascience.com/conditional-random-field-tutorial-in-pytorch-ca0d04499463/) ## [Conditional Random Field Tutorial in PyTorch 🔥](https://towardsdatascience.com/conditional-random-field-tutorial-in-pytorch-ca0d04499463/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Let's say I have two identical dice, but one is fair and the other is... [Freddy Boulton](https://towardsdatascience.com/author/freddyalfonsoboulton/) May 3, 20189 min read * [![](../images/012.webp)](https://towardsdatascience.com/gpt-6-astra-just-hit-openais-highest-cybersecurity-risk-level/) ## [GPT-6 Astra Just Hit OpenAI's Highest Cybersecurity Risk Level](https://towardsdatascience.com/gpt-6-astra-just-hit-openais-highest-cybersecurity-risk-level/) [Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/) Why this matters less for what Astra can do and more for what every other model hasn't been tested for [Benjamin Nweke](https://towardsdatascience.com/author/benjaminnweke/) September 21, 20267 min read * [![](../images/013.png)](https://towardsdatascience.com/the-ultimate-guide-to-finding-outliers-in-your-time-series-data-part-1-1bf81e09ade4-2/) ## [The Ultimate Guide to Finding Outliers in Your Time-Series Data (Part 1)](https://towardsdatascience.com/the-ultimate-guide-to-finding-outliers-in-your-time-series-data-part-1-1bf81e09ade4-2/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Effective statistical methods and tools for outlier detection in time-series analysis [Sara Nobrega](https://towardsdatascience.com/author/saranobregafn/) May 21, 202419 min read * [![](../images/014.png)](https://towardsdatascience.com/announcement-tensorflow-2-0-is-coming-7e17e8e333e3/) ## [Announcement: TensorFlow 2.0 is coming!](https://towardsdatascience.com/announcement-tensorflow-2-0-is-coming-7e17e8e333e3/) [Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/) And TensorFlow 1.x is going to be... [Admond Lee](https://towardsdatascience.com/author/admond1994/) November 23, 20183 min read * [![](../images/015.gif)](https://towardsdatascience.com/empowering-llms-to-think-deeper-by-erasing-thoughts/) ## [Empowering LLMs to Think Deeper by Erasing Thoughts](https://towardsdatascience.com/empowering-llms-to-think-deeper-by-erasing-thoughts/) [Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/) We examine the fundamental limits of CoT-based test-time scaling and explore efficient methods for deep reasoning in LLMs. [Chenxiao Yang](https://towardsdatascience.com/author/chenxiao.yang/) May 12, 20259 min read * [![](../images/016.png)](https://towardsdatascience.com/llms-weapons-of-mass-disinformation-4def0dc3dc7/) ## [LLMs, new Weapons of Mass Disinformation?](https://towardsdatascience.com/llms-weapons-of-mass-disinformation-4def0dc3dc7/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) You thought the 2016 Brexit and US presidential campaigns were bad? Think again. [Anthony Mensier](https://towardsdatascience.com/author/anthony.mensier/) May 29, 202318 min read * [YouTube](https://www.youtube.com/c/TowardsDataScience) * [X](https://x.com/TDataScience) * [LinkedIn](https://www.linkedin.com/company/towards-data-science/) * [Threads](https://www.threads.net/@towardsdatascience) * [Bluesky](https://bsky.app/profile/towardsdatascience.com) [](https://towardsdatascience.com/) Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/) © Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network) #### Recommended Articles Close * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ "Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ "The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ "Does Your Company Have a Data Strategy?") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4970/image?etag=1780565225463&width=500&height=500) Productivity Tips, Data Career Insights, and Other Recent Must-Reads ](https://towardsdatascience.com/productivity-tips-data-career-insights-and-other-recent-must-reads-5c47eb7a2fe0/ "Productivity Tips, Data Career Insights, and Other Recent Must-Reads")