{ "source_file": "0930", "source_platform": "研究机构", "author": "Hugging Face Blog", "title_listed": "Vision Language Models (Better, faster, stronger)", "keywords": "视觉语言模型,VLM,多模态", "domain": "图像处理", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://huggingface.co/blog/vlms-2025", "publish_date": "2025-05", "corpus_stage": "预训练/SFT", "notes": "视觉语言模型最新进展", "id": "fbfcd4a5fc9c", "collected_at": "2026-10-02T15:24:26", "title": "Vision Language Models (Better, faster, stronger)", "body_markdown": "Back to Articles\nVision Language Models (Better, faster, stronger)\nPublished\n\t\t\t\t\tMay 12, 2025\nUpdate on GitHub\nUpvote\n619\n+613\nmerve\nmerve\nFollow\nSergio Paniego\nsergiopaniego\nFollow\nAritra Roy Gosthipaty\nariG23498\nFollow\nPedro Cuenca\npcuenq\nFollow\nAndres Marafioti\nandito\nFollow\nMotivation\nVision Language Models (VLMs) are the talk of the town. In\na previous blog post\n(from\nApril 2024\n), we talked a lot about VLMs. A major chunk was about\nLLaVA\n, the first\nsuccessful\nand\neasily reproducible\nopen-source vision language model, along with tips on how to discover, evaluate, and fine-tune open models.\nSince then, so much has changed. Models have become\nsmaller yet more powerful\n. We've seen the rise of new architectures and capabilities (reasoning, agency, long video understanding, etc.). In parallel, entirely new paradigms, such as multimodal Retrieval Augmented Generation (RAG) and multimodal agents have taken shape.\nIn this blog post, we’ll take a look back and unpack everything that happened with vision language models the past year. You’ll discover key changes, emerging trends, and notable developments.\nWe highly recommend reading the first blog post if you want a good primer on how vision language models work.\nTable of Contents\nNew Model Trends\nAny-to-any models\nReasoning models\nSmol Yet Capable Models\nMixture-of-Experts as Decoders\nVision Language Action Models\nSpecialized Capabilities\nObject Detection, Segmentation, Counting with Vision Language Models\nMultimodal Safety Models\nMultimodal RAG: retrievers, rerankers\nMultimodal Agents\nVideo Language Models\nNew Alignment Techniques for Vision Language Models\nNew Benchmarks\nMMT-Bench\nMMMU-Pro\nExtra: Our Model Picks\nUseful Resources\nNew model trends\nIn this section, we will look at the new types of VLMs. While some are absolutely new, others are improved versions of previous research.\nAny-to-any models\nAny-to-any models, as the name suggests, are models that can take in any modality and output any modality (image, text, audio). They do it by aligning the modalities, where an input from one modality can be translated to another (e.g. the word “dog” would be associated with an image of a dog, or with the utterance of the word).\nThese models have multiple encoders (one for each modality) and then fuse the embeddings together to create a shared representation space. The decoders (multiple or single) use the shared latent space as input and decode into the modality of choice. Earliest attempt to build any-to-any models is\nChameleon by Meta\n, which can take in image and text and output image and text. Meta didn’t release the image generation capability in this model, so Alpha-VLLM released\nLumina-mGPT\n, which has built image generation on top of Chameleon.\nThe latest and most capable any-to-any model,\nQwen 2.5 Omni\n(figure below) is a good example to understand the architecture of an any-to-any model.\nQwen2.5-Omni employs a novel \"Thinker-Talker\" architecture, where the \"Thinker\" handles text generation, and the \"Talker\" produces natural speech responses in a streaming manner.\nMiniCPM-o 2.6\n, an 8B parameter multimodal model is capable of understanding and generating content across vision, speech, and language modalities.\nJanus-Pro-7B\n, introduced by DeepSeek AI, is a unified multimodal model that excels in both understanding and generating content across modalities. It features a decoupled visual encoding architecture, separating the processes for understanding and generation.\nWe suspect an uptick in the number of such models in the coming years. It is a well-known intuition that multimodal learning is the only way we can learn deep representations better. We have curated some any-to-any models and demos in\nthis collection\n.\nReasoning Models\nReasoning models are models that can solve complex problems. We saw them first with large language models, and now vision language models. Until 2025, there was only one open-source multimodal reasoning model,\nQVQ-72B-preview\nby Qwen. It was an experimental model that was developed by the Alibaba Qwen team and came with many disclaimers.\nThis year there’s another player,\nKimi-VL-A3B-Thinking\nby the Moonshot AI team. It consists of MoonViT (SigLIP-so-400M) as the image encoder and a Mixture-of-Experts (MoE) decoder with 16B total parameters and only 2.8B active parameters. The model is a long chain-of-thought fine-tuned and further aligned (reinforcement learning) version of the Kimi-VL base vision language model. You can try the model\nhere\n.\nThe authors also released an instruction fine-tuned version called\nKimi-VL-A3B-Instruct\n.\nThe model can take in long videos, PDFs, screenshots and more. It has agentic capabilities as well.\nSmol yet Capable Models\nThe community used to scale intelligence through the number of parameters, and then high-quality synthetic data. After a certain point, the benchmarks saturated and scaling models had diminishing returns. The community went to shrink larger models through various methods, like distillation. This makes sense because it reduces compute costs, simplifies deployment, and unlocks use cases like local execution, enhancing data privacy.\nWhen we say small vision language models we often refer to models with less than 2B parameters that can be run on consumer GPUs. SmolVLM is a good example model family for smaller vision language models. Instead of shrinking larger models, the authors went all the way and tried to fit models into tiny number of parameters like 256M, 500M and 2.2B. SmolVLM2, for instance, attempted to solve video understanding in these sizes and found 500M to be a good trade-off. At Hugging Face, we have built an iPhone application, HuggingSnap, to demonstrate that these model sizes can achieve video understanding on consumer devices.\nAnother striking model is\ngemma3-4b-it\nby Google DeepMind. It’s particularly exciting as it’s one of the smallest multimodal models to have 128k token context window, and supports 140+ languages. The model comes with the Gemma 3 family of models, with its largest model ranking first on Chatbot Arena at the time. The largest model was then distilled to a 1B variant.\nLastly, although not the smallest,\nQwen2.5-VL-3B-Instruct\nis worth noting. The model can do various tasks ranging from localization (object detection and pointing), to document understanding, to agentic tasks; with context length up to 32k tokens.\nYou can use small models through MLX and Llama.cpp integrations. For MLX, assuming you have it installed, you can get started with SmolVLM-500M-Instruct with this one liner:\npython3 -m mlx_vlm.generate --model HuggingfaceTB/SmolVLM-500M-Instruct --max-tokens 400 --temp 0.0 --image https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm_example.jpg --prompt\n\"What is in this image?\"\nYou can get started with using\ngemma-3-4b-it\nmodel in GGUF format with llama.cpp through CLI with this one-liner:\nllama-mtmd-cli -hf ggml-org/gemma-3-4b-it-GGUF\nYou can also serve the same model as follows.\nllama-server -hf ggml-org/gemma-3-4b-it-GGUF\nWe would like to give a shoutout to\nmoondream2\nand\nFlorence-2\nas they're the earliest attempts for smallest vision language models. In this blog, we are covering primarily newer models (mostly models that came out after April 2024).\nMixture-of-Experts as Decoders\nMixture of Expert (MoEs) models offer an\nalternative\nto\ndense\narchitectures by dynamically selecting and activating only the most relevant sub-models, termed \"experts\", to process a given input data segment. This selective activation (done by a router) mechanism has demonstrated the potential to substantially enhance model performance and operational efficiency while utilizing fewer computational resources.\nMoEs are faster at inference than their similar parameter-dense counterparts because of the selective activation of a smaller slice of the network. They also converge quickly during training. Every good thing comes with a cost, as MoEs need more memory cost due to the entire model being on the GPU, even if a smaller chunk is used.\nIn the widely adopted Transformer architecture, MoE layers are most commonly integrated by replacing the standard Feed-Forward Network (FFN) layers within each Transformer block. Dense networks use the entire model to run an inference, while similarly sized MoE networks selectively activate some experts. This helps in better compute utilization and faster inference.\nVision language models that have mixture-of-experts decoders seem to have enhanced performance. For instance, Kimi-VL as of now is the most advanced open reasoning model that has a mixture-of-experts decoder. Mixture-of-Experts show promising results with\nMoE-LLaVA\n's focus on efficiency and hallucination reduction and\nDeepSeek-VL2\n's broad multimodal capabilities too. The latest version of Llama (\nLlama 4\n) is an MoE with vision capabilities. MoE as a decoder is a promising research area, and we suspect an increase in models like these.\nTo get a nice understanding of MoEs we recommend reading\nthis fantastic article\n.\nVision-Language-Action Models\nVLMs are even making their mark in the field of robotics! There, they are known as Vision-language-action models (VLA). But don't be fooled, those are mainly VLMs with a little moustache and hat. VLAs take images and text instructions, and return text indicating actions for the robot to take directly. VLAs extend vision language models by adding action and state tokens to interact with and control physical environments. These extra tokens represent the system’s internal state (how it perceives the environment), actions (what it does based on commands), and time-related information (like the order of steps in a task). These tokens are appended to the vision language input to generate actions or policy.\nVLAs are usually fine-tuned on top of a base VLM. Some people extend this definition further and define VLAs as any model interacting visually with a real or digital world. In this definition, VLAs can do UI navigation or be used in agentic workflows. But many people believe those applications fall in the VLM domain.\nGreat examples of VLAs are\nπ0\nand π0-FAST, the first robotics foundation models by Physical Intelligence, ported to Hugging Face’s LeRobot library. These models are trained across 7 robotics platforms and 68 unique tasks. They show strong zero-shot and fine-tuned performance on complex, real-world activities such as laundry folding, table bussing, grocery bagging, box assembly, and object retrieval.\nGR00T N1\nis NVIDIA’s open VLA foundation model for generalist humanoid robots. It understands images and language, and turns them into actions, like moving its arms or following instructions, thanks to a system that combines smart reasoning with real-time movement control. GR00T N1 also builds on the LeRobot dataset format, the open standard created to simplify sharing and training on robot demonstrations.\nTaken from\nthe paper\nNow that we’ve looked at the latest VLM model innovations, let’s explore how more established capabilities have evolved.\nSpecialized Capabilities\nObject Detection, Segmentation, Counting with Vision Language Models\nAs we’ve seen in earlier sections, VLMs enable\ngeneralization\nover traditional computer vision tasks. Models can now take in images and a variety of prompts, such as open-ended text, and output structured text with localization tokens (for detection, segmentation and more).\nLast year,\nPaliGemma\nwas the first model to attempt solving these tasks. The model takes in an image and text, where text is a description of an object of interest, along with a task prefix. The text prompt looks like “segment striped cat” or “detect bird on the roof”.\nFor detection, the model outputs the bounding box coordinates as\ntokens\n. For segmentation, on the other hand, the model outputs detection tokens and segmentation tokens. These segmentation tokens aren’t all the segmented pixel coordinates, but codebook indices that are decoded by a variational autoencoder trained to decode these tokens into valid segmentation masks (as shown in the figure below).\nMany models have been introduced to do localization tasks after PaliGemma. Late last year, an upgraded version of PaliGemma, PaliGemma 2, appeared with the same capabilities and better performance. Another model that came later was Molmo by Allen AI, which can point to instances with dots and count object instances.\nQwen2.5-VL can also detect, point to, and count objects, and this includes UI elements as objects too!\nMultimodal Safety Models\nVision language models in production require\nfiltering\ninputs and outputs to prevent jailbreaks and harmful outputs for compliance. Harmful content varies from inputs with violence to sexually explicit content. That’s where multimodal safety models come in: they are used before and after vision language models to filter their inputs and outputs. They are just like LLM safety models but with additional image input.\nIn early 2025, Google introduced the first open multimodal safety model,\nShieldGemma 2\n. It is built on ShieldGemma, the text-only safety model. This model takes in images and content policies and returns whether an image is safe for a given policy.\nPolicy\nrefers to a criterion in which the image is inappropriate. ShieldGemma 2 can also be used to filter outputs of image generation models.\nLlama Guard 4\nby Meta, is a dense multimodal and multilingual safety model. It is densely pruned from Llama 4 Scout (a multimodal mixture-of-experts) with safety fine tuning.\nThe model can be used for text-only and multimodal inference. The model can also take in vision language model outputs, complete conversations, and filter them before sending them to the user.\nMultimodal RAG: retrievers, rerankers\nNow let’s look at how Retrieval Augmented Generation has evolved in the multimodal space. RAG for complex documents, usually formatted in PDF, is processed in three steps:\nparsing the document completely into text\npassing the plain text and the query to a retriever and a reranker to get the most relevant document\npassing the relevant context and query to an LLM\nA traditional PDF parser consists of multiple elements to preserve the structure and visual elements in the document, such as\nlayout\n,\ntables\n,\nimages\n,\ncharts,\nall rendered into a markdown. But this setup can be hard to maintain.\nWith the rise of vision language models, this issue was addressed: there are now multimodal retrievers and rerankers.\nMultimodal retrievers take a stack of PDFs and a query as input and return the most relevant page numbers along with their confidence scores. The scores represent how likely the page contains the answer to the query, or how relevant the query is to the page. This bypasses the brittle parsing step.\nThe most relevant pages are then fed to the vision language model along with the query, and the VLM generates the answer.\nThere are two main\nmultimodal retriever architectures\n:\nDocument Screenshot Embedding (DSE, MCDSE)\nColBERT-like models (ColPali, ColQwen2, ColSmolVLM)\nDSE models consist of a text encoder and an image encoder, returning a single vector per query. The returned scores are softmax over the dot products of embeddings. They return a single vector per passage.\nTaken from\nthe paper\nColBERT-like models, like ColPali, are also dual encoder models, with a twist: ColPali has a vision language model as an image encoder, and a large language model as a text encoder. These models are inherently not encoders, but the models output embeddings, which are then passed to a “MaxSim”. The outputs are multiple vectors, one for each token, unlike DSE. In MaxSim, the similarity between each text token embedding and each image patch embedding is calculated, and this approach captures nuances better. Due to this reason, ColBERT-like models are less cost-efficient, have better performance.\nBelow you can see the indexing latency for ColPali. Since it’s just a single model, it’s also easier to maintain.\nTaken from\nthe paper\nOn Hugging Face Hub, you can find these models under the task “\nVisual Document Retrieval\n”.\nThe most popular benchmark for this task is ViDoRe, which consists of documents in\nEnglish\nand\nFrench\n, with documents varying from financial reports, scientific figures to administrative documents. Each example of ViDoRe has the document as image, a query and potential answers. The documents matched with the queries help with contrastive pre-training, so the ViDoRe train set is used to train new models.\nMultimodal Agents\nVision language models unlock many agentic workflows from chatting with documents to computer use. Here we will cover the latter since it requires more advanced agentic capabilities. Recently, there have been many vision language models releases that understand and operate over UIs. The latest one is UI-TARS-1.5 by ByteDance, which showed great results in operating over browser, computer and phone use. It can also do gameplay with reasoning, and operate in open world games. Another impactful release of this year is MAGMA-8B, it’s a foundation model for both UI navigation and physical interaction with the real world. Moreover, Qwen2.5-VL (especially its\n32B variant\nas it is further trained on agentic tasks) and\nKimi-VL reasoning model\nare good in GUI agentic tasks.\nAt the beginning of 2025, we introduced smolagents, a new lightweight agentic library that implements the ReAct framework. Shortly after, we implemented vision language support for the library. This integration took place on two use cases:\nAt the beginning of the run, provide images for once. This is useful for document AI with tool use.\nDynamically retrieve images. This is useful for cases such as GUI control with VLM agents, where the agent repeatedly takes screenshots.\nThe library provides building blocks for the users to build their own agentic workflows with image understanding. We provide different scripts and single-line CLI commands to get the users started easily.\nFor the first case, assume we want an agent to describe documents (which is not very agentic, but good for minimal use cases). You can initialize the CodeAgent (an agent that writes its own code!) like the following:\nagent = CodeAgent(tools=[], model=model)\n# no need for tools\nagent.run(\n\"Describe these documents:\"\n, images=[document_1, document_2, document_3])\nFor the latter use case where we need an agent to get screenshots, we can define a callback to be executed at the end of each\nActionStep\n. For your own use case where you need to get images dynamically, modify the callback however you’d like. We will not define it here in detail for simplicity. Optionally, you can read the blog post and the script itself at the end of this blog post. For now, let’s see how we initialize the agent with callbacks and browser control steps.\ndef\nsave_screenshot\n(\nmemory_step: ActionStep, agent: CodeAgent\n) ->\nNone\n:\n\"\"\"\nTakes screenshots and writes to observations.\n\"\"\"\npng_bytes = driver.get_screenshot_as_png()\n memory_step.observations_images = [image.copy()]\n# persist images to memory_step\nurl_info =\nf\"Current url:\n{driver.current_url}\n\"\nmemory_step.observations = (\n url_info\nif\nmemory_step.observations\nis\nNone\nelse\nmemory_step.observations +\n\"\\n\"\n+ url_info\n )\nreturn\nagent = CodeAgent(\n tools=[go_back, close_popups, search_item_ctrl_f],\n# pass navigation tools\nmodel=model,\n additional_authorized_imports=[\n\"helium\"\n],\n step_callbacks=[save_screenshot],\n# pass callback\n)\nYou can simply try the whole example by running the following CLI command. It starts an agent with access to control over the web browser, powered by a vision language model to accomplish web automation tasks (please replace with the website you’d like to navigate to).\nwebagent\n\"go to xyz.com/men, get to sale section, click the first clothing item you see. Get the product details, and the price, return them. note that I'm shopping from France\"\nsmolagents provides different model types, such as local transformers models, open-source models served using Inference Providers, or endpoints closed-source model providers. We encourage the use of open-source models as many agentic workflows currently require reasoning, which benefits from models with a large number of parameters. Qwen 2.5 VL as of April 2025 is a good candidate for agentic workflows, as the model is further trained on agentic tasks.\nVideo Language Models\nMost vision language models these days can handle videos, because videos can be represented as a sequence of frames. However, video understanding is tricky because of the temporal relationship between frames and the large amount of frames, so different techniques are used to select a representative set of video frames.\nSince last year, the community has weighed on different approaches and tricks to solve this problem.\nA good example is the\nLongVU model\nby Meta. It downsamples video frames by passing them to DINOv2 to pick the most similar ones to remove them, and then the model further refines frames by picking the most relevant frames according to the text query, where both the text and the frames are projected to the same space and similarity is calculated.\nQwen2.5VL\ncan handle long context and is adapted to dynamic FPS rates, as the model is trained with videos with different frame rates. Through extended multimodal RoPE, it understands the absolute time positions of frames, and can handle different rates and still understand the speed of the events happening in real life. Another model is\nGemma 3\n, which can accept video frames interleaved with timestamps in text prompt, e.g. “Frame 00.00: ..”, and is very performant for video understanding tasks.\nTaken from\nthe paper\nNew Alignment Techniques for Vision Language Models\nPreference optimization\nis an alternative fine-tuning approach for language models that can also be extended to vision language models. Instead of relying on fixed labels, this method focuses on comparing and ranking candidate responses based on preferences. The\ntrl\nlibrary offers support for direct preference optimization (DPO), including for VLMs.\nBelow is an example of how a preference dataset for DPO of a VLM fine-tuning is structured. Each entry consists of an image + question pair and two corresponding answers: one chosen and one rejected. The VLM is fine-tuned to generate responses aligned with the preferred (chosen) answer.\nAn example dataset for this procedure is\nRLAIF-V\n, which contains over 83000 annotated samples formatted according to the structure described above. Each entry includes a list of images (usually one), a prompt, a chosen answer, and a rejected answer, just as expected by the DPOTrainer.\nThere is a\nRLAIF-V formatted\ndataset, which is already formatted accordingly. Below is an example of a single sample:\n{\n'images'\n:\n[\n\n]\n,\n'prompt'\n:\n[\n{\n\"content\"\n:\n[\n{\n\"text\"\n:\nnull\n,\n\"type\"\n:\n\"image\"\n}\n,\n{\n\"text\"\n:\n\"What should this catcher be using?\"\n,\n\"type\"\n:\n\"text\"\n}\n]\n,\n\"role\"\n:\n\"user\"\n}\n]\n,\n'rejected'\n:\n[\n{\n\"content\"\n:\n[\n{\n\"text\"\n:\n\"The catcher, identified by the number...\"\n,\n\"type\"\n:\n\"text\"\n}\n]\n,\n\"role\"\n:\n\"assistant\"\n}\n]\n,\n'chosen'\n:\n[\n{\n\"content\"\n:\n[\n{\n\"text\"\n:\n\"The catcher in the image should be using a baseball glove...\"\n,\n\"type\"\n:\n\"text\"\n}\n]\n,\n\"role\"\n:\n\"assistant\"\n}\n]\n}\nOnce the dataset is prepared, you can use the\nDPOConfig\nand\nDPOTrainer\nclasses from the trl library to configure and launch the fine-tuning process.\nBelow is an example configuration using\nDPOConfig\n:\nfrom\ntrl\nimport\nDPOConfig\n\ntraining_args = DPOConfig(\n output_dir=\n\"smolvlm-instruct-trl-dpo-rlaif-v\"\n,\n bf16=\nTrue\n,\n gradient_checkpointing=\nTrue\n,\n per_device_train_batch_size=\n1\n,\n per_device_eval_batch_size=\n1\n,\n gradient_accumulation_steps=\n32\n,\n num_train_epochs=\n5\n,\n dataset_num_proc=\n8\n,\n# tokenization will use 8 processes\ndataloader_num_workers=\n8\n,\n# data loading will use 8 workers\nlogging_steps=\n10\n,\n report_to=\n\"tensorboard\"\n,\n push_to_hub=\nTrue\n,\n save_strategy=\n\"steps\"\n,\n save_steps=\n10\n,\n save_total_limit=\n1\n,\n eval_steps=\n10\n,\n# Steps interval for evaluation\neval_strategy=\n\"steps\"\n,\n)\nTo train your model using\nDPOTrainer\n, you can optionally provide a reference model to compute the reward difference. If you’re using Parameter-Efficient Fine-Tuning (PEFT), you may omit the reference model by setting\nref_model=None\n.\nfrom\ntrl\nimport\nDPOTrainer\n\ntrainer = DPOTrainer(\n model=model,\n ref_model=\nNone\n,\n args=training_args,\n train_dataset=train_dataset,\n eval_dataset=test_dataset,\n peft_config=peft_config,\n tokenizer=processor\n)\n\ntrainer.train()\nNew benchmarks\nBenchmarks have also evolved significantly over the past year. In our previous blog, we described MMMU and MMBench as two emerging benchmarks for evaluating Vision Language Models. With the rapid progress in the field, the models have saturated on these benchmarks, and we need better evaluation tools. To achieve this, we need tools that assess specific capabilities, on top of general purpose benchmarks.\nNow, we highlight two general-purpose benchmarks that stand out: MMT-Bench and MMMU-Pro.\nMMT-Bench\nMMT-Bench\nis designed to assess VLMs across a wide range of multimodal tasks that require expert knowledge, precise visual recognition, localization, reasoning, and planning. The benchmark includes 31325 multi-choice visual questions from various multimodal scenarios, with image, text, video, and point cloud modalities. With 32 different meta-tasks with 162 subtasks, it covers a variety of tasks, including OCR, Visual Recognition, or Visual-Language Retrieval.\nMMMU-Pro\nMMMU-Pro\nis a better version of the original MMMU benchmark. It also evaluates advanced AI models’ true understanding capabilities across multiple modalities.\nIt is more complex than MMMU, e.g. it has a vision-only input setting and an increase in the number of candidate options from 4 to 10. The benchmark also incorporates real-world simulation, with vision-only questions derived from screenshots or photos captured within a simulated display, featuring varying backgrounds, font styles, and sizes to mimic real-world conditions.\nExtra: Our model picks\nHere are our picks for some highlighted models. There are many models that we like, the ones below are the latest.\nModel Name\nSizes\nWhy we love it\nQwen2.5-VL\nfrom 3B to 72B\nGreat versatile model with agentic capabilities, math and more\nRolmOCR\n7B\nVery performant OCR model\nKimi-VL-Thinking\n16B MoE with 3B active parameters\nBest reasoning model\nSmolVLM2\n256M, 500M (our favorite!), 2.2B\nSmallest video language model\nLlama 4 Scout & Maverick\n109B/400B MoE with 17B active parameters\nLoooooong context\nMolmo\n1B, 7B, 72B and MoE with 1B active parameters\nFully open model with localization capabilities on top\nAaaaand that’s it! We hope you found this blog post useful to catch up with all the things that happened this past year. We are looking forward to seeing all the things you’ll build with the models in this blog. 🤗 Below we provide some links on more in-depth explanations to each topic in this blog post.\nWe thank\nVaibhav Srivastav\nand\nPablo Montalvo Leroux\nfor their review on this blog.\nUseful Resources\nHere’s a compilation of blogs where we went through the items in the blog post in-depth.\nModels, datasets and more mentioned in this blog\nMultimodal Safety:\nLlama Guard 4 Blog\nDPO in VLMs:\nPreference Optimization for Vision Language Models with TRL\nSmolagents with VLM support:\nWe just gave sight to smolagents\nAgents Course section for Vision Agents using smolagents:\nVision Agents with smolagents\nGemma 3 Model Release:\nWelcome Gemma 3: Google's all new multimodal, multilingual, long context open LLM\nPaliGemma 2 Model Release:\nWelcome PaliGemma 2 – New vision language models by Google\nPi0 release by Hugging Face\nMultimodal retrieval:\nVisually Multilingual: Introducing mcdse-2b\nMultimodal retrieval:\nColPali: Efficient Document Retrieval with Vision Language Models\nVideo Language Modelling:\nSmolVLM2: Bringing Video Understanding to Every Device\nMinimal training of VLM with vanilla PyTorch:\nGitHub - huggingface/nanoVLM: The simplest, fastest repository for training/finetuning small-sized VLMs.\nModels mentioned in this article\n9\nDatasets mentioned in this article\n2\nSpaces mentioned in this article\n1\nPapers mentioned in this article\n4\nCollections mentioned in this article\n9\nMore Articles from our Blog\naudio\nvision\nllm\nGemma 3n fully available in the open-source ecosystem!\n+4\n123\nJune 26, 2025\nmultimodal\nnlp\ncommunity\nTraining and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers\n83\nApril 16, 2026\nCommunity\nharpreetsahota\nMay 12, 2025\nOverall, this is a really great article that gives a primer on VLMS. I just wanted to chime in and say that Florence2 is also a really good model and one of my favourites for starting with unlabeled datasets. Another shout out to Moondream2 as well, which I find runs quite fast even on CPU/MPS\nSee translation\n1 reply\n·\n🔥\n9\n9\n🤝\n2\n2\n❤️\n1\n1\n➕\n1\n1\n+\nmerve\nArticle author\nMay 13, 2025\n•\nedited May 13, 2025\nhey Harpreet, thanks a lot! We wanted to cover primarily what came out after last April-May and latest developments, we'll give a shoutout to it. We already cover moondream2 in this one:\nhttps://huggingface.co/blog/vlms\nSee translation\n❤️\n3\n3\n+\nalielfilali01\nMay 13, 2025\nAnother striking model is gemma3-1b-it by Google DeepMind. It’s particularly exciting as it’s one of the smallest multimodal models to have 32k token context window, and supports 140+ languages. The model comes with the Gemma 3 family of models, with its largest model ranking first on Chatbot Arena at the time. The largest model was then distilled to a 1B variant.\nAre you sure\ngoogle/gemma-3-1b-it\nis multimodal ?\nSee translation\n1 reply\n·\nmerve\nArticle author\nMay 13, 2025\n•\nedited May 13, 2025\nWe have fixed with 4B, sorry you're correct! the snippet actually belongs to 4b and we should've linked to 4b\nSee translation\n🤗\n2\n2\n+\nalielfilali01\nMay 13, 2025\nGREAT piece ! Really AMAZING !\nWell written & structured (as usual from you guys) and serves as a reference to get back to from time to time ...\nThanks folks\nSee translation\n1 reply\n·\n❤️\n4\n4\n+\nmerve\nArticle author\nMay 13, 2025\nI hope it was useful!\nSee translation\n🧠\n2\n2\n❤️\n1\n1\n+\nfcakyon\nMay 13, 2025\nFantastic recap, thanks for the work!\nHowever I still prefer Florence2 and Moondream in production due to their smaller sizes and wide range of supported output formats 👌🏻\nSee translation\n2 replies\n·\n❤️\n4\n4\n+\nmerve\nArticle author\nMay 13, 2025\n•\nedited May 13, 2025\nhello Fatih, teşekkürler! We wanted to cover primarily what came out after last April-May and latest developments, we'll give a shoutout to it. We have already covered them in this blog\nhttps://huggingface.co/blog/vlms\nSee translation\n🤝\n2\n2\n🤗\n1\n1\n+\nExpand 1\n\t\t\t\t\t\treply\nreverentelusarca\nMay 13, 2025\nI agree with Kimi-VL, I wonder can we get the GGUF version since llama.cpp supports vision as well\nSee translation\n❤️\n3\n3\n+\nReply\n1ucky1uke\nMay 13, 2025\nThank you for the great article. There seems to be an issue in the Multimodal RAG section where the same graphic is displayed twice?\nSee translation\n1 reply\n·\n👍\n2\n2\n❤️\n1\n1\n+\nsergiopaniego\nArticle author\nMay 13, 2025\nThanks for reporting! It’s now updated 🙌\nSee translation\nnzarif\nMay 14, 2025\nThank you for the great article. It was very informative and well-organized> I wonder why there was no mention of InternVL. When comparing smaller VLM, InternVL does pretty well afaik.\nSee translation\n1 reply\n·\n🚀\n3\n3\n❤️\n1\n1\n+\nmerve\nArticle author\nMay 15, 2025\nthank you so much for your kind words! I think in general Qwen-VL variants (inc InternVL) do well yes :)\nSee translation\n👍\n1\n1\n+\nLallo67\nMay 14, 2025\nThank you so much for the helpful clarification. I used Kimi-VL in a tree recognition demo by chance, and I was genuinely impressed by how well it performed. Now things are starting to make a lot more sense. 😊\nSee translation\n2 replies\n·\n❤️\n2\n2\n+\nmerve\nArticle author\nMay 15, 2025\nvery happy it's of use to you, indeed I love Kimi-VL as well! super impressive with the size etc especially :)\nSee translation\n🚀\n3\n3\n+\nExpand 1\n\t\t\t\t\t\treply\nJilt\nMay 15, 2025\nLooking forward to blog post on truly multimodal models (Omni models) in around 6 months :)\nSee translation\n🚀\n4\n4\n❤️\n3\n3\n+\nReply\nsealical\nMay 16, 2025\nFantastic overview of the VLM landscape! One recent development worth noting is the Skywork R1V series, which demonstrates impressive multimodal reasoning capabilities.\nSkywork R1V introduced efficient cross-modal transfer of reasoning LLMs to vision, while R1V2 further advances this with hybrid reinforcement learning (MPO + GRPO). R1V2 achieves state-of-the-art open-source results: 73.6 on MMMU, 62.6 on OlympiadBench, and 78.9 on AIME24 - often outperforming much larger models.\nWhat's particularly interesting is R1V2's solution to the \"vanishing advantages\" problem in GRPO through their Selective Sample Buffer mechanism, which maintains training efficiency as model responses converge.\nBoth models are fully open-sourced and would fit well in your reasoning VLM section alongside Kimi-VL and QVQ. Thanks for this comprehensive resource!\nSee translation\n👍\n4\n4\n+\nReply\nJcZane\nMay 16, 2025\nIt's a readable and excellent project for a beginner like me. By reading this, I can quickly hands on some ideas I wanna do.\nSee translation\n1 reply\n·\n🔥\n5\n5\n+\nmerve\nArticle author\nMay 20, 2025\nVery happy it's useful for you! 💗\nSee translation\n❤️\n2\n2\n+\ndeleted\nMay 22, 2025\n•\nThis comment has been hidden\njinlovespho\nMay 27, 2025\nwow!! thx!! :)\n🔥\n3\n3\n😎\n2\n2\n+\nReply\nbella22334\nAug 18, 2025\nReally insightful breakdown . What excites me the most about VLMs is how quickly they’re evolving from static perception tools into interactive learning systems. The way they’re now handling multimodal RAG and long video understanding feels very similar to how structured learning modules like LETRS help people break down and master complex subjects step by step.\nSee translation\n👀\n1\n1\n+\nReply\nrish8200\nNov 12, 2025\nGreat article, we need another update given the latest progress in the VLM space, with models like qwen3 and more!\nSee translation\n😎\n2\n2\n+\nReply\nashrafb04\n9 days ago\nGreat article!! I think it is time for another update given that it is over a year now.\nSee translation\nReply\nEdit\nPreview\nUpload images, audio, and videos by dragging in the text input, pasting, or\nclicking here\n.\nTap or paste here to upload images\nComment\n·\nSign up\nor\nlog in\nto comment\nUpvote\n619\n+607\nModels mentioned in this article\n9\nDatasets mentioned in this article\n2\nSpaces mentioned in this article\n1\nPapers mentioned in this article\n4\nCollections mentioned in this article\n9", "metadata": { "title": "Vision Language Models (Better, faster, stronger)", "description": "We’re on a journey to advance and democratize artificial intelligence through open source and open science.", "keywords": null, "author": null, "og:title": "Vision Language Models (Better, faster, stronger)", "og:description": "We’re on a journey to advance and democratize artificial intelligence through open source and open science.", "og:type": "website", "og:url": "https://huggingface.co/blog/vlms-2025", "og:image": "https://huggingface.co/blog/assets/vlms2/vlms2.png", "twitter:card": "summary_large_image", "twitter:site": "@huggingface", "twitter:image": "https://huggingface.co/blog/assets/vlms2/vlms2.png" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/fbfcd4a5fc9c/code/body-000.txt", "chars": 237, "language": "text" }, { "file": "attachments/fbfcd4a5fc9c/code/body-001.txt", "chars": 49, "language": "text" }, { "file": "attachments/fbfcd4a5fc9c/code/body-002.txt", "chars": 47, "language": "text" }, { "file": "attachments/fbfcd4a5fc9c/code/body-003.txt", "chars": 145, "language": "text" }, { "file": "attachments/fbfcd4a5fc9c/code/body-004.txt", "chars": 691, "language": "text" }, { "file": "attachments/fbfcd4a5fc9c/code/body-005.txt", "chars": 181, "language": "text" }, { "file": "attachments/fbfcd4a5fc9c/code/body-006.txt", "chars": 518, "language": "text" }, { "file": "attachments/fbfcd4a5fc9c/code/body-007.txt", "chars": 632, "language": "text" }, { "file": "attachments/fbfcd4a5fc9c/code/body-008.txt", "chars": 247, "language": "text" } ], "images": [ { "status": "ok", "url": "https://huggingface.co/avatars/9e5d384fa04a44ef068178344e4f30c8.svg", "bytes": 1282, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-6cccfb9336c8aa96e43a.svg", "src": "https://huggingface.co/avatars/9e5d384fa04a44ef068178344e4f30c8.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RapueQQiWcsGEVSb51cQ-.jpeg", "bytes": 16730, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-009eec1d492bdb53bd61.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RapueQQiWcsGEVSb51cQ-.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/11c35454ea9b1b8fa830c348a0822503.svg", "bytes": 397, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-c39704b67a8d31e4c0c4.svg", "src": "https://huggingface.co/avatars/11c35454ea9b1b8fa830c348a0822503.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/64409d69518271b0d1c033a6/xxr8UEceXqyNVMETWSiOu.jpeg", "bytes": 6228, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-8896db05e9853aba705f.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/64409d69518271b0d1c033a6/xxr8UEceXqyNVMETWSiOu.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg", "bytes": 5452, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-1d8aabd4d93a54fb563b.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/cvtBI7qAk9vDrIUkq7hd4.png", "bytes": 8406, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-dad59198280c74315e57.png", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/cvtBI7qAk9vDrIUkq7hd4.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/qwen-omni.png", "bytes": 842391, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/005.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/qwen-omni.png", "alt": "Qwen-Omni", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/kimi-vl.png", "bytes": 1150005, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/006.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/kimi-vl.png", "alt": "kimi-vl", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/pi0.png", "bytes": 2252492, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/007.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/pi0.png", "alt": "pi0", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/pg2-seg.png", "bytes": 638409, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/008.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/pg2-seg.png", "alt": "PaliGemma3", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/molmo-pointing.png", "bytes": 1768182, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/009.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/molmo-pointing.png", "alt": "molmo", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/qwen3-gui.png", "bytes": 1233782, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/010.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/qwen3-gui.png", "alt": "Qwen2.5VL", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/llama-guard.png", "bytes": 1112756, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/011.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/llama-guard.png", "alt": "Llama Guard 4", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/rag-1.png", "bytes": 174652, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/012.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/rag-1.png", "alt": "Traditional Parsing", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/rag-2.png", "bytes": 191246, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/013.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/rag-2.png", "alt": "Multimodal RAG", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/dse.png", "bytes": 483647, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/014.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/dse.png", "alt": "DSE", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/colpali.png", "bytes": 130302, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/015.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/colpali.png", "alt": "ColPali", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/video.png", "bytes": 848058, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/016.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/video.png", "alt": "Video LMs", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/mrope.png", "bytes": 992657, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/017.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/mrope.png", "alt": "MRoPE", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/dpo.png", "bytes": 792630, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/018.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/dpo.png", "alt": "DPO", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/mmt-bench.png", "bytes": 2242937, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/019.png", "src": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/mmt-bench.png", "alt": "MMT-Bench", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/blog/assets/gemma3n/thumbnail.png", "bytes": 56540, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-6e02d3f4f0ded3b61329.png", "src": "https://huggingface.co/blog/assets/gemma3n/thumbnail.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/608aabf24955d2bfc3cd99c6/-YxmtpzEmf3NKOTktODRP.jpeg", "bytes": 9712, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/002.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/608aabf24955d2bfc3cd99c6/-YxmtpzEmf3NKOTktODRP.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1617264212503-603d25b75f9d390ab190b777.jpeg", "bytes": 9492, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/003.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1617264212503-603d25b75f9d390ab190b777.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/61929226ded356549e20c5da/ONUjP2S5fUWd07BiFXm0i.jpeg", "bytes": 3442, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/001.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/61929226ded356549e20c5da/ONUjP2S5fUWd07BiFXm0i.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1655385361868-61b85ce86eb1f2c5e6233736.jpeg", "bytes": 21134, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-c53427db6869fa24f06c.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1655385361868-61b85ce86eb1f2c5e6233736.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/blog/assets/train-sentence-transformers/st-hf-thumbnail.png", "bytes": 129194, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-3f59e95de0cdb130c49c.png", "src": "https://huggingface.co/blog/assets/train-sentence-transformers/st-hf-thumbnail.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/6317233cc92fd6fee317e030/cJHSvvimr1kqgQfHOjO5n.png", "bytes": 5138, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-4846b3d76d1e92e2d185.png", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/6317233cc92fd6fee317e030/cJHSvvimr1kqgQfHOjO5n.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/630904f2c038bf42d56d9d11/S8mYgFpPSHYOiifBnNfwG.jpeg", "bytes": 9394, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-847876035b03ddbef912.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/630904f2c038bf42d56d9d11/S8mYgFpPSHYOiifBnNfwG.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/6141a88b3a0ec78603c9e784/DJsxSmWV39M33JFheLobC.jpeg", "bytes": 15094, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/000.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/6141a88b3a0ec78603c9e784/DJsxSmWV39M33JFheLobC.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/626237d9bbcbd1c34f1bb231/EJrOjvAL-68qMCYdnvOrq.png", "bytes": 6472, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-b77626182a32c6130270.png", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/626237d9bbcbd1c34f1bb231/EJrOjvAL-68qMCYdnvOrq.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/617e7dbd129c9e67703ffe62/nJD_rI351t2c2QN_BpUBz.jpeg", "bytes": 7642, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-f3f5b7129e6dfbcc5b3d.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/617e7dbd129c9e67703ffe62/nJD_rI351t2c2QN_BpUBz.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/661d56bdbca423783d3184d9/QC4_jclW_osxUIhKPrvj1.jpeg", "bytes": 6624, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-fac4cbda5aef03cf08ee.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/661d56bdbca423783d3184d9/QC4_jclW_osxUIhKPrvj1.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/e98dc0753d77bdc12eb5a00d81e13b2b.svg", "bytes": 1282, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-5a69d290cc4432689104.svg", "src": "https://huggingface.co/avatars/e98dc0753d77bdc12eb5a00d81e13b2b.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-uploads.huggingface.co/production/uploads/635a7dadbd6f7742b4c39d60/q06gfzNWt1mhiI1Ydhj50.png", "bytes": 167178, "reused": true, "local_path": "attachments/fbfcd4a5fc9c/images/020.png", "src": "https://cdn-uploads.huggingface.co/production/uploads/635a7dadbd6f7742b4c39d60/q06gfzNWt1mhiI1Ydhj50.png", "alt": "grafik.png", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/8c8eba3ab2310a0954f7d7998b8ef466.svg", "bytes": 397, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-309fcfb66f5a032885c2.svg", "src": "https://huggingface.co/avatars/8c8eba3ab2310a0954f7d7998b8ef466.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/64e72bec31f7cea634c44a2f/Ag18iGb3KGuiGGAYKaEbV.jpeg", "bytes": 20048, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-7cacc8800f57bdcab2e9.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/64e72bec31f7cea634c44a2f/Ag18iGb3KGuiGGAYKaEbV.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/35423176841331948f8956fda156617f.svg", "bytes": 397, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-71079922216f2fb0da8b.svg", "src": "https://huggingface.co/avatars/35423176841331948f8956fda156617f.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/c22e3a77ff84b3b87c16cff2469f6d3d.svg", "bytes": 1283, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-8cc086d349eb7dac8076.svg", "src": "https://huggingface.co/avatars/c22e3a77ff84b3b87c16cff2469f6d3d.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/a3f0d5b5d0dc919c2eef19815ed52724.svg", "bytes": 1281, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-244120b0d5cd6fef68be.svg", "src": "https://huggingface.co/avatars/a3f0d5b5d0dc919c2eef19815ed52724.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/d7b507be0175a61a8fc21176eea45001.svg", "bytes": 397, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-1e8e2d9cd46e07b0e79d.svg", "src": "https://huggingface.co/avatars/d7b507be0175a61a8fc21176eea45001.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/lvaSE3w1ETZ9XjdvEADqN.png", "bytes": 10416, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-e70110e31342a7f8e619.png", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/lvaSE3w1ETZ9XjdvEADqN.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/fc4be73fbb8adee5aaad39d6371545f5.svg", "bytes": 1284, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-fba0cd6b8c496d2a17ad.svg", "src": "https://huggingface.co/avatars/fc4be73fbb8adee5aaad39d6371545f5.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/141e6f75f40eb2d0f9790500ae215840.svg", "bytes": 1283, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-15acb86172392f5585bb.svg", "src": "https://huggingface.co/avatars/141e6f75f40eb2d0f9790500ae215840.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/faa8a52af4e8eb181ac85f18c50fe094.svg", "bytes": 397, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-a3460a318c6e0e2c9bcc.svg", "src": "https://huggingface.co/avatars/faa8a52af4e8eb181ac85f18c50fe094.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/JMMWB5H9F-y354NZDZZjc.png", "bytes": 7368, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-8fb41ecb20510b36f5ba.png", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/JMMWB5H9F-y354NZDZZjc.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/1678809388600-noauth.png", "bytes": 9970, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-5213f03c2b1ba82c21d5.png", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/1678809388600-noauth.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/6632fcf688f75d987d60795c/HkKgCf6VihpzjzTA84pwn.jpeg", "bytes": 15376, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-6a1250869b3e308c4472.jpeg", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/6632fcf688f75d987d60795c/HkKgCf6VihpzjzTA84pwn.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn-avatars.huggingface.co/v1/production/uploads/641c9b6492cd25302997b305/RzjGLOMDSKiHii4cf_jYo.png", "bytes": 13164, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-0f984f3083a1ccf88e42.png", "src": "https://cdn-avatars.huggingface.co/v1/production/uploads/641c9b6492cd25302997b305/RzjGLOMDSKiHii4cf_jYo.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://huggingface.co/avatars/5ccc9219546e388155ce6d4f06d568e7.svg", "bytes": 1281, "reused": false, "local_path": "attachments/fbfcd4a5fc9c/images/body-7e582da9c50bc49ae6e2.svg", "src": "https://huggingface.co/avatars/5ccc9219546e388155ce6d4f06d568e7.svg", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://huggingface.co/blog", "label": "Back to Articles", "type": "a" }, { "href": "https://github.com/huggingface/blog/blob/main/vlms-2025.md", "label": "Update on GitHub", "type": "a" }, { "href": "https://huggingface.co/login?next=%2Fblog%2Fvlms-2025", "label": "Upvote 619", "type": "a" }, { "href": "https://huggingface.co/jnasimi", "label": "", "type": "a" }, { "href": "https://huggingface.co/ksj047", "label": "", "type": "a" }, { "href": "https://huggingface.co/XuLiu008", "label": "", "type": "a" }, { "href": "https://huggingface.co/zhouzaida", "label": "", "type": "a" }, { "href": "https://huggingface.co/taesiri", "label": "", "type": "a" }, { "href": "https://huggingface.co/duoan", "label": "", "type": "a" }, { "href": "https://huggingface.co/merve", "label": "", "type": "a" }, { "href": "https://huggingface.co/sergiopaniego", "label": "", "type": "a" }, { "href": "https://huggingface.co/ariG23498", "label": "", "type": "a" }, { "href": "https://huggingface.co/pcuenq", "label": "", "type": "a" }, { "href": "https://huggingface.co/andito", "label": "", "type": "a" }, { "href": "https://huggingface.co/blog/vlms", "label": "a previous blog post", "type": "a" }, { "href": "https://huggingface.co/papers/2304.08485", "label": "LLaVA", "type": "a" }, { "href": "https://huggingface.co/blog/smolvlm", "label": "smaller yet more powerful", "type": "a" }, { "href": "https://huggingface.co/collections/facebook/chameleon-668da9663f80d483b4c61f58", "label": "Chameleon by Meta", "type": "a" }, { "href": "https://huggingface.co/collections/Alpha-VLLM/lumina-mgpt-family-66ae48a59a973eeae4513848", "label": "Lumina-mGPT", "type": "a" }, { "href": "https://huggingface.co/collections/Qwen/qwen25-omni-67de1e5f0f9464dc6314b36e", "label": "Qwen 2.5 Omni", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/qwen-omni.png", "label": "", "type": "a" }, { "href": "https://huggingface.co/openbmb/MiniCPM-o-2_6", "label": "MiniCPM-o 2.6", "type": "a" }, { "href": "https://huggingface.co/deepseek-ai/Janus-Pro-7B", "label": "Janus-Pro-7B", "type": "a" }, { "href": "https://huggingface.co/collections/merve/any-to-any-models-6822042ee8eb7fb5e38f9b62", "label": "this collection", "type": "a" }, { "href": "https://huggingface.co/Qwen/QVQ-72B-Preview", "label": "QVQ-72B-preview", "type": "a" }, { "href": "https://huggingface.co/moonshotai/Kimi-VL-A3B-Thinking", "label": "Kimi-VL-A3B-Thinking", "type": "a" }, { "href": "https://huggingface.co/spaces/moonshotai/Kimi-VL-A3B-Thinking", "label": "here", "type": "a" }, { "href": "https://huggingface.co/moonshotai/Kimi-VL-A3B-Instruct", "label": "Kimi-VL-A3B-Instruct", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/kimi-vl.png", "label": "", "type": "a" }, { "href": "https://huggingface.co/google/gemma-3-4b-it", "label": "gemma3-4b-it", "type": "a" }, { "href": "https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct", "label": "Qwen2.5-VL-3B-Instruct", "type": "a" }, { "href": "https://huggingface.co/collections/google/gemma-3-release-67c6c6f89c4f76621268bb6d", "label": "gemma-3-4b-it", "type": "a" }, { "href": "https://huggingface.co/vikhyatk/moondream2", "label": "moondream2", "type": "a" }, { "href": "https://huggingface.co/collections/microsoft/florence-6669f44df0d87d9c3bfb76de", "label": "Florence-2", "type": "a" }, { "href": "https://huggingface.co/papers/2401.15947", "label": "MoE-LLaVA", "type": "a" }, { "href": "https://huggingface.co/deepseek-ai/deepseek-vl2", "label": "DeepSeek-VL2", "type": "a" }, { "href": "https://huggingface.co/collections/meta-llama/llama-4-67f0c30d9fe03840bc9d0164", "label": "Llama 4", "type": "a" }, { "href": "https://huggingface.co/blog/moe", "label": "this fantastic article", "type": "a" }, { "href": "https://huggingface.co/lerobot/pi0", "label": "π0", "type": "a" }, { "href": "https://huggingface.co/nvidia/GR00T-N1-2B", "label": "GR00T N1", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/pi0.png", "label": "", "type": "a" }, { "href": "https://www.physicalintelligence.company/download/pi0.pdf", "label": "the paper", "type": "a" }, { "href": "https://huggingface.co/blog/paligemma", "label": "PaliGemma", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/pg2-seg.png", "label": "", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/molmo-pointing.png", "label": "", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/qwen3-gui.png", "label": "", "type": "a" }, { "href": "https://huggingface.co/google/shieldgemma-2-4b-it", "label": "ShieldGemma 2", "type": "a" }, { "href": "https://huggingface.co/spaces/merve/llama-guard-4", "label": "Llama Guard 4", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/llama-guard.png", "label": "", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/rag-1.png", "label": "", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/rag-2.png", "label": "", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/dse.png", "label": "", "type": "a" }, { "href": "https://arxiv.org/pdf/2406.11251", "label": "the paper", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/colpali.png", "label": "", "type": "a" }, { "href": "https://arxiv.org/pdf/2407.01449", "label": "the paper", "type": "a" }, { "href": "https://huggingface.co/models?pipeline_tag=visual-document-retrieval&sort=trending", "label": "Visual Document Retrieval", "type": "a" }, { "href": "https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct", "label": "32B variant", "type": "a" }, { "href": "https://huggingface.co/collections/moonshotai/kimi-vl-a3b-67f67b6ac91d3b03d382dd85", "label": "Kimi-VL reasoning model", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/video.png", "label": "", "type": "a" }, { "href": "https://huggingface.co/collections/Vision-CAIR/longvu-67181d2debabfc1eb050c21d", "label": "LongVU model", "type": "a" }, { "href": "https://huggingface.co/collections/Qwen/qwen25-vl-6795ffac22b334a837c0f9a5", "label": "Qwen2.5VL", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/mrope.png", "label": "", "type": "a" }, { "href": "https://arxiv.org/pdf/2502.13923", "label": "the paper", "type": "a" }, { "href": "https://huggingface.co/docs/trl/en/index", "label": "trl", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/dpo.png", "label": "", "type": "a" }, { "href": "https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset", "label": "RLAIF-V", "type": "a" }, { "href": "https://huggingface.co/datasets/HuggingFaceH4/rlaif-v_formatted", "label": "RLAIF-V formatted", "type": "a" }, { "href": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm2/mmt-bench.png", "label": "", "type": "a" }, { "href": "https://huggingface.co/papers/2404.16006", "label": "MMT-Bench", "type": "a" }, { "href": "https://huggingface.co/papers/2409.02813", "label": "MMMU-Pro", "type": "a" }, { "href": "https://huggingface.co/reach-vb", "label": "Vaibhav Srivastav", "type": "a" }, { "href": "https://huggingface.co/Molbap", "label": "Pablo Montalvo Leroux", "type": "a" }, { "href": "https://huggingface.co/collections/sergiopaniego/vision-language-models-2025-update-682206d8ed0728be05dbf901", "label": "Models, datasets and more mentioned in this blog", "type": "a" }, { "href": "https://huggingface.co/blog/llama-guard-4", "label": "Llama Guard 4 Blog", "type": "a" }, { "href": "https://huggingface.co/blog/dpo_vlm", "label": "Preference Optimization for Vision Language Models with TRL", "type": "a" }, { "href": "https://huggingface.co/blog/smolagents-can-see", "label": "We just gave sight to smolagents", "type": "a" }, { "href": "https://huggingface.co/learn/agents-course/unit2/smolagents/vision_agents", "label": "Vision Agents with smolagents", "type": "a" }, { "href": "https://huggingface.co/blog/gemma3", "label": "Welcome Gemma 3: Google's all new multimodal, multilingual, long context open LLM", "type": "a" }, { "href": "https://huggingface.co/blog/paligemma2", "label": "Welcome PaliGemma 2 – New vision language models by Google", "type": "a" }, { "href": "https://huggingface.co/blog/pi0", "label": "Pi0 release by Hugging Face", "type": "a" }, { "href": "https://huggingface.co/blog/marco/announcing-mcdse-2b-v1", "label": "Visually Multilingual: Introducing mcdse-2b", "type": "a" }, { "href": "https://huggingface.co/blog/manu/colpali", "label": "ColPali: Efficient Document Retrieval with Vision Language Models", "type": "a" }, { "href": "https://huggingface.co/blog/smolvlm2", "label": "SmolVLM2: Bringing Video Understanding to Every Device", "type": "a" }, { "href": "https://github.com/huggingface/nanoVLM", "label": "GitHub - huggingface/nanoVLM: The simplest, fastest repository for training/finetuning small-sized VLMs.", "type": "a" }, { "href": "https://huggingface.co/blog/gemma3n", "label": "audio vision llm Gemma 3n fully available in the open-source ecosystem! +4 123 June 26, 2025", "type": "a" }, { "href": "https://huggingface.co/blog/train-multimodal-sentence-transformers", "label": "multimodal nlp community Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers 83 Ap", "type": "a" }, { "href": "https://huggingface.co/harpreetsahota", "label": "harpreetsahota", "type": "a" }, { "href": "https://huggingface.co/alielfilali01", "label": "alielfilali01", "type": "a" }, { "href": "https://huggingface.co/fcakyon", "label": "fcakyon", "type": "a" }, { "href": "https://huggingface.co/reverentelusarca", "label": "reverentelusarca", "type": "a" }, { "href": "https://huggingface.co/1ucky1uke", "label": "1ucky1uke", "type": "a" }, { "href": "https://cdn-uploads.huggingface.co/production/uploads/635a7dadbd6f7742b4c39d60/q06gfzNWt1mhiI1Ydhj50.png", "label": "", "type": "a" }, { "href": "https://huggingface.co/nzarif", "label": "nzarif", "type": "a" }, { "href": "https://huggingface.co/Lallo67", "label": "Lallo67", "type": "a" }, { "href": "https://huggingface.co/Jilt", "label": "Jilt", "type": "a" }, { "href": "https://huggingface.co/sealical", "label": "sealical", "type": "a" }, { "href": "https://huggingface.co/JcZane", "label": "JcZane", "type": "a" }, { "href": "https://huggingface.co/jinlovespho", "label": "jinlovespho", "type": "a" }, { "href": "https://huggingface.co/bella22334", "label": "bella22334", "type": "a" }, { "href": "https://huggingface.co/rish8200", "label": "rish8200", "type": "a" }, { "href": "https://huggingface.co/ashrafb04", "label": "ashrafb04", "type": "a" }, { "href": "https://huggingface.co/join?next=%2Fblog%2Fvlms-2025", "label": "Sign up", "type": "a" }, { "href": "https://huggingface.co/MichaelCreator82Mentor", "label": "", "type": "a" }, { "href": "https://huggingface.co/AmK1304", "label": "", "type": "a" }, { "href": "https://huggingface.co/MarkWang", "label": "", "type": "a" }, { "href": "https://huggingface.co/KaneWilliams", "label": "", "type": "a" }, { "href": "https://huggingface.co/Tamicktom", "label": "", "type": "a" }, { "href": "https://huggingface.co/winvswon78", "label": "", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:53:30Z", "source_url": "https://huggingface.co/blog/vlms-2025", "papers": [ { "status": "ok", "url": "https://www.physicalintelligence.company/download/pi0.pdf", "bytes": 7873003, "reused": false, "local_path": "papers/repair/ff7c13cc0eecd640fa8e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2406.11251", "bytes": 6800537, "reused": false, "local_path": "papers/repair/17de3235c227dc5702d9.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2407.01449", "bytes": 8260302, "reused": false, "local_path": "papers/repair/30d6a8872212da86ccbf.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2502.13923", "bytes": 2219929, "reused": false, "local_path": "papers/repair/7a171ab94f95cbf96657.pdf" } ], "actual_video_files": [], "body_chars": 34786, "expected_body_images": 50, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/fbfcd4a5fc9c/pages/source.html", "local_html": "attachments/fbfcd4a5fc9c/pages/article.html", "body_base_path": "attachments/fbfcd4a5fc9c/pages/", "body_markdown_path": "attachments/fbfcd4a5fc9c/pages/article.md" }