{ "source_file": "0930", "source_platform": "Medium", "author": "Shawhin Talebi", "title_listed": "Fine-tuning Multimodal Embedding Models", "keywords": "多模态,CLIP,微调", "domain": "机器学习", "content_type": "深度技术解析", "is_paid_flag": "部分免费/会员墙", "url": "https://towardsdatascience.com/fine-tuning-multimodal-embedding-models-bf007b1c5da5/", "publish_date": "2025-02", "corpus_stage": "预训练/SFT", "notes": "", "id": "9b184704bb79", "collected_at": "2026-10-01T22:22:11", "title": "Fine-tuning Multimodal Embedding Models | Towards Data Science", "body_markdown": "Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\n# Fine-tuning Multimodal Embedding Models\nAdapting CLIP to YouTube Data (with Python Code)\n[Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\nJanuary 31, 202510 min read\nThis is the 4th article in a larger series on [multimodal AI](https://shawhin.medium.com/list/multimodal-ai-fe9521d0e77a). In the previous post, we discussed [multimodal RAG](https://towardsdatascience.com/multimodal-rag-process-any-file-type-with-ai-e6921342c903) systems, which can retrieve and synthesize information from different data modalities (e.g. text, images, audio). There, we saw how we could implement such a system using CLIP. One issue with this approach, however, is that vector search results from a general-purpose embedding model (like CLIP) **may perform poorly in domain-specific use cases**. In this article, I'll discuss how we can mitigate these issues via fine-tuning multimodal embedding models.\n![Photo by Markus Winkler on Unsplash](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/0Y524lLKsF5spVr0K-scaled.jpg)\nPhoto by [Markus Winkler](https://unsplash.com/@markuswinkler?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\n* * *\n**Multimodal embeddings** represent multiple data modalities in the same vector space such that similar concepts are co-located. A visual example of this is shown below, where semantically **similar items** (e.g. a picture of a dog and its corresponding caption) **are close** , while **dissimilar items** (e.g. a picture of a cat and a caption describing a dog) **are far apart**.\n![Stock photos from Canva. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/1c6UUQUk10H61dy4lCjsrtA.png)\nStock photos from Canva. Image by author.\nA popular multimodal embedding model is CLIP, which was trained on a massive corpus of image-caption pairs using [contrastive learning](https://towardsdatascience.com/multimodal-embeddings-an-introduction-5dc36975966f#09fe). The key insight from **CLIP** was that such a model **unlocks 0-shot abilities such as image classification, search, and captioning** [1].\nOne limitation here is that CLIP's 0-shot abilities **may not transfer well to domains involving specialized information** e.g. architectural drawings, medical imaging, and technical jargon. In such cases, we can improve CLIP's performance through fine-tuning.\n## **Fine-tuning CLIP**\nFine-tuning involves **adapting a model to a particular use case through additional training**. This is powerful because it enables us to build on top of existing state-of-the-art models to develop powerful specialized models with relatively small data.\nWe can do this with CLIP through the following key steps.\n 1. Collect text-image training pairs\n 2. Pre-process training data\n 3. Define Evals\n 4. Fine-tune the model\n 5. Evaluate the model\n\n\nI will discuss each of these steps in the context of a concrete example. If you are curious about what this looks like for text embedding (i.e. text-text pairs), I did that in a previous [blog post](https://medium.com/@shawhin/fine-tuning-text-embeddings-f913b882b11c).\n> [**Fine-Tuning Text Embeddings For Domain-Specific Search**](https://shawhin.medium.com/fine-tuning-text-embeddings-f913b882b11c)\n## **Example: Fine-tuning CLIP on YouTube Titles and Thumbnails**\nHere, I will fine-tune CLIP on titles and thumbnails from my [YouTube channel](https://www.youtube.com/@ShawhinTalebi). At the end of this, we will have a model that can take title-thumbnail pairs and return a similarity score. This can be used for practical applications such as **matching title ideas to an existing thumbnai** l or **performing search over a thumbnail library**.\nThe [example code](https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/4-ft-mm-embeddings), [dataset](https://huggingface.co/datasets/shawhin/yt-title-thumbnail-pairs), and [fine-tuned model](https://huggingface.co/shawhin/clip-title-thumbnail-embeddings) are freely available on GitHub and the Hugging Face Hub, respectively. You can use this code and data to train your own models. If you end up publishing any work using this dataset, please cite the original source :)\n[GitHub Repo](https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai/4-ft-mm-embeddings) | [Dataset](https://huggingface.co/datasets/shawhin/yt-title-thumbnail-pairs) | [Fine-tuned Model](https://huggingface.co/shawhin/clip-title-thumbnail-embeddings)\n* * *\n## **Step 1: Collect Text-Image Training Pairs**\nThe first (and most important) step of any fine-tuning process is data collection. Here, I extracted title-thumbnail pairs from my channel in a 2-step process.\nFirst, I used YouTube's search API to **extract the video IDs** for all the videos on my channel. Second, I used YouTube's video API to **extract the title and thumbnail URL** of each of my long-form videos (i.e. longer than 3 min).\npython\n\n```\n# imports\nfrom top_secret import my_key\nimport requests\nfrom isodate import parse_duration\n\nimport pandas as pd\nimport numpy as np\nfrom sentence_transformers import SentenceTransformer\nfrom datasets import DatasetDict, Dataset\n```\n\npython\n\n```\nchannel_id = 'UCa9gErQ9AE5jT2DZLjXBIdA' # my YouTube channel ID\npage_token = None # initialize page token\nurl = 'https://www.googleapis.com/youtube/v3/search' # YouTube search API \n\n# extract video data across multiple search result pages\nvideo_id_list = []\n\nwhile page_token != 0:\n params = {\n \"key\": my_key, \n 'channelId': channel_id, \n 'part': [\"snippet\",\"id\"], \n 'order': \"date\", \n 'maxResults':50, \n 'pageToken': page_token\n }\n response = requests.get(url, params=params)\n\n for raw_item in dict(response.json())['items']:\n\n # only execute for youtube videos\n if raw_item['id']['kind'] != \"youtube#video\":\n continue\n\n # grab video ids\n video_id_list.append(raw_item['id']['videoId'])\n\n try:\n # grab next page token\n page_token = dict(response.json())['nextPageToken']\n except:\n # if no next page token kill while loop\n page_token = 0\n```\n\nNote that you will need a YouTube API key to run the above Python code, which you can create using the [Google Cloud Console](https://console.cloud.google.com/). To adapt this to your channel, you just need to change the _channel _id_ variable.\npython\n\n```\n# extract video titles and thumbnails\nurl = \"https://www.googleapis.com/youtube/v3/videos\"\nvideo_data_list = []\n\nfor video_id in video_id_list:\n\n params = {\n \"part\": [\"snippet\",\"contentDetails\"],\n \"id\": video_id, \n \"key\": my_key, \n }\n response = requests.get(url, params=params)\n\n raw_dict = dict(response.json())['items'][0]\n\n # only process videos longer than 3 minutes\n iso_duration = raw_dict['contentDetails'][\"duration\"]\n if parse_duration(iso_duration).total_seconds() < 180:\n continue\n\n # extract video data\n video_data = {}\n video_data['video_id'] = video_id\n video_data['title'] = raw_dict['snippet']['title']\n video_data['thumbnail_url'] = raw_dict['snippet']['thumbnails']['high']['url']\n\n # append data to list\n video_data_list.append(video_data)\n```\n\nAs an additional step, I **created negative thumbnail-title pairs**. We can use these during the training process to not only guide the model with examples of which embedding should be close together (i.e. positive pair), but also which embedding should be far apart (i.e. negative pairs).\nTo do this, I computed the similarity between all possible title pairs using the sentence transformer library. Then for each positive pair, I matched the least similar title as a negative example (ensuring there were no duplicates).\npython\n\n```\n# store data in dataframe\ndf = pd.DataFrame(video_data_list)\n\n# Load the model\nmodel = SentenceTransformer(\"all-mpnet-base-v2\")\n\n# Encode all titles\nembeddings = model.encode(df['title'].to_list())\n\n# compute similarities\nsimilarities = model.similarity(embeddings, embeddings)\n\n# match least JDs least similar to positive match as the negative match\nsimilarities_argsorted = np.argsort(similarities.numpy(), axis=1)\nnegative_pair_index_list = []\n\nfor i in range(len(similarities)):\n\n # Start with the smallest similarity index for the current row\n j = 0\n index = int(similarities_argsorted[i][j])\n\n # Ensure the index is unique\n while index in negative_pair_index_list:\n j += 1 # Move to the next smallest index\n index = int(similarities_argsorted[i][j]) # Fetch next smallest index\n\n negative_pair_index_list.append(index)\n\n# add negative pairs to df\ndf['title_neg'] = df['title'].iloc[negative_pair_index_list].values\n```\n\nFinally, I created a **train-valid-test split** and pushed the dataset to the Hugging Face Hub.\nmakefile\n\n```\n# Shuffle the dataset\ndf = df.sample(frac=1, random_state=42).reset_index(drop=True)\n\n# Split into train, validation, and test sets\ntrain_frac = 0.7\nvalid_frac = 0.15\ntest_frac = 0.15\n\n# define train and validation size\ntrain_size = int(train_frac * len(df))\nvalid_size = int(valid_frac * len(df))\n\n# create train, validation, and test datasets\ndf_train = df[:train_size]\ndf_valid = df[train_size:train_size + valid_size]\ndf_test = df[train_size + valid_size:]\n\n# Convert the pandas DataFrames back to Hugging Face Datasets\ntrain_ds = Dataset.from_pandas(df_train)\nvalid_ds = Dataset.from_pandas(df_valid)\ntest_ds = Dataset.from_pandas(df_test)\n\n# Combine into a DatasetDict\ndataset_dict = DatasetDict({\n 'train': train_ds,\n 'valid': valid_ds,\n 'test': test_ds\n})\n```\n\npython\n\n```\n# push data to hub\ndataset_dict.push_to_hub(\"shawhin/yt-title-thumbnail-pairs\")\n```\n\n## **Step 2: Pre-process Training Pairs**\nAlthough we have all the data we need for fine-tuning, it is still not a suitable format for training. More specifically, we need to **convert our image URLs to PIL image objects** and **organize our data into (anchor, positive,** **negative) triplets** , **** i.e., a thumbnail, its corresponding title, and negative title, respectively.\nWe can process all three data splits (i.e. train, valid, and test) in the following way using the Hugging Face Datasets library.\npython\n\n```\nfrom PIL import Image\n\n# load dataset\ndataset = load_dataset(\"shawhin/yt-title-thumbnail-pairs\")\n\n# define preprocessing function\ndef preprocess(batch):\n \"\"\"\n Preprocessing data without augmentations for test set\n \"\"\"\n # get images from urls\n image_list = [Image.open(requests.get(url, stream=True).raw) \n for url in batch[\"thumbnail_url\"]]\n\n # return columns with standard names\n return {\n \"anchor\": image_list, \n \"positive\": batch[\"title\"], \n \"negative\": batch[\"title_neg\"]\n }\n\n# remove columns not relevant to training\ncolumns_to_remove = [col for col in dataset['train'].column_names \n if col not in ['anchor', 'positive', 'negative']]\n# apply transformations\ndataset = dataset.map(preprocess, batched=True, \n remove_columns=columns_to_remove)\n```\n\nIt's important that we order our columns as (anchor, positive, negative) triplets because **this is the format expected by the loss function** we will use during training (which I learned the hard way).\n## **Step 3: Define Evals**\nTraining involves optimizing a model's parameters to minimize a loss function. However, this value (i.e. a contrastive loss) is rarely helpful in **assessing the model's performance on a downstream task** (e.g. matching titles to thumbnails).\nA quantity that is more insightful, in this case, is the model's ability to correctly **match a given thumbnail to the correct title** among several candidates. This is denoted **Recall@1**.\nWe can implement an evaluator compatible with the Sentence Transformers library to compute this metric. Since the code is quite long, I won't paste it here, but the curious reader can find it in Cell 12 of [this notebook](https://github.com/ShawhinT/YouTube-Blog/blob/main/multimodal-ai/4-ft-mm-embeddings/2-finetune_clip_sbert.ipynb).\npython\n\n```\n# function to create new evaluator given data split\ndef create_recall_evaluator(set_name, k=1):\n \"\"\"\n Create triplet evaluator for \"train\", \"valid\", or \"test\" split\n \"\"\"\n\n return ImageTextRetrievalEvaluator(\n images=dataset[f\"{set_name}\"][\"anchor\"],\n texts=dataset[f\"{set_name}\"][\"positive\"],\n name=f\"yt-title-thumbnail-{set_name}\",\n k=k\n )\n\n# Create new evaluator with Recall@k\nevaluator_recall_train = create_recall_evaluator(\"train\", k=1)\nevaluator_recall_valid = create_recall_evaluator(\"valid\", k=1)\n\nprint(\"Train:\", evaluator_recall_train(model))\nprint(\"Valid:\", evaluator_recall_valid(model))\n\n# >> Train: {'yt-title-thumbnail-train_Recall@1': 0.660377358490566}\n# >> Valid: {'yt-title-thumbnail-valid_Recall@1': 0.6363636363636364}\n```\n\nWe can see the model already has decent performance out-of-the-box, with correct titles being matched 66% of the time.\n## **Step 4: Fine-tune the Model**\nThere are **3 key things** we must do before training the model. Namely, choose which parameters to train, pick a loss function, and set hyperparameters.\n### **Trainable Parameters**\nThe key limitation of this project is that I've only posted 76 YouTube videos (as of writing this). With the validation and test splits, this leaves **only 53 examples for training**.\nSince we have so few training examples, **limiting the number of parameters we train is a good idea**. In this case, I only train the final projection layer of the model, which maps the text and image embeddings into a shared vector space. This is about 1M parameters total.\npython\n\n```\n# import model\nfrom sentence_transformers import SentenceTransformer\nmodel = SentenceTransformer(\"sentence-transformers/clip-ViT-L-14\")\n\n# pick specific layers to train (note: you can add more layers to this list)\ntrainable_layers_list = ['projection']\n\n# Apply freezing configuration\nfor name, param in model.named_parameters():\n\n # freeze all params\n param.requires_grad = False\n\n # unfreeze layers in trainable_layers_list\n if any(layer in name for layer in trainable_layers_list):\n param.requires_grad = True\n```\n\npython\n\n```\n# Count total and trainable parameters\ntotal_params = sum(p.numel() for p in model.parameters())\ntrainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n\nprint(f\"Total parameters: {total_params:,}\")\nprint(f\"Trainable parameters: {trainable_params:,}\")\nprint(f\"% of trainable parameters: {100*trainable_params/total_params:.2f}%\")\n\n# >> Total parameters: 427,616,513\n# >> Trainable parameters: 1,376,256\n# >> % of trainable parameters: 0.32%\n```\n\n### **Loss function**\nHere, I use the [Multiple Negatives Ranking Loss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) from the Sentence Transformers library (which works with single negatives like in this case). It works by **maximizing the similarity between positive pairs** while **minimizing the similarity between negative pairs**. Here's what the loss function looks like for the single negative case [2].\n![Mulitple negatives loss function \\(with only 1 negative\\). Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/15Rw76x_hN5cQUz5VAwILcQ.png)\nMulitple negatives loss function (with only 1 negative). Image by author.\npython\n\n```\nfrom sentence_transformers.losses import MultipleNegativesRankingLoss\n\n# define loss\nloss = MultipleNegativesRankingLoss(model)\n```\n\n### **Hyperparameters**\nFor hyperparameters, I experimented with a handful of choices manually and picked the choice with the best validation loss and Recall@1 performance. Here are the final choices.\npython\n\n```\nfrom sentence_transformers import SentenceTransformerTrainingArguments\n\n# hyperparameters\nnum_epochs = 2\nbatch_size = 16\nlr = 1e-4\nfinetuned_model_name = \"clip-title-thumbnail-embeddings\"\n\ntrain_args = SentenceTransformerTrainingArguments(\n output_dir=f\"models/{finetuned_model_name}\",\n num_train_epochs=num_epochs,\n per_device_train_batch_size=batch_size,\n per_device_eval_batch_size=batch_size,\n learning_rate=lr,\n # Evaluation settings\n eval_strategy=\"epoch\",\n eval_steps=1,\n logging_steps=1,\n)\n```\n\nWith our loss and hyperparameters defined, we can train the model using the SentenceTransformersTrainer().\npython\n\n```\nfrom sentence_transformers import SentenceTransformerTrainer\n\ntrainer = SentenceTransformerTrainer(\n model=model,\n args=train_args,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"valid\"],\n loss=loss,\n evaluator=[evaluator_recall_train, evaluator_recall_valid],\n)\ntrainer.train()\n```\n\n**Model training is an** **iterative process** where you may explore dozens of models for different choices of trainable parameters, loss functions, and hyperparameters.\nHowever, I highly recommend **keeping these experiments as simple as possible**. If you find yourself spending too much time tweaking training args to get your model to converge, there's probably something fundamentally wrong with your data (speaking from experience 😅 ).\n## **Step 5: Evaluate the Model**\nAs a final step, we can evaluate the model's Recall@1 score on the testing set. These data were not used for training or hyperparameter tuning, so it gives us an unbiased assessment of the model.\npython\n\n```\nevaluator_recall_test = create_recall_evaluator(\"test\")\n\nprint(\"Train:\", evaluator_recall_train(model))\nprint(\"Valid:\", evaluator_recall_valid(model))\nprint(\"Test:\", evaluator_recall_test(model))\n\n# >> Train: {'yt-title-thumbnail-train_Recall@1': 0.8490566037735849}\n# >> Valid: {'yt-title-thumbnail-valid_Recall@1': 0.9090909090909091}\n# >> Test: {'yt-title-thumbnail-test_Recall@1': 0.75}\n```\n\nWe see that the model performs well across all three datasets with **75% Recall@1 on the test set**. In other words, 75% of the time, the model correctly matches a given thumbnail to its original title. Additionally, the recall for the validation dataset increases by 27%!\n## **What's Next?**\nMultimodal embedding models, like CLIP, unlock countless 0-shot use cases such as image classification and retrieval. Here, we saw how we can fine-tune such a model to adapt it to a specialized domain (i.e. my YouTube titles and thumbnails).\nAlthough CLIP is a small model by today's standards (~500M parameters) and our training dataset was tiny, **the final model still demonstrated strong performance on this task**. This highlights the power of fine-tuning.\nIf you have any questions or suggestions for future content, let me know in the comments :)\n**More on Multimodal AI 👇**\n> [**Multimodal AI**](https://shawhin.medium.com/list/fe9521d0e77a)\n* * *\n**🗞 ️ Get exclusive access to AI resources and project ideas** : \n**🧑‍🎓 Learn AI in 6 weeks by building it:** h[ttps://maven.com/shaw-talebi/ai-builders-bootcamp?promoCode=AI25](https://maven.com/shaw-talebi/ai-builders-bootcamp?promoCode=AI25)\n## References\n[1] [arXiv:2103.00020](https://arxiv.org/abs/2103.00020) **[cs.CV]**\n[2] [arXiv:1705.00652](https://arxiv.org/abs/1705.00652) **[cs.CL]**\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657074748_utev3i.jpeg)\nWritten By\n## [Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[See all from Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[AI](https://towardsdatascience.com/tag/ai/)[Fine Tuning](https://towardsdatascience.com/tag/fine-tuning/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Sentence Transformers](https://towardsdatascience.com/tag/sentence-transformers/)[Transformers](https://towardsdatascience.com/tag/transformers/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-multimodal-embedding-models-bf007b1c5da5%2F&title=Fine-tuning%20Multimodal%20Embedding%20Models)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-multimodal-embedding-models-bf007b1c5da5%2F&title=Fine-tuning%20Multimodal%20Embedding%20Models)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-multimodal-embedding-models-bf007b1c5da5%2F&text=Fine-tuning%20Multimodal%20Embedding%20Models)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/06/1KmVHS7miXDwxMc1ZRMKXqQ.gif)](https://towardsdatascience.com/building-a-real-time-chat-application-with-nlp-super-powers-ce800e19cb2b/)\n## [Building a Real Time Chat Application with NLP Super Powers!](https://towardsdatascience.com/building-a-real-time-chat-application-with-nlp-super-powers-ce800e19cb2b/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA Chat App with Sentiment Analysis and Tone Detection using TensorFlow JS Deep Learning API, IBM Cloud, Node.JS, Web Sockets, and React\n[Deval Parikh](https://towardsdatascience.com/author/devalpp/)\nJune 1, 20215 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/11/181oVjlRcufPVKjZ2kuWXqg-scaled.jpeg)](https://towardsdatascience.com/a-full-end-to-end-deployment-of-a-machine-learning-algorithm-into-a-live-production-environment-3d9971ade188/)\n## [A Full End-to-End Deployment of a Machine Learning Algorithm into a Live Production Environment](https://towardsdatascience.com/a-full-end-to-end-deployment-of-a-machine-learning-algorithm-into-a-live-production-environment-3d9971ade188/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nHow to use scikit-learn, pickle, Flask, Microsoft Azure and ipywidgets to deploy a Python machine learning algorithm into a live...\n[Graham Harrison](https://towardsdatascience.com/author/grahamharrison-86487/)\nNovember 24, 202111 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/1xZne_oOiNwAqpxWFp2voxA.jpeg)](https://towardsdatascience.com/how-much-memory-is-your-ml-code-consuming-98df64074c8f/)\n## [How Much Memory is your ML Code Consuming?](https://towardsdatascience.com/how-much-memory-is-your-ml-code-consuming-98df64074c8f/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nLearn how to quickly check the memory footprint of your machine learning function/module with one line of command.\n[Tirthajyoti Sarkar](https://towardsdatascience.com/author/tirthajyoti/)\nMay 23, 20219 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/01/1SkluBJG8tGwU6RsCC0sOIw.jpeg)](https://towardsdatascience.com/machine-learning-cats-and-dogs-breeds-classifier-b26a9df45000/)\n## [Machine Learning Cats and Dogs Breeds Classifier](https://towardsdatascience.com/machine-learning-cats-and-dogs-breeds-classifier-b26a9df45000/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nLearn how to create a cat and dog breeds classifier using a machine learning model in Python to predict the breed of your pet\n[Mariana Santos](https://towardsdatascience.com/author/im.marianaas/)\nJanuary 11, 202213 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/18_odawTgvYDNAZMLr6xhCg.png)](https://towardsdatascience.com/routing-in-rag-driven-applications-a685460a7220/)\n## [Routing in RAG Driven Applications](https://towardsdatascience.com/routing-in-rag-driven-applications-a685460a7220/)\n[Programming](https://towardsdatascience.com/category/programming/)\nDirecting the application flow based on query intent\n[Sami Maameri](https://towardsdatascience.com/author/ssmaameri/)\nMay 9, 20249 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/1m5Rq5JMxdDy-Ml7y5qeKeg-scaled.jpeg)](https://towardsdatascience.com/pycaret-and-streamlit-how-to-create-and-deploy-data-science-web-app-273d205271a3/)\n## [PyCaret and Streamlit: How to Create and Deploy Data Science Web App](https://towardsdatascience.com/pycaret-and-streamlit-how-to-create-and-deploy-data-science-web-app-273d205271a3/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nBuilding a wine classifier with PyCaret and using Streamlit to create and deploy the web app\n[Ruben Winastwan](https://towardsdatascience.com/author/marcellusruben/)\nDecember 22, 202010 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/0U3m05SEZe-nPsKaa-scaled.jpg)](https://towardsdatascience.com/understand-your-data-in-real-time-1f6d9f6937e5/)\n## [Understand your Data in Real-Time](https://towardsdatascience.com/understand-your-data-in-real-time-1f6d9f6937e5/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nwith bytewax and ydata-profiling\n[Miriam Santos](https://towardsdatascience.com/author/miriam.santos/)\nJuly 20, 20238 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2017/09/1AFximUznOGWAM2Iw3hmA_Q.jpeg)](https://towardsdatascience.com/very-simple-python-script-for-extracting-most-common-words-from-a-story-1e3570d0b9d0/)\n## [Very simple Python script for extracting most common words from a story](https://towardsdatascience.com/very-simple-python-script-for-extracting-most-common-words-from-a-story-1e3570d0b9d0/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nOften one needs to quickly examine and plot most frequent words in a large text file like a story. How to do it?\n[Tirthajyoti Sarkar](https://towardsdatascience.com/author/tirthajyoti/)\nSeptember 23, 20172 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/03/0SLLfW0yMNyEGZJpD.png)](https://towardsdatascience.com/variance-reduction-on-steroids-introducing-mlrate-be328cd71a03/)\n## [Variance reduction on steroids... introducing MLRATE](https://towardsdatascience.com/variance-reduction-on-steroids-introducing-mlrate-be328cd71a03/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nMLRATE uses machine learning to reach statistical power quicker when experimenting: we provide examples from Wayve, an autonomous vehicle...\n[Chris Kelly](https://towardsdatascience.com/author/chrisckelly/)\nMarch 11, 20229 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\nword word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word\n \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3642/image?etag=1780830904480&width=500&height=500) Check Your Biases ](https://towardsdatascience.com/check-your-biases-ba2685a7f799/ \"Check Your Biases\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4973/image?etag=1782232567105&width=500&height=500) Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads ](https://towardsdatascience.com/machine-learning-tricks-learning-with-chatgpt-and-other-recent-must-reads-f277648a1921/ \"Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads\")\n\n\n", "metadata": { "title": "Fine-tuning Multimodal Embedding Models | Towards Data Science", "description": "Adapting CLIP to YouTube Data (with Python Code)", "keywords": null, "author": "Shaw Talebi", "og:locale": "en_US", "og:type": "article", "og:title": "Fine-tuning Multimodal Embedding Models", "og:description": "Adapting CLIP to YouTube Data (with Python Code)", "og:url": "https://towardsdatascience.com/fine-tuning-multimodal-embedding-models-bf007b1c5da5/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/0Y524lLKsF5spVr0K-scaled.jpg", "og:image:width": "2560", "og:image:height": "1707", "og:image:type": "image/jpeg", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Shaw Talebi", "twitter:label2": "Est. reading time", "twitter:data2": "10 min read", "article:published_time": "2025-01-31T22:31:23.000Z", "article:modified_time": "2025-01-31T22:31:23.000Z", "article:author": "https://towardsdatascience.com/author/shawhin/" }, "paywall": { "listed_paid": true, "detected": false, "marker": "", "public_only": true, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "", "retried": true }, "code_blocks": [ { "language": "txt", "file": "attachments/9b184704bb79/code/block_000.txt", "chars": 227 }, { "language": "txt", "file": "attachments/9b184704bb79/code/block_001.txt", "chars": 965 }, { "language": "txt", "file": "attachments/9b184704bb79/code/block_002.txt", "chars": 832 }, { "language": "txt", "file": "attachments/9b184704bb79/code/block_003.txt", "chars": 955 }, { "language": "txt", "file": "attachments/9b184704bb79/code/block_004.txt", "chars": 773 }, { "language": "txt", "file": "attachments/9b184704bb79/code/block_005.txt", "chars": 80 }, { "language": "txt", "file": "attachments/9b184704bb79/code/block_006.txt", "chars": 872 }, { "language": "txt", "file": "attachments/9b184704bb79/code/block_007.txt", "chars": 787 }, { "language": "txt", "file": "attachments/9b184704bb79/code/block_008.txt", "chars": 532 }, { "language": "txt", "file": "attachments/9b184704bb79/code/block_009.txt", "chars": 466 }, { "language": "txt", "file": "attachments/9b184704bb79/code/block_010.txt", "chars": 128 }, { "language": "txt", "file": "attachments/9b184704bb79/code/block_011.txt", "chars": 525 }, { "language": "txt", "file": "attachments/9b184704bb79/code/block_012.txt", "chars": 306 }, { "language": "txt", "file": "attachments/9b184704bb79/code/block_013.txt", "chars": 391 } ], "images": [ { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/0Y524lLKsF5spVr0K-scaled.jpg", "alt": "Photo by Markus Winkler on Unsplash", "local_path": "attachments/9b184704bb79/images/000.jpg", "bytes": 562544, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/1c6UUQUk10H61dy4lCjsrtA.png", "alt": "Stock photos from Canva. Image by author.", "local_path": "attachments/9b184704bb79/images/001.png", "bytes": 119967, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/15Rw76x_hN5cQUz5VAwILcQ.png", "alt": "Mulitple negatives loss function (with only 1 negative). Image by author.", "local_path": "attachments/9b184704bb79/images/002.png", "bytes": 21719, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/06/1KmVHS7miXDwxMc1ZRMKXqQ.gif", "alt": "", "local_path": "attachments/9b184704bb79/images/003.gif", "bytes": 3396204, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/11/181oVjlRcufPVKjZ2kuWXqg-scaled.jpeg", "alt": "", "local_path": "attachments/9b184704bb79/images/004.jpeg", "bytes": 351636, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/1xZne_oOiNwAqpxWFp2voxA.jpeg", "alt": "", "local_path": "attachments/9b184704bb79/images/005.jpeg", "bytes": 202019, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/01/1SkluBJG8tGwU6RsCC0sOIw.jpeg", "alt": "", "local_path": "attachments/9b184704bb79/images/006.jpeg", "bytes": 69737, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/18_odawTgvYDNAZMLr6xhCg.png", "alt": "", "local_path": "attachments/9b184704bb79/images/007.png", "bytes": 73136, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/1m5Rq5JMxdDy-Ml7y5qeKeg-scaled.jpeg", "alt": "", "local_path": "attachments/9b184704bb79/images/008.jpeg", "bytes": 153812, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/0U3m05SEZe-nPsKaa-scaled.jpg", "alt": "", "local_path": "attachments/9b184704bb79/images/009.jpg", "bytes": 296006, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2017/09/1AFximUznOGWAM2Iw3hmA_Q.jpeg", "alt": "", "local_path": "attachments/9b184704bb79/images/010.jpeg", "bytes": 57148, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/03/0SLLfW0yMNyEGZJpD.png", "alt": "", "local_path": "attachments/9b184704bb79/images/011.png", "bytes": 899283, "downloaded": true } ], "audio": [], "videos": [], "links": [], "retry_at": "2026-10-03T08:00:10", "repair_version": 2, "repaired_at": "2026-10-14T11:11:49Z", "source_url": "https://towardsdatascience.com/fine-tuning-multimodal-embedding-models-bf007b1c5da5/", "delivery_status": "excluded_paid", "discovery_status": "excluded_paid", "missing_reason": "按原约定排除付费/会员全文" }