{ "source_file": "0930", "source_platform": "Medium", "author": "Sascha Kirch", "title_listed": "DINO - A Foundation Model for Computer Vision", "keywords": "", "domain": "图像处理", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://towardsdatascience.com/dino-a-foundation-model-for-computer-vision-4cb08e821b18/", "publish_date": "2023-09-27", "corpus_stage": "", "notes": "", "id": "6609a57ffa9a", "collected_at": "2026-10-02T00:50:45", "title": "DINO - A Foundation Model for Computer Vision | Towards Data Science", "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# DINO - A Foundation Model for Computer Vision\nEmerging Properties in Self-Supervised Vision Transformers by M. Caron et. al.\n[Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\nSeptember 27, 202313 min read\n### [🚀 Sascha's Paper Club](https://towardsdatascience.com/tagged/saschas-paper-club)\nIt is an exciting decade for computer vision. Great successes from the natural language domain are transferred to the vision domain including the introduction of the ViT (vision transformer) and lately large-scale self-supervised pre-training techniques have made headlines under the name of foundation models.\nToday we are looking into a framework called DINO (self **DI** stillation, **N** O labels), a visual foundation model built on interesting properties of ViTs. It is also the predecessor of one of today's best performing foundation models: [DINOv2](https://arxiv.org/abs/2304.07193).\n![Image created from publication by Sascha Kirch](../images/000.png)\nImage created from [publication](https://arxiv.org/abs/2104.14294) by [Sascha Kirch](https://medium.com/@SaschaKirch)\n> **Paper:** [Emerging Properties in Self-Supervised Vision Transformers](https://arxiv.org/abs/2104.14294), by [Mathilde Caron](https://arxiv.org/search/cs?searchtype=author&query=Caron%2C+M) et.al., 29. Apr. 2021\n> **Resources:** [GitHub](https://github.com/facebookresearch/dino) -[ Blog Post](https://ai.meta.com/blog/dino-paws-computer-vision-with-self-supervised-transformers-and-10x-more-efficient-training/)\n> **Category:** foundation model, computer vision, vision transformer, knowledge distillation, similarity learning, self-supervised learning\n> [**Other Walkthroughs**](https://medium.com/@SaschaKirch/list/paper-walkthroughs-by-sascha-kirch-89c7847da8e2)**:** [[BYOL](https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c?sk=fc5a3b3a556088181d8726226862252c)] - [[CLIP](https://towardsdatascience.com/the-clip-foundation-model-7770858b487d?sk=a7b10ba1d0c3a20ecd4adb8200a48500)] - [[GLIP](https://towardsdatascience.com/glip-introducing-language-image-pre-training-to-object-detection-5ddb601873aa?sk=4f0acb404a38d342b7669f861c013a05)] - [[Segment Anything](https://towardsdatascience.com/segment-anything-promptable-segmentation-of-arbitrary-objects-f28958c5612d?sk=bd1311a6d8b1e0e6d3369d536dba0700)] - [[DINO](https://towardsdatascience.com/segment-anything-promptable-segmentation-of-arbitrary-objects-f28958c5612d?sk=bd1311a6d8b1e0e6d3369d536dba0700)] - [[Depth Anything](https://medium.com/towards-data-science/depth-anything-a-foundation-model-for-monocular-depth-estimation-8a7920b5c9cc?sk=fc6197edd68e6137c3396c83e50f65cb)] - [[DDPM](https://towardsdatascience.com/the-rise-of-diffusion-models-a-new-era-of-generative-deep-learning-3ef4779f6e1b?sk=8c178422a977c6f49ec24b13502be4fd)]\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\n## Outline\n 1. Context & Background\n 2. Method\n 3. Experiments\n 4. Ablations\n 5. Conclusion\n 6. Further Readings & Resources\n\n\n## Context & Background\nThe year is 2021, April to be precise. It has been four years since the release of the transformer model with [attention is all you need](https://arxiv.org/abs/1706.03762). Self-supervised pre-training is long being practiced in NLP by models such as [BERT](https://arxiv.org/pdf/1810.04805.pdf) and the term foundation model is not yet known for the next few months until the release of _\"_[_on the opportunities and Risks of Foundation Models_](https://arxiv.org/abs/2108.07258) _\"_. Six months earlier the [Vision transformer (ViT)](https://arxiv.org/abs/2010.11929) was first published on arxiv and it is still one month until ICLR 2021 where it will be presented.\n> Let that sink in for a moment: ViT had its debut on arxiv.org in October 2020 and was presented on ICLR2021 in May 2021. DINO was released on arxiv in April 2021. So, one month before it was actually presented on a conference. This would mean they only had 5 months if they had started right away to come up with the project's idea, compile a team, lay out the theoretical foundation, train the model, perform experiments and ablations, and write the paper. No wonder PhD students these days feel constantly anxious. At least that's what's happening to me sometimes _😅_\nWhile ViTs were very competitive with convolutional networks, they are demanding in terms of computational resources and amount of training data.\nThe authors of DINO made a simple observation: the success of transformers in NLP was coupled with self-supervised pre-training and current self-supervised methods in the vision domain are built from convnets, like e.g. [BYOL](https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c?source=friends_link&sk=fc5a3b3a556088181d8726226862252c).\n> [**BYOL -The Alternative to Contrastive Self-Supervised Learning**](https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c)\nInspired by [BYOL](https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c?source=friends_link&sk=fc5a3b3a556088181d8726226862252c) and the [mean teacher](https://arxiv.org/abs/1703.01780), the authors came up with a framework to train a ViT in a self-supervised fashion and found:\n 1. Self-supervised ViT features explicitly contain the scene layout and, in particular, object boundaries.\n 2. Self-supervised ViT features perform particularly well with a basic nearest neighbors classifier (k-NN) without any fine-tuning, linear classifier nor data augmentation.\n\n\nIn contrast to BYOL and mean teacher, DINO implements a knowledge-distillation framework consisting of a student and teacher model that acts upon different views of the same image and adds extra measures to deal with inherent instabilities of similarity-learning approaches, where solutions often collapse.\nAn interesting finding of the underlying vision transformer architecture (ViT) is that when trained with unsupervised learning techniques its features contain explicit information about the semantic segmentation of an image. One can simply visualize the self-attention map of selected heads of the multi-head attention layer as shown in the video bellow:\n![Fig. 1: Self-attention maps for selected heads. Source](../images/001.png)\nFig. 1: Self-attention maps for selected heads. [Source](https://arxiv.org/abs/2104.14294)\nLet us unpack another layer of abstraction and let's try to understand how DINO implements its framework, tackles instabilities and how it performs compared to previous methods!\n> [**Paper Walkthroughs by Sascha Kirch**](https://medium.com/@SaschaKirch/list/89c7847da8e2)\n* * *\n## Method\nThe DINO framework shares the same overall structure with other similarity-learning frameworks like [BYOL](https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c?source=friends_link&sk=fc5a3b3a556088181d8726226862252c) or the [mean teacher](https://arxiv.org/abs/1703.01780) but also with knowledge distillation. Let's first have a look on how DINO does it and the differentiate between the other frameworks.\n![Fig. 2: DINO architecture. Source + annotations by Sascha Kirch](../images/002.png)\nFig. 2: DINO architecture. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\n### **Networks and Update Rule**\nLet's start from the middle. DINO implements two networks with the exact same architecture but a different set of weights. Those are the student and the teacher. The student is trained with back propagation and the teacher updates its weights with an exponential moving average of its own weights and those of the student.\n![Equation 1: Update rule of the teacher's weights. Source + annotations by Sascha Kirch](../images/003.png)\nEquation 1: Update rule of the teacher's weights. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\nBackbones are either a ResNet50 or [DeiT](https://arxiv.org/abs/2012.12877) (which is a [ViT](https://arxiv.org/abs/2010.11929) adapted for knowledge distillation). An MLP-based projection head is connected to the backbone to reduce the dimensionality of the features, but is removed for inference.\n_**Nice, but which model is used for inference: student or teacher?**_ - Well that's a good question and funny enough not a single word is mentioned in the paper. Intuitively you might think the student, at least I did at first. But as we will see later, the teacher outperforms the student throughout the training. The only hint beside the better performance is that in the code implementation the teacher checkpoint is the default one for the evaluation of for example [video segmentation](https://github.com/facebookresearch/dino/blob/7c446df5b9f45747937fb0d72314eb9f7b66930a/eval_video_segmentation.py#L257C9-L257C9), [linear probing](https://github.com/facebookresearch/dino/blob/7c446df5b9f45747937fb0d72314eb9f7b66930a/eval_linear.py#L264C34-L264C34) and [k-NN](https://github.com/facebookresearch/dino/blob/7c446df5b9f45747937fb0d72314eb9f7b66930a/eval_knn.py#L203C32-L203C32). Since this parameter can be changed though, I cannot tell you with certainty.\n### **Inputs and Outputs**\nFrom an input image _x_ different views _x1_ and _x2_ are created by cropping and applying image augmentations like in [BYOL](https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c?source=friends_link&sk=fc5a3b3a556088181d8726226862252c) (e.g. color jitter, Gaussian blur and solarization). The technique used for cropping is called[ multi-crop](https://arxiv.org/abs/2006.09882) where multiple crops of different sizes are generated to save memory while providing more data. Small crops are called local views and consist of 96x96 pixels that are exclusively feed into the student. Larger crops are called global views and consists of 224x224 pixels that are exclusively fed into the teacher. As we will see later in the ablation section, 2 global views and 10 local views have been used during training.\n> NOTE: The paper is a bit confusing regarding the multi-crop technique because neither the provided pseudo-code nor the architecture shown in Fig. 3 above reflect it. The pseudo code even suggests that x1 and x2 are feed into both, the student and the teacher like in BYOL, which is not the case when using multi-crop.\nIn contrast to similarity learning where the objective is to maximize the similarity of embeddings, DINO minimizes the cross-entropy between the teacher's and the student's output distribution. As indicated by the equation bellow, the cross-entropy is calculated for each pair of global and local views and is then summed up.\n![Equation 2: Optimization objective. Source + annotations by Sascha Kirch](../images/004.png)\nEquation 2: Optimization objective. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\n _**And what do the models output?**_ - Like in similarity learning, the student and the teacher output an embedding for a given image, rather than a prediction score. Like in knowledge distillation, the output is transformed via a SoftMax transformation into a probability distribution. The SoftMax has a temperature parameter that controls the smoothing or sharpening of the resulting distribution. This temperature plays a crucial role in knowledge distillation because it allows to control the balance between transferring general knowledge and fine-grained details from a teacher network to a student network, making the distillation process more effective for different tasks.\n![Fig. 3: Effect of temperature value on the SoftMax output. Illustration by Sascha Kirch created with this python notebook](../images/005.png)\nFig. 3: Effect of temperature value on the SoftMax output. Illustration by [Sascha Kirch](https://medium.com/@SaschaKirch) created with [this python notebook](https://github.com/sascha-kirch/ML_Notebooks/blob/main/Softmax_Temperature.ipynb)\nI created a notebook for you so you can investigate the impact of the temperature on the resulting distribution:\n> [**ML_Notebooks/Softmax_Temperature.ipynb at main · sascha-kirch/ML_Notebooks**](https://github.com/sascha-kirch/ML_Notebooks/blob/main/Softmax_Temperature.ipynb)\n### **Avoiding Collapse**\nAs mentioned earlier, student and teacher have the exact same architecture. This kind of setup is unstable (if no counter measures are implemented) and might result in collapsing solutions, where all features are mapped to a certain region in the latent space, e.g. a single point in the worst case. BYOL addressed this issue with an extra prediction head for only one of the models introducing an asymmetry. Since DINO has symmetric models another trick is required: centering and sharpening. Both are applied to the teacher network only. Centering is a technique that prevents a single dimension in the latent space to dominate, by adding a bias term _c_ to the teachers output _g(x) = g(x)+c_ , where\n![Equation 3: Update rule of the centering term. Source + annotations by Sascha Kirch](../images/006.png)\nEquation 3: Update rule of the centering term. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\nWhile centering has a positive effect, it also encourages the output to collapse into a uniform distribution. Sharpening has the opposite effect hence applying both balances their effect and stabilizes training. Sharpening is achieved by using a smaller temperature in the SoftMax (see Fig. 3) for the teacher as for the student.\nTo avoid collapsing the hyperparameter _m_ from equation 3 and the temperature of the teacher are crucial. In their ablation study in the appendix section the authors show that _m=0.9...0.999_ works best and the temperature value is linearly increased from _0.04_ to _0.07_ during warm-up.\n### **What does DINO do? Knowledge Distillation or Similarity Learning?**\nThe answer is a little bit of both!\nWhile knowledge distillation usually distils knowledge from an already trained, larger and more accurate teacher model into a smaller student model, it could also be seen as some sort of similarity learning because it encourages the student network to produce predictions that are similar to those of the teacher. In similarity learning, the two models are usually trained jointly and often align their latent space predictions rather than probability distributions.\nSince the authors of DINO phrase their objective as knowledge distillation, let's have a look on some differences compared with \"standard\" knowledge distillation:\n 1. DINO's teacher is not available a priori but \"trained\" alongside the student. It can even be considered as a co-distillation since knowledge is also distilled from the student into the teacher.\n 2. DINO's teacher and student are not acting on the same input but on different views of the image cropped to different sizes.\n 3. DINO uses different temperatures in the SoftMax of both models to perform sharpening.\n 4. DINO calculates the cross-entropy over the temperature-scaled SoftMax of the embeddings rather than prediction scores.\n\n\nAnd how is it similar to knowledge distillation?:\n 1. DINO consists of a student and a teacher network, where the teacher performs better than the student as we will see in the experiments.\n 2. Rather than maximizing a similarity metric, DINO minimizes the cross-entropy loss of a temperature scaled SoftMax output.\n\n\n> [**Get an email whenever Sascha Kirch publishes 🚀**](https://medium.com/@SaschaKirch/subscribe)[ _Get an email whenever Sascha Kirch publishes 🚀 Looking to learn more about deep learning or simply stay up to dat_e...medium.com](https://medium.com/@SaschaKirch/subscribe)\n* * *\n## Experiments\nThe paper presents a vast number of experiments. They pre-train the model on ImageNet, a commonly used dataset in representation learning.\nFor the evaluation, common techniques usually either train a linear classifier on top of frozen features or fine-tune the model to new downstream tasks, where the parameters of the model are adapted.\nThe authors of DINO claim that those techniques are very sensitive to hyperparameters which makes comparisons unfair and hard to reproduce. Hence, they propose to use a simple nearest neighbor clustering algorithm on the features of the pre-trained model.\n### Linear and k-NN Classification on ImageNet\nIn a this experiment the models are tested on their image classification accuracy on ImageNet. A variety of self-supervised pre-trained models are tested with either a ResNet or a ViT backbone. The classification is done either with linear probing or k-NN clustering.\n![Table 1: Linear and k-NN classification on ImageNet. Source + annotations by Sascha Kirch](../images/007.png)\nTable 1: Linear and k-NN classification on ImageNet. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\nI guess the key take-aways are:\n 1. K-NN performs better on ViT features than on ResNet features.\n 2. Decreasing patch size in the ViT has larger improvement as larger backbone, but at the cost of slower inference.\n\n\n### Video Instance Segmentation\nAn important experiment has been the video segmentation task, since the paper is about the ViT's capability to capture semantic segmentation in its features when trained with unsupervised methods. Or let's say that's what is claimed 😁\n![Table 2: Video Instance segmentation. Source + annotations by Sascha Kirch](../images/008.png)\nTable 2: Video Instance segmentation. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\nObserving those results I am missing two further experiments:\n 1. It would be nice to see a comparison of a supervised ResNet50 and a self-supervised ResNet50 in the DINO framework to support their claim that the ViT is superior to the ResNet architecture.\n 2. It would also be great to see the same set of ViT backbones for supervised as for self-supervised to see the impact on patch-size and model size.\n\n\nBut as I always say: asking questions is easy 😁 In real-world projects the authors often face resource constraints and project deadlines so not every single little detail can be covered!\n### Probing the Self-Attention Map\nIn this experiment the authors investigated the self-attention maps of different heads in the multi-head self-attention layers of the ViT. They visualize the attention maps from selected heads from the last layer of ViT-S/8, those of the learned [CLS] token to be precise.\n![Fig. 4: Attention maps from selected heads. Source + annotations by Sascha Kirch](../images/009.png)\nFig. 4: Attention maps from selected heads. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\n### Other Experiments\nIn other experiments, DINO improved compared against the supervised baseline. Those tasks include image retrieval and copy detection.\n* * *\n## Ablations\nFor their ablation study the authors experiment with the ViT-S model.\n### Importance of Patch Size\nRecall that a vision transformer inputs a patchified version of an input image, transforms each patch into a token and then applies a transformer with its self-attention mechanism. This was a trick by the authors of ViT to reduce the compute requirements for trading-off performance, making transformers applicable to image data.\nDINO claims that smaller size of the patches increases the performance while decreasing the throughput (number of images that can be processed per second), which is exactly what ViT claims.\n![Fig. 5: Impact of patch size on accuracy and throughput. Source + annotations by Sascha Kirch](../images/010.png)\nFig. 5: Impact of patch size on accuracy and throughput. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\nIntuitively I'd say it is no surprise since you increase the input resolution and you end up with more tokens to attend to, so you end up with a fine-grained attention map.\n### Different Teacher Update Rules\nThe teacher in DINO is updated by calculating the exponential moving average from the updated student and the current teacher. This is the \"momentum encoder\" approach they refer to.\nUsing a momentum encoder and plotting the accuracy of the teacher and the student during training, the teacher performs better throughout the entire process. From this we can hypothesize:\n 1. the teacher can provide a strong learning signal to the student.\n 2. an improving student improves the teacher due to the EMA update rule (co-distillation).\n 3. One can use the Teacher as final model which has better performance but the same architecture as the student, hence no change in compute requirements.\n\n\n![Fig. 6: Teacher performance. Source + annotations by Sascha Kirch](../images/011.png)\nFig. 6: Teacher performance. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\nThey also experiment with 3 other update rules: copying the weights from the student to the teacher, use the student weights from the previous iteration of the optimizer and use the student weights from the previous epoch.\n### Multi-Crop vs. Time and GPU Memory\nAs mentioned earlier, DINO inputs multiple cropped views of the same image and feeds the global views into the teacher and the local views into the student. In this ablation, the authors experiment with different amounts of local views and report the impact on performance, training time and peak memory per GPU.\n![Table 3: Multi-Crop vs. Time and GPU Memory. Source + annotations by Sascha Kirch](../images/012.png)\nTable 3: Multi-Crop vs. Time and GPU Memory. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\n### Avoiding Collapse\nIn this ablation the authors evaluated the role of their stabilizing measures to avoid collapsing solutions: centering and sharpening.\nTo do so, they decomposed the cross-entropy into an entropy term and a Kullback-Leibler (KL) divergence term. KL divergence is a measure of difference of two probability distributions. If KL is 0, two distributions are considered equal.\nThe intuition behind this is the following: if the KL divergence of the output distribution of the teacher and the student is constant throughout the training, there is no learning signal for updating the weights of the student.\n![Fig. 7: Analysis of collapsing solutions. Source + annotations by Sascha Kirch](../images/013.png)\nFig. 7: Analysis of collapsing solutions. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\n### Effect of Batch Size\nAn interesting property is that DINO can be trained with small batch sizes without a large drop in performance. This was actually one of BYOL's motivation, a paper DINO builds upon, to be less dependent on batch size compared to contrastive self-supervised learning.\n![Table 4: Batch size vs. accuracy. Source + annotations by Sascha Kirch](../images/body-2551c805750759b87878.png)\nTable 4: Batch size vs. accuracy. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\nContrastive methods like CLIP and GLIP provide a lot of negative samples for a given positive sample to avoid collapsing solutions. The more negative samples per optimizer update step (hence per batch) the better it works.\n* * *\n## Conclusion\nIn conclusion, DINO is a knowledge-distillation framework. It is a visual foundation model that exploits interesting properties of ViTs and is the predecessor of one of today's best-performing foundation models, DINOv2. DINO's framework consists of a student and teacher model that acts upon different views of the same image and adds extra measures to deal with inherent instabilities of similarity-learning approaches. The experiments show that DINO outperforms other self-supervised pre-trained models on various tasks.\n* * *\n## Further Readings & Resources\n### Papers\nIn the meantime an improved version of DINO has been released:\n 1. [DINOv2: Learning Robust Visual Features without Supervision](https://arxiv.org/abs/2304.07193)\n 2. [Blog post DINOv2 by Meta](https://ai.meta.com/blog/dino-v2-computer-vision-self-supervised-learning/)\n\n\n* * *\n### Paper Walkthroughs\nYou might also like my other paper walkthroughs covering concepts we discussed in this article:\n> [**The CLIP Foundation Model**](https://towardsdatascience.com/the-clip-foundation-model-7770858b487d)\n> [**GLIP: Introducing Language-Image Pre-Training to Object Detection**](https://towardsdatascience.com/glip-introducing-language-image-pre-training-to-object-detection-5ddb601873aa)\n> [**BYOL -The Alternative to Contrastive Self-Supervised Learning**](https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c)\n> [**Segment Anything - Promptable Segmentation of Arbitrary Objects**](https://towardsdatascience.com/segment-anything-promptable-segmentation-of-arbitrary-objects-f28958c5612d)\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-2648fe230469ec3ba2b1.png)\nWritten By\n## [Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\n[See all from Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Computer Vision](https://towardsdatascience.com/tag/computer-vision/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Knowledge Distillation](https://towardsdatascience.com/tag/knowledge-distillation/)[Saschas Paper Club](https://towardsdatascience.com/tag/saschas-paper-club/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fdino-a-foundation-model-for-computer-vision-4cb08e821b18%2F&title=DINO%20-%20A%20Foundation%20Model%20for%20Computer%20Vision)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fdino-a-foundation-model-for-computer-vision-4cb08e821b18%2F&title=DINO%20-%20A%20Foundation%20Model%20for%20Computer%20Vision)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fdino-a-foundation-model-for-computer-vision-4cb08e821b18%2F&text=DINO%20-%20A%20Foundation%20Model%20for%20Computer%20Vision)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](../images/014.png)](https://towardsdatascience.com/nlp-splitting-text-into-sentences-7bbce222ef17/)\n## [NLP: Splitting Text into Sentences](https://towardsdatascience.com/nlp-splitting-text-into-sentences-7bbce222ef17/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nLanguage agnostic model for transforming text into sentences\n[Moran Beladev](https://towardsdatascience.com/author/moranbeladev90/)\nAugust 1, 20214 min read\n * [![](../images/015.jpg)](https://towardsdatascience.com/deep-reinforcement-learning-hands-on-for-optimized-ad-placement-b402ffa47245/)\n## [Deep Reinforcement Learning hands-on for Optimized Ad Placement](https://towardsdatascience.com/deep-reinforcement-learning-hands-on-for-optimized-ad-placement-b402ffa47245/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nDeep Reinforcement technique is used in this article to optimize Ad placement on a website to maximize the probability of user clicks and...\n[NandaKishore Joshi](https://towardsdatascience.com/author/nandakishorej8/)\nOctober 28, 20218 min read\n * [![](../images/016.jpg)](https://towardsdatascience.com/top-python-libraries-for-image-augmentation-in-computer-vision-2566bed0533e/)\n## [Top Python libraries for Image Augmentation in Computer Vision](https://towardsdatascience.com/top-python-libraries-for-image-augmentation-in-computer-vision-2566bed0533e/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nFeaturing the best augmentation libraries (along with sample codes) for your next computer vision project\n[Kenneth Leung](https://towardsdatascience.com/author/kennethleungty/)\nAugust 2, 20215 min read\n * [![](../images/017.png)](https://towardsdatascience.com/denoising-radar-satellite-images-with-python-has-never-been-so-easy-f445241002a5/)\n## [Denoising Radar Satellite Images with Python Has Never Been So Easy](https://towardsdatascience.com/denoising-radar-satellite-images-with-python-has-never-been-so-easy-f445241002a5/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nPresentation of the latest release of deepdespeckling\n[Hadrien Mariaccia](https://towardsdatascience.com/author/brash6/)\nApril 23, 20247 min read\n * [![](../images/018.png)](https://towardsdatascience.com/deep-learning-illustrated-part-4-recurrent-neural-networks-d0121f27bc74/)\n## [Deep Learning Illustrated, Part 4: Recurrent Neural Networks](https://towardsdatascience.com/deep-learning-illustrated-part-4-recurrent-neural-networks-d0121f27bc74/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAn illustrated and intuitive guide on the inner workings of an RNN and the Softmax Activation Function\n[Shreya Rao](https://towardsdatascience.com/author/shreya.rao/)\nJune 11, 202415 min read\n * [![](../images/019.jpeg)](https://towardsdatascience.com/a-must-have-tool-for-every-data-scientist-5e7c76f1916f/)\n## [A Must-Have Tool for Every Data Scientist](https://towardsdatascience.com/a-must-have-tool-for-every-data-scientist-5e7c76f1916f/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nTensorDash - An app to remotely monitor your Machine Learning models\n[Arun](https://towardsdatascience.com/author/cleanpegasus/)\nOctober 21, 20204 min read\n * [![](../images/020.png)](https://towardsdatascience.com/residual-networks-in-computer-vision-ee118d3be68f/)\n## [Residual Networks in Computer Vision](https://towardsdatascience.com/residual-networks-in-computer-vision-ee118d3be68f/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nDeep Learning application using Tensorflow and Keras Deep Convolutional Neural Networks changed the research landscape...\n[Luís Roque](https://towardsdatascience.com/author/luisroque/)\nMay 27, 202113 min read\n * [![](../images/021.png)](https://towardsdatascience.com/how-to-leverage-neural-style-transfer-to-create-stunning-images-c30eb4a9007a/)\n## [How to Leverage Neural Style Transfer to Create Stunning Images](https://towardsdatascience.com/how-to-leverage-neural-style-transfer-to-create-stunning-images-c30eb4a9007a/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nAI that can paint images\n[Nour Islam Mokhtari](https://towardsdatascience.com/author/nourislam/)\nDecember 6, 20214 min read\n * [![](../images/022.jpeg)](https://towardsdatascience.com/deep-neural-networks-how-to-define-73d87bf36421/)\n## [Deep neural networks: How to define?](https://towardsdatascience.com/deep-neural-networks-how-to-define-73d87bf36421/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nCorrectly defining what makes a neural network deep is important. This post proposes an alternative definition to deep neural networks.\n[Valdivino Santiago Júnior](https://towardsdatascience.com/author/vald.santiago/)\nSeptember 29, 20218 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n", "metadata": { "title": "DINO - A Foundation Model for Computer Vision | Towards Data Science", "description": "Emerging Properties in Self-Supervised Vision Transformers by M. Caron et. al.", "keywords": null, "author": "Sascha Kirch", "og:locale": "en_US", "og:type": "article", "og:title": "DINO - A Foundation Model for Computer Vision", "og:description": "Emerging Properties in Self-Supervised Vision Transformers by M. Caron et. al.", "og:url": "https://towardsdatascience.com/dino-a-foundation-model-for-computer-vision-4cb08e821b18/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1gR9pdqZRH2bREu8B5NByhA.png", "og:image:width": "1242", "og:image:height": "620", "og:image:type": "image/png", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Sascha Kirch", "twitter:label2": "Est. reading time", "twitter:data2": "13 min read", "article:published_time": "2023-09-27T18:31:18.000Z", "article:modified_time": "2023-09-27T18:31:18.000Z", "article:author": "https://towardsdatascience.com/author/saschakirch/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [ { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:37:57Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:37:57Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:37:57Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:38:36Z" } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/c/TowardsDataScience", "kind": "channel_reference", "status": "reference_only" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/deep-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/saschakirch/" }, { "type": "internal", "href": "https://towardsdatascience.com/tagged/saschas-paper-club" }, { "type": "internal", "href": "https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c?sk=fc5a3b3a556088181d8726226862252c" }, { "type": "internal", "href": "https://towardsdatascience.com/the-clip-foundation-model-7770858b487d?sk=a7b10ba1d0c3a20ecd4adb8200a48500" }, { "type": "internal", "href": "https://towardsdatascience.com/glip-introducing-language-image-pre-training-to-object-detection-5ddb601873aa?sk=4f0acb404a38d342b7669f861c013a05" }, { "type": "internal", "href": "https://towardsdatascience.com/segment-anything-promptable-segmentation-of-arbitrary-objects-f28958c5612d?sk=bd1311a6d8b1e0e6d3369d536dba0700" }, { "type": "internal", "href": "https://towardsdatascience.com/the-rise-of-diffusion-models-a-new-era-of-generative-deep-learning-3ef4779f6e1b?sk=8c178422a977c6f49ec24b13502be4fd" }, { "type": "internal", "href": "https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c?sk=fc5a3b3a556088181d8726226862252c&source=friends_link" }, { "type": "internal", "href": "https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c" }, { "type": "internal", "href": "https://towardsdatascience.com/the-clip-foundation-model-7770858b487d" }, { "type": "internal", "href": "https://towardsdatascience.com/glip-introducing-language-image-pre-training-to-object-detection-5ddb601873aa" }, { "type": "internal", "href": "https://towardsdatascience.com/segment-anything-promptable-segmentation-of-arbitrary-objects-f28958c5612d" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/deep-dives/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://arxiv.org/abs/2304.07193" }, { "type": "external", "href": "https://arxiv.org/abs/2104.14294" }, { "type": "external", "href": "https://medium.com/@SaschaKirch" }, { "type": "external", "href": "https://arxiv.org/search/cs?query=Caron%2C+M&searchtype=author" }, { "type": "external", "href": "https://github.com/facebookresearch/dino" }, { "type": "external", "href": "https://ai.meta.com/blog/dino-paws-computer-vision-with-self-supervised-transformers-and-10x-more-efficient-training/" }, { "type": "external", "href": "https://medium.com/@SaschaKirch/list/paper-walkthroughs-by-sascha-kirch-89c7847da8e2" }, { "type": "external", "href": "https://medium.com/towards-data-science/depth-anything-a-foundation-model-for-monocular-depth-estimation-8a7920b5c9cc?sk=fc6197edd68e6137c3396c83e50f65cb" }, { "type": "external", "href": "https://roadmap.sh/machine-learning" }, { "type": "external", "href": "https://arxiv.org/abs/1706.03762" }, { "type": "external", "href": "https://arxiv.org/pdf/1810.04805.pdf" }, { "type": "external", "href": "https://arxiv.org/abs/2108.07258" }, { "type": "external", "href": "https://arxiv.org/abs/2010.11929" }, { "type": "external", "href": "https://arxiv.org/abs/1703.01780" }, { "type": "external", "href": "https://medium.com/@SaschaKirch/list/89c7847da8e2" }, { "type": "external", "href": "https://arxiv.org/abs/2012.12877" }, { "type": "external", "href": "https://github.com/facebookresearch/dino/blob/7c446df5b9f45747937fb0d72314eb9f7b66930a/eval_video_segmentation.py" } ], "repair_version": 2, "repaired_at": "2026-10-20T17:24:00Z", "source_url": "https://towardsdatascience.com/dino-a-foundation-model-for-computer-vision-4cb08e821b18/", "delivery_status": "unverified", "discovery_status": "http_403", "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片", "repair_error": "", "actual_video_files": [], "expected_saved_body_images": 4, "image_discovery": "saved_markdown", "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# DINO - A Foundation Model for Computer Vision\nEmerging Properties in Self-Supervised Vision Transformers by M. Caron et. al.\n[Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\nSeptember 27, 202313 min read\n### [🚀 Sascha's Paper Club](https://towardsdatascience.com/tagged/saschas-paper-club)\nIt is an exciting decade for computer vision. Great successes from the natural language domain are transferred to the vision domain including the introduction of the ViT (vision transformer) and lately large-scale self-supervised pre-training techniques have made headlines under the name of foundation models.\nToday we are looking into a framework called DINO (self **DI** stillation, **N** O labels), a visual foundation model built on interesting properties of ViTs. It is also the predecessor of one of today's best performing foundation models: [DINOv2](https://arxiv.org/abs/2304.07193).\n![Image created from publication by Sascha Kirch](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1gR9pdqZRH2bREu8B5NByhA.png)\nImage created from [publication](https://arxiv.org/abs/2104.14294) by [Sascha Kirch](https://medium.com/@SaschaKirch)\n> **Paper:** [Emerging Properties in Self-Supervised Vision Transformers](https://arxiv.org/abs/2104.14294), by [Mathilde Caron](https://arxiv.org/search/cs?searchtype=author&query=Caron%2C+M) et.al., 29. Apr. 2021\n> **Resources:** [GitHub](https://github.com/facebookresearch/dino) -[ Blog Post](https://ai.meta.com/blog/dino-paws-computer-vision-with-self-supervised-transformers-and-10x-more-efficient-training/)\n> **Category:** foundation model, computer vision, vision transformer, knowledge distillation, similarity learning, self-supervised learning\n> [**Other Walkthroughs**](https://medium.com/@SaschaKirch/list/paper-walkthroughs-by-sascha-kirch-89c7847da8e2)**:** [[BYOL](https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c?sk=fc5a3b3a556088181d8726226862252c)] - [[CLIP](https://towardsdatascience.com/the-clip-foundation-model-7770858b487d?sk=a7b10ba1d0c3a20ecd4adb8200a48500)] - [[GLIP](https://towardsdatascience.com/glip-introducing-language-image-pre-training-to-object-detection-5ddb601873aa?sk=4f0acb404a38d342b7669f861c013a05)] - [[Segment Anything](https://towardsdatascience.com/segment-anything-promptable-segmentation-of-arbitrary-objects-f28958c5612d?sk=bd1311a6d8b1e0e6d3369d536dba0700)] - [[DINO](https://towardsdatascience.com/segment-anything-promptable-segmentation-of-arbitrary-objects-f28958c5612d?sk=bd1311a6d8b1e0e6d3369d536dba0700)] - [[Depth Anything](https://medium.com/towards-data-science/depth-anything-a-foundation-model-for-monocular-depth-estimation-8a7920b5c9cc?sk=fc6197edd68e6137c3396c83e50f65cb)] - [[DDPM](https://towardsdatascience.com/the-rise-of-diffusion-models-a-new-era-of-generative-deep-learning-3ef4779f6e1b?sk=8c178422a977c6f49ec24b13502be4fd)]\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\n## Outline\n 1. Context & Background\n 2. Method\n 3. Experiments\n 4. Ablations\n 5. Conclusion\n 6. Further Readings & Resources\n\n\n## Context & Background\nThe year is 2021, April to be precise. It has been four years since the release of the transformer model with [attention is all you need](https://arxiv.org/abs/1706.03762). Self-supervised pre-training is long being practiced in NLP by models such as [BERT](https://arxiv.org/pdf/1810.04805.pdf) and the term foundation model is not yet known for the next few months until the release of _\"_[_on the opportunities and Risks of Foundation Models_](https://arxiv.org/abs/2108.07258) _\"_. Six months earlier the [Vision transformer (ViT)](https://arxiv.org/abs/2010.11929) was first published on arxiv and it is still one month until ICLR 2021 where it will be presented.\n> Let that sink in for a moment: ViT had its debut on arxiv.org in October 2020 and was presented on ICLR2021 in May 2021. DINO was released on arxiv in April 2021. So, one month before it was actually presented on a conference. This would mean they only had 5 months if they had started right away to come up with the project's idea, compile a team, lay out the theoretical foundation, train the model, perform experiments and ablations, and write the paper. No wonder PhD students these days feel constantly anxious. At least that's what's happening to me sometimes _😅_\nWhile ViTs were very competitive with convolutional networks, they are demanding in terms of computational resources and amount of training data.\nThe authors of DINO made a simple observation: the success of transformers in NLP was coupled with self-supervised pre-training and current self-supervised methods in the vision domain are built from convnets, like e.g. [BYOL](https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c?source=friends_link&sk=fc5a3b3a556088181d8726226862252c).\n> [**BYOL -The Alternative to Contrastive Self-Supervised Learning**](https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c)\nInspired by [BYOL](https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c?source=friends_link&sk=fc5a3b3a556088181d8726226862252c) and the [mean teacher](https://arxiv.org/abs/1703.01780), the authors came up with a framework to train a ViT in a self-supervised fashion and found:\n 1. Self-supervised ViT features explicitly contain the scene layout and, in particular, object boundaries.\n 2. Self-supervised ViT features perform particularly well with a basic nearest neighbors classifier (k-NN) without any fine-tuning, linear classifier nor data augmentation.\n\n\nIn contrast to BYOL and mean teacher, DINO implements a knowledge-distillation framework consisting of a student and teacher model that acts upon different views of the same image and adds extra measures to deal with inherent instabilities of similarity-learning approaches, where solutions often collapse.\nAn interesting finding of the underlying vision transformer architecture (ViT) is that when trained with unsupervised learning techniques its features contain explicit information about the semantic segmentation of an image. One can simply visualize the self-attention map of selected heads of the multi-head attention layer as shown in the video bellow:\n![Fig. 1: Self-attention maps for selected heads. Source](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1xkYvzDjq_bnmemXh_QJ8mw.png)\nFig. 1: Self-attention maps for selected heads. [Source](https://arxiv.org/abs/2104.14294)\nLet us unpack another layer of abstraction and let's try to understand how DINO implements its framework, tackles instabilities and how it performs compared to previous methods!\n> [**Paper Walkthroughs by Sascha Kirch**](https://medium.com/@SaschaKirch/list/89c7847da8e2)\n* * *\n## Method\nThe DINO framework shares the same overall structure with other similarity-learning frameworks like [BYOL](https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c?source=friends_link&sk=fc5a3b3a556088181d8726226862252c) or the [mean teacher](https://arxiv.org/abs/1703.01780) but also with knowledge distillation. Let's first have a look on how DINO does it and the differentiate between the other frameworks.\n![Fig. 2: DINO architecture. Source + annotations by Sascha Kirch](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1_PU3n2ASJcZaw-jSFUPf3w.png)\nFig. 2: DINO architecture. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\n### **Networks and Update Rule**\nLet's start from the middle. DINO implements two networks with the exact same architecture but a different set of weights. Those are the student and the teacher. The student is trained with back propagation and the teacher updates its weights with an exponential moving average of its own weights and those of the student.\n![Equation 1: Update rule of the teacher's weights. Source + annotations by Sascha Kirch](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1wxlRX7bhEo6rBEhrmbljew.png)\nEquation 1: Update rule of the teacher's weights. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\nBackbones are either a ResNet50 or [DeiT](https://arxiv.org/abs/2012.12877) (which is a [ViT](https://arxiv.org/abs/2010.11929) adapted for knowledge distillation). An MLP-based projection head is connected to the backbone to reduce the dimensionality of the features, but is removed for inference.\n_**Nice, but which model is used for inference: student or teacher?**_ - Well that's a good question and funny enough not a single word is mentioned in the paper. Intuitively you might think the student, at least I did at first. But as we will see later, the teacher outperforms the student throughout the training. The only hint beside the better performance is that in the code implementation the teacher checkpoint is the default one for the evaluation of for example [video segmentation](https://github.com/facebookresearch/dino/blob/7c446df5b9f45747937fb0d72314eb9f7b66930a/eval_video_segmentation.py#L257C9-L257C9), [linear probing](https://github.com/facebookresearch/dino/blob/7c446df5b9f45747937fb0d72314eb9f7b66930a/eval_linear.py#L264C34-L264C34) and [k-NN](https://github.com/facebookresearch/dino/blob/7c446df5b9f45747937fb0d72314eb9f7b66930a/eval_knn.py#L203C32-L203C32). Since this parameter can be changed though, I cannot tell you with certainty.\n### **Inputs and Outputs**\nFrom an input image _x_ different views _x1_ and _x2_ are created by cropping and applying image augmentations like in [BYOL](https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c?source=friends_link&sk=fc5a3b3a556088181d8726226862252c) (e.g. color jitter, Gaussian blur and solarization). The technique used for cropping is called[ multi-crop](https://arxiv.org/abs/2006.09882) where multiple crops of different sizes are generated to save memory while providing more data. Small crops are called local views and consist of 96x96 pixels that are exclusively feed into the student. Larger crops are called global views and consists of 224x224 pixels that are exclusively fed into the teacher. As we will see later in the ablation section, 2 global views and 10 local views have been used during training.\n> NOTE: The paper is a bit confusing regarding the multi-crop technique because neither the provided pseudo-code nor the architecture shown in Fig. 3 above reflect it. The pseudo code even suggests that x1 and x2 are feed into both, the student and the teacher like in BYOL, which is not the case when using multi-crop.\nIn contrast to similarity learning where the objective is to maximize the similarity of embeddings, DINO minimizes the cross-entropy between the teacher's and the student's output distribution. As indicated by the equation bellow, the cross-entropy is calculated for each pair of global and local views and is then summed up.\n![Equation 2: Optimization objective. Source + annotations by Sascha Kirch](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1pKDO4PTemeFeh47yBtKt2Q.png)\nEquation 2: Optimization objective. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\n _**And what do the models output?**_ - Like in similarity learning, the student and the teacher output an embedding for a given image, rather than a prediction score. Like in knowledge distillation, the output is transformed via a SoftMax transformation into a probability distribution. The SoftMax has a temperature parameter that controls the smoothing or sharpening of the resulting distribution. This temperature plays a crucial role in knowledge distillation because it allows to control the balance between transferring general knowledge and fine-grained details from a teacher network to a student network, making the distillation process more effective for different tasks.\n![Fig. 3: Effect of temperature value on the SoftMax output. Illustration by Sascha Kirch created with this python notebook](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1J-Cz2Bd38CzbYzFxDS8KTg.png)\nFig. 3: Effect of temperature value on the SoftMax output. Illustration by [Sascha Kirch](https://medium.com/@SaschaKirch) created with [this python notebook](https://github.com/sascha-kirch/ML_Notebooks/blob/main/Softmax_Temperature.ipynb)\nI created a notebook for you so you can investigate the impact of the temperature on the resulting distribution:\n> [**ML_Notebooks/Softmax_Temperature.ipynb at main · sascha-kirch/ML_Notebooks**](https://github.com/sascha-kirch/ML_Notebooks/blob/main/Softmax_Temperature.ipynb)\n### **Avoiding Collapse**\nAs mentioned earlier, student and teacher have the exact same architecture. This kind of setup is unstable (if no counter measures are implemented) and might result in collapsing solutions, where all features are mapped to a certain region in the latent space, e.g. a single point in the worst case. BYOL addressed this issue with an extra prediction head for only one of the models introducing an asymmetry. Since DINO has symmetric models another trick is required: centering and sharpening. Both are applied to the teacher network only. Centering is a technique that prevents a single dimension in the latent space to dominate, by adding a bias term _c_ to the teachers output _g(x) = g(x)+c_ , where\n![Equation 3: Update rule of the centering term. Source + annotations by Sascha Kirch](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1-neYHZMWE9AbqBpMmNYtew.png)\nEquation 3: Update rule of the centering term. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\nWhile centering has a positive effect, it also encourages the output to collapse into a uniform distribution. Sharpening has the opposite effect hence applying both balances their effect and stabilizes training. Sharpening is achieved by using a smaller temperature in the SoftMax (see Fig. 3) for the teacher as for the student.\nTo avoid collapsing the hyperparameter _m_ from equation 3 and the temperature of the teacher are crucial. In their ablation study in the appendix section the authors show that _m=0.9...0.999_ works best and the temperature value is linearly increased from _0.04_ to _0.07_ during warm-up.\n### **What does DINO do? Knowledge Distillation or Similarity Learning?**\nThe answer is a little bit of both!\nWhile knowledge distillation usually distils knowledge from an already trained, larger and more accurate teacher model into a smaller student model, it could also be seen as some sort of similarity learning because it encourages the student network to produce predictions that are similar to those of the teacher. In similarity learning, the two models are usually trained jointly and often align their latent space predictions rather than probability distributions.\nSince the authors of DINO phrase their objective as knowledge distillation, let's have a look on some differences compared with \"standard\" knowledge distillation:\n 1. DINO's teacher is not available a priori but \"trained\" alongside the student. It can even be considered as a co-distillation since knowledge is also distilled from the student into the teacher.\n 2. DINO's teacher and student are not acting on the same input but on different views of the image cropped to different sizes.\n 3. DINO uses different temperatures in the SoftMax of both models to perform sharpening.\n 4. DINO calculates the cross-entropy over the temperature-scaled SoftMax of the embeddings rather than prediction scores.\n\n\nAnd how is it similar to knowledge distillation?:\n 1. DINO consists of a student and a teacher network, where the teacher performs better than the student as we will see in the experiments.\n 2. Rather than maximizing a similarity metric, DINO minimizes the cross-entropy loss of a temperature scaled SoftMax output.\n\n\n> [**Get an email whenever Sascha Kirch publishes 🚀**](https://medium.com/@SaschaKirch/subscribe)[ _Get an email whenever Sascha Kirch publishes 🚀 Looking to learn more about deep learning or simply stay up to dat_e...medium.com](https://medium.com/@SaschaKirch/subscribe)\n* * *\n## Experiments\nThe paper presents a vast number of experiments. They pre-train the model on ImageNet, a commonly used dataset in representation learning.\nFor the evaluation, common techniques usually either train a linear classifier on top of frozen features or fine-tune the model to new downstream tasks, where the parameters of the model are adapted.\nThe authors of DINO claim that those techniques are very sensitive to hyperparameters which makes comparisons unfair and hard to reproduce. Hence, they propose to use a simple nearest neighbor clustering algorithm on the features of the pre-trained model.\n### Linear and k-NN Classification on ImageNet\nIn a this experiment the models are tested on their image classification accuracy on ImageNet. A variety of self-supervised pre-trained models are tested with either a ResNet or a ViT backbone. The classification is done either with linear probing or k-NN clustering.\n![Table 1: Linear and k-NN classification on ImageNet. Source + annotations by Sascha Kirch](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1HR7nZIUhdiW3KjoGfJhDKg.png)\nTable 1: Linear and k-NN classification on ImageNet. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\nI guess the key take-aways are:\n 1. K-NN performs better on ViT features than on ResNet features.\n 2. Decreasing patch size in the ViT has larger improvement as larger backbone, but at the cost of slower inference.\n\n\n### Video Instance Segmentation\nAn important experiment has been the video segmentation task, since the paper is about the ViT's capability to capture semantic segmentation in its features when trained with unsupervised methods. Or let's say that's what is claimed 😁\n![Table 2: Video Instance segmentation. Source + annotations by Sascha Kirch](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1LmWLdxOjcSrezCYEPAmNiA.png)\nTable 2: Video Instance segmentation. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\nObserving those results I am missing two further experiments:\n 1. It would be nice to see a comparison of a supervised ResNet50 and a self-supervised ResNet50 in the DINO framework to support their claim that the ViT is superior to the ResNet architecture.\n 2. It would also be great to see the same set of ViT backbones for supervised as for self-supervised to see the impact on patch-size and model size.\n\n\nBut as I always say: asking questions is easy 😁 In real-world projects the authors often face resource constraints and project deadlines so not every single little detail can be covered!\n### Probing the Self-Attention Map\nIn this experiment the authors investigated the self-attention maps of different heads in the multi-head self-attention layers of the ViT. They visualize the attention maps from selected heads from the last layer of ViT-S/8, those of the learned [CLS] token to be precise.\n![Fig. 4: Attention maps from selected heads. Source + annotations by Sascha Kirch](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1Nw2L-wXPRJeuQjkFJWJrVA.png)\nFig. 4: Attention maps from selected heads. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\n### Other Experiments\nIn other experiments, DINO improved compared against the supervised baseline. Those tasks include image retrieval and copy detection.\n* * *\n## Ablations\nFor their ablation study the authors experiment with the ViT-S model.\n### Importance of Patch Size\nRecall that a vision transformer inputs a patchified version of an input image, transforms each patch into a token and then applies a transformer with its self-attention mechanism. This was a trick by the authors of ViT to reduce the compute requirements for trading-off performance, making transformers applicable to image data.\nDINO claims that smaller size of the patches increases the performance while decreasing the throughput (number of images that can be processed per second), which is exactly what ViT claims.\n![Fig. 5: Impact of patch size on accuracy and throughput. Source + annotations by Sascha Kirch](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1ndm1P9DjiP8cSXSjAJBmYA.png)\nFig. 5: Impact of patch size on accuracy and throughput. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\nIntuitively I'd say it is no surprise since you increase the input resolution and you end up with more tokens to attend to, so you end up with a fine-grained attention map.\n### Different Teacher Update Rules\nThe teacher in DINO is updated by calculating the exponential moving average from the updated student and the current teacher. This is the \"momentum encoder\" approach they refer to.\nUsing a momentum encoder and plotting the accuracy of the teacher and the student during training, the teacher performs better throughout the entire process. From this we can hypothesize:\n 1. the teacher can provide a strong learning signal to the student.\n 2. an improving student improves the teacher due to the EMA update rule (co-distillation).\n 3. One can use the Teacher as final model which has better performance but the same architecture as the student, hence no change in compute requirements.\n\n\n![Fig. 6: Teacher performance. Source + annotations by Sascha Kirch](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1UP3zSztLkfADE8FfgFLA4w.png)\nFig. 6: Teacher performance. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\nThey also experiment with 3 other update rules: copying the weights from the student to the teacher, use the student weights from the previous iteration of the optimizer and use the student weights from the previous epoch.\n### Multi-Crop vs. Time and GPU Memory\nAs mentioned earlier, DINO inputs multiple cropped views of the same image and feeds the global views into the teacher and the local views into the student. In this ablation, the authors experiment with different amounts of local views and report the impact on performance, training time and peak memory per GPU.\n![Table 3: Multi-Crop vs. Time and GPU Memory. Source + annotations by Sascha Kirch](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1ER0Fu1-nZsEeZPUGm6lcdQ.png)\nTable 3: Multi-Crop vs. Time and GPU Memory. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\n### Avoiding Collapse\nIn this ablation the authors evaluated the role of their stabilizing measures to avoid collapsing solutions: centering and sharpening.\nTo do so, they decomposed the cross-entropy into an entropy term and a Kullback-Leibler (KL) divergence term. KL divergence is a measure of difference of two probability distributions. If KL is 0, two distributions are considered equal.\nThe intuition behind this is the following: if the KL divergence of the output distribution of the teacher and the student is constant throughout the training, there is no learning signal for updating the weights of the student.\n![Fig. 7: Analysis of collapsing solutions. Source + annotations by Sascha Kirch](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1K36z8yxiWqPD5zpVzpo_Qg.png)\nFig. 7: Analysis of collapsing solutions. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\n### Effect of Batch Size\nAn interesting property is that DINO can be trained with small batch sizes without a large drop in performance. This was actually one of BYOL's motivation, a paper DINO builds upon, to be less dependent on batch size compared to contrastive self-supervised learning.\n![Table 4: Batch size vs. accuracy. Source + annotations by Sascha Kirch](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1v_OFt8ThjEOWd3oKLuHoTg.png)\nTable 4: Batch size vs. accuracy. [Source](https://arxiv.org/abs/2104.14294) + annotations by [Sascha Kirch](https://medium.com/@SaschaKirch)\nContrastive methods like CLIP and GLIP provide a lot of negative samples for a given positive sample to avoid collapsing solutions. The more negative samples per optimizer update step (hence per batch) the better it works.\n* * *\n## Conclusion\nIn conclusion, DINO is a knowledge-distillation framework. It is a visual foundation model that exploits interesting properties of ViTs and is the predecessor of one of today's best-performing foundation models, DINOv2. DINO's framework consists of a student and teacher model that acts upon different views of the same image and adds extra measures to deal with inherent instabilities of similarity-learning approaches. The experiments show that DINO outperforms other self-supervised pre-trained models on various tasks.\n* * *\n## Further Readings & Resources\n### Papers\nIn the meantime an improved version of DINO has been released:\n 1. [DINOv2: Learning Robust Visual Features without Supervision](https://arxiv.org/abs/2304.07193)\n 2. [Blog post DINOv2 by Meta](https://ai.meta.com/blog/dino-v2-computer-vision-self-supervised-learning/)\n\n\n* * *\n### Paper Walkthroughs\nYou might also like my other paper walkthroughs covering concepts we discussed in this article:\n> [**The CLIP Foundation Model**](https://towardsdatascience.com/the-clip-foundation-model-7770858b487d)\n> [**GLIP: Introducing Language-Image Pre-Training to Object Detection**](https://towardsdatascience.com/glip-introducing-language-image-pre-training-to-object-detection-5ddb601873aa)\n> [**BYOL -The Alternative to Contrastive Self-Supervised Learning**](https://towardsdatascience.com/byol-the-alternative-to-contrastive-self-supervised-learning-5d0a26983d7c)\n> [**Segment Anything - Promptable Segmentation of Arbitrary Objects**](https://towardsdatascience.com/segment-anything-promptable-segmentation-of-arbitrary-objects-f28958c5612d)\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657088944_hlxvw4.png)\nWritten By\n## [Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\n[See all from Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Computer Vision](https://towardsdatascience.com/tag/computer-vision/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Knowledge Distillation](https://towardsdatascience.com/tag/knowledge-distillation/)[Saschas Paper Club](https://towardsdatascience.com/tag/saschas-paper-club/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fdino-a-foundation-model-for-computer-vision-4cb08e821b18%2F&title=DINO%20-%20A%20Foundation%20Model%20for%20Computer%20Vision)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fdino-a-foundation-model-for-computer-vision-4cb08e821b18%2F&title=DINO%20-%20A%20Foundation%20Model%20for%20Computer%20Vision)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fdino-a-foundation-model-for-computer-vision-4cb08e821b18%2F&text=DINO%20-%20A%20Foundation%20Model%20for%20Computer%20Vision)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/1OjK0I3yl04ZU_JD94LQpHw.png)](https://towardsdatascience.com/nlp-splitting-text-into-sentences-7bbce222ef17/)\n## [NLP: Splitting Text into Sentences](https://towardsdatascience.com/nlp-splitting-text-into-sentences-7bbce222ef17/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nLanguage agnostic model for transforming text into sentences\n[Moran Beladev](https://towardsdatascience.com/author/moranbeladev90/)\nAugust 1, 20214 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/0dLXAqsZ47-b8zx1m-scaled.jpg)](https://towardsdatascience.com/deep-reinforcement-learning-hands-on-for-optimized-ad-placement-b402ffa47245/)\n## [Deep Reinforcement Learning hands-on for Optimized Ad Placement](https://towardsdatascience.com/deep-reinforcement-learning-hands-on-for-optimized-ad-placement-b402ffa47245/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nDeep Reinforcement technique is used in this article to optimize Ad placement on a website to maximize the probability of user clicks and...\n[NandaKishore Joshi](https://towardsdatascience.com/author/nandakishorej8/)\nOctober 28, 20218 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/0IMWlQD91jd04ISqN-scaled.jpg)](https://towardsdatascience.com/top-python-libraries-for-image-augmentation-in-computer-vision-2566bed0533e/)\n## [Top Python libraries for Image Augmentation in Computer Vision](https://towardsdatascience.com/top-python-libraries-for-image-augmentation-in-computer-vision-2566bed0533e/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nFeaturing the best augmentation libraries (along with sample codes) for your next computer vision project\n[Kenneth Leung](https://towardsdatascience.com/author/kennethleungty/)\nAugust 2, 20215 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/1Cw0v6NAPh0E-Zx18kGr0yQ.png)](https://towardsdatascience.com/denoising-radar-satellite-images-with-python-has-never-been-so-easy-f445241002a5/)\n## [Denoising Radar Satellite Images with Python Has Never Been So Easy](https://towardsdatascience.com/denoising-radar-satellite-images-with-python-has-never-been-so-easy-f445241002a5/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nPresentation of the latest release of deepdespeckling\n[Hadrien Mariaccia](https://towardsdatascience.com/author/brash6/)\nApril 23, 20247 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/06/1WQcWBdB_yODxMSeZjlxIdA.png)](https://towardsdatascience.com/deep-learning-illustrated-part-4-recurrent-neural-networks-d0121f27bc74/)\n## [Deep Learning Illustrated, Part 4: Recurrent Neural Networks](https://towardsdatascience.com/deep-learning-illustrated-part-4-recurrent-neural-networks-d0121f27bc74/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAn illustrated and intuitive guide on the inner workings of an RNN and the Softmax Activation Function\n[Shreya Rao](https://towardsdatascience.com/author/shreya.rao/)\nJune 11, 202415 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/1mpyrgqwMjfclV2oN1U2VIA-scaled.jpeg)](https://towardsdatascience.com/a-must-have-tool-for-every-data-scientist-5e7c76f1916f/)\n## [A Must-Have Tool for Every Data Scientist](https://towardsdatascience.com/a-must-have-tool-for-every-data-scientist-5e7c76f1916f/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nTensorDash - An app to remotely monitor your Machine Learning models\n[Arun](https://towardsdatascience.com/author/cleanpegasus/)\nOctober 21, 20204 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/15zx9M6xk1YQng05Rj7pqmA.png)](https://towardsdatascience.com/residual-networks-in-computer-vision-ee118d3be68f/)\n## [Residual Networks in Computer Vision](https://towardsdatascience.com/residual-networks-in-computer-vision-ee118d3be68f/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nDeep Learning application using Tensorflow and Keras Deep Convolutional Neural Networks changed the research landscape...\n[Luís Roque](https://towardsdatascience.com/author/luisroque/)\nMay 27, 202113 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/1fE-Nub-x4oZJ86F8zR0nQ.png)](https://towardsdatascience.com/how-to-leverage-neural-style-transfer-to-create-stunning-images-c30eb4a9007a/)\n## [How to Leverage Neural Style Transfer to Create Stunning Images](https://towardsdatascience.com/how-to-leverage-neural-style-transfer-to-create-stunning-images-c30eb4a9007a/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nAI that can paint images\n[Nour Islam Mokhtari](https://towardsdatascience.com/author/nourislam/)\nDecember 6, 20214 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/09/15ZG2iekZrG_WCVVelaws9A-scaled.jpeg)](https://towardsdatascience.com/deep-neural-networks-how-to-define-73d87bf36421/)\n## [Deep neural networks: How to define?](https://towardsdatascience.com/deep-neural-networks-how-to-define-73d87bf36421/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nCorrectly defining what makes a neural network deep is important. This post proposes an alternative definition to deep neural networks.\n[Valdivino Santiago Júnior](https://towardsdatascience.com/author/vald.santiago/)\nSeptember 29, 20218 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n", "body_base_path": "attachments/6609a57ffa9a/pages/", "body_markdown_path": "attachments/6609a57ffa9a/pages/archived-body.md", "local_html": "attachments/6609a57ffa9a/pages/archived-body.html", "body_source": "prior_capture" }