保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Data Science

Platonic Representation: Are AI Deep Network Models Converging?

Are Artificial Intelligence models evolving towards a unified representation of reality? The Platonic Representation Hypothesis says yes! Hesam Sheikh May 23, 20248 min read

Are AI Deep Network Models Converging?

Are artificial intelligence models evolving towards a unified representation of reality? The Platonic Representation Hypothesis says ML models are converging.

A recent MIT paper has come to my attention for its impressive claim: AI models are converging, even across different modalities - vision and language. "We argue that representations in AI models, particularly deep networks, are converging " is how The Platonic Representation Hypothesis paper begins. But how can different models, trained on different datasets and for different use cases converge? What has led to this convergence? ✨This is a paid article. If you're not a Medium member, you can read this for free in my newsletter:Qiubyte . Platos allegory of the cave by Jan Saenredam public domain. Plato's allegory of the cave by Jan Saenredam (public domain). Learn this step by step with the interactive AI and Data Scientist, AI Engineer and Machine Learning roadmaps.

1. The Platonic Representation Hypothesis

We argue that there is a growing similarity in how datapoints are represented in different neural network models. This similarity spans across different model architectures, training objectives, and even data modalities. The Platonic Representation Hypothesis. The visual representation, X, and the textual one Y, are both projections of a common reality, Z. source: Paper The Platonic Representation Hypothesis. The visual representation, X, and the textual one Y , are both projections of a common reality, Z. (source: Paper)

introduction

The paper's central argument is that models of various origins and modalities are converging to a representation of reality - the joint distribution over the events of the world that generate the data we observe and use to train the models. The authors argue that this convergence towards a platonic representation is driven by the underlying structure and nature of the data that models are trained on, and by the growing complexity and capability of the models themselves. As models encounter various datasets and wider applications, they require a representation that captures the fundamental properties commonly found in all data types. An Illustration of The Allegory of the Cave, from Platos Republic art by 4edges, source: Wikipedia An Illustration of The Allegory of the Cave, from Plato's Republic (art by 4edges, source: Wikipedia)

2. Are AI Models Converging?

AI models of various scales, even built on diverse architecture and trained for different tasks, are showing signs of convergence in how they represent data. As these models grow in size and complexity and the feeding data becomes larger and varied, their methods of processing data begin to align. Do models trained on different data modalities - vision or text, also converge? The answer could be yes!

2.1 Vision Models that Talk

This alignment spans over visual and textual data - the paper later confirms that the limitations of this theory are that it's focused on these two modularities and not other modalities such as audio, or robotics perception of the world. One of the cases [1] to support this is LLaVA, which shows projecting visual features into language features using a 2-layer MLP, resulting in state-of-the-art results. Outline of how LLaVA maps visual features to a Language Model. source: LLaVA, CC-BY Outline of how LLaVA maps visual features to a Language Model. (source: LLaVA, CC-BY)

2.2 Language Models that See

Another interesting example is A Vision Check-up for Language Models [2] which explores the extent to which large language models understand and process visual data. The study uses code as a bridge between images and text, as a novel approach to feed visual data to LLMs. The paper reveals that LLMs can generate images by code that while may not look realistic, still contain enough visual information to train vision models. Can a language model see? source Can a language model see? (source)

2.3 Bigger Models, Bigger Alignment

The alignment of different models is correlated with their scale. As an example, models trained on CIFAR-10 classification that are bigger, show greater alignment with each other, compared to smaller models. This means that with the current trend of building models in the order of 10s and now 100s of billions, these giants will be even more aligned.

"all strong models are alike, each weak model is weak in its own way."

3. Why are AI Models Converging?

The learning process of an AI model, f ∗ is the trained model, 𝐹 F is the function class, 𝐿 L is the loss function depending on the model 𝑓 f and an input 𝑥 x from the dataset, 𝑅 R represents the regularization function, and 𝐸 E denotes the expectation over the dataset. Each color represents one of the causes of this convergence. source: Paper The learning process of an AI model, f ∗ is the trained model, 𝐹 F is the function class, 𝐿 L is the loss function depending on the model 𝑓 f and an input 𝑥 x from the dataset, 𝑅 R represents the regularization function, and 𝐸 E denotes the expectation over the dataset. Each color represents one of the causes of this convergence. (source: Paper) In training an AI model, there are elements that contribute most to why AI models converge:

3.1 Tasks are Becoming General

As models are trained to solve tasks that are more and more general simultaneously, the size of their solution space becomes smaller and more constrained. More generality means trying to learn data points that are closer to reality. The more tasks a model can solve, it is forced to learn a disjoint representation that is useful in solving all of those tasks. source: Paper The more tasks a model can solve, it is forced to learn a disjoint representation that is useful in solving all of those tasks. (source: Paper) The Platonic Representation Hypothesis paper formulates this as The Multitask Scaling Hypothesis:

"There are fewer representations that are competent for N tasks than there are for M < N tasks. As we train more general models that solve more tasks at once, we should expect fewer possible solutions." In other words, the solution to a complex problem is much more narrow than the solution to an easy problem. As we are training models that are more and more general on gigantic internet-wide datasets across different modalities, you can only imagine how constrained the solution space will be.

3.2 Models are Getting Bigger

As the capacity of models increases, through more sophisticated architectures, larger datasets, or more complex training algorithms, these models develop representations that are more similar to each other. Bigger hypothesis spaces are more likely to converge on a solution, rather than smaller spaces. source: Paper Bigger hypothesis spaces are more likely to converge on a solution, rather than smaller spaces. (source: Paper) While The Platonic Representation Hypothesis paper doesn't offer proofs or examples for this hypothesis that they call The Capacity Hypothesis - that "Bigger models are more likely to converge to a shared representation than smaller models", it seems trivial that bigger models at least have more capacity to come up with mutual solution spaces than small models. As AI models scale, thanks to their depth and complexity, they have a greater capacity for abstraction. This allows them to capture underlying concepts and patterns of the data and wave off noise or outliers, thus arriving at a representation that is more generalized and possibly closer to the real world.

3.3 The Simplicity Bias

Imagine training two large-scale neural networks on two separate tasks: one model must be able to recognize faces from images, and another is trained to interpret the emotions of faces. Initially, these two tasks might seem unrelated - but would you be surprised to see both models converge on similar ways of representing facial features? After all, it all comes down to an accurate identification and interpretation of key facial landmarks (eyes, nose, mouth, etc). Deep Neural Networks tend towards simpler functions. source: Paper Deep Neural Networks tend towards simpler functions. (source: Paper) Several literature points out a tendency of deep neural networks to find simpler and more general solutions [3,4,5]. In other words, deep networks favor simple solutions. Often called The Simplicity Bias the paper formulates it as such:

Deep networks are biased toward finding simple fits to the data, and the bigger the model, the stronger the bias. Therefore, as models get bigger, we should expect convergence to a smaller solution space. Why do neural networks show this behavior? Networks show simplicity bias mostly because of the fundamental properties of the learning algorithms used to train them. Algorithms tend to favor simpler, more generalizable models as a way to prevent overfitting and enhance generalization. During training, simpler models are more likely to emerge because by capturing the dominant patterns in the data, they minimize the loss function more efficiently. Simplicity bias acts as a natural regulator during training. It pushes models toward an optimal way of representing and processing data, which is both general across tasks and simple enough to be efficiently learned and applied, and this increases the chance of models learning mutual hypothesis spaces.

4. Implications of This Convergence

So what if models are converging? First of all, this shows that data across different modalities can be more useful than thought before. Fine-tuning vision models from pre-trained LLMs or vice-versa could yield surprisingly good results. Another implication pointed out by the paper is that "Scaling may reduce hallucination and bias". The argument is that as models scale, they can learn from a larger and more diverse dataset, which helps them develop a more accurate and robust understanding of the world. This enhanced understanding allows them to make predictions and generate outputs that are not only more reliable but also less biased. VISION models converge as COMPETENCE increases. source: Paper VISION models converge as COMPETENCE increases. (source: Paper)

5. A Pinch of Salt

When it comes to the arguments posed by the paper, you have to consider some limitations, almost all of which are also addressed by the paper as well. Firstly, the paper assumes a bijective projection of reality in which one real-world concept Z, has projections X and Y that can be learned. However, some concepts are uniquely inherent in one modularity. Sometimes, language can express a concept or feeling that many images can't, and in the same way, language can fail to take the place of an image in describing a visual concept. Secondly, as mentioned before, the paper focuses on two modalities: vision and language. Thirdly, the argument that "AI models are Converging" only holds for multi-task AI models and not specific ones, such as ADAS or Sentiment Analysis models. Lastly, while the paper shows that the alignment of different models increases , it doesn't indicate the models' representations become similar. The score of alignment between larger models is indeed higher than smaller ones, but still, a score of 0.16/1.00 leaves some open questions to the research.

🌟 Join +1000 people learning about Python🐍 , ML/MLOps/AI🤖, Data Science📈 , and LLM 🗯 follow me and check out my X/Twitter , where I keep you updated Daily**.**

QiuByte | Hesam Sheikh | Substack Thanks for reading, — Hesam

[1] Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In NeurIPS, 2023. [2] Sharma, P., Rott Shaham, T., Baradad, M., Fu, S., Rodriguez-Munoz, A., Duggal, S., Isola, P., and Torralba, A. A vision check-up for language models. In arXiv preprint, 2024. [3]H. Shah, K. Tamuly, The Pitfalls of Simplicity Bias in Neural Networks, 2020, https://arxiv.org/abs/2006.07710 [4] A brief note on Simplicity Bias [5] Deep Neural Networks are biased, at initialisation, towards simple functions

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Hesam Sheikh

See all from Hesam Sheikh Aritificial IntelligenceData ScienceDeep Neural NetworksLarge Language ModelsMachine Learning Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Classification Problems and Exploring Decision Boundaries

Machine Learning Classification problems are one of the main business problems where organizations can harness the power of data science to create... Ivo Bernardo August 14, 20198 min read

How I Went From Being a Sales Engineer to Deep Learning / Computer Vision Research Engineer

Artificial Intelligence Guidance, tips and suggestions that I would have loved to receive when I was in the process of my career change to DL/ML/CV. Manpreet Singh Minhas January 9, 202111 min read

How to Get Yourself a Mentor in Data Science

Artificial Intelligence No, you don't go and ask them to mentor you. Arunn Thevapalan November 30, 20207 min read

User Churn Prediction

Machine Learning Modern data warehousing and Machine Learning Mike Shakhomirov December 23, 202312 min read

Coding the Smart Wordle Solver Python Version (Part 2)

Programming An extensive comparison between the Java and Python implementation of the Smart Wordle Solver Daniel García Solla December 20, 202219 min read

COVID-19's Effect on the Data Science Field and What We Should Do About It

Data Science Developing new data scientists will require change, and it is vitally important for organizations to be thoughtful and creative about the... Dr. Xuxu Wang May 26, 20204 min read

t-SNE: Behind the Math

Machine Learning Being one of the most talked about dimensionality reduction algorithms in the recent years, especially for visualizations, I thought I'd... Sushanth Sreenivasa August 15, 20204 min read

Approximating stochastic functions

Machine Learning A generic approach for training probabilistic machine learning models Nicolas Arroyo Duran November 14, 202018 min read

Unsupervised Text Classification with Lbl2Vec

NLP An introduction to embedding-based classification of unlabeled text documents Tim Schopf December 23, 20216 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close