{
  "source_file": "0930",
  "source_platform": "研究机构",
  "author": "Anthropic Research",
  "title_listed": "Toy Models of Superposition",
  "keywords": "叠加,玩具模型,可解释性",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://transformer-circuits.pub/2022/toy_model/",
  "publish_date": "2022-09",
  "corpus_stage": "预训练/SFT",
  "notes": "叠加现象玩具模型",
  "id": "2397a4352560",
  "collected_at": "2026-10-02T15:34:53",
  "title": "Toy Models of Superposition",
  "body_markdown": "Transformer Circuits Thread\nToy Models of Superposition\nAuthors\nNelson Elhage\n∗\n,\nTristan Hume\n∗\n,\nCatherine Olsson\n∗\n,\nNicholas Schiefer\n∗\n,\nTom Henighan\n,\nShauna Kravec,\nZac Hatfield-Dodds\n,\nRobert Lasenby,\nDawn Drain,\nCarol Chen,\nRoger Grosse,\nSam McCandlish,\nJared Kaplan,\nDario Amodei,\nMartin Wattenberg\n∗\n,\nChristopher Olah\n‡\nAffiliations\nAnthropic,\nHarvard\nPublished\nSept 14, 2022\n* Core Research Contributor;\n‡ Correspondence to\ncolah@anthropic.com\n;\nAuthor contributions statement below\n.\nIt would be very convenient if the individual neurons of artificial neural networks corresponded to cleanly interpretable features of the input. For example, in an “ideal” ImageNet classifier, each neuron would fire only in the presence of a specific visual feature, such as the color red, a left-facing curve, or a dog snout. Empirically, in models we have studied, some of the neurons do cleanly map to features. But it isn't always the case that features correspond so cleanly to neurons, especially in large language models where it actually seems rare for neurons to correspond to clean features. This brings up many questions. Why is it that neurons sometimes align with features and sometimes don't? Why do some models and tasks have many of these clean neurons, while they're vanishingly rare in others?\nIn this paper, we use toy models — small ReLU networks trained on synthetic data with sparse input features — to investigate how and when models represent more features than they have dimensions. We call this phenomenon\nsuperposition\n. When features are sparse, superposition allows compression beyond what a linear model would do, at the cost of \"interference\" that requires nonlinear filtering.\nConsider a toy model where we train an embedding of five features of varying importance\nWhere “importance” is a scalar multiplier on mean squared error loss.\nin two dimensions, add a ReLU afterwards for filtering, and vary the sparsity of the features. With dense features, the model learns to represent an orthogonal basis of the most important two features (similar to what Principal Component Analysis might give us), and the other three features are not represented. But if we make the features sparse, this changes:\nThis figure and a few others can be reproduced using the\ntoy model framework Colab notebook\nin our\nGithub repo\nNot only can models store additional features in superposition by tolerating some interference, but we'll show that, at least in certain limited cases,\nmodels can perform computation while in superposition\n. (In particular, we'll show that models can put simple circuits computing the absolute value function in superposition.) This leads us to hypothesize that\nthe\nneural networks we observe in practice are in some sense noisily simulating larger, highly sparse networks\n. In other words, it's possible that models we train can be thought of as doing “the same thing as” an imagined much-larger model, representing the exact same features but with no interference.\nFeature superposition isn't a novel idea. A number of previous interpretability papers have considered it\n, and it's very closely related to the long-studied topic of compressed sensing in mathematics\n, as well as the ideas of distributed, dense, and population codes in neuroscience\nand deep learning\n. What, then, is the contribution of this paper?\nFor interpretability researchers, our main contribution is providing a direct demonstration that superposition occurs in artificial neural networks given a relatively natural setup, suggesting this may also occur in practice. That is, we show a case where interpreting neural networks as having sparse structure in superposition isn't just a useful post-hoc interpretation, but actually the \"ground truth\" of a model. We offer a theory of when and why this occurs, revealing a\nphase diagram\nfor superposition. This\nexplains\nwhy neurons are sometimes \"monosemantic\" responding to a single feature, and sometimes \"polysemantic\"\nresponding to many unrelated features. We also discover that, at least in our toy model, superposition exhibits\ncomplex geometric structure\n.\nBut our results may also be of broader interest. We find preliminary evidence that superposition may be linked to adversarial examples and grokking, and might also suggest a theory for the performance of mixture of experts models. More broadly, the toy model we investigate has unexpectedly rich structure, exhibiting\nphase changes\n, a\ngeometric structure\nbased on uniform polytopes,\n\"energy level\"-like jumps\nduring training, and a\nphenomenon\nwhich is qualitatively similar to the fractional quantum Hall effect in physics, among other striking phenomena. We originally investigated the subject to gain understanding of cleanly-interpretable neurons in larger models, but we've found these toy models to be surprisingly interesting in their own right.\nKey Results From Our Toy Models\nIn our toy models, we are able to demonstrate that:\nSuperposition is a real, observed phenomenon\n.\nBoth monosemantic and polysemantic neurons can form.\nAt least some kinds of computation can be performed in superposition.\nWhether features are stored in superposition is\ngoverned by a phase change\n.\nSuperposition organizes features into geometric structures\nsuch as digons, triangles, pentagons, and tetrahedrons.\nOur toy models are simple ReLU networks, so it seems fair to say that neural networks exhibit these properties in at least some regimes, but it's very unclear what to generalize to real networks.\nDefinitions and Motivation: Features, Directions, and Superposition\nIn our work, we often think of neural networks as having\nfeatures of the input\nrepresented as\ndirections\nin activation space\n. This isn't a trivial claim. It isn't obvious what kind of structure we should expect neural network representations to have. When we say something like \"word embeddings have a gender direction\" or \"vision models have curve detector neurons\", one is implicitly making strong claims about the structure of network representations.\nDespite this, we believe this kind of \"linear representation hypothesis\" is supported both by significant empirical findings and theoretical arguments. One might think of this as two separate properties, which we'll explore in more detail shortly:\nDecomposability:\nNetwork representations can be described in terms of independently understandable features.\nLinearity:\nFeatures are represented by direction.\nIf we hope to reverse engineer neural networks, we\nneed\na property like decomposability. Decomposability is what\nallows us to reason about the model\nwithout fitting the whole thing in our heads! But it's not enough for things to be decomposable: we need to be able to access the decomposition somehow. In order to do this, we need to\nidentify\nthe individual features within a representation. In a linear representation, this corresponds to determining which directions in activation space correspond to which independent features of the input.\nSometimes, identifying feature directions is very easy because features seem to correspond to neurons. For example, many neurons in the early layers of InceptionV1 clearly correspond to features (e.g. curve detector neurons\n). Why is it that we sometimes get this extremely helpful property, but in other cases don't? We hypothesize that there are really two countervailing forces driving this:\nPrivileged Basis:\nOnly some representations have a\nprivileged basis\nwhich encourages features to align with basis directions (i.e. to correspond to neurons).\nSuperposition:\nLinear representations can represent more features than dimensions, using a strategy we call\nsuperposition\n. This can be seen as neural networks\nsimulating larger networks\n. This pushes features\naway\nfrom corresponding to neurons.\nSuperposition has been hypothesized in previous work\n, and in some cases, assuming something like superposition has been shown to help find interpretable structure\n. However, we're not aware of feature superposition having been unambiguously demonstrated to occur in neural networks before (\ndemonstrates a closely related phenomenon of model superposition). The goal of this paper is to change that, demonstrating superposition and exploring how it interacts with privileged bases. If superposition occurs in networks, it deeply influences what approaches to interpretability research make sense, so unambiguous demonstration seems important.\nThe goal of this section will be to motivate these ideas and unpack them in detail.\nIt's worth noting that many of the ideas in this section have close connections to ideas in other lines of interpretability research (especially disentanglement), neuroscience (distributed representations, population codes, etc), compressed sensing, and many other lines of work. This section will focus on articulating our perspective on the problem. We'll discuss these other lines of work in detail in\nRelated Work\n.\nEmpirical Phenomena\nWhen we talk about \"features\" and how they're represented, this is ultimately theory building around several observed empirical phenomena. Before describing how we conceptualize those results, we'll simply describe some of the major results motivating our thinking:\nWord Embeddings\n– A famous result by\nMikolov et al.\nfound that word embeddings appear to have directions which correspond to semantic properties, allowing for embedding arithmetic vectors such as\nV(\"king\") - V(\"man\") + V(\"woman\") = V(\"queen\")\n(\nbut see\n).\nLatent Spaces\n– Similar \"vector arithmetic\" and interpretable direction results have also been found for generative adversarial networks (e.g.\n).\nInterpretable Neurons\n– There is a significant body of results finding neurons which appear to be interpretable (\nin RNNs\n;\nin CNNs\n;\nin GANs\n), activating in response to some understandable property. This work has faced some skepticism\n. In response, several papers have aimed to give extremely detailed accounts of a few specific neurons, in the hope of dispositively establishing examples of neurons which truly detect some understandable property (notably Cammarata\net al.\n, but also\n).\nUniversality\n– Many analogous neurons responding to the same properties can be found across networks\n.\nPolysemantic Neurons\n– At the same time, there are also many neurons which appear to not respond to an interpretable property of the input, and in particular, many\npolysemantic neurons\nwhich appear to respond to unrelated mixtures of inputs\n.\nAs a result, we tend to think of neural network representations as being composed of\nfeatures\nwhich are\nrepresented as directions\n. We'll unpack this idea in the following sections.\nWhat are Features?\nOur use of the term \"feature\" is motivated by the interpretable properties of the input we observe neurons (or word embedding directions) responding to. There's a rich variety of such observed properties!\nIn the context of vision, these have ranged from low-level neurons like\ncurve detectors\nand\nhigh-low frequency detectors\n, to more complex neurons like\noriented dog-head detectors\nor\ncar detectors\n, to extremely abstract neurons corresponding to\nfamous people\n,\nemotions\n,\ngeographic regions\n, and\nmore\n. In language models, researchers have found word embedding directions such as a male-female or singular-plural direction\n, low-level neurons disambiguating words that occur in multiple languages, much more abstract neurons, and \"action\" output neurons that help produce certain words\n.\nWe'd like to use the term \"feature\" to encompass all these properties.\nBut even with that motivation, it turns out to be quite challenging to create a satisfactory definition of a feature. Rather than offer a single definition we're confident about, we consider three potential working definitions:\nFeatures as arbitrary functions.\nOne approach would be to define features as any function of the input (as in\n). But this doesn't quite seem to fit our motivations. There's something special about these features that we're observing: they seem to in some sense be fundamental abstractions for reasoning about the data, with the same features forming reliably across models. Features also seem identifiable: cat and car are two features while cat+car and cat-car seem like mixtures of features rather than features in some important sense.\nFeatures as interpretable properties.\nAll the features we described are strikingly understandable to humans. One could try to use this for a definition: features are the presence of human understandable \"concepts\" in the input. But it seems important to allow for features we might not understand. If AlphaFold discovers some important chemical structure for predicting protein folding, it very well might not be something we initially understand!\nNeurons in Sufficiently Large Models.\nA final approach is to define features as properties of the input which a sufficiently large neural network will reliably dedicate a neuron to representing.\nThis definition is trickier than it seems. Specifically, something is a feature if there\nexists\na large enough model size such that it gets a dedicated neuron. This creates a kind \"epsilon-delta\" like definition. Our present understanding – as we'll see in later sections – is that arbitrarily large models can still have a large fraction of their features be in superposition. However, for any given feature, assuming the feature importance curve isn't flat, it should eventually be given a dedicated neuron. This definition can be helpful in saying that something\nis\na feature – curve detectors are a feature because you find them across a range of models larger than some minimal size – but unhelpful for the much more common case of features we only hypothesize about or observe in superposition.\nFor example, curve detectors appear to reliably occur across sufficiently sophisticated vision models, and so are a feature. For interpretable properties which we presently only observe in polysemantic neurons, the hope is that a sufficiently large model would dedicate a neuron to them. This definition is slightly circular, but avoids the issues with the earlier ones.\nWe've written this paper with the final \"neurons in sufficiently large models\" definition in mind. But we aren't overly attached to it, and actually think it's probably important to not prematurely attach to a definition.\nA famous book by Lakatos\nillustrates the importance of uncertainty about definitions and how important rethinking definitions often is in the context of research.\nFeatures as Directions\nAs we've mentioned in previous sections, we generally think of\nfeatures as being represented by\ndirections\n. For example, in word embeddings, \"gender\" and \"royalty\" appear to correspond to directions, allowing arithmetic like\nV(\"king\") - V(\"man\") + V(\"woman\") = V(\"queen\")\n. Examples of interpretable neurons are also cases of features as directions, since the amount a neuron activates corresponds to a basis direction in the representation\nLet's call a neural network representation\nlinear\nif features correspond to directions in activation space. In a linear representation, each feature\nf_i\nhas a corresponding representation direction\nW_i\n. The presence of multiple features\nf_1, f_2…\nactivating with values\nx_{f_1}, x_{f_2}…\nis represented by\nx_{f_1}W_{f_1} + x_{f_2}W_{f_2}...\n. To be clear, the features being represented are almost certainly nonlinear functions of the input. It's only the map from features to activation vectors which is linear. Note that whether something is a linear representation depends on what you consider to be the features.\nWe don't think it's a coincidence that neural networks empirically seem to have linear representations. Neural networks are built from linear functions interspersed with non-linearities. In some sense, the linear functions are the vast majority of the computation (for example, as measured in FLOPs). Linear representations are the natural format for neural networks to represent information in! Concretely, there are three major benefits:\nLinear representations are the natural outputs of obvious algorithms a layer might implement.\nIf one sets up a neuron to pattern match a particular weight template, it will fire more as a stimulus matches the template better and less as it matches it less well.\nLinear representations make features \"linearly accessible.\"\nA typical neural network layer is a linear function followed by a non-linearity. If a feature in the previous layer is represented linearly, a neuron in the next layer can \"select it\" and have it consistently excite or inhibit that neuron. If a feature were represented non-linearly, the model would not be able to do this in a single step.\nStatistical Efficiency\n.\nRepresenting features as different directions may allow\nnon-local generalization\nin models with linear transformations (such as the weights of neural nets), increasing their statistical efficiency relative to models which can only locally generalize. This view is especially advocated in some of Bengio's writing (e.g.\n). A more accessible argument can be found in\nthis blog post\n.\nIt is possible to construct non-linear representations, and retrieve information from them, if you use multiple layers (although even these examples can be seen as linear representations with more exotic features). We provide an example in the appendix. However, our intuition is that non-linear representations are generally inefficient for neural networks.\nOne might think that a linear representation can only store as many features as it has dimensions, but it turns out this isn't the case! We'll see that the phenomenon we call\nsuperposition\nwill allow models to store more features – potentially many more features – in linear representations.\nFor discussion on how this view of features squares with a conception of features as being multidimensional manifolds, see the appendix “What about Multidimensional Features?”.\nPrivileged vs Non-privileged Bases\nEven if features are encoded as directions, a natural question to ask is which directions? In some cases, it seems useful to consider the basis directions, but in others it doesn't. Why is this?\nWhen researchers study word embeddings, it doesn't make sense to analyze basis directions. There would be no reason to expect a basis dimension to be different from any other possible direction. One way to see this is to imagine applying some random linear transformation\nM\nto the word embedding, and apply\nM^{-1}\nto the following weights. This would produce an identical model where the basis dimensions are totally different. This is what we mean by a\nnon-privileged basis\n. Of course, it's possible to study activations without a privileged basis, you just need to identify interesting directions to study somehow, such as creating a gender direction in a word embedding by taking the difference vector between \"man\" and \"woman\".\nBut many neural network layers are not like this. Often, something about the architecture makes the basis directions special, such as applying an activation function. This \"breaks the symmetry\", making those directions special, and potentially encouraging features to align with the basis dimensions. We call this a privileged basis, and call the basis directions \"neurons.\" Often, these neurons correspond to interpretable features.\nFrom this perspective, it only makes sense to ask if a\nneuron\nis interpretable when it is in a privileged basis. In fact, we typically reserve the word \"neuron\" for basis directions which are in a privileged basis. (See longer discussion\nhere\n.)\nNote that having a privileged basis doesn't guarantee that features will be basis-aligned – we'll see that they often aren't! But it's a minimal condition for the question to even make sense.\nThe Superposition Hypothesis\nEven when there is a privileged basis, it's often the case that neurons are \"polysemantic\", responding to several unrelated features. One explanation for this is the\nsuperposition hypothesis\n. Roughly, the idea of superposition is that neural networks \"want to represent more features than they have neurons\", so they exploit a property of high-dimensional spaces to simulate a model with many more neurons.\nSeveral results from mathematics suggest that something like this might be plausible:\nAlmost Orthogonal Vectors.\nAlthough it's only possible to have\nn\northogonal vectors in an\nn\n-dimensional space, it's possible to have\n\\exp(n)\nmany \"almost orthogonal\" (\n<\\epsilon\ncosine similarity) vectors in high-dimensional spaces. See the\nJohnson–Lindenstrauss lemma\n.\nCompressed sensing.\nIn general, if one projects a vector into a lower-dimensional space, one can't reconstruct the original vector. However, this changes if one knows that the original vector is sparse. In this case, it is often possible to recover the original vector.\nConcretely, in the superposition hypothesis, features are represented as almost-orthogonal directions in the vector space of neuron outputs. Since the features are only almost-orthogonal, one feature activating looks like other features slightly activating. Tolerating this \"noise\" or \"interference\" comes at a cost. But for neural networks with highly sparse features, this cost may be outweighed by the benefit of being able to represent more features! (Crucially, sparsity greatly reduces the costs since sparse features are rarely active to interfere with each other, and non-linear activation functions create opportunities to filter out small amounts of noise.)\nOne way to think of this is that a small neural network may be able to noisily \"simulate\" a sparse larger model:\nAlthough we've described superposition with respect to neurons, it can also occur in representations with an unprivileged basis, such as a word embedding. Superposition simply means that there are more features than dimensions.\nSummary: A Hierarchy of Feature Properties\nThe ideas in this section might be thought of in terms of four progressively more strict properties that neural network representations might have.\nDecomposability:\nNeural network activations which are\ndecomposable\ncan be decomposed into features, the meaning of which is not dependent on the value of other features. (This property is ultimately the most important – see the role of decomposition in defeating the curse of dimensionality.)\nLinearity:\nFeatures correspond to directions. Each feature\nf_i\nhas a corresponding  representation direction\nW_i\n. The presence of multiple features\nf_1, f_2…\nactivating with values\nx_{f_1}, x_{f_2}…\nis represented by\nx_{f_1}W_{f_1} + x_{f_2}W_{f_2}...\n.\nSuperposition vs Non-Superposition:\nA linear representation exhibits superposition if\nW^TW\nis not invertible. If\nW^TW\nis invertible, it does not exhibit superposition.\nBasis-Aligned:\nA representation is basis aligned if all\nW_i\nare one-hot basis vectors. A representation is partially basis aligned if all\nW_i\nare sparse. This requires a privileged basis.\nThe first two (decomposability and linearity) are properties we hypothesize to be widespread, while the latter (non-superposition and basis-aligned) are properties we believe only sometimes occur.\nDemonstrating Superposition\nIf one takes the superposition hypothesis seriously, a natural first question is whether neural networks can actually noisily represent more features than they have neurons. If they can't, the superposition hypothesis may be comfortably dismissed.\nThe intuition from linear models would be that this isn't possible: the best a linear model can do is to store the principal components. But we'll see that adding just a slight nonlinearity can make models behave in a radically different way! This will be our first demonstration of superposition. (It will also be an object lesson in the complexity of even very simple neural networks.)\nExperiment Setup\nOur goal is to explore whether a neural network can project a high dimensional vector\nx \\in R^n\ninto a lower dimensional vector\nh\\in R^m\nand then recover it.\nThis experiment setup could also be viewed as an autoencoder reconstructing\nx\n.\nThe Feature Vector (\nx\n)\nWe begin by describing the high-dimensional vector\nx\n: the activations of our idealized, disentangled larger model. We call each element\nx_i\na \"feature\" because we're imagining features to be perfectly aligned with neurons in the hypothetical larger model. In a vision model, this might be a Gabor filter, a curve detector, or a floppy ear detector. In a language model, it might correspond to a token referring to a specific famous person, or a clause being a particular kind of description.\nSince we don't have any ground truth for features, we need to create\nsynthetic data\nfor\nx\nwhich simulates any important properties we believe features have from the perspective of modeling them. We make three major assumptions:\nFeature Sparsity:\nIn the natural world, many features seem to be sparse in the sense that they only rarely occur. For example, in vision, most positions in an image don't contain a horizontal edge, or a curve, or a dog head\n. In language, most tokens don't refer to Martin Luther King or aren't part of a clause describing music\n. This idea goes back to classical work on vision and the statistics of natural images (see e.g. Olshausen, 1997, the section \"Why Sparseness?\"\n). For this reason, we will choose a sparse distribution for our features.\nMore Features Than Neurons:\nThere are an enormous number of potentially useful features a model might represent.\nA vision model of sufficient generality might benefit from representing every species of plant and animal and every manufactured object which it might potentially see. A language model might benefit from representing each person who has ever been mentioned in writing. These are only scratching the surface of plausible features, but already there seem more than any model has neurons. In fact, large language models demonstrably do in fact know about people of very modest prominence – presumably more such people than they have neurons. This point is a common argument in discussion of the plausibility of \"grandmother neurons\" in neuroscience, but seems even stronger for artificial neural networks.\nThis imbalance between features and neurons in real models seems like it must be a central tension in neural network representations.\nFeatures Vary in Importance:\nNot all features are equally useful to a given task. Some can reduce the loss more than others. For an ImageNet model, where classifying different species of dogs is a central task, a floppy ear detector might be one of the most important features it can have. In contrast, another feature might only very slightly improve performance.\nFor computational reasons, we won't focus on it in this article, but we often imagine an infinite number of features with importance asymptotically approaching zero.\nConcretely, our synthetic data is defined as follows: The input vectors\nx\nare synthetic data intended to simulate the properties we believe the true underlying features of our task have. We consider each dimension\nx_i\nto be a \"feature\". Each one has an associated sparsity\nS_i\nand importance\nI_i\n. We let\nx_i=0\nwith probability\nS_i\n, but it is otherwise uniformly distributed between\n[0,1]\n.\nThe choice to have features distributed uniformly is arbitrary. An exponential or power law distribution would also be very natural.\nIn practice, we focus on the case where all features have the same sparsity,\nS_i = S\n.\nThe Model (\nx \\to x'\n)\nWe will actually consider two models, which we motivate below. The first \"linear model\" is a well understood baseline which does not exhibit superposition. The second \"ReLU output model\" is a very simple model which does exhibit superposition. The two models vary only in the final activation function.\nLinear Model\nh~=~Wx\nx'~=~W^Th~+~b\nx' ~=~W^TWx ~+~ b\nReLU Output Model\nh~=~Wx\nx'~=~\\text{ReLU}(W^Th+b)\nx' ~=~\\text{ReLU}(W^TWx + b)\nWhy these models?\nThe superposition hypothesis suggests that each feature in the higher-dimensional model corresponds to a direction in the lower-dimensional space. This means we can represent the down projection as a linear map\nh=Wx\n. Note that each column\nW_i\ncorresponds to the direction in the lower-dimensional space that represents a feature\nx_i\n.\nTo recover the original vector, we'll use the transpose of the same matrix\nW^T\n. This has the advantage of avoiding any ambiguity regarding what direction in the lower-dimensional space really corresponds to a feature. It also seems relatively mathematically principled\nRecall that\nW^T = W^{-1}\nif\nW\nis orthonormal. Although\nW\ncan't be literally orthonormal, our intuition from compressed sensing is that it will be \"almost orthonormal\" in the sense of Candes & Tao\n.\n, and empirically works.\nWe also add a bias. One motivation for this is that it allows the model to set features it doesn't represent to their expected value. But we'll see later that the ability to set a negative bias is important for superposition for a second set of reasons – roughly, it allows models to discard small amounts of noise.\nThe final step is whether to add an activation function. This turns out to be critical to whether superposition occurs. In a real neural network, when features are actually used by the model to do computation, there will be an activation function, so it seems principled to include one at the end.\nThe Loss\nOur loss is weighted mean squared error weighted by the feature importances,\nI_i\n, described above:\nL = \\sum_x \\sum_i I_i (x_i - x'_i)^2\nBasic Results\nOur first experiment will simply be to train a few ReLU output models with different sparsity levels and visualize the results. (We'll also train a linear model – if optimized well enough, the linear model solution does not depend on sparsity level.)\nThe main question is how to visualize the results. The simplest way is to visualize\nW^TW\n(a features by features matrix) and\nb\n(a feature length vector). Note that features are arranged from most important to least, so the results have a fairly nice structure. Here's an example of what this type of visualization might look like, for a small model (\nn=20; ~m=5;\n) which behaves in the \"expected linear model-like\" way, only representing as many features as it has dimensions:\nBut the thing we really care about is this hypothesized phenomenon of superposition – does the model represent \"extra features\" by storing them non-orthogonally? Is there a way to get at it more explicitly? Well, one question is just how many features the model learns to represent. For any feature, whether or not it is represented is determined by\n||W_i||\n, the norm of its embedding vector.\nWe'd also like to understand whether a given feature shares its dimension with other features. For this, we calculate\n\\sum_{j\\neq i} (\\hat{W_i}\\cdot W_j)^2\n, projecting all other features onto the direction vector of\nW_i\n. It will be\n0\nif the feature is orthogonal to other features (dark blue below). On the other hand, values\n\\geq 1\nmean that there is some group of other features which can activate\nW_i\nas strongly as feature\ni\nitself!\nWe can visualize the model we looked at previously this way:\nNow that we have a way to visualize models, we can start to actually do experiments.  We'll start by considering models with only a few features (\nn=20; ~m=5;~ I_i=0.7^i\n). This will make it easy to visually see what happens. We consider a linear model, and several ReLU-output models trained on data with different feature sparsity levels:\nAs our standard intuitions would expect, the linear model always learns the top-\nm\nmost important features, analogous to learning the top principal components. The ReLU output model behaves the same on dense features (\n1-S=1.0\n), but as sparsity increases, we see superposition emerge.\nThe model represents more features by having them not be orthogonal to each other.\nIt starts with less important features, and gradually affects the most important ones. Initially this involves arranging them in antipodal pairs, where one feature’s representation vector is exactly the negative of the other’s, but we observe it gradually transition to other geometric structures as it represents more features.  We'll discuss feature geometry further in the later section,\nThe Geometry of Superposition\n.\nThe results are qualitatively similar for models with more features and hidden dimensions. For example, if we consider a model with\nm=20\nhidden dimensions and\nn=80\nfeatures (with importance increased to\nI_i=0.9^i\nto account for having more features), we observe essentially a rescaled version of the visualization above:\nMathematical Understanding\nIn the previous section, we observed a surprising empirical result: adding a ReLU to the output of our model allowed a radically different solution –\nsuperposition\n– which doesn't occur in linear models.\nThe model where it occurs is still quite mathematically simple. Can we analytically understand why superposition is occurring? And for that matter, why does adding a single non-linearity make things so different from the linear model case? It turns out that we can get a fairly satisfying answer, revealing that our model is governed by balancing two competing forces –\nfeature benefit\nand\ninterference\n– which will be useful intuition going forwards. We'll also discover a connection to the famous Thomson Problem in chemistry.\nLet's start with the linear case. This is well understood by prior work! If one wants to understand why linear models don't exhibit superposition, the easy answer is to observe that linear models essentially perform PCA. But this isn't fully satisfying: if we set aside all our knowledge and intuition about linear functions for a moment, why exactly is it that superposition can't occur?\nA deeper understanding can come from the results of Saxe et al.\nwho study the learning dynamics of\nlinear neural networks\n– that is, neural networks without activation functions. Such models are ultimately linear functions, but because they are the composition of multiple linear functions the dynamics are potentially quite complex. The punchline of their paper reveals that neural network weights can be thought of as optimizing a simple closed-form solution. We can tweak their problem to be a bit more similar to our linear case,\nWe have the model be\nx' = W^TWx\n, but leave\nx\nGaussianly distributed as in Saxe.\nrevealing the following equation:\nThe Saxe results reveal that there are fundamentally two competing forces which control learning dynamics in the considered model. Firstly, the model can attain a better loss by representing more features (we've labeled this \"feature benefit\"). But it also gets a worse loss if it represents more than it can fit orthogonally due to \"interference\" between features.\nAs a brief aside, it's interesting to contrast the linear model interference,\n\\sum_{i\\neq j}|W_i \\cdot W_J|^2\n, to the notion of\ncoherence\nin compressed sensing,\n\\max_{i\\neq j}|W_i \\cdot W_J|\n. We can see them as the\nL^2\nand\nL^\\infty\nnorms of the same vector.\nIn fact, this makes it never worthwhile for the linear model to represent more features than it has dimensions.\nTo prove that superposition is never optimal in a linear model, solve for the gradient of the loss being zero or consult Saxe et al.\nCan we achieve a similar kind of understanding for the ReLU output model? Concretely, we'd like to understand\nL=\\int_x ||I(x-\\text{ReLU}(W^TWx+b))||^2 d\\textbf{p}(x)\nwhere\nx\nis distributed such that\nx_i=0\nwith probability\nS\n.\nThe integral over\nx\ndecomposes into a term for each sparsity pattern according to the binomial expansion of\n((1\\!-\\!S)+S)^n\n. We can group terms of the sparsity together, rewriting the loss as\nL = (1\\!-\\!S)^n L_n +\\ldots+\n(1\\!-\\!S)S^{n-1} L_1+ S^n L_0\n, with each\nL_k\ncorresponding to the loss when the input is a\nk\n-sparse vector. Note that as\nS\\to 1\n,\nL_1\nand\nL_0\ndominate. The\nL_0\nterm, corresponding to the loss on a zero vector, is just a penalty on positive biases,\n\\sum_i \\text{ReLU}(b_i)^2\n. So the interesting term is\nL_1\n, the loss on\n1\n-sparse vectors:\nThis new equation is vaguely similar to the famous\nThomson problem\nin chemistry. In particular, if we assume uniform importance and that there are a fixed number of features with\n||W_i|| = 1\nand the rest have\n||W_i|| = 0\n, and that\nb_i = 0\n, then the feature benefit term is constant and the interference term becomes a generalized Thomson problem – we're just packing points on the surface of the sphere with a slightly unusual energy function. (We'll see this can be a productive analogy when we resume our empirical investigation in the following sections!)\nAnother interesting property is that ReLU makes negative interference free in the 1-sparse case. This explains why the solutions we've seen prefer to only have negative interference when possible. Further, using a negative bias can convert small positive interferences into essentially being negative interferences.\nWhat about the terms corresponding to less sparse vectors? We leave explicitly writing these out to the reader, but the main idea is that there are multiple compounding interferences, and the \"active features\" can experience interference. In a\nlater section\n, we'll see that features often organize themselves into sparse interference graphs such that only a small number of features interfere with another feature – it's interesting to note that this reduces the probability of compounding interference and makes the 1-sparse loss term more important relative to others.\nSuperposition as a Phase Change\nThe results in the previous section seem to suggest that there are three outcomes for a feature when we train a model: (1) the feature may simply not be learned; (2) the feature may be learned, and represented in superposition; or (3) the model may represent a feature with a dedicated dimension. The transitions between these three outcomes seem sharp. Possibly, there's some kind of phase change.\nHere, we use “phase change” in the generalized sense of “discontinuous change”, rather than in the more technical sense of a discontinuity arising in the limit of infinite system size.\nOne way to understand this better is to explore if there's something like a \"phase diagram\" from physics, which could help us understand when a feature is expected to be in one of these regimes.  Although we can see hints of this in\nour previous experiment\n, it's hard to really isolate what's going on because many features are changing at once and there may be interaction effects. As a result, we set up the following experiment to better isolate the effects.\nAs an initial experiment, we consider models with 2 features but only 1 hidden layer dimension. We still consider the ReLU output model,\n\\text{ReLU}(W^T W x - b)\n. The first feature has an importance of 1.0. On one axis, we vary the importance of the 2nd \"extra\" feature from 0.1 to 10. On the other axis, we vary the sparsity of all features from 1.0 to 0.01. We then plot whether the 2nd \"extra\" feature is not learned, learned in superposition, or learned and represented orthogonally. To reduce noise, we train ten models for each point and average over the results, discarding the model with the highest loss.\nWe can compare this to a theoretical \"toy model of the toy model\" where we can get closed form solutions for the loss of different weight configurations as a function of importance and sparsity. There are three natural ways to store 2 features in 1 dimension:\nW=[1,0]\n(ignore\n[0,1]\n, throwing away the extra feature),\nW=[0,1]\n(ignore\n[1,0]\n, throwing away the first feature to give the extra feature a dedicated dimension), and\nW=[1,-1]\n(store the features in superposition, losing the ability to represent\n[1,1]\n, the combination of both features at the same time). We call this last solution “antipodal” because the two basis vectors\n[1, 0]\nand\n[0, 1]\nare mapped in opposite directions. It turns out we can analytically determine the loss for these solutions (details can be found in\nthis notebook\n).\nAs expected, sparsity is necessary for superposition to occur, but we can see that it interacts in an interesting way with relative feature importance. But most interestingly, there appears to be a real phase change, observed in both the empirical and theoretical diagrams! The optimal weight configuration discontinuously changes in magnitude and superposition. (In the theoretical model, we can analytically confirm that there's a first-order phase change: there's crossover between the functions, causing a discontinuity in the derivative of the optimal loss.)\nWe can ask this same question of embedding three features in two dimensions. This problem still has a single \"extra feature\" (now the third one) we can study, asking what happens as we vary its importance relative to the other two and change sparsity.\nFor the theoretical model, we now consider four natural solutions. We can describe solutions by asking \"what feature direction did\nW\nignore?\" For example,\nW\nmight just not represent the extra feature – we'll write this\nW \\perp [0, 0, 1]\n. Or\nW\nmight ignore one of the other features,\nW \\perp [1, 0, 0]\n. But the interesting thing is that there are two ways to use superposition to make antipodal pairs. We can put the \"extra feature\" in an antipodal pair with one of the others (\nW \\perp [0, 1, 1]\n) or put the other two features in superposition and give the extra feature a dedicated dimension (\nW \\perp [1, 1, 0]\n). Details on the closed form losses for these solutions can be found in\nthis notebook\n. We do not consider a last solution of putting all the features in joint superposition,\nW \\perp [1, 1, 1]\n.\nThese diagrams suggest that there really is a phase change between different strategies for encoding features. However, we'll see in the next section that there's much more complex structure this preliminary view doesn't capture.\nThe Geometry of Superposition\nWe've seen that superposition can allow a model to represent extra features, and that the number of extra features increases as we increase sparsity. In this section, we'll investigate this relationship in more detail, discovering an unexpected geometric story: features seem to organize themselves into geometric structures such as pentagons and tetrahedrons! In some ways, the structure described in this section seems \"too elegant to be true\" and we think there's a good chance it's at least partly idiosyncratic to the toy model we're investigating. But it seems worth investigating because if anything about this generalizes to real models, it may give us a lot of leverage in understanding their representations.\nWe'll start by investigating\nuniform superposition\n, where all features are identical: independent, equally important and equally sparse. It turns out that uniform superposition has a surprising connection to the geometry of uniform polytopes! Later, we'll move on to investigate\nnon-uniform superposition\n, where features are not identical. It turns out that this can be understood, at least to some extent, as a deformation of uniform superposition.\nUniform Superposition\nAs mentioned above, we begin our investigation with uniform superposition, where all features have the same importance and sparsity. We'll see later that this case has some unexpected structure, but there's also a much more basic reason to study it: it's much easier to reason about than the non-uniform case, and has fewer variables we need to worry about in our experiments.\nWe'd like to understand what happens as we change feature sparsity,\nS\n. Since all features are equally important, we will assume without loss of generality\nScaling the importance of all features by the same amount simply scales the loss, and does not change the optimal solutions.\nthat each feature has importance\nI_i = 1\n. We'll study a model with\nn=400\nfeatures and\nm=30\nhidden dimensions, but it turns out the number of features and hidden dimensions doesn't matter very much. In particular, it turns out that the number of input features\nn\ndoesn't matter as long as it's much larger than the number of hidden dimensions,\nn \\gg m\n. And it also turns out that the number of hidden dimensions doesn't really matter as long as we're interested in the ratio of features learned to hidden features. Doubling the number of hidden dimensions just doubles the number of features the model learns.\nA convenient way to measure the number of features the model has learned is to look at the Frobenius norm,\n||W||_F^2\n. Since\n||W_i||^2\\simeq 1\nif a feature is represented and\n||W_i||^2\\simeq 0\nif it is not, this is roughly the number of features the model has learned to represent. Conveniently, this norm is basis-independent, so it still behaves nicely in the dense regime\nS=0\nwhere the feature basis isn't privileged by anything and the model represents features with arbitrary directions instead.\nWe'll plot\nD^* = m / ||W||_F^2\n, which we can think of as the \"dimensions per feature\":\nSurprisingly, we find that this graph is \"sticky\" at\n1\nand\n1/2\n. (This very vaguely resembles the fractional quantum Hall effect – see e.g.\nthis diagram\n.) Why is this? On inspection, the\n1/2\n\"sticky point\" seems to correspond to a precise geometric arrangement where features come in \"antipodal pairs\", each being exactly the negative of the other, allowing two features to be packed into each hidden dimension. It appears that antipodal pairs are so effective that the model preferentially uses them over a wide range of the sparsity regime.\nIt turns out that antipodal pairs are just the tip of the iceberg. Hiding underneath this curve are a number of extremely specific geometric configurations of features.\nFeature Dimensionality\nIn the previous section, we saw that there's a sticky regime where the model has \"half a dimension per feature\" in some sense. This is an average statistical property of the features the model represents, but it seems to hint at something interesting. Is there a way we could understand what \"fraction of a dimension\" a specific feature gets?\nWe'll define the\ndimensionality\nof the\ni\nth feature,\nD_i\n, as:\nD_i ~=~ \\frac{||W_i||^2}{\\sum_j (\\hat{W_i} \\cdot W_j)^2}\nwhere\nW_i\nis the weight vector column associated with the\ni\nth feature, and\n\\hat{W_i}\nis the unit version of that vector.\nIntuitively, the numerator represents the extent to which a given feature is represented, while the denominator is \"how many features share the dimension it is embedded in\" by projecting each feature onto its dimension. In the antipodal case, each feature participating in an antipodal pair will have a dimensionality of\nD = 1 / (1+1) = 1/2\nwhile features which are not learned will have a dimensionality of\n0\n. Empirically, it seems that the dimensionality of all features add up to the number of embedding dimensions when the features are \"packed efficiently\" in some sense.\nWe can now break the above plot down on a per-feature basis. This reveals many more of these \"sticky points\"! To help us understand this better, we're going to create a scatter plot annotated with some additional information:\nWe start with the line plot we had in the previous section.\nWe overlay this with a scatter plot of the individual feature dimensionalities for each feature in the models at each sparsity level.\nThe feature dimensionalities cluster at certain fractions, so we draw lines for those. (It turns out that each fraction corresponds to a specific weight geometry – we'll discuss this shortly.)\nWe visualize the weight geometries for a few models with a \"feature geometry graph\" where each feature is a node and edge weights are based on the absolute value of the dot product feature embedding vectors. So features are connected if they aren't orthogonal.\nLet's look at the resulting plot, and then we'll try to figure out what it's showing us:\nWhat is going on with the points clustering at specific fractions?? We'll see shortly that the model likes to create specific weight geometries and kind of jumps between the different configurations.\nIn the previous section, we developed a theory of superposition as a phase change. But everything on this plot between 0 (not learning a feature) and 1 (dedicating a dimension to a feature) is superposition. Superposition is what happens when features have fractional dimensionality. That is to say – superposition isn't just one thing!\nHow can we relate this to our original understanding of the phase change? We often think of water as only having three phases: ice, water and steam. But this is a simplification: there are actually many phases of ice, often corresponding to different crystal structures (eg. hexagonal vs cubic ice). In a vaguely similar way, neural network features seem to also have many other phases within the general category of \"superposition.\"\nWhy these geometric structures?\nIn the previous diagram, we found that there are distinct lines corresponding to dimensionality of: ¾ (tetrahedron), ⅔ (triangle), ½ (antipodal pair), ⅖ (pentagon), ⅜ (square antiprism), and 0 (feature not learned). We believe there would also be a 1 (dedicated dimension for a feature) line if not for the fact that basis features are indistinguishable from other directions in the dense regime.\nSeveral of these configurations may jump out as solutions to the famous\nThomson problem\n. (In particular,\nsquare antiprisms\nare much less famous than cubes and are primarily of note for their\nrole in molecular geometry\ndue to being a Thomson problem solution.) As we saw earlier, there is a very real sense in which our model can be understood as solving a generalized version of the Thomson problem. When our model chooses to represent a feature, the feature is embedded as a point on an\nm\n-dimensional sphere.\nA second clue as to what's going on is that there are lines for the Thomson solutions which are uniform polyhedra (e.g. tetrahedron), but there seem to be split lines where we'd expect to see non-uniform solutions (e.g. instead of a ⅗ line for triangular bipyramids we see a co-occurrence of points at ⅔ for triangles and points at ½ for antipodes). In a uniform polyhedron, all vertices have the same geometry, and so if we embed features as them each feature has the same dimensionality. But if we embed features as a non-uniform polyhedron, different features will have more or less interference with others.\nIn particular, many of the Thomson solutions can be understood as\ntegum products\n(an operation which constructs polytopes  by embedding two polytopes in orthogonal subspaces) of smaller uniform polytopes. (In the earlier graph visualizations of feature geometry, two subgraphs are disconnected if and only if they are in different tegum factors.) As a result, we should expect their dimensionality to actually correspond to the underlying factor uniform polytopes.\nThis also suggests a possible reason why we observe 3D Thomson problem solutions, despite the fact that we're actually studying a higher dimensional version of the problem. Just as many 3D Thomson solutions are tegum products of 2D and 1D solutions, perhaps higher dimensional solutions are often tegum products of 1D, 2D, and 3D solutions.\nThe orthogonality of factors in tegum products has interesting implications. For the purposes of superposition, it means that there can't be any \"interference\" across tegum-factors. This may be preferred by the toy model: having many features interfere simultaneously could be really bad for it. (See related discussion in\nour earlier mathematical analysis\n.)\nAside: Polytopes and Low-Rank Matrices\nAt this point, it's worth making explicit that there's a correspondence between\npolytopes\nand\nsymmetric, positive-definite, low-rank matrices\n(i.e. matrices of the form\nW^TW\n). This correspondence underlies the results we saw in the previous section, and is generally useful for thinking about superposition.\nIn some ways, the correspondence is trivial. If one has a rank-\nm\nn\\!\\times\\!n\n-matrix of the form\nW^TW\n, then\nW\nis a\nn\\!\\times\\!m\n-matrix. We can interpret the columns of\nW\nas\nn\npoints in a\nm\n-dimensional space. The place where this starts to become interesting is that it makes it clear that\nW^TW\nis driven by the geometry. In particular, we can see how the off-diagonal terms are driven by the geometry of the points.\nPut another way, there's an exact correspondence between polytopes and strategies for superposition. For example, every strategy for putting three features in superposition in a 2-dimensional space corresponds to a triangle, and every triangle corresponds to such a strategy. From this perspective, it doesn't seem surprising that if we have three equally important and equally sparse features, the optimal strategy is an equilateral triangle.\nThis correspondence also goes the other direction. Suppose we have a rank\n(n\\!-\\!i)\n-matrix of the form\nW^TW\n. We can characterize it by the dimensions\nW\ndid not\nrepresent – that is, which directions are orthogonal to\nW\n? For example, if we have a\n(n\\!-\\!1)\n-matrix, we might ask what single direction did\nW\nnot represent? This is especially informative if we assume that\nW^TW\nwill be as \"identity-like\" as possible, given the constraint of not representing certain vectors.\nIn fact, given such a set of orthogonal vectors, we can construct a polytope by starting with\nn\nbasis vectors and projecting them to a space orthogonal to the given vectors. For example, if we start in three dimensions and then project such that\nW \\perp (1,1,1)\n, we get a triangle. More generally, setting\nW \\perp (1,1,1,...)\ngives us a\nregular\nn\n-simplex\n. This is interesting because it's in some sense the \"minimal possible superposition.\" Assuming that features are equally important and sparse, the best possible direction to not represent is the fully dense vector\n(1,1,1,...)\n!\nNon-Uniform Superposition\nSo far, this section has focused on the geometry of uniform superposition, where all features are of equal importance, equal sparsity, and independent. The model is essentially solving a variant of the Thomson problem. Because all features are the same, solutions corresponding to uniform polyhedra get especially low loss. In this subsection, we'll study non-uniform superposition, where features are somehow not uniform. They may vary in importance and sparsity, or have a correlational structure that makes them not independent. This distorts the uniform geometry we saw earlier.\nIn practice, it seems like superposition in real neural networks will be non-uniform, so developing an understanding of it seems important. Unfortunately, we're far from a comprehensive theory of the geometry of non-uniform superposition at this point. As a result, the goal of this section will merely be to highlight some of the more striking phenomena we observe:\nFeatures varying in importance or sparsity\ncauses smooth deformation of polytopes as the imbalance builds, up until a critical breaking point at which they snap to another polytope.\nCorrelated features\nprefer to be orthogonal, often forming in different tegum factors. As a result, correlated features may form an orthogonal local basis. When they can't be orthogonal, they prefer to be side-by-side. In some cases correlated features merge into a single feature: this hints at some kind of interaction between \"superposition-like behavior\" and \"PCA-like behavior\".\nAnti-correlated features\nprefer to be in the same tegum factor when superposition is necessary. They prefer to have negative interference, ideally being antipodal.\nWe attempt to illustrate these phenomena with some representative experiments below.\nPerturbing a Single Feature\nThe simplest kind of non-uniform superposition is to vary one feature and leave the others uniform. As an experiment, let's consider an experiment where we represent\nn=5\nfeatures in\nm=2\ndimensions. In the uniform case, with importance\nI=1\nand activation density\n1-S=0.05\n, we get a regular pentagon. But if we vary one point – in this case we'll make it more or less sparse – we see the pentagram\nstretch\nto account for the new value. If we make it denser, activating more frequently (yellow) the other features repel from it, giving it more space. On the other hand, if we make it sparser, activating less frequently (blue) it takes less space and other points push towards it.\nIf we make it sufficiently sparse, there's a phase change, and it collapses from a pentagon to a pair of digons with the sparser point at zero. The phase change corresponds to loss curves corresponding to the two different geometries crossing over. (This observation allows us to directly confirm that it is genuinely a first order phase change.)\nTo visualize the solutions, we canonicalize them, rotating them to align with each other in a consistent manner.\nThese results seem to suggest that, at least in some cases, non-uniform superposition can be understood as a\ndeformation of uniform superposition\nand\njumping between uniform superposition configurations\nrather than a totally different regime. Since uniform superposition has a lot of understandable structure, but real world superposition is almost certainly non-uniform, this seems very promising!\nThe reason pentagonal solutions are not on the unit circle is because models reduce the effect of positive interference, setting a slight negative bias to cut off noise and setting their weights to\n||W_i|| = 1 / (1-b_i)\nto compensate. Distance from the unit circle can be interpreted as primarily driven by the amount of positive interference.\nA note for reimplementations: optimizing with a two-dimensional hidden space makes this easier to study, but the actual optimization process turns out to be really challenging from gradient descent – a lot harder than even just having three dimensions. Getting clean results required fitting each model multiple times and taking the solution with the lowest loss. However, there's a silver lining to this: visualizing the sub-optimal solutions on a scatter plot as above allows us to see the loss curves for different geometries and gain greater insight into the phase change.\nCorrelated and Anticorrelated Features\nA more complicated form of non-uniform superposition occurs when there are correlations between features. This seems essential for understanding superposition in the real world, where many features are correlated or anti-correlated.\nFor example, one very pragmatic question to ask is whether we should expect polysemantic neurons to group the same features together across models. If the groupings were random, you could use this to detect polysemantic neurons, by comparing across models! However, we'll see that correlational structure strongly influences which features are grouped together in superposition.\nThe behavior seems to be quite nuanced, with a kind of \"order of preferences\" for how correlated features behave in superposition. The model ideally represents correlated features orthogonally, in separate tegum factors with no interactions between them. When that fails, it prefers to arrange them so that they're as close together as possible – it prefers positive interference between correlated features over negative interference. Finally, when there isn't enough space to represent all the correlated features, it will collapse them and represent their principal component instead! Conversely, when features are anti-correlated, models prefer to have them interfere, especially with negative interference. We'll demonstrate this with a few experiments below.\nSetup for Exploring Correlated and Anticorrelated Features\nThroughout this section we'll refer to \"correlated feature sets\" and \"anticorrelated feature sets\".\nCorrelated Feature Sets.\nOur correlated feature sets can be thought of as \"bundles\" of co-occurring features. One can imagine a highly idealized version of what might happen in an image classifier: there could be a bundle of features used to identify animals (fur, ears, eyes) and another bundle used to identify buildings (corners, windows, doors). Features from one of these bundles are likely to appear together. Mathematically, we represent this by linking the choice of whether all the features in a correlated feature set are zero or not together. Recall that we\noriginally defined\nour synthetic distribution to have features be zero with probability\nS\nand otherwise uniformly distributed between [0,1]. We simply have the same sample determine whether they're zero.\nAnticorrelated Feature Sets.\nOne could also imagine anticorrelated features which are extremely unlikely to occur together. To simulate these, we'll have anticorrelated feature sets where only one feature in the set can be active at a time. To simulate this, we'll have the feature set be entirely zero with probability\nS\n, but then only have one randomly selected feature in the set be uniformly sampled from [0,1] if it's active, with the others being zero.\nOrganization of Correlated and Anticorrelated Features\nFor our initial investigation, we simply train a number of small toy models with correlated and anti-correlated features and observe what happens. To make this easy to study, we limit ourselves to the\nm=2\ncase where we can explicitly visualize the weights as points in 2D space. In general, such solutions can be understood as a collection of points on a unit circle. To make solutions easy to compare, we rotate and flip solutions to align with each other.\nLocal Almost-Orthogonal Bases\nIt turns out that the tendency of models to arrange correlated features to be orthogonal is actually quite a strong phenomenon. In particular, for larger models, it seems to generate a kind of \"local almost-orthogonal basis\" where, even though the model as a whole is in superposition, the correlated feature sets considered in isolation are (nearly) orthogonal and can be understood as having very little superposition.\nTo investigate this, we train a larger model with two sets of correlated features and visualize\nW^TW\n.\nIf this result holds in real neural networks, it suggests we might be able to make a kind of \"local non-superposition\" assumption, where for certain sub-distributions we can assume that the activating features are not in superposition. This could be a powerful result, allowing us to confidently use methods such as PCA which might not be principled to generally use in the context of superposition.\nCollapsing of Correlated Features\nOne of the most interesting properties is that there seems to be a trade off with Principal Components Analysis (PCA) and superposition. If there are two correlated features\na\nand\nb\n, but the model only has capacity to represent one, the model will represent their principal component\n(a+b)/\\sqrt{2}\n, a sparse variable that has more impact on the loss than either individually, and ignore the second principal component\n(a-b)/\\sqrt{2}\n.\nAs an experiment, we consider six features, organized into three sets of correlated pairs. Features in each correlated pair are represented by a given color (red, green, and blue). The correlation is created by having both features always activate together – they're either both zero or neither zero. (The exact non-zero values they take when they activate is uncorrelated.)\nAs we vary the sparsity of the features, we find that in the very sparse regime, we observe superposition as expected, with features arranged in a hexagon and correlated features side-by-side. As we decrease sparsity, the features progressively \"collapse\" into their principal components. In very dense regimes, the solution becomes equivalent to PCA.\nThese results seem to hint that PCA and superposition are in some sense complementary strategies which trade off with one another. As features become more correlated, PCA becomes a better strategy. As features become sparser, superposition becomes a better strategy. When features are both sparse and correlated, mixtures of each strategy seem to occur. It would be nice to more deeply understand this space of tradeoffs.\nIt's also interesting to think about this in the context of continuous\nequivariant features\n, such as features which occur in different rotations.\nSuperposition and Learning Dynamics\nThe focus of this paper is how superposition contributes to the functioning of fully trained neural networks, but as a brief detour it's interesting to ask how our toy models – and the resulting superposition – evolve over the course of training.\nThere are several reasons why these models seem like a particularly interesting case for studying learning dynamics. Firstly, unlike most neural networks, the fully trained models converge to a simple but non-trivial structure that rhymes with an emerging thread of evidence that neural network learning dynamics might have geometric weight structure that we can understand. One might hope that understanding the final structure would make it easier for us to understand the evolution over training. Secondly, superposition hints at surprisingly discrete structure (regular polytopes of all things!). We'll find that the underlying learning dynamics are also surprisingly discrete, continuing an emerging trend of evidence that neural network learning might be less continuous than it seems. Finally, since superposition has significant implications for interpretability, it would be nice to understand how it emerges over training – should we expect models to use superposition early on, or is it something that only emerges later in training, as models struggle to fit more features in?\nUnfortunately, we aren't able to give these questions the detailed investigation they deserve within the scope of this paper. Instead, we'll limit ourselves to a couple particularly striking phenomena we've noticed, leaving more detailed investigation for future work.\nPhenomenon 1: Discrete \"Energy Level\" Jumps\nPerhaps the most striking phenomenon we've noticed is that the learning dynamics of toy models with large numbers of features appear to be dominated by \"energy level jumps\" where features jump between different feature dimensionalities. (Recall that a feature's\ndimensionality\nis the fraction of a dimension dedicated to representing a feature.)\nLet's consider the problem setup we studied when investigating the geometry of uniform superposition in the\nprevious section\n, where we have a large number of features of equal importance and sparsity. As we saw previously, the features ultimately arrange themselves into a small number of polytopes with fractional dimensionalities.\nA natural question to ask is what happens to these feature dimensionalities over the course of training. Let's pick one model where all the features converge into digons and observe. In the first plot, each colored line corresponds to the dimensionality of a single feature. The second plot shows how the loss curve changes over the same duration.\nNote how the dimensionality of some features \"jump\" between different values and swap places. As this happens, the loss curve also undergoes a sudden drop (a very small one at the first jump, and a larger one at the second jump).\nThese results make us suspect that seemingly smooth decreases of the loss curve in larger models are in fact composed of many small jumps of features between different configurations. (For similar results of sudden mechanistic changes, see Olsson\net al.\n's induction head phase change\n, and Nanda and Lieberum's results on phase changes in modular arithmetic\n. More broadly, consider the phenomenon of grokking\n.)\nPhenomenon 2: Learning as Geometric Transformations\nMany of our toy model solutions can be understood as corresponding to geometric structures. This is especially easy to see and study when there are only\nm=3\nhidden dimensions, since we can just directly visualize the feature embeddings as points in 3D space forming a polyhedron.\nIt turns out that, at least in some cases, the learning dynamics leading to these structures can be understood as a sequence of simple, independent geometric transformations!\nOne particularly interesting example of this phenomenon occurs in the context of\ncorrelated features\n, as studied in the previous section. Consider the problem of representing\nn=6\nfeatures in superposition within\nm=3\ndimensions. If we have the\n6\nfeatures be\n2\nsets of\n3\ncorrelated features, we observe a really interesting pattern. The learning proceeds in distinct regimes which are visible in the loss curve, with each regime corresponding to a distinct geometric transformation:\n(Although the last solution – an octahedron with features from different correlated sets arranged in antipodal pairs – seems to be a strong attractor, the learning trajectory visualized above appears to be one of a few different learning trajectories that attract the model. The different trajectories vary at step\nC\n: sometimes the model gets pulled directly into the antiprism configuration from the start or organizes features into antipodal pairs. Presumably this depends on which feature geometry the model is closest to when step\nB\nends.)\nThe learning dynamics we observe here seem directly related to previous findings on simple models.\nfound that two-layer neural networks, in early stages of training, tend to learn a linear approximation to a problem. Although the technicalities of our data generation process do not precisely match the hypotheses of their theorem, it seems likely that the same basic mechanism is at work. In our case, we see the toy network learns a linear PCA solution before moving to a better nonlinear solution. A second related finding comes from\n, who looked at hierarchical sets of features, with a data generation process similar to the one we consider. They find empirically that certain networks (nonlinear and deep linear) “split” embedding vectors in a manner very much like what we observed. They also provide a theoretical analysis in terms of the underlying dynamical system. A key difference is that they focus on the topology—the branching structure of the emerging feature representations—rather than the geometry. Despite this difference, it seems likely that their analysis could be generalized to our case.\nRelationship to Adversarial Robustness\nAlthough we're most interested in the implications of superposition for interpretability, there appears to be a connection to adversarial examples. If one gives it a little thought, this connection can actually be quite intuitive.\nIn a model without superposition, the end-to-end weights for the first feature are:\n(W^TW)_0 ~~=~~ (1,~ 0,~ 0,~ 0,~ ...)\nBut in a model with superposition, it's something like:\n(W^TW)_0 ~~=~~ (1,~ \\epsilon,~ -\\epsilon,~ \\epsilon,~ ...)\nThe\n\\epsilon\nentries (which are solely an artifact of superposition \"interference\") create an obvious way for an adversary to attack the most important feature. Note that this may remain true even in the infinite data limit: the optimal behavior of the model fit to sparse infinite data is to use superposition to represent more features, leaving it vulnerable to attack.\nTo test this, we generated L2 adversarial examples (allowing a max L2 attack norm of 0.1 of the average input norm). We originally generated attacks with gradient descent, but found that for extremely sparse examples where ReLU neurons are in the zero regime 99% of the time, attacks were difficult, effectively due to gradient masking\n. Instead, we found it worked better to analytically derive adversarial attacks by considering the optimal L2 attacks for each feature (\n\\lambda (W^TW)_i / ||(W^TW)_i||_2\n) and taking the one of these attacks which most harms model performance.\nWe find that vulnerability to adversarial examples sharply increases as superposition forms (increasing by >3×), and that the level of vulnerability closely tracks the number of features per dimension (the reciprocal of\nfeature dimensionality\n).\nWe're hesitant to speculate about the extent to which superposition is responsible for adversarial examples in practice. There are compelling theories for why adversarial examples occur without reference to superposition (\ne.g.\n). But it is interesting to note that if one wanted to try to argue for a \"superposition maximalist stance\", it does seem like many interesting phenomena related to adversarial examples can be predicted from superposition. As seen above, superposition can be used to explain why adversarial examples exist. It also predicts that adversarially robust models would have worse performance, since making models robust would require giving up superposition and representing less features. It predicts that more adversarially robust models might be more interpretable (\nsee e.g.\n). Finally, it could arguably predict that adversarial examples transfer (\nsee e.g.\n) if the arrangement of features in superposition is heavily influenced by which features are correlated or anti-correlated (see\nearlier results on this\n). It might be interesting for future work to see how far the hypothesis that superposition is a significant contributor to adversarial examples can be driven.\nIn addition to observing that superposition can cause models to be vulnerable to adversarial examples, we briefly experimented with adversarial training to see if the relationship could be used in the other direction to reduce superposition. To keep training reasonably efficient, we used the analytic optimal attack against a random feature. We found that this did reduce superposition, but attacks had to be made unreasonably large (80% input L2 norm) to fully eliminate it, which didn't seem satisfying. Perhaps stronger adversarial attacks would work better. We didn't explore this further since the increased cost and complexity of adversarial training made us want to prioritize other lines of attack on superposition first.\nSuperposition in a Privileged Basis\nSo far, we've explored superposition in a model\nwithout a privileged basis\n. We can rotate the hidden activations arbitrarily and, as long as we rotate all the weights, have the exact same model behavior. That is, for any ReLU output model with weights\nW\n, we could take an arbitrary orthogonal matrix\nO\nand consider the model\nW' = OW\n. Since\n(OW)^T(OW) = W^TW\n, the result would be an identical model!\nModels without a privileged basis are elegant, and can be an interesting analogue for certain neural network representations which don't have a privileged basis – word embeddings, or the transformer residual stream. But we'd also (and perhaps primarily) like to understand neural network representations where there are neurons which do impose a privileged basis, such as transformer MLP layers or conv net neurons.\nOur goal in this section is to explore the simplest toy model which gives us a privileged basis. There are at least two ways we could do this: we could add an activation function or apply L1 regularization to the hidden layer. We'll focus on adding an activation function, since the representation we are most interested in understanding is hidden layers with neurons, such as the transformer MLP layer.\nThis gives us the following \"ReLU hidden layer\" model:\nh~=~\\text{ReLU}(Wx)\nx'~=~\\text{ReLU}(W^Th+b)\nWe'll train this model on the same data as before.\nAdding a ReLU to the hidden layer radically changes the model from an interpretability perspective. The key thing is that while\nW\nin our previous model was challenging to interpret (recall that we visualized\nW^TW\nrather than\nW\n),\nW\nin the ReLU hidden layer model can be directly interpreted, since it connects features to basis-aligned neurons.\nWe'll discuss this in much more detail shortly, but here's a comparison of weights resulting from a linear hidden layer model and a ReLU hidden layer model:\nRecall that we think of basis elements in the input as \"features,\" and basis elements in the middle layer as \"neurons\". Thus\nW\nis a map from features to neurons.\nWhat we see in the above plot is that\nthe\nfeatures are aligning with neurons\nin a structured way\n! Many of the neurons are simply dedicated to representing a feature! (This is the critical property that justifies why neuron-focused interpretability approaches – such as much of the work in the original Circuits thread – can be effective in some circumstances.)\nLet's explore this in more detail.\nVisualizing Superposition in Terms of Neurons\nHaving a privileged basis opens up new possibilities for visualizing our models. As we saw above, we can simply inspect\nW\n. We can also make a per-neuron stacked bar plot where, for every neuron, we visualize its weights as a stack of rectangles on top of each other:\nEach column in the stack plot visualizes one column of\nW\n.\nEach rectangle represents one weight entry, with height corresponding to the absolute value.\nThe color of each rectangle corresponds to the feature it acts on (i.e. which row of\nW\nit's in).\nNegative values go below the x-axis.\nThe order of the rectangles is not significant.\nThis stack plot visualization can be nice as models get bigger. It also makes polysemantic neurons obvious: they simply correspond to having more than one weight.\nWe'll now visualize a ReLU hidden layer toy model with\nn=10;~ m=5; I^i = 0.75^i\nand varying feature sparsity levels. We chose a very small model (only 5 neurons) both for ease of visualization, and to circumvent some issues with this toy model we'll discuss below.\nHowever, we found that these small models were harder to optimize. For each model shown, we trained 1000 models and visualized the one with the lowest loss. Although the typical solutions are often similar to the minimal loss solutions shown, selecting the minimal loss solutions reveals even more structure in how features align with neurons. It also reveals that there are ranges of sparsity values where the optimal solution for all models trained on data with that sparsity has the same weight configurations.\nThe solutions are visualized below, both visualizing the raw\nW\nand a neuron stacked bar plot. We color features in the stacked bar plot based on whether they're in superposition, and color neurons as being monosemantic or polysemantic depending on whether they store more than one feature. Neuron order was chosen by hand (since it's arbitrary).\nThe most important thing to pay attention to is how\nthere's a shift from monosemantic to polysemantic neurons as sparsity increases\n. Monosemantic neurons do exist in some regimes! Polysemantic neurons exist in others. And they can both exist in the same model! Moreover, while it's not quite clear how to formalize this, it looks a great deal like there's a neuron-level phase change, mirroring the feature phase changes we saw earlier.\nIt's also interesting to examine the structure of the polysemantic solutions, which turn out to be surprisingly structured and neuron-aligned. Features typically correspond to\nsets of neurons\n(monosemantic neurons might be seen as the special case where features only correspond to singleton sets). There's also structure in how polysemantic neurons are. They transition from monosemantic, to only representing a few features, to gradually representing more. However, it's unclear how much of this is generalizable to real models.\nLimitations of The ReLU Hidden Layer Toy Model Simulating Identity\nUnfortunately, the toy model described in this section has a significant weakness, which limits the regimes in which it shows interesting results. The issue is that the model doesn't benefit from the ReLU hidden layer – it has no role except limiting how the model can encode information. If given any chance, the model will circumvent it. For example, given a hidden layer bias, the model will set all the biases to be positive, shifting the neurons into a positive regime where they behave linearly. If one removes the bias, but gives the model enough features, it will simulate a bias by averaging over many features. The model will only use the ReLU activation function if absolutely forced, which is a significant mark against studying this toy model.\nWe'll introduce a model without this issue in the next section, but wanted to study this model as a simpler case study.\nComputation in Superposition\nSo far, we've shown that neural networks can store sparse features in superposition and then recover them. But we actually believe superposition is more powerful than this – we think that neural networks can\nperform computation entirely in superposition\nrather than just using it as storage. This model will also give us a more principled way to study a\nprivileged basis\nwhere features align with basis dimensions.\nTo explore this, we consider a new setup where we imagine our input and output layer to be the layers of our hypothetical disentangled model, but have our hidden layer be a smaller layer we're imagining to be the observed model which might use superposition. We'll then try to compute a simple non-linear function and explore whether it can use superposition to do this. Since the model will have (and need to use) the hidden layer non-linearity, we'll also see features align with a privileged basis.\nSpecifically, we'll have the model compute\ny=\\text{abs}(x)\n. Absolute value is an appealing function to study because there's a very simple way to compute it with ReLU neurons:\n\\text{abs}(x) = \\text{ReLU}(x) + \\text{ReLU}(-x)\n. This simple structure will make it easy for us to study the geometry of how the hidden layer is leveraged to do computation.\nSince this model\nneeds\nReLU to compute absolute value, it doesn't have the issues the model in the previous section had with trying to avoid the activation function.\nExperiment Setup\nThe input feature vector,\nx\n, is still sparse, with each feature\nx_i\nhaving probability\nS_i\nof being\n0\n. However, since we want to have the model compute absolute value, we need to allow it to take on non-positive values for this to be a non-trivial task. As a result, if it is non-zero, its value is now sampled uniformly from\n[-1,1]\n. The target output\ny\nis\ny=\\text{abs}(x)\n.\nFollowing the previous section, we'll consider the \"ReLU hidden layer\" toy model variant, but no longer tie the two weights to be identical:\nh = \\text{ReLU}(W_1x)\ny' = \\text{ReLU}(W_2h+b)\nThe loss is still the mean squared error weighted by feature importances\nI_i\nas before.\nBasic Results\nWith this model, it's a bit less straightforward to study how individual features get embedded; because of the ReLU on the hidden layer, we can't just study\nW_2^TW_1\n. And because\nW_2\nand\nW_1\nare now learned independently, we can't just study columns of\nW_1\n. We believe that with some manipulation we could recover much of the simplicity of the earlier model by considering \"positive features\" and \"negative features\" independently, but we're going to focus on another perspective instead.\nAs we saw in the previous section, having a hidden layer activation function means that it makes sense to visualize the weights in terms of neurons. We can visualize\nW\ndirectly or as a neuron stack plot as we did before. We can also visualize it as a graph, which can sometimes be helpful for understanding computation.\nLet's look at what happens when we train a model with\nn=3\nfeatures to perform absolute value on\nm=6\nhidden layer neurons. Without superposition, the model needs two hidden layer neurons to implement absolute value on one feature.\nThe resulting model – modulo a subtle issue about rescaling input and output weights\nNote that there's a degree of freedom for the model in learning\nW_1\n: We can rescale any hidden unit by scaling its row of\nW_1\nby\n\\alpha\n, and its column of\nW_2\nby\n\\alpha^{-1}\n, and arrive at the same model. For consistency in the visualization, we rescale each hidden unit before visualizing so that the largest-magnitude weight to that neuron from\nW_1\nhas magnitude\n1\n.\n– performs absolute value exactly as one might expect. For each input feature\nx_i\n, it constructs a \"positive side\" neuron\n\\text{ReLU}(x_i)\nand a \"negative side\" neuron\n\\text{ReLU}(-x_i)\n. It then adds these together to compute absolute value:\nSuperposition vs Sparsity\nWe've seen that – as expected – our toy model can learn to implement absolute value. But can it use superposition to compute absolute value for more features? To test this, we train models with\nn=100\nfeatures and\nm=40\nneurons and a feature importance curve\nI_i = 0.8^i\n, varying feature sparsity.\nThese specific values were chosen to illustrate the phenomenon we're interested in: the absolute value model learns more easily when there are more neurons, but we wanted to keep the numbers small enough that it could be easily visualized.\nA couple of notes on visualization: Since we're primarily interested in understanding superposition and polysemantic neurons, we'll show a stacked weight plot of the absolute values of weights. The features are colored by superposition. To make the diagrams easier to read, neurons are faintly colored based on how polysemantic they are (as judged by eye based on the plots). Neuron order is sorted by the importance of the largest feature.\nMuch like we saw in the ReLU hidden layer models, these results demonstrate that activation functions, under the right circumstances, create a privileged basis and cause features to align with basis dimensions. In the dense regime, we end up with each neuron representing a single feature, and we can read feature values directly off of neuron activations.\nHowever, once the features become sufficiently sparse, this model, too, uses superposition to represent more features than it has neurons. This result is notable because it demonstrates the ability of neural networks to\nperform computation\neven on data that is represented in superposition.\nOne question you might ask is whether we can quantify the ability of superposition to enable extra computation by examining the loss. Unfortunately, we can't easily do this. Superposition occurs when we change the task, making it sparser. As a result, the losses of models with different amounts of superposition are not comparable – they're measuring the loss on different tasks!\nRemember that the model is required to use the hidden layer ReLU in order to compute an absolute value; gradient descent manages to find solutions that usefully approximate the computation even when each neuron encodes a mix of multiple features.\nFocusing on the intermediate sparsity regimes, we find several additional qualitative behaviors that we find fascinatingly reminiscent of behavior that has been observed in real, full-scale neural networks:\nTo begin, we find that in some regimes,\nmany\nof the model's neurons will encode pure features, but a subset of them will be highly polysemantic. This is similar to the\nphase change\nwe saw earlier in the ReLU output model. However, in that case, the phase change was with respect to features, with more important features not being put in superposition. In this experiment, the neurons don't have any intrinsic importance, but we see that the neurons representing the most important features (on the left) tend to be monosemantic.\nWe find this to bear a suggestive resemblance to\nsome previous work in vision models\n, which found some layers that contained \"mostly pure\" feature neurons, but with some neurons representing additional features on a different scale.\nWe also note that many neurons appear to be associated with a single \"primary\" feature – encoded by a relatively large weight – coupled with one or more \"secondary\" features encoded with smaller-magnitude weights to that neuron. If we were to observe the activations of such a neuron over a range of input examples, we would find that the largest activations of that neuron were all or nearly-all associated with the presence of the \"primary\" feature, but that the lower-magnitude activations were much more polysemantic.\nIntriguingly, that description closely matches what researchers have found in previous work on language models\n– many neurons appear interpretable when we examine their strongest activations over a dataset, but can be shown on further investigation to activate for other meanings or patterns, often at a lower magnitude. While only suggestive, the ability of our toy model to reproduce these qualitative features of larger neural networks offers an exciting hint that these models are illuminating general phenomena.\nThe Asymmetric Superposition Motif\nIf neural networks can perform computation in superposition, a natural question is to ask how exactly they're doing so. What does that look like mechanically, in terms of the weights? In this subsection, we'll (mostly) work through one such model and see an interesting motif of\nasymmetric superposition\n. (We use the term \"motif\" in the\nsense\nof the original circuit thread, inspired by its use in systems biology\n.)\nThe model we're trying to understand is shown below on the left, visualized as a neuron weight stack plot, with features corresponding to colors. The model is only doing a limited amount of superposition, and many of the weights can be understood as simply implementing absolute value in the expected way.\nHowever, there are a few neurons doing something else…\nThese other neurons implement two instances of asymmetric superposition and inhibition. Each instance consists of two neurons:\nOne neuron does\nasymmetric superposition\n. In normal superposition, one might store features with equal weights (eg.\nW=[1,-1]\n) and then have equal output weights (\nW=[1,1]\n). In asymmetric superposition, one stores the features with different magnitudes (eg.\nW=[2,-\\frac{1}{2}]\n) and then has reciprocal output weights (eg.\nW=[\\frac{1}{2}, 2]\n). This causes one feature to heavily interfere with the other, but avoid the other interfering with the first!\nTo avoid the consequences of that interference, the model has another neuron heavily inhibit the feature in the case where there would have been positive interference. This essentially converts positive interference (which could greatly increase the loss) into negative interference (which has limited consequences due to the output ReLU).\nThere are a few other weights this doesn't explain. (We believe they're effectively small conditional biases.) But this asymmetric superposition and inhibition pattern appears to be the primary story.\nThe Strategic Picture of Superposition\nAlthough superposition is scientifically interesting, much of our interest comes from a pragmatic motivation: we believe that superposition is deeply connected to the challenge of using interpretability to make claims about the safety of AI systems. In particular, it is a clear challenge to the most promising path we see to be able to say that neural networks won't perform certain harmful behaviors or to catch \"unknown unknowns\" safety problems. This is because superposition is deeply linked to the ability to identify and enumerate over all features in a model, and the ability to enumerate over all features would be a powerful primitive for making claims about model behavior.\nWe begin this section by describing how \"solving superposition\" in a certain sense is equivalent to many strong interpretability properties which might be useful for safety. Next, we'll describe three high level strategies one might take to \"solving superposition.\" Finally, we'll describe a few other additional strategic considerations.\nSafety, Interpretability, & \"Solving Superposition\"\nWe'd like a way to have confidence that models will never do certain behaviors such as \"deliberately deceive\" or \"manipulate.\" Today, it's unclear how one might show this, but we believe a promising tool would be the ability to\nidentify and enumerate over all features\n. The ability to have a universal quantifier over the fundamental units of neural network computation is a significant step towards saying that certain types of circuits don't exist.\nUltimately we want to say that a model doesn't implement some class of behaviors. Enumerating over all features makes it easy to say a feature doesn't exist (e.g. \"there is no 'deceptive behavior' feature\") but that isn't quite what we want. We expect models that need to represent the world to represent unsavory behaviors. But it may be possible to build more subtle claims such as \"all 'deceptive behavior' features do not participate in circuits X, Y and Z.\"\nIt also seems like a powerful tool for addressing \"unknown unknowns\", since it's a way that one can fully cover network behavior, in a sense.\nHow does this relate to superposition? It turns out that the ability to enumerate over features is deeply intertwined with superposition. One way to see this is to imagine a neural network with a privileged basis and without superposition (like the monosemantic neurons found in early InceptionV1, e.g.\n): features would simply correspond to neurons, and you could enumerate over features by enumerating over neurons.\nSuperposition also makes it harder to find interpretable directions in a model without a privileged basis. Without superposition, one could try to do something like the Gram–Schmidt process, progressively identifying interpretable directions and then removing them to make future features easier to identify. But with superposition, one can't simply remove a direction even if one knows that it is a feature direction.\nThe connection also goes the other way: if one has the ability to enumerate over features, one can perform compressed sensing using the feature directions to (with high probability) \"unfold\" a superposition model's activations into those of a larger, non-superposition model.\nFor this reason, we'll call any method that gives us the ability to enumerate over features – and equivalently, unfold activations – a \"solution to superposition\".\nAny solution is on the table, from creating models that just don't have superposition, to identifying what directions correspond to features after the fact. We'll discuss the space of possibilities shortly.\nWe've motivated \"solving superposition\" in terms of feature enumeration, but it's worth noting that it's equivalent to (or necessary for) many other interpretability properties one might care about:\nDecomposing Activation Space.\nThe most fundamental challenge of any interpretability agenda is to defeat the curse of dimensionality. For mechanistic interpretability, this ultimately reduces to\nwhether we can decompose activation space\ninto independently understandable components, analogous to how computer program memory can be decomposed into variables. Identifying features is what allows us to decompose the model in terms of them.\nDescribing Activations in Terms of Pure Features.\nOne of the most obvious casualties of superposition is that we can't describe activations in terms of pure features. When features are relatively basis aligned, we can take an activation – say the activations for a dog head in a vision model – and decompose them into individual underlying features, like a floppy ear, short golden fur, and a snout. (See the \"semantic dictionary\" interface in\nBuilding Blocks\n.) Solving superposition would allow us to do this for every model.\nUnderstanding Weights (ie. Circuit Analysis).\nNeural network weights can typically only be understood when they're connecting together understandable features. All the circuit analysis seen in the original circuit thread (see especially\n), was fundamentally only possible because the weights connected non-polysemantic neurons. We need to solve superposition for this to work in general.\nEven very basic approaches become perilous with superposition.\nIt isn't just sophisticated approaches to interpretability which are harmed by superposition. Even very basic methods one might consider become unreliable. For example, if one is concerned about language models exhibiting manipulative behavior, one might ask if an input has a significant cosine similarity to the representations of other examples of deceptive behavior. Unfortunately, superposition means that cosine similarity has the potential to be misleading, since unrelated features start to be embedded with positive dot products to each other. However, if we solve superposition, this won't be an issue – either we'll have a model where features align with neurons, or a way to use compressed sensing to lift features to a space where they no longer have positive dot products.\nThree Ways Out\nAt a very high level, there seem to be three potential approaches to resolving superposition:\nCreate models without superposition.\nFind an overcomplete basis\nthat describes how features are represented in models with superposition.\nHybrid approaches\nin which one changes models, not resolving superposition, but making it easier for a second stage of analysis to find an overcomplete basis that describes it.\nOur sense is that all of these approaches are possible if one doesn't care about having a competitive model. For example, we believe it's possible to accomplish any of these for the toy models described in this paper. However, as one starts to consider serious neural networks, let alone modern large language models, all of these approaches begin to look very difficult. We'll outline the challenges we see for each approach in the following sections.\nWith that said, it's worth highlighting one bright spot before we focus on the challenges. You might have believed that superposition was something you could never fully get rid of, but that doesn't seem to be the case. All our results seem to suggest that superposition and polysemanticity are phases with sharp transitions. That is, there may exist a regime for every model where it has no superposition or polysemanticity. The question is largely whether the cost of getting rid of or otherwise resolving superposition is too high.\nApproach 1: Creating Models Without Superposition\nIt's actually quite easy to get rid of superposition in the toy models described in this paper, albeit at the cost of a higher loss. Simply apply an L1 regularization term to the hidden layer activations (i.e. add\n\\lambda ||h||_1\nto the loss). This actually has a nice interpretation in terms of killing features below a certain importance threshold, especially if they're not basis aligned. Generalizing this to real neural networks isn't trivial, but we expect it can be done. (This approach would be similar to work attempting to use sparsity to encourage basis-aligned word embeddings\n.)\nHowever, it seems likely that models are significantly benefitting from superposition. Roughly, the sparser features are, the more features can be squeezed in per neuron. And many features in language models seem very sparse! For example, language models know about individuals with only modest public presences, such as several of the authors of this paper. Presumably we only occur with frequency significantly less than one in a million tokens. As a result, it may be the case that superposition effectively makes models much bigger.\nAll of this paints a picture where getting rid of superposition may be fairly achievable, but doing so will have a large performance cost. For a model with a fixed number of neurons, superposition helps – potentially a lot.\nBut this is only true if the constraint is thought of in terms of neurons. That is, a superposition model with\nn\nneurons likely has the same performance as a significantly larger monosemantic model with\nkn\nneurons. But neurons aren't the fundamental constraint: flops are. In the most common model architectures, flops and neurons have a strict correspondence, but this doesn't have to be the case and it's much less clear that superposition is optimal in the broader space of possibilities.\nOne family of models which change the flop-neuron relationship are Mixture of Experts (MoE) models (\nsee review\n). The intuition is that most neurons are for specialized circumstances and don't need to activate most of the time. For example, German-specific neurons don't need to activate on French text. Harry Potter neurons don't need to activate on scientific papers. So MoE models organize neurons into blocks or experts, which only activate a small fraction of the time. This effectively allows the model to have\nk\ntimes more neurons for a similar flop budget, given the constraint that only\n1/k\nof the neurons activate in a given example and that they must activate in a block. Put another way, MoE models can recover neuron sparsity as free flops, as long as the sparsity is organized in certain ways.\nIt's unclear how far this can be pushed, especially given difficult engineering constraints. But there's an obvious lower bound, which is likely too optimistic but is interesting to think about: what if models only expended flops on neuron activations, and recovered the compute of all non-activating neurons? In this world, it seems unlikely that superposition would be optimal: you could always split a polysemantic neuron into dedicated neurons for each feature with the same cost, except for the cases where there would have been interference that hurt the model anyways. Our preliminary investigations comparing various types of superposition in terms of \"loss reduction per activation frequency\" seem to suggest that superposition is not optimal on these terms, although it asymptotically becomes as good as dedicated feature dimensions. Another way to think of this is that superposition exploits a gap between the sparsity of neurons and the sparsity of the underlying features; MoE eats that same gap, and so we should expect MoE models to have less superposition.\nTo be clear, MoE models are already well studied, and we don't think this changes the capabilities case for them. (If anything, superposition offers a theory for why MoE models have not proven more effective for capabilities when the case for them seems so initially compelling!) But if one's goal is to create competitive models that don't have superposition, MoE models become interesting to think about. We don't necessarily think that they specifically are the right path forward – our goal here has been to use them as an example of why we think it remains plausible there may be ways to build competitive superposition-free models.\nApproach 2: Finding an Overcomplete Basis\nThe opposite strategy of creating a superposition-free model is to take a regular model, which has superposition, and find an overcomplete basis describing how features are embedded after the fact. This appears to be a relatively standard\nsparse coding\n(or\ndictionary learning\n) problem, where we want to take the activations of neural network layers and find out which directions correspond to features.\nMore formally, given a matrix\nH \\sim [d,m] ~=~[h_0, h_1, …]\nof hidden layer activations\nh \\sim [m]\nsampled over\nd\nstimuli, if we believe there are\nn\nunderlying features, we can try to find matrices\nA\\sim [d,n]\nand\nB \\sim [n,m]\nsuch that\nA\nis sparse.\nThis approach has been explored by some prior work\n.\nThe advantage of this is that we don't need to worry about whether we're damaging model performance. On the other hand, many other things are harder:\nIt’s no longer easy to know how many features you have to enumerate.\nA monosemantic model represents a feature per neuron, but when finding an overcomplete basis there’s an additional challenge of identifying how many features to use for it.\nSolutions are no longer integrated into the surface computational structure.\nNeural networks can be understood in terms of their surface structure – neurons, attention heads, etc – and virtual structure that implicitly emerge (\ne.g.\nvirtual attention heads\n). A model described by an overcomplete basis has \"virtual neurons\": there's a further gap between the surface and virtual structure.\nIt's a different, major engineering challenge.\nSeriously attempting to solve superposition by applying sparse coding to real neural nets suggests a\nmassive\nsparse coding problem. For truly large language models, one would be starting with something like a millions (neurons) by billions (tokens) matrix and then trying to do an extremely overcomplete factorization, perhaps trying to factor it to be a thousand or more times\nlarger\n. This is a major engineering challenge which is different from the standard distributed training challenges ML labs are set up for.\nInterference is no longer pushing in your favor.\nIf you try to train models without superposition, interference between features is pushing the training process to have less superposition. If you instead try to decode superposition after the fact, whatever amount of superposition is \"baked in\" by the training process and you don't have part of the objective pushing in your favor.\nApproach 3: Hybrid Approaches\nIn addition to approaches which address superposition purely at training time, or purely after the fact, it may be possible to take \"hybrid approaches\" which do a mixture. For example, even if one can't change models without superposition, it may be possible to produce models with\nless\nsuperposition, which are then easier to decode.\nIn particular, it seems like we should expect to be able to reduce superposition at least a little bit with essentially no effect on performance, just by doing something like L1 regularization without any architectural changes.  Note that models should have a level of superposition where the derivative of loss with respect to the amount of superposition is zero – otherwise, they'd use more or less superposition. As a result, there should be at least some margin within which we can reduce the amount of superposition without affecting model performance.\nAlternatively, it may be possible for architecture changes to make finding an overcomplete basis easier or more computationally tractable in large models, separately from trying to reduce superposition.\nAdditional Considerations\nPhase Changes as Cause For Hope.\nIs totally getting rid of superposition a realistic hope? One could easily imagine a world where it can only be asymptotically reduced, and never fully eliminated. While the results in this paper seem to suggest that superposition is hard to get rid of because it's actually very useful, the upshot of it corresponding to a phase change is that there's a regime\nwhere it totally doesn't exist\n. If we can find a way to push models in the non-superposition regime, it seems likely it can be totally eliminated.\nAny superposition-free model would be a powerful tool for research.\nWe believe that most of the research risk is in whether one can make\nperformant\nsuperposition free models, rather than whether it's possible to make superposition free models at all. Of course, ultimately, we need to make performant models. But a non-performant superposition free model could still be a very useful research tool for studying superposition in normal models. At present, it's challenging to study superposition in models because we have no ground truth for what the features are. (This is also the reason why the toy models described in this paper can be studied – we do know what the features are!) If we had a superposition-free model, we may be able to use it as a ground truth to study superposition in regular models.\nLocal bases are not enough.\nEarlier, when we considered\nthe geometry of non-uniform superposition\n, we observed that models often form\nlocal orthogonal bases\n, where co-occurring features are orthogonal. This suggests a strategy for locally understanding models on sufficiently narrow sub-distributions. However, if our goal is to eventually make useful statements about the safety of models, we need mechanistic accounts that hold for the full distribution (and off distribution). Local bases seem unlikely to give this to us.\nDiscussion\nTo What Extent Does Superposition Exist in Real Models?\nWhy are we interested in toy models? We believe they are useful proxies for studying the superposition we suspect might exist in real neural networks. But how can we know if they're actually a useful toy model? Our best validation is whether their predictions are consistent with empirical observations regarding polysemanticity. To the best of our knowledge they are. In particular:\nPolysemantic neurons exist.\nPolysemantic neurons form in our third model, just as they are observed in a wide range of neural networks.\nNeurons are sometimes \"cleanly interpretable\" and sometimes \"polysemantic\", often in the same layer.\nOur third model exhibits both polysemantic and non-polysemantic neurons, often at the same time. This is analogous to how real neural networks often have a mixture of polysemantic and non-polysemantic neurons in the same layer.\nInceptionV1 has more polysemantic neurons in later layers.\nEmpirically, the fraction of neurons which are polysemantic in InceptionV1 increases with depth. One natural explanation is that as features become higher-level the stimuli they detect become rarer and thus sparser (for example, in vision, a high-level floppy ear feature is less common than a low-level Gabor filter's edge). A major prediction of our model is that superposition and polysemanticity increase as sparsity increases.\nEarly Transformer MLP neurons are extremely polysemantic.\nOur experience is that neurons in the first MLP layer in Transformer language models are often extremely polysemantic. If the goal of the first MLP layer is to distinguish between different interpretations of the same token (eg. \"die\" in English vs German vs Dutch vs Afrikaans), such features would be very sparse and our toy model would predict lots of polysemanticity.\nThis doesn't mean that everything about our toy model reflects real neural networks. Our intuition is that some of the phenomena we observe (superposition, monosemantic vs polysemantic neurons, perhaps the relationship to adversarial examples) are likely to generalize, while other phenomena (especially the geometry and learning dynamics results) are much more uncertain.\nOpen Questions\nThis paper has shown that the superposition hypothesis is true in certain toy models. But if anything, we're left with many more questions about it than we had at the start. In this final section, we review some of the questions which strike us as most important: what do we know, and would we like for future work to clarify?\nIs there a statistical test for catching superposition?\nHow can we control whether superposition and polysemanticity occur?\nPut another way, can we change the phase diagram such that features don't fall into the superposition regime? Pragmatically, this seems like the most important question. L1 regularization of activations, adversarial training, and changing the activation function all seem promising.\nAre there any models of superposition which have a closed-form solution?\nSaxe et al.\ndemonstrate that it's possible to create nice closed-form solutions for linear neural networks. We made some progress towards this for the\nn=2; m=1\nReLU output model (and Tom McGrath makes further progress in his comment), but it would be nice to solve this more generally.\nHow realistic are these toy models?\nTo what extent do they capture the important properties of real models with respect to superposition? How can we tell?\nCan we estimate the feature importance curve or feature sparsity curve of real models?\nIf one takes our toy models seriously, the most important properties for understanding the problem are the feature importance and sparsity curves. Is there a way we can estimate them for real models? (Likely, this would involve training models of varying sizes or amounts of regularization, observing the loss and neuron sparsities, and trying to infer something.)\nShould we expect superposition to go away if we just scale enough?\nWhat assumptions about the feature importance curve and sparsity would need to be true for that to be the case? Alternatively, should we expect superposition to remain a constant fraction of represented features, or even to increase as we scale?\nAre we measuring the maximally principled things?\nFor example, what is the most principled definition of superposition / polysemanticity?\nHow important are polysemantic neurons?\nIf X% of the model is interpretable neurons and 1-X% are polysemantic, how much should we believe we understand from understanding the X% interpretable neurons? (See also the \"feature packing principle\" suggested above.)\nHow many features should we expect to be stored in superposition?\nThis was briefly discussed in the previous section. It seems like results from compressed sensing should be able to give us useful upper-bounds, but it would be nice to have a clearer understanding – and perhaps tighter bounds!\nDoes the apparent phase change we observe in features/neurons have any connection to phase changes in\ncompressed sensing\n?\nHow does superposition relate to non-robust features?\nAn interesting\npaper by Gabriel Goh\n(\narchive.org backup\n) explores features in a linear model in terms of the principal components of the data. It focuses on a trade off between \"usefulness\" and \"robustness\" in the principal component features, but it seems like one could also relate it to the interpretability of features. How much would this perspective change if one believed the superposition hypothesis – could it be that the useful, non-robust features are an artifact of superposition?\nTo what extent c\nan neural networks \"do useful computation\" on features in superpositio\nn?\nIs the absolute value problem representative of computation in superposition generally, or idiosyncratic? What class of computation is amenable to being performed in superposition? Does it require a sparse structure to the computation?\nHow does superposition change if features are not independent?\nCan superposition pack features more efficiently if they are anti-correlated?\nCan models effectively use nonlinear representations?\nWe suspect models will tend not to use them, but further experimentation could provide good evidence. See the appendix on nonlinear compression. For example investigating the representations used by autoencoders with multi-layer encoders and decoders with really small bottlenecks on random uncorrelated data.\nRelated Work\nInterpretable Features\nOur work is inspired by research exploring the features that naturally occur in neural networks. Many models form at least some interpretable features. Word embeddings have semantic directions (\nsee\n). There is evidence of interpretable neurons in RNNs (\ne.g.\n), convolutional neural networks (\nsee generally e.g.\n;\nindividual neuron families\n), and in some limited cases, transformer language models (\nsee detailed discussion in our previous paper\n). However this work has also found many \"polysemantic\" neurons which are\nnot\ninterpretable as a single concept\n.\nSuperposition\nThe earliest reference to superposition in artificial neural networks that we're aware of is Arora\net al.\n's work\n, who suggest that the word embeddings of words with multiple different word senses may be superpositions of the vectors for the distinct meanings. Arora extend this idea to there being many sparse \"atoms of discourse\" in superposition, an idea which was generalized to other kinds of embedding vectors and explored in more detail by Goh\n.\nIn parallel with this, investigations of individual neurons in models with privileged bases were beginning to grapple with \"polysemantic\" neurons which respond to unrelated inputs\n. A natural hypothesis was that these polysemantic neurons are disambiguated by the combined activation of other neurons. This line of thinking eventually became the \"superposition hypothesis\" for circuits\n.\nSeparate from all of this, Cheung\net al.\nexplore a slightly different idea one might describe as \"model level\" superposition: can neural network parameters represent multiple completely independent models? Their investigation is motivated by catastrophic forgetting, but seems quite related to the questions investigated in this paper. Model level superposition can be seen as feature level superposition for highly correlated sets of features, similar to the\n\"almost orthogonal bases\" experiment\nwe considered above.\nDisentanglement\nThe goal of learning\ndisentangled representations\narises from Bengio\net al.\n's influential position paper on representation learning\n: \"we would like our representations to\ndisentangle the factors of variation\n…  to learn representations that separate the various explanatory sources.\" Since then, a literature has developed motivated by this goal, tending to focus on creating generative models which separate out major factors of variation in their latent spaces. This research touches on questions related to superposition, but is also quite different in a number of ways.\nConcretely, disentanglement research often explores whether one can train a VAE or GAN where basis dimensions correspond to the major features one might use to describe the problem (e.g. rotation, lighting, gender… as relevant). Early work often focused on semi-supervised approaches where the features were known in advance, but fully unsupervised approaches started to develop around 2016\n.\nPut another way, the goal of disentanglement might be described as\nimposing\na strong privileged basis on representations which are rotationally invariant by default. This helps get at ways in which the questions of polysemanticity and superposition are a bit different from disentanglement. Consider that when we deal with neurons, rather than embeddings, we have a privileged basis\nby default\n. It varies by model, but many neurons just cleanly respond to features. This means that polysemanticity arises as a kind of anomalous behavior, and superposition arises as a hypothesis for explaining it. The question then isn't how to impose a privileged basis, but how to remove superposition as a fundamental problem to accessing features.\nOf course, if the superposition hypothesis is true, there are still a number of connections to disentanglement. On the one hand, it seems likely superposition occurs in the latent spaces of generative models, even though that isn't an area we've investigated. If so, it may be that superposition is a major reason why disentanglement is difficult. Superposition may allow generative models to be much more effective than they would otherwise be without. Put another way, disentanglement often assumes a small number of important latent variables to explain the data. There are clearly examples of such variables, like the orientation of objects – but what if a large number of sparse, rare, individually unimportant features are collectively very important? Superposition would be the natural way for models to represent this.\nA more subtle issue is that GANs and VAEs often assume that their latent space is Gaussianly distributed. Sparse latent variables are very non-Gaussian, but the central limit theorem means that the superposition of many such variables will gradually look more Gaussian. So the latent spaces of some generative models may in fact force models to use superposition!\nOn the other hand, one could imagine ideas from the disentanglement literature being useful in creating architectures that resist superposition by creating an even more strongly privileged basis.\nCompressed Sensing\nThe toy problems we consider are quite similar to the problems considered in the field of\ncompressed sensing\n, which is also known as compressive sensing and sparse recovery. However, there are some important differences:\nCompressed sensing recovers vectors by solving an optimization problem using general techniques, while our toy model must use a neural network layer. Compressed sensing algorithms are, in principle, much more powerful than our toy model.\nCompressed sensing works using the number of non-zero entries as the measure of sparsity, while we use the probability that each dimension is zero as the sparsity. These are not wholly unrelated: concentration of measure implies that our vectors have a bounded number of non-zero entries with high probability.\nCompressed sensing requires that the embedding matrix (usually called the measurement matrix) have a certain “incoherent” structure\nsuch as the restricted isometry property\nor nullspace property\n. Our toy model learns the embedding matrix, and will often simply ignore many input dimensions to make others easier to recover.\nFeatures in our toy model have different \"importances\", which means the model will often prefer to be able to recover “important” features more accurately, at the cost of not being able to recover “less important” features at all.\nIn general, our toy model is solving a similar problem using\nless powerful\nmethods than compressed sensing algorithms, especially because the computational model is so much more restricted (to just a single linear transformation and a non-linearity) compared to the arbitrary computation that might be used by a compressed sensing algorithm.\nAs a result, compressed sensing lower bounds—which give lower bounds on the dimension of the embedding such that recovery is still possible—can be interpreted as giving an upper bound on the amount of superposition in our toy model. In particular, in various compressed sensing settings, one can recover an\nn\n-dimensional\nk\n-sparse vector from an\nm\ndimensional projection if and only if\nm = \\Omega(k \\log (n/k))\n. While the connection is not entirely straightforward, we apply one such result to the toy model\nin the appendix\n.\nAt first, this bound appears to allow a number of features that is exponential in\nm\nto be packed into the\nm\n-dimensional embedding space. However, in our setting, the integer\nk\nfor which all vectors have at most\nk\nnon-zero entries is determined by the fixed density parameter\nS\nas\nk = O((1 - S)n)\n. As a result, our bound is actually\nm = \\Omega(-n (1 - S) \\log(1 - S))\n. Therefore, the number of features is linear in\nm\nbut modulated by the sparsity.\nNote that this has a nice information-theoretic interpretation:\n\\log(1 - S)\nis the surprisal of a given dimension being non-zero, and is multiplied by the expected number of non-zeros.\nThis is good news if we are hoping to eliminate superposition as a phenomenon! However, these bounds also allow for the amount of superposition to increase dramatically with sparsity – hopefully this is an artifact of the techniques in the proofs and not an inherent barrier to reducing or eliminating superposition.\nA striking parallel between our toy model and compressed sensing is the existence of\nphase changes\n.\nNote that in the compressed sensing case, the phase transition is in the limit as the number of dimensions becomes large – for finite-dimensional spaces, the transition is fast but not discontinuous.\nIn compressed sensing, if one considers a two-dimensional space defined by the sparsity and dimensionality of the vectors, there are sharp phase changes where the vector can almost surely be recovered in one regime and almost surely not in the other\n. It isn't immediately obvious how to connect these phase changes in compressed sensing – which apply to recovery of the entire vector, rather than one particular component – to the phase changes we observe in features and neurons. But the parallel is suspicious.\nAnother interesting line of work has tried to build useful sparse recovery algorithms using neural networks\n. While we find it useful for analysis purposes to view the toy model as a sparse recovery algorithm, so that we may apply sparse recovery lower bounds, we do not expect that the toy model is useful for the problem of sparse recovery. However, there may be an exciting opportunity to relate our understanding of the phenomenon of superposition to these and other techniques.\nSparse Coding and Dictionary Learning\nSparse Coding\nstudies the problem of finding a sparse representation of dense data. One can think of it as being like compressed sensing, except the matrix projecting sparse vectors into the lower dimensional space is also unknown. This topic goes by many different names including sparse coding (most common in neuroscience), dictionary learning (in computer science), and sparse frame design (in mathematics). For a general introduction, we refer readers to a textbook by Michael Elad\n.\nClassic sparse coding algorithms take an expectation-maximization approach (this includes Olshausen\net al\n's early work\n, the MOD algorithm\n, and the k-SVD algorithm\n). More recently, new methods based on gradient descent and autoencoders have begun building on these ideas\n.\nFrom our perspective, sparse coding is interesting because it's probably the most natural mathematical formulation of trying to \"solve superposition\" by discovering which directions correspond to features.\nInterestingly, this is the reverse of how sparse coding is typically thought of in neuroscience. Neuroscience often thinks of biological neurons as sparse coding their inputs, whereas we're interested in applying it the opposite direction, to find features in superposition over neurons.\nBut can we actually use these methods to solve superposition in practice? Previous work has attempted to use sparse coding to find sparse structure\n. More recently,\nresearch\nby Sharkey\net al\nfollowing up on the original publication of this paper has had preliminary success in extracting features out of superposition in toy models using a sparse autoencoder. In general, we're only in the very preliminary investigations of using sparse coding and dictionary learning in this way, but the situation seems quite optimistic. See the section\nApproach 2: Finding an Overcomplete Basis\nfor more discussion.\nTheories of Neural Coding and Representation\nOur work explores representations in artificial “neurons”. Neuroscientists study similar questions in biological neurons. There are a variety of theories for how information could be encoded by a group of neurons. At one extreme is a\nlocal code\n, in which every individual stimulus is represented by a separate neuron. At the other extreme is a\nmaximally-dense distributed code\n, in which the information-theoretic capacity of the population is fully utilized, and every neuron in the population plays a necessary role in representing every input.\nOne challenge in comparing our work with the neuroscience literature is that a “distributed representation” seems to mean different things. Consider an overly-simplified example of a population of neurons, each taking a binary value of active or inactive, and a stimulus set of sixteen items: four shapes, with four colors  (example borrowed from\n). A “local code” would be one with a “red triangle” neuron, a “red square” neuron, and so on. In what sense could the representation be made more “distributed”? One sense is by representing\nindependent features\nseparately — e.g. four “shape” neurons and four “color” neurons. A second sense is by representing\nmore items than neurons\n— i.e. using a binary code over four neurons to encode 2^4 = 16 stimuli. In our framework, these senses correspond to\ndecomposability\n(representing stimuli as compositions of independent features) and\nsuperposition\n(representing more features than neurons, at cost of interference if features co-occur).\nDecomposability doesn’t necessarily mean each feature gets its own neuron. Instead, it could be that each feature corresponds to a “direction in activation-space”\nWe haven’t encountered a specific term in the distributed coding literature that corresponds to this hypothesis specifically, although the idea of a “direction in activation-space” is common in the literature, which may be due to ignorance on our part. We call this hypothesis\nlinearity.\n, given scalar “activations” (which in biological neurons would be firing rate). Then, only if there is a\nprivileged basis\n, “feature neurons” are incentivized to develop. In biological neurons, metabolic considerations are often hypothesized to induce a privileged basis, and thus a “sparse code”. This would be expected if the nervous system’s energy expenditure increases linearly or sublinearly with firing rate.\nExperimental evidence seems to support this\nAdditionally, neurons are the units by which biological neural networks can implement non-linear transformations, so if a feature needs to be non-linearly transformed, a “feature neuron” is a good way to achieve that.\nAny decomposable linear code that uses orthogonal feature vectors is functionally equivalent from the viewpoint of a linear readout. So, a code can both be “maximally distributed” — in the sense that every neuron participates in representing every input, making each neuron extremely polysemantic — and also have no more features than it has dimensions. In this conception, it’s clear that a code can be fully “distributed” and also have no\nsuperposition\n.\nA notable difference between our work, and the neuroscience literature we have encountered, is that we consider as a central concept the likelihood that features co-occur with some probability.\nA related, but different, concept in the neuroscience literature is the “binding problem”\nin which e.g. a red triangle is a co-occurrence of exactly one shape and exactly one color, which is not a representational challenge, but a binding problem arises if a decomposed code needs to represent simultaneously also a blue square — which shape feature goes with which color feature? Our work does not engage with the binding question, merely treating this as a co-occurrence of “blue”, “red”, “triangle”, and “square”.\nA “maximally-dense distributed code” makes the most sense in the case where items never co-occur; if the network only needs to represent one item at a time, it can tolerate a very extreme degree of superposition. By contrast, a network that could plausibly need to represent all the items at once can do so without interference between the items if it uses a code with no superposition. One example of high feature co-occurrence could be encoding spatial frequency in a receptive field; these visual neurons need to be able to represent white noise, which has energy at all frequencies. An example of limited co-occurrence could be a motor “reach” task to discrete targets, far enough apart that only one can be reached at a time.\nOne hypothesis in neuroscience is that highly compressed representations might have an important use in long-range communication between brain areas\n. Under this theory, sparse representations are used within a brain area to do computation, and then are compressed for transmission across a small number of axons. Our experiments with\nthe absolute value toy model\nshows that networks can do useful computation even under a code with a moderate degree of superposition. This suggests that all neural codes, not just those used for efficient communication, could plausibly be “compressed” to some degree; the regional code might not necessarily need to be decompressed to a fully sparse one.\nIt's worth noting that the term \"distributed representation\" is also used in deep learning, and has the same ambiguities of meaning there. Our sense is that some influential early works (\ne.g.\n) may have primarily meant the \"independent features are represented independently\"\ndecomposability\nsense, but we believe that other work intends to suggest something similar to what we call superposition.\nAdditional Connections\nAfter publishing the original version of this paper, a number of readers generously brought to our attention additional connections to prior work. We don't have a sufficiently deep understanding of this work to offer a detailed review, but we offer a brief overview below:\nVector Symbolic Architectures and Hyperdimensional Computing (\nsee reviews\n) are models from theoretical neuroscience of how neural systems can manipulate symbols. Many of the core ideas of how quasi-orthogonal vectors and the \"blessings of dimensionality\" enable computation are closely related to our notions of superposition.\nFrames (\nsee review\n) are a generalization of the idea of a mathematical basis. The way superposition encodes features in lower dimensional spaces might be seen as frames, at least in some cases. In particular, the \"Mercedes-Benz Frame\" is equivalent to the triangular geometry superposition we sometimes observe.\nAlthough we discuss compressed sensing and sparse coding above, it's worth noting that this only scratches the surface of research on how sparse vectors can be encoded in lower dimensional dense vectors, and there's a large body of additional work not captured by these topics.\nComments & Replications\nInspired by the original\nCircuits Thread\nand\nDistill's Discussion Article experiment\n, the authors invited several external researchers who we had previously discussed our preliminary results with to comment on this work. Their comments are included below.\nReplication & Forthcoming Paper\nKshitij Sachan\nis a research intern at\nRedwood Research\n.\nRedwood Research has been working on toy models of polysemanticity, inspired by Anthropic's work. We plan to separately publish our results, and during our research we replicated many of the experiments in this paper. Specifically, we replicated all plots in the\nDemonstrating Superposition\nand\nSuperposition as a Phase Change\nsections (visualizations of the relu models with different sparsities and the phase diagrams) as well as the plot in\nThe Geometry of Superposition – Uniform Superposition\n. We found the phase diagrams look quite different depending on the activation function, suggesting that in this toy model some activation functions induce more polysemanticity than others.\nOriginal Authors' Response:\nRedwood's further analysis of the superposition phase change significantly advanced our own understanding of the issue – we're very excited for their analysis to be shared with the world. We also appreciate the independent replication of our basic results.\nUpdate:\nThe research by Redwood mentioned in the previous comment,\nPolysemanticity and Capacity in Neural Networks\n(\nAlignment Forum\n,\nArxiv\n) is out! They study a slightly different toy model, and get some really interesting results. Highlights include analytical traction on understanding a variant of the toy model, understanding superposition in terms of constrained optimization, and analysis of the role different activation functions play.\nReplication & Further Results\nTom McGrath\nis a research scientist at\nDeepMind\n.\nThe results in this paper are an important contribution – they really further our theoretical understanding of a phenomenon that may be central to interpretability research and understanding network representations more generally. It’s surprising that such simple settings can produce these rich phenomena. We’ve reproduced the experiments in the\nDemonstrating Superposition\nand\nSuperposition as a Phase Change\nsections and have a minor additional result to contribute.\nIt is possible to exactly solve the expected loss for the\nn=2\n,\nm=1\ncase of the basic\nReLU output toy model\n(ignoring bias terms). The derivation is mathematically simple but somewhat long-winded: the ‘tricks’ are to (1) represent the sparse portion of the input distribution with delta functions, and (2) replace the ReLu with a restriction of the domain of integration:\n\\int_D \\text{ReLU}(f(x)) dx = \\int_{D \\cap f(x)>0} f(x) dx\nMaking this substitution renders the integral analytically tractable, which allows us to plot the full loss surface and solve for the loss minima directly. We show some example loss surfaces below:\nAlthough many of these loss surfaces (Figure 1a, 1b) have minima qualitatively similar to one of the network weights used in the section\nSuperposition as a Phase Change\n, we also find a new phase where\nW_1\\simeq W_2 \\simeq \\frac{1}{\\sqrt{2}}\n: weights are similar rather than antipodal. This ‘confused feature’ regime occurs when sparsity is low and both features are important (Figure 1c). (This is slightly similar to the behavior described in\nThe Geometry of Superposition – Collapsing of Correlated Features\n, but occurs without the features being correlated!) Further, although the solutions we find are often qualitatively similar to the weights used in\nSuperposition as a Phase Change\n, they can be quantitatively different, as Figure 1a shows. The transition from Figure 1a to Figure 1b is continuous: the minima moves smoothly in weight space as the degree of sparsity alters. This explains the ‘blurry’ region around the triple point in the phase diagram.\nAs Figure 1c shows, some combinations of sparsity and relative feature importance lead to loss surfaces with two minima (once the symmetry\n(W_1, W_2) \\to (-W_1, -W_2)\nhas been accounted for). If this pattern holds for larger values of\nn\nand\nm\n(and we see no reason why it would not) this could account for the\nDiscrete \"Energy Level\" Jumps phenomenon\nas solutions hop between minima. In some cases (e.g. when parameters approach those needed for a phase transition) the global minimum can have a considerably smaller basin of attraction than local minima. The transition between the antipodal and confused-feature solutions appears to be discontinuous.\nOriginal Authors' Response:\nThis closed form analysis of the\nn=2, m=1\ncase is fascinating. We hadn't realized that\nW_1\\simeq W_2 \\simeq \\frac{1}{\\sqrt{2}}\ncould be a solution without correlated features! The clarification of the \"blurry behavior\" and the observation about local minima are also very interesting. More generally, we're very grateful for the independent replication of our core results.\nReplication\nJeffrey Wu\nand\nDan Mossing\nare members of the Alignment team at\nOpenAI\n.\nWe are very excited about these toy models of polysemanticity. This work sits at a rare intersection of being plausibly very important for training more interpretable models and being very simple and elegant. The results have been surprisingly easy to replicate – we have reproduced (with very little fuss) plots similar to those in the\nDemonstrating Superposition – Basic Results\n,\nGeometry – Feature Dimensionality\n, and\nLearning Dynamics – Discrete \"Energy Level\" Jumps\nsections.\nOriginal Authors' Response:\nWe really appreciate this replication of our basic results. Some of our findings were quite surprising to us, and this gives us more confidence that they aren't the result of an idiosyncratic quirk or bug in our implementations.\nReplication\nSpencer Becker-Kahn\nis a senior research scholar at the Future of Humanity Institute and a SERI Machine Learning Alignment Theory Scholar.\nAfter seeing preliminary results, I independently replicated some of the key graphs from\nDemonstrating Superposition – Basic Results\nand, using very small toy models, produced a series of plots consistent with the conceptual picture emerging in\nGeometry – Feature Dimensionality\nand\nSuperposition and Learning Dynamics\n.\nSee also\nTwitter thread\n.\nEngineering Monosemanticity in Toy Models\nAdam Jermyn\nis an independent researcher focused on AI alignment and interpretability. He was previously a Research Fellow at the Flatiron Institute’s Center for Computational Astrophysics.\nEvan Hubinger\nis a Research Fellow at MIRI.\nNicholas Schiefer\nis a member of the technical staff at Anthropic and an author of the original paper.\nInspired by the results in this paper and the\nprevious paper\nintroducing the SoLU activation, we have been investigating whether changes to the model architecture or training process can reduce superposition in toy models. After replicating several of these results independently, we made the following extensions in that direction:\nA modified toy model that tries to more accurately represent the non-toy case where sparse features are projected down into a non-sparse input to the model with fewer dimensions.\nA “many neuron” architecture that gives the model enough capacity to avoid polysemanticity entirely.\nA training and initialization method that actually makes these toy models monosemantic.\nA systematic exploration of the polysemantic neurons that tend to appear in these models, which guided our explorations of the training methods.\nAt least in some limits, this suggests there may not be a price to be paid for monosemanticity. Detailed results can be found in our paper,\nEngineering Monosemanticity in Toy Models\n(\nAlignment Forum\n,\nArXiV\n).\nFractional Dimensionality and \"Pressure\"\nTom Henighan\nand\nChris Olah\nare authors of the original paper.\nIn the\n\"Feature Dimensionality\" section\n, we found that features organized into clean polytopes when there are more features than can be easily represented in the embedding dimensions.\nWe briefly investigated this further and found that the number of features competing to be represented significantly influences this phenomenon. Cleaner structure often seems to emerge when there is more \"pressure\" – more features competing to be represented. This is especially true at high sparsity levels. Additionally, training longer seems to also produce cleaner structure.\nMore investigation would be needed to really understand this phenomenon.\nReplication\nMarius Hobbhahn\nis a PhD student at the University of Tuebingen.\nI replicated the\n“Basic Results”\nin\nSection 2 (“Demonstrating Superposition”)\nand all of\nSection 7 (“Superposition in a Privileged Basis”)\nof the “Toy Models of Superposition” paper. All of my findings are identical to the ones described in the paper. I replicated most findings in the follow-up\n“Superposition, Memorization, and Double Descent”\npaper.\nThe details of my replication can be found in my write-up\n“More Findings on Memorization and Double Descent”\n.\nExtracting Features with Sparse Autoencoders\nLee Sharkey\n,\nDan Braun\n, and\nBeren Millidge\nare researchers at Conjecture.\nThe results from this paper, and the strategic picture it paints, inspired our\npreliminary follow-up work\nthat aimed to address some of the challenges described in the section titled '\nApproach 2: Finding an overcomplete basis\n'.\nBefore studying the activations of real neural networks, where we're not sure what the 'ground truth' features are, we studied a toy example. We generated a set of toy ground truth features and created a dataset using sparse combinations of them. We found that a one-layer sparse autoencoder with an\nL_1\npenalty on its hidden activations learned features that were almost identical to the ground truth features. This is encouraging because it indicates that a relatively simple method should be able to recover the features used by neural networks, inasmuch as they have them.\nFor the toy dataset, we knew how many ground truth features there were. But we ultimately want to count the features used by real neural networks, where the number of features is unknown. We explored three ways to count the features in the toy dataset: a) Counting dead neurons in the autoencoders; b) looking at autoencoder losses; and c) comparing the features learned by autoencoders of different sizes. We found indications that these methods might be suitable to count the number of features in superposition in real neural data.\nWe also applied our method to real activations from a small language model. Our initial, preliminary investigations led to inconclusive results, possibly resulting from having used autoencoders that were either too small or undertrained. At the time of writing, investigations are ongoing.\nLinear Representation in Othello\nNeel Nanda\nis an external researcher in mechanistic interpretability. This is a description of his blog post,\nActually, Othello-GPT Has A Linear Emergent World Representation\n.\nI describe a natural experiment of the\nlinear representation hypothesis\ndescribed in this paper – the idea that features correspond to directions in neural networks.\nBackground\n: Martin Wattenberg (an author on this paper) and colleagues\nrecently found\nthat, if you train transformer language model trained to predict the next token in synthetic Othello games (where each move is a randomly chosen legal move), it forms an emergent model of the board state (despite only being trained to predict the next move!). They showed that the state of the board (whether each cell was empty, black or white) could be recovered with high accuracy by a one hidden layer MLP probe. They further showed that you could use the world model to causally intervene on the model’s residual stream. By choosing another board state, and changing the residual stream (with gradient descent) such that the probe indicates that new board state, they caused the model to output legal moves in the new board state, even if the edited board state was impossible to reach via legal Othello play!\nPre-Registered Hypothesis\n: The probing and causal intervention together provided strong evidence that the model had learned to represent features corresponding to the state of each square on the board. Yet, notably, linear probes were\nnot\nable to recover the board state. Since linear features should be recoverable with a one-layer probe, and the causal intervention suggests the model both computes and uses the board state, this seemed like significant evidence against the linear representation hypothesis.\nHowever, Chris Olah (an author on this paper) argued the model might still be representing features linearly if it used a different set of features, and that the probe and causal intervention may be picking up on this different set of features. This created an informal pre-registered prediction of the hypothesis which was contrary to the evidence at the time.\nResults\n: I independently came to the same conclusion as Chris and investigated\nthe Othello playing model\n. I found that the model\ndoes\nform an emergent model of the board state that is linearly represented and can be extracted with a linear probe. But that as the model plays both black and white moves, the model represents the state of a cell as whether it has “my colour” vs “opponent’s colour”. Further, I found circumstantial evidence that these features are used by the model, as we can linearly intervene on the residual stream using the directions given by the probe to edit the represented board state, and the model plays legal moves in the new board state.\nI consider these results to be notable, as the paper’s results provided evidence against the linear representation hypothesis and the hypothesis faced genuine risk of falsification. And the hypothesis made non-trivial predictions that were contrary to where the evidence pointed, but these turned out to be true. This is both a proof of concept that there are underlying principles of neural networks which have predictive power about models, and a natural experiment supporting the linear representation hypothesis.\nI think that there is further work interpreting the Othello playing model that could test other hypotheses in this paper and our broader conceptual frameworks about neural networks and transformers, such as by looking for monosemantic vs superposed neurons in its MLP layers. The model is both complex enough to be interesting and expose principles of how transformers learn algorithms, yet the algorithmic nature of the task and existence of the probe suggests that finding circuits will be tractable. I elaborate on what I consider promising directions of future work in\na follow-up post\n.\nLeverage Score and Feature Dimensionality\nFred Zhang\nis a PhD student in the Theory Group of the EECS Department at UC Berkeley.\nIn the\nGeometry of Superposition\nsection, the paper defines a notion of feature dimensionality,\nD_i ~=~ \\frac{||W_i||^2}{\\sum_j (\\hat{W_i} \\cdot W_j)^2},\nwhere\nW_i\nis the\ni\nth feature and\n\\hat{W_i}=W_i / \\| W_i \\|_2\nis its normalized version. For simplicity of notation, we assume that there are\nn\nfeatures, each being a\nd\n-dimensional real vector.\nFollowing this definition, the paper makes the remark that \"\nEmpirically\n, it seems that the dimensionality of all features add up to the number of embedding dimensions when the features are 'packed efficiently' in some sense.\" In this comment, I point out a natural,\ntheoretical\nexplanation of this observation. The argument is via the notion of\nleverage score\nin matrix approximation. I’ll define it first, then explain how it connects to feature dimensionality.\nAt a conceptual level, leverage score is a measure of the importance of a row in composing the row space of a matrix. For instance, if a row is orthogonal to all other rows, its leverage score is 1, meaning that it’s maximally important. This is natural, since removing it would decrease the rank of the matrix and completely change the row space. Formally, if\nW\nbe an\nn\nby\nd\nmatrix (think of it as tall and thin, so\nn > d\n), then the leverage score of\ni\nth row\nW_i\nis\n\\tau_i = \\max_{x : \\|x\\|_2 = 1} \\frac{\\langle W_i , x \\rangle^2}{\\sum_{j=1}^n \\langle W_j , x \\rangle^2}.\nNotice that the denominator term equals\n\\| Wx \\|_2^2\n. Hence, the leverage score of\nW_i\nis the maximum contribution\n\\langle W_i, x\\rangle^2\ncan make towards\n|| Wx ||^2\n(over all possible directions\nx\n). This is a fractional number between 0 and 1. Also observe that this indeed has the nice property that if\nW_i\nis orthogonal to all other rows,\n\\tau_i\nequals 1, because all except the\ni\nth term within the summation of the denominator is 0.\nThree quick remarks on this definition:\nIt coincides with the notion of\nmaximal data dimensionality\nin a comment by Chris Olah on\nData Dimensionality of MNIST\nin the follow-up work of\nSuperposition, Memorization, and Double Descent\n.\nThere are other equivalent definitions of leverage score; see section 3.3 of\nhere\n.\nLeverage score arises in numerical linear algebra. It is well-known that sampling rows proportional to their leverage score yields an almost square matrix that approximates the original, potentially very tall matrix. For a formal statement, see Theorem 17 of\nhttps://arxiv.org/abs/1411.4357\n.\nReturning to my main point, another nice fact about leverage score is they sum up to the rank of the matrix. In the tall and thin case above, they sum up to d (if the matrix is full-rank). Given that, it is natural this paper makes the empirical observation that the sum of\nD_i\nroughly equals the embedding dimension\nd\n, if the vectors are \"efficiently packed\". Specifically, one formal way of stating that they are packed efficiently is that they are roughly in isotropic position, that is, the variance\n\\langle W_j, x\\rangle^2\nalong any direction\nx\nis close to\n1\n. In other words, the covariance structure of the embedding vectors is close to identity. (Geometrically, it means they are well spread out. This would be the case, for example, if the vectors form a regular simplex, just as it happens in the experiments of this paper.) Then in the above definition of leverage score, each summand within the denominator is 1 for any\nx\n. So to maximize the numerator, we just take\nx\nto be\nW_i\nitself—and this precisely recovers the definition of feature dimensionality\nD_i\n. Thus, in this well-spread-out case, these two notions are roughly the same, and they both sum up to\nd\n.\nCode\nWe provide a notebook to reproduce some of the core diagrams in this article\nhere\n. (It isn't comprehensive, since we needed to rewrite code for our experiments to run outside our codebase.) We provide a\nseparate notebook\nfor the theoretical phase change diagrams.\nNote that the reproductions by other researchers mentioned in comments above were not based on this code, but are instead fully independent replications with clean code from the description in an early draft of this article.\nAcknowledgments\nWe're extremely grateful to a number of colleagues across several organizations for their invaluable support in our writing of this paper.\nJeff Wu, Daniel Mossing, Tom McGrath, and Kshitij Sachan did independent replications of many of our experiments, greatly increasing our confidence in our results. Kshitij Sachan's and Tom McGrath's additional investigations and insightful questions both pushed us to clarify our understanding of the superposition phase change (both as reflected in this paper, and in further understanding which we learned from them not captured here). Buck Shlegeris, Adam Scherlis, and Adam Jermyn shared valuable insights into the mathematical nature of the toy problem and related work.  Adam Jermyn also coined the term \"virtual neurons.\"\nGabriel Goh, Neel Nanda, Vladimir Mikulik, and Nick Cammarata gave detailed feedback which improved the paper, in addition to being motivating. Alex Dimakis, Piotr Indyk, Dan Yamins generously took time to discuss these results with us and give advice on how they might connect to their area of expertise. Finally, we benefited from the feedback and comments of James Bradbury, Sebastian Farquhar, Shan Carter, Patrick Mineault, Alex Tamkin, Paul Christiano, Evan Hubinger, Ian McKenzie, and Sid Black. We're additionally grateful to Trenton Bricken and Manjari Narayan for referring us to valuable related work we originally missed. Thanks to Ken Kahn for typo corrections.\nFinally, we're very grateful to all our colleagues at Anthropic for their advice and support: Daniela Amodei, Jack Clark, Tom Brown, Ben Mann, Nick Joseph, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Timothy Telleen-Lawton, Anna Chen, Yuntao Bai, Jeffrey Ladish, Deep Ganguli, Liane Lovitt, Nova DasSarma, Jia Yuan Loke, Jackson Kernion, Tom Conerly, Scott Johnston, Jamie Kerr, Sheer El Showk, Stanislav Fort, Rebecca Raible, Saurav Kadavath, Rune Kvist, Jarrah Bloomfield, Eli Tran-Johnson, Rob Gilson, Guro Khundadze, Filipe Dobreira, Ethan Perez, Sam Bowman, Sam Ringer, Sebastian Conybeare, Jeeyoon Hyun, Michael Sellitto, Jared Mueller, Joshua Landau, Cameron McKinnon, Sandipan Kundu, Jasmine Brazilek, Da Yan, Robin Larson, Noemí Mercado, Anna Goldie, Azalia Mirhoseini, Jennifer Zhou, Erick Galankin, James Sully, Dustin Li, James Landis.\nAuthor Contributions\nBasic Results\n– The basic toy model results demonstrating the existence of superposition were done by Nelson Elhage and Chris Olah. Chris suggested the toy model and Nelson ran the experiments.\nPhase Change\n– Chris Olah ran the empirical phase change experiments, with help from Nelson Elhage. Martin Wattenberg introduced the theoretical model where exact losses for specific weight configurations can be computed.\nGeometry\n– The uniform superposition geometry results were discovered by Nelson Elhage and Nicholas Schiefer, with help from Chris Olah. Nelson discovered the original\nm/||W||_F^2\nmysterious \"stickiness\". Chris introduced the definition of feature dimensionality. Nicholas and Nelson then investigated the polytopes that formed. As for non-uniform superposition, Martin Wattenberg performed the initial investigations of the resulting geometry, focusing on the behavior of correlated features. Chris extended this with an investigation of the role of relative feature importance and sparsity.\nLearning Dynamics\n– Nelson Elhage discovered the \"energy level jump\" phenomenon, in collaboration with Nicholas Schiefer and Chris Olah. Martin Wattenberg discovered the \"geometric transformations\" phenomenon.\nAdversarial Examples\n– Chris Olah and Catherine Olsson found evidence of a connection between superposition and adversarial examples.\nSuperposition with a Privileged Basis / Doing Computation\n– Chris Olah did the basic investigation of superposition in a privileged basis. Nelson Elhage, with help from Chris, investigated the \"absolute value\" model which provided a more principled demonstration of superposition and showed that computation could be done while in superposition. Nelson discovered the \"asymmetric superposition\" motif.\nTheory\n– The theoretical picture articulated over the course of this paper (especially in the \"mathematical understanding\" section) was developed in conversations between all authors, but especially Chris Olah, Jared Kaplan, Martin Wattenberg, Nelson Elhage, Tristan Hume, Tom Henighan, Catherine Olsson, Nicholas Schiefer, Dawn Drain, Shauna Kravec, Roger Grosse, Robert Lasenby, and Sam McCandlish. Jared introduced the strategy of rewriting the loss by grouping terms with the number of active features. Both Jared and Martin independently noticed the value of investigating the\nn=2; m=1\ncase as the simplest case to understand. Nicholas and Dawn clarified our understanding of the connection to compressed sensing.\nStrategic Picture\n– The strategic picture articulated in this paper – What does superposition mean for interpretability and safety? What would a suitable solution be? How might one solve it? – developed in extensive conversations between authors, and in particular Chris Olah, Tristan Hume, Nelson Elhage, Dario Amodei, Jared Kaplan. Nelson Elhage recognized the potential importance of \"enumerative safety\", further articulated by Dario. Tristan brainstormed extensively about ways one might solve superposition and pushed Chris on this topic.\nWriting\n– The paper was primarily drafted by Chris Olah, with some sections by Nelson Elhage, Tristan Hume, Martin Wattenberg, and Catherine Olsson. All authors contributed to editing, with particularly significant contributions from Zac Hatfield Dodds, Robert Lasenby, Kipply Chen, and Roger Grosse.\nIllustration\n– The paper was primarily illustrated by Chris Olah, with help from Tristan Hume, Nelson Elhage, and Catherine Olsson.\nCitation Information\nPlease cite as:\nElhage, et al., \"Toy Models of Superposition\", Transformer Circuits Thread, 2022.\nBibTeX Citation:\n@article{elhage2022superposition,\n   title={Toy Models of Superposition},\n   author={Elhage, Nelson and Hume, Tristan and Olsson, Catherine and Schiefer, Nicholas and Henighan, Tom and Kravec, Shauna and Hatfield-Dodds, Zac and Lasenby, Robert and Drain, Dawn and Chen, Carol and Grosse, Roger and McCandlish, Sam and Kaplan, Jared and Amodei, Dario and Wattenberg, Martin and Olah, Christopher},\n   year={2022},\n   journal={Transformer Circuits Thread},\n   note={https://transformer-circuits.pub/2022/toy_model/index.html}\n}\nNonlinear Compression\nThis paper focuses on the assumption that representations are linear. But what if models don’t use linear feature directions to represent information? What might such a thing concretely look like?\nNeural networks have nonlinearities that make it theoretically possible to compress information even more compactly than a linear superposition. There are reasons we think models are unlikely to pervasively use nonlinear compression schemes:\nThe model needs to decompress things before it can compute with them naturally: Most of the computation in the model is linear, so this kind of compression is likely only worth it to save space in the residual stream across many layers before being decompressed to be computed with linearly again.\nThey’re probably difficult to learn: Nonlinear compression schemes may require finely tuned approximations of discontinuities, and for the compression and decompression to line up, and may be difficult for gradient descent to learn.\nThey probably take enough neurons that the benefit over superposition isn’t worth it:\nRepresenting the piecewise linear functions in the simple example with ReLU neurons using the universal function approximation result that each line segment takes two neurons, would require 12 neurons per Z segment, so only starts to beat linear compression at a combined 36 neurons for the compression and decompression.\nThis comparison has only one hidden dimension and dense features, which is somewhat of a degenerate case for superposition. Superposition is much more powerful for compression of sparse features in many dimensions. We suspect in large models the scaling is in favor of superposition, although this is just intuition and it’s possible that scaling nonlinear compression is competitive.\nRegardless of whether large models end up using nonlinear compression, it should be possible to view directions being used with nonlinear compression as linear feature directions and reverse engineer the computation being used for compression like any other circuit. If this kind of encoding is pervasive throughout the network then it may merit some kind of automated decoding. It shouldn’t pose a fundamental challenge to interpretability unless the model learns a scheme for doing complex computation while staying in a complicated nonlinear representation, which we suspect is unlikely.\nTo help provide intuition, the simplest example of what a nonlinear compression scheme might look like is compressing two [0,1) dimensions\nx\nand\ny\ninto a single [0,1) dimension\nt\n:\nt = \\frac{\\lfloor Zx \\rfloor + y}{Z}\nThis works by quantizing the\nx\ndimension using some integer\nZ\nsuch that the floating point precision of\nt\nis split between\nx\nand\ny\n. This particular function needs the discontinuous floor function to compute, and the discontinuous fmod function to invert, but models can’t compute discontinuous functions. However it’s possible to replace the discontinuities with steep linear segments that are only some epsilon value wide.\nWe can compare the mean squared error loss on random uniform dense values of\nx\nand\ny\nand see that even with epsilons as large as\n0.1\nand\nZ\nvalues as small as 3 the nonlinear compression outperforms linear compression such as picking one of the dimensions or using the average:\nConnection between compressed sensing lower bounds and the toy model\nHere, we formalize the relationship between a compressed sensing lower bound and the toy model.\nLet\nT(x) : \\mathbb{R}^n \\to \\mathbb{R}^n\nbe the complete toy model autoencoder defined by\nT(x) = \\mathrm{ReLU}(W_2 W_1 x - b)\nfor an\nm \\times n\nmatrix\nW_1\nand an\nn \\times m\nmatrix\nW_2\n.\nWe derive the following theorem:\nTheorem 1.\nSuppose that the toy model recovers all\nx\nwith\nT(x)\nsuch that\n\\|T(x) - x\\|_2 \\leq \\varepsilon\nfor sufficiently small\n\\varepsilon\nand\nW_1\nhas the\n(\\delta, k)\nrestricted isometry property. The inner dimension of the projection matrix\nW\nis\nm = \\Omega(k log(n/k))\n.\nWe prove this result by framing our toy model as a compressed sensing algorithm. The primary barrier to doing so is that our optimization only searches for vectors that are close in\n\\ell_2\ndistance to the original vector and may not itself be exactly\nk\n-sparse. The following lemma resolves this concern through a denoising step:\nLemma 1.\nSuppose that we have a toy model\nT(x)\nwith the properties in Theorem 1. Then there exists a compressed sensing algorithm\nf(y) : \\mathbb{R}^m \\to \\mathbb{R}^n\nfor the measurement matrix\nW_1\n.\nProof.\nWe construct\nf(y)\nas follows. First, compute\n\\tilde{x} = \\mathrm{ReLU}(W_2 y - b)\n, as in\nT(x)\n. This produces the vector\n\\tilde{x} = T(x)\nand so by supposition\n\\|T(x) - x\\|_2 \\leq \\varepsilon\n. Next, we threshold\n\\tilde{x}\nto obtain\n\\tilde{x}'\nby dropping all but its\nk\nlargest entries. Lastly, we solve the optimization problem:\n\\min_{x'} \\|x' - \\tilde{x}'\\|\nsubject to\nW_1 x' = y\n, which is convex because\nx'\nand\n\\tilde{x}'\nhave the same support. For sufficiently small\n\\varepsilon\n(specifically,\n\\varepsilon\nsmaller than the\n(k + 1)\nth largest entry in\nx\n), both\n\\tilde{x}\nand the nearest\nk\n-sparse vector to\nx\nhave the same support, and so the convex optimization problem has a unique solution: the nearest\nk\nsparse vector to\nx\n. Therefore,\nf\nis a compressed sensing algorithm for\nW_1\nwith approximation factor 1.\n\\qed\n.\nLastly, we use the deterministic compressed sensing lower bound of Do Ba, Indyk, Price, and Woodruff\n:\nTheorem 2 (Corollary 3.1 in\n).\nGiven a\nk \\times n\nmatrix\nA\nwith the restricted isometry property, a sparse recovery algorithm finds a\nk\n-sparse approximation\n\\hat{x}\nof\nx \\in \\mathbb{R}^n\nfrom\nAx\nsuch that\n\\|x - \\hat{x}\\|_1 \\leq C(k) \\min_{x', \\|x'\\|_0 \\leq k} \\|x - x'\\|_1\nfor an approximation factor\nC(k)\n. If\nC(k) = O(1)\n, then a sparse recovery algorithm exists only if\nm = \\Omega(k \\log (n/k))\n.\nTheorem 1 follows directly from Lemma 1 and Theorem 2.",
  "metadata": {
    "title": "Toy Models of Superposition",
    "description": null,
    "keywords": null,
    "author": null
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/2397a4352560/code/body-000.txt",
      "chars": 81,
      "language": "text"
    },
    {
      "file": "attachments/2397a4352560/code/body-001.txt",
      "chars": 527,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://transformer-circuits.pub/",
      "label": "Transformer Circuits Thread",
      "type": "a"
    },
    {
      "href": "https://nelhage.com/",
      "label": "Nelson Elhage ∗",
      "type": "a"
    },
    {
      "href": "https://tomhenighan.com/",
      "label": "Tom Henighan",
      "type": "a"
    },
    {
      "href": "https://zhd.dev/",
      "label": "Zac Hatfield-Dodds",
      "type": "a"
    },
    {
      "href": "https://www.bewitched.com/",
      "label": "Martin Wattenberg",
      "type": "a"
    },
    {
      "href": "https://colah.github.io/",
      "label": "Christopher Olah ‡",
      "type": "a"
    },
    {
      "href": "https://www.anthropic.com/",
      "label": "Anthropic,",
      "type": "a"
    },
    {
      "href": "https://www.harvard.edu/",
      "label": "Harvard",
      "type": "a"
    },
    {
      "href": "https://transformer-circuits.pub/2022/toy_model/colah@anthropic.com",
      "label": "colah@anthropic.com",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/github/anthropics/toy-models-of-superposition/blob/main/toy_models.ipynb",
      "label": "toy model framework Colab notebook",
      "type": "a"
    },
    {
      "href": "https://github.com/anthropics/toy-models-of-superposition",
      "label": "Github repo",
      "type": "a"
    },
    {
      "href": "https://transformer-circuits.pub/2022/mech-interp-essay/index.html",
      "label": "allows us to reason about the model",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2020/circuits/curve-detectors/",
      "label": "curve detectors",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2020/circuits/frequency-edges/",
      "label": "high-low frequency detectors",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2020/circuits/zoom-in/",
      "label": "oriented dog-head detectors",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2021/multimodal-neurons/",
      "label": "famous people",
      "type": "a"
    },
    {
      "href": "https://colah.github.io/posts/2014-07-NLP-RNNs-Representations/",
      "label": "this blog post",
      "type": "a"
    },
    {
      "href": "https://transformer-circuits.pub/2022/solu/index.html",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Johnson%E2%80%93Lindenstrauss_lemma",
      "label": "Johnson–Lindenstrauss lemma",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Mutual_coherence_(linear_algebra)",
      "label": "coherence",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Thomson_problem",
      "label": "Thomson problem",
      "type": "a"
    },
    {
      "href": "https://github.com/wattenberg/superposition/blob/main/Exploring_Exact_Toy_Models.ipynb",
      "label": "this notebook",
      "type": "a"
    },
    {
      "href": "https://www.researchgate.net/profile/Charles-Dunkl/publication/318205131/figure/fig3/AS:512607505350656@1499226561051/Fractional-quantum-Hall-effect.png",
      "label": "this diagram",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Square_antiprism",
      "label": "square antiprisms",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Square_antiprismatic_molecular_geometry",
      "label": "role in molecular geometry",
      "type": "a"
    },
    {
      "href": "https://polytope.miraheze.org/wiki/Tegum_product",
      "label": "tegum products",
      "type": "a"
    },
    {
      "href": "https://polytope.miraheze.org/wiki/Simplex",
      "label": "regular n -simplex",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2020/circuits/equivariance/",
      "label": "equivariant features",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2018/building-blocks/",
      "label": "Building Blocks",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2019/advex-bugs-discussion/response-3/",
      "label": "paper by Gabriel Goh",
      "type": "a"
    },
    {
      "href": "https://web.archive.org/web/20190811044354/https://distill.pub/2019/advex-bugs-discussion/response-3/",
      "label": "archive.org backup",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Compressed_sensing",
      "label": "compressed sensing",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Sparse_dictionary_learning",
      "label": "Sparse Coding",
      "type": "a"
    },
    {
      "href": "https://www.lesswrong.com/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition",
      "label": "research",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2020/circuits/",
      "label": "Circuits Thread",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2019/advex-bugs-discussion/",
      "label": "Distill's Discussion Article experiment",
      "type": "a"
    },
    {
      "href": "https://kshitijsachan.com/",
      "label": "Kshitij Sachan",
      "type": "a"
    },
    {
      "href": "https://www.redwoodresearch.org/",
      "label": "Redwood Research",
      "type": "a"
    },
    {
      "href": "https://www.alignmentforum.org/posts/kWp4R9SYgKJFHAufB/polysemanticity-and-capacity-in-neural-networks",
      "label": "Alignment Forum",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2210.01892",
      "label": "Arxiv",
      "type": "a"
    },
    {
      "href": "https://tommcgrath.github.io/",
      "label": "Tom McGrath",
      "type": "a"
    },
    {
      "href": "https://www.deepmind.com/",
      "label": "DeepMind",
      "type": "a"
    },
    {
      "href": "https://www.wuthejeff.com/",
      "label": "Jeffrey Wu",
      "type": "a"
    },
    {
      "href": "https://dmossing.github.io/",
      "label": "Dan Mossing",
      "type": "a"
    },
    {
      "href": "https://openai.com/",
      "label": "OpenAI",
      "type": "a"
    },
    {
      "href": "https://spencerbeckerkahn.wixsite.com/stbk",
      "label": "Spencer Becker-Kahn",
      "type": "a"
    },
    {
      "href": "https://twitter.com/sbeckerkahn/status/1572255478666649602",
      "label": "Twitter thread",
      "type": "a"
    },
    {
      "href": "https://adamjermyn.com/",
      "label": "Adam Jermyn",
      "type": "a"
    },
    {
      "href": "https://evhub.github.io/",
      "label": "Evan Hubinger",
      "type": "a"
    },
    {
      "href": "https://nicholasschiefer.com/",
      "label": "Nicholas Schiefer",
      "type": "a"
    },
    {
      "href": "https://www.alignmentforum.org/posts/LvznjZuygoeoTpSE6/engineering-monosemanticity-in-toy-models",
      "label": "Alignment Forum",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2211.09169.pdf",
      "label": "ArXiV",
      "type": "a"
    },
    {
      "href": "https://www.mariushobbhahn.com/",
      "label": "Marius Hobbhahn",
      "type": "a"
    },
    {
      "href": "https://transformer-circuits.pub/2023/toy-double-descent/index.html",
      "label": "“Superposition, Memorization, and Double Descent”",
      "type": "a"
    },
    {
      "href": "https://www.lesswrong.com/posts/KzwB4ovzrZ8DYWgpw/more-findings-on-memorization-and-double-descent",
      "label": "“More Findings on Memorization and Double Descent”",
      "type": "a"
    },
    {
      "href": "https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition",
      "label": "preliminary follow-up work",
      "type": "a"
    },
    {
      "href": "https://transformer-circuits.pub/2022/toy_model/index.html",
      "label": "Approach 2: Finding an overcomplete basis",
      "type": "a"
    },
    {
      "href": "https://neelnanda.io/othello",
      "label": "Actually, Othello-GPT Has A Linear Emergent World Representation",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2210.13382.pdf",
      "label": "recently found",
      "type": "a"
    },
    {
      "href": "https://www.neelnanda.io/othello",
      "label": "the Othello playing model",
      "type": "a"
    },
    {
      "href": "https://www.alignmentforum.org/posts/qgK7smTvJ4DB8rZ6h/othello-gpt-future-work-i-am-excited-about",
      "label": "a follow-up post",
      "type": "a"
    },
    {
      "href": "https://fredzhang.me/",
      "label": "Fred Zhang",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1408.5099",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1411.4357",
      "label": "https://arxiv.org/abs/1411.4357",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:37Z",
  "source_url": "https://transformer-circuits.pub/2022/toy_model/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2210.01892",
      "bytes": 3166657,
      "reused": false,
      "local_path": "papers/repair/1b9894c4828553ff9df8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2211.09169.pdf",
      "bytes": 5061904,
      "reused": true,
      "local_path": "papers/repair/d0db5fc8264f6d96d6fc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2210.13382.pdf",
      "bytes": 4064326,
      "reused": false,
      "local_path": "papers/repair/dcc2886eb440e79d842d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1408.5099",
      "bytes": 293915,
      "reused": false,
      "local_path": "papers/repair/4459a5ff965169239082.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1411.4357",
      "bytes": 969649,
      "reused": false,
      "local_path": "papers/repair/5542dbec76650f7fd737.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 163637,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/2397a4352560/pages/source.html",
  "local_html": "attachments/2397a4352560/pages/article.html",
  "body_base_path": "attachments/2397a4352560/pages/",
  "body_markdown_path": "attachments/2397a4352560/pages/article.md"
}