{
  "source_file": "0930",
  "source_platform": "技术平台",
  "author": "Distill",
  "title_listed": "Visualizing Neural Networks with the Grand Tour",
  "keywords": "Grand Tour,神经网络,可视化",
  "domain": "机器学习",
  "content_type": "图解教程",
  "is_paid_flag": "否",
  "url": "https://distill.pub/2020/grand-tour/",
  "publish_date": "2020-03",
  "corpus_stage": "预训练/SFT",
  "notes": "Grand Tour神经网络可视化",
  "id": "02c83ff36532",
  "collected_at": "2026-10-02T17:54:37",
  "title": "Visualizing Neural Networks with the Grand Tour",
  "body_markdown": "Visualizing Neural Networks with the Grand Tour\nThe Grand Tour\nin action. This visualization shows the behavior of the final 10-dimensional layer\n    of a neural network as it is trained on the MNIST dataset. With this technique, it is possible to see interesting training behavior. For example, the network appears to learn to classify digits\n1\nand\n7\nin an almost discontinuous manner, after training epochs\n14\nand\n21\nrespectively.\nAuthors\nAffiliations\nMingwei Li\nUniversity of Arizona\nZhenge Zhao\nUniversity of Arizona\nCarlos Scheidegger\nUniversity of Arizona\nPublished\nMarch 16, 2020\nDOI\n10.23915/distill.00025\nThe Grand Tour\nis a classic visualization technique for high-dimensional point clouds that\nprojects\na high-dimensional dataset into two dimensions.\n\n  Over time, the Grand Tour smoothly animates its projection so that every possible view of the dataset is (eventually) presented to the viewer.\n\n  Unlike modern nonlinear projection methods such as t-SNE\nand UMAP\n, the Grand Tour is fundamentally a\nlinear\nmethod.\n\n  In this article, we show how to leverage the linearity of the Grand Tour to enable a number of capabilities that are uniquely useful to visualize the behavior of neural networks.\n  \n  Concretely, we present three use cases of interest: visualizing the training process as the network weights change, visualizing the layer-to-layer behavior as the data goes through the network and visualizing both how adversarial examples\nare crafted and how they fool a neural network.\nIntroduction\nDeep neural networks often achieve best-in-class performance in supervised learning contests such as the ImageNet Large Scale Visual Recognition Challenge (ILSVRC)\n.\n  \n  Unfortunately, their decision process is notoriously hard to interpret\n, and their training process is often hard to debug\n.\n  \n  In this article, we present a method to visualize the responses of a neural network which leverages properties of deep neural networks and properties of the\nGrand Tour\n.\n\n  Notably, our method enables us to more directly reason about the relationship between\nchanges in the data\nand\nchanges in the resulting visualization\n.\n\n  As we will show, this data-visual correspondence is central to the method we present, especially when compared to other non-linear projection methods like UMAP and t-SNE.\nTo understand a neural network, we often try to observe its action on input examples (both real and synthesized)\n.\n  \n  These kinds of visualizations are useful to elucidate the activation patterns of a neural network for a single example, but they might offer less insight about the relationship between different examples, different states of the network as it’s being trained, or how the data in the example flows through the different layers of a single network.\n  \n  Therefore, we instead aim to enable visualizations of the\ncontext around\nour objects of interest: what is the difference between the present training epoch and the next one? How does the classification of a network converge (or diverge) as the image is fed through the network?\n\n  Linear methods are attractive because they are particularly easy to reason about.\n\n  The Grand Tour works by generating a random, smoothly changing rotation of the dataset, and then projecting the data to the two-dimensional screen: both are linear processes.\n\n  Although deep neural networks are clearly not linear processes, they often confine their nonlinearity to a small set of operations, enabling us to still reason about their behavior.\n\n  Our proposed method better preserves context by providing more\n  consistency: it should be possible to know\nhow the visualization\n  would change, if the data had been different in a particular\n  way\n.\nWorking Examples\nTo illustrate the technique we will present, we trained deep neural\n  network models (DNNs) with 3 common image classification datasets:\n  MNIST\nMNIST\ncontains grayscale images of 10 handwritten digits\nImage credit to\nhttps://en.wikipedia.org/wiki/File:MnistExamples.png\n,\n  fashion-MNIST\nFashion-MNIST\ncontains grayscale images of 10 types of fashion items:\nImage credit to\nhttps://towardsdatascience.com/multi-label-classification-and-class-activation-map-on-fashion-mnist-1454f09f5925\nand CIFAR-10\nCIFAR-10\ncontains RGB images of 10 classes of objects\nImage credit to\nhttps://www.cs.toronto.edu/~kriz/cifar.html\n. \n  While our architecture is simpler and smaller than current DNNs, it’s still indicative of modern networks, and is complex enough to demonstrate both our proposed techniques and shortcomings of typical approaches.\nThe following figure presents a simple functional diagram of the neural network we will use throughout the article. The neural network is a sequence of linear (both convolutional\nA convolution calculates weighted sums of regions in the input. \n    In neural networks, the learnable weights in convolutional layers are referred to as the kernel.\n    For example\nImage credit to\nhttps://towardsdatascience.com/gentle-dive-into-math-behind-convolutional-neural-networks-79a07dd44cf9\n.\nSee also\nConvolution arithmetic\n.\nand fully-connected\nA fully-connected layer computes output neurons as weighted sum of input neurons. In matrix form, it is a matrix that linearly transforms the input vector into the output vector.\n), max-pooling, and ReLU\nFirst introduced by Nair and Hinton\n, ReLU calculates\nf\n(\nx\n)\n=\nm\na\nx\n(\n0\n,\nx\n)\nf(x)=max(0,x)\nf\n(\nx\n)\n=\nm\na\nx\n(\n0\n,\nx\n)\nfor each entry in a vector input. Graphically, it is a hinge at the origin:\nImage credit to\nhttps://pytorch.org/docs/stable/nn.html#relu\nlayers, culminating in a softmax\nSoftmax function calculates\nS\n(\ny\ni\n)\n=\ne\ny\ni\nΣ\nj\n=\n1\nN\ne\ny\nj\nS(y_i)=\\frac{e^{y_i}}{\\Sigma_{j=1}^{N} e^{y_j}}\nS\n(\ny\ni\n​\n)\n=\nΣ\nj\n=\n1\nN\n​\ne\ny\nj\n​\ne\ny\ni\n​\n​\nfor each entry (\ny\ni\ny_i\ny\ni\n​\n) in a vector input (\ny\ny\ny\n). For example,\nImage credit to\nhttps://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/\nlayer.\nNeural network opened. The colored blocks are building-block functions (i.e. neural network layers), the gray-scale heatmaps are either the input image or intermediate activation vectors after some layers.\nEven though neural networks are capable of incredible feats of classification, deep down, they really are just pipelines of relatively simple functions.\n  For images, the input is a 2D array of scalar values for gray scale images or RGB triples for colored images.\n  When needed, one can always flatten the 2D array into an equivalent (\nw\n⋅\nh\n⋅\nc\nw \\cdot h \\cdot c\nw\n⋅\nh\n⋅\nc\n) -dimensional vector.\n  Similarly, the intermediate values after any one of the functions in composition, or activations of neurons after a layer, can also be seen as vectors in\nR\nn\n\\mathbb{R}^n\nR\nn\n, where\nn\nn\nn\nis the number of neurons in the layer. \n  The softmax, for example, can be seen as a 10-vector whose values are positive real numbers that sum up to 1.\n  This vector view of data in neural network not only allows us represent complex data in a mathematically compact form, but also hints us on how to visualize them in a better way.\nMost of the simple functions fall into two categories: they are either linear transformations of their inputs (like fully-connected layers or convolutional layers), or relatively simple non-linear functions that work component-wise (like sigmoid activations\nSigmoid calculates\nS\n(\nx\n)\n=\ne\nx\ne\nx\n+\n1\nS(x)=\\frac{e^{x}}{e^{x}+1}\nS\n(\nx\n)\n=\ne\nx\n+\n1\ne\nx\n​\nfor each entry (\nx\nx\nx\n) in a vector input. Graphically, it is an S-shaped curve.\nImage credit to\nhttps://en.wikipedia.org/wiki/Sigmoid_function\nor ReLU activations).\n  Some operations, notably max-pooling\nMax-pooling calculates maximum of a region in the input. For example\nImage credit to\nhttps://towardsdatascience.com/gentle-dive-into-math-behind-convolutional-neural-networks-79a07dd44cf9\nand softmax, do not fall into either categories. We will come back to this later.\nThe above figure helps us look at a single image at a time; however, it does not provide much context to understand the relationship between layers, between different examples, or between different class labels. For that, researchers often turn to more sophisticated visualizations.\nUsing Visualization to Understand DNNs\nLet’s start by considering the problem of visualizing the training process of a DNN.\n  When training neural networks, we optimize parameters in the function to minimize a scalar-valued loss function, typically through some form of gradient descent.\n  We want the loss to keep decreasing, so we monitor the whole history of training and testing losses over rounds of training (or “epochs”), to make sure that the loss decreases over time. \n  The following figure shows a line plot of the training loss for the MNIST classifier.\nAlthough its general trend meets our expectation as the loss steadily decreases, we see something strange around epochs 14 and 21: the curve goes almost flat before starting to drop again.\n  What happened? What caused that?\nIf we separate input examples by their true labels/classes and plot the\nper-class\nloss like above, we see that the two drops were caused by the classses 1 and 7; the model learns different classes at very different times in the training process. \n  Although the network learns to recognize digits 0, 2, 3, 4, 5, 6, 8 and 9 early on, it is not until epoch 14 that it starts successfully recognizing digit 1, or until epoch 21 that it recognizes digit 7.\n  If we knew ahead of time to be looking for class-specific error rates, then this chart works well. But what if we didn’t really know what to look for?\nIn that case, we could consider visualizations of neuron activations (e.g. in the last softmax layer) for\nall\nexamples at once, looking\n  to find patterns like class-specific behavior, and other patterns besides.\n  Should there be only two neurons in that layer, a simple two-dimensional scatter plot would work.\n  However, the points in the softmax layer for our example datasets are 10 dimensional (and in larger-scale classification problems this number can be much larger).\n  We need to either show two dimensions at a time (which does not scale well as the number of possible charts grows quadratically),\n  or we can use\ndimensionality reduction\nto map the data into a two dimensional space and show them in a single plot.\nThe State-of-the-art Dimensionality Reduction is Non-linear\nModern dimensionality reduction techniques such as t-SNE and UMAP are capable of impressive feats of summarization, providing two-dimensional images where similar points tend to be clustered together very effectively.\n  However, these methods are not particularly good to understand the behavior of neuron activations at a fine scale.\n  Consider the aforementioned intriguing feature about the different learning rate that the MNIST classifier has on digit 1 and 7: the network did not learn to recognize digit 1 until epoch 14, digit 7 until epoch 21.\n  We compute t-SNE, Dynamic t-SNE\n, and UMAP projections of the epochs where the phenomenon we described happens.\n  Consider now the task of identifying this class-specific behavior during training. As a reminder, in this case, the strange behavior happens with digits 1 and 7, around epochs 14 and 21 respectively.\n  While the behavior is not particularly subtle&emdash;digit goes from misclassified to correctly classified&emdash; it is quite hard to notice it in any of the plots below. \n  Only on careful inspection we can notice that (for example) in the UMAP plot, the digit 1 which clustered in the bottom in epoch 13 becomes a new tentacle-like feature in epoch 14.\nSoftmax activations of the MNIST classifier with non-linear dimensionality reduction. Use the buttons on the right to highlight digits 1 and 7 in the plot, or drag rectangles around the charts to select particular point subsets to highlight in the other charts.\nOne reason that non-linear embeddings fail in elucidating this phenomenon is that, for the particular change in the data, the fail the principle of\ndata-visual correspondence\n. More concretely, the principle states that specific visualization tasks should be modeled as functions that change the data; the visualization sends this change from data to visuals, and\n  we can study the extent to which the visualization changes are easily perceptible.\n  Ideally, we want the changes in data and visualization to\nmatch in magnitude\n: a barely noticeable change in visualization should be due to the smallest possible change in data, and a salient change in visualization should reflect a significant one in data.\n  Here, a significant change happened in only a\nsubset\nof data (e.g. all points of digit 1 from epoch 13 to 14), but\nall\npoints in the visualization move dramatically.\n  For both UMAP and t-SNE, the position of each single point depends non-trivially on the whole data distribution in such embedding algorithms.\n  This property is not ideal for visualization because it fails the data-visual correspondence, making it hard to\ninfer\nthe underlying change in data from the change in the visualization.\nNon-linear embeddings that have non-convex objectives also tend to be sensitive to initial conditions.\n  For example, in MNIST, although the neural network starts to stabilize on epoch 30, t-SNE and UMAP still generate quite different projections between epochs 30, 31 and 32 (in fact, all the way to 99).\n  Temporal regularization techniques (such as Dynamic t-SNE) mitigate these consistency issues, but still suffer from other interpretability issues\n.\nNow, let’s consider another task, that of identifying classes which the neural network tends to confuse.\n  For this example, we will use the Fashion-MNIST dataset and classifier, and consider the confusion among sandals, sneakers and ankle boots.\n  If we know ahead of time that these three classes are likely to confuse the classifier, then we can directly design an appropriate linear projection, as can be seen in the last row of the following figure (we found this particular projection using both the Grand Tour and the direct manipulation technique we later describe). The pattern in this case is quite salient, forming a triangle.\n  T-SNE, in contrast, incorrectly separates the class clusters (possibly because of an inappropriately-chosen hyperparameter).\n  UMAP successfully isolates the three classes, but even in this case it’s not possible to distinguish between three-way confusion for the classifier in epochs 5 and 10 (portrayed in a linear method by the presence of points near the center of the triangle), and multiple two-way confusions in later epochs (evidences by an “empty” center).\nThree-way confusion in fashion-MNIST. Notice that in contrast to non-linear methods, a carefully-constructed linear projection can offer a better visualization of the classifier behavior.\nLinear Methods to the Rescue\nWhen given the chance, then, we should prefer methods for which changes in the data produce predictable, visually salient changes in the result, and linear dimensionality reductions often have this property.\n  Here, we revisit the linear projections described above in an interface where the user can easily navigate between different training epochs.\n  In addition, we introduce another useful capability which is only available to linear methods, that of direct manipulation.\n  Each linear projection from\nn\nn\nn\ndimensions to\n2\n2\n2\ndimensions can be represented by\nn\nn\nn\n2-dimensional vectors which have an intuitive interpretation: they are the vectors that the\nn\nn\nn\ncanonical basis vector in the\nn\nn\nn\n-dimensional space will be projected to.\n  In the context of projecting the final classification layer, this is especially simple to interpret: they are the destinations of an input that is classified with 100% confidence to any one particular class.\n  If we provide the user with the ability to change these vectors by dragging around user-interface handles, then users can intuitively set up new linear projections.\nThis setup provides additional nice properties that explain the salient patterns in the previous illustrations.\n  For example, because projections are linear and the coefficients of vectors in the classification layer sum to one, classification outputs that are halfway confident between two classes are projected to vectors that are halfway between the class handles.\nFrom\nthis linear projection\n, we can easily identify the learning of\ndigit 1\non\nepoch 14\nand\ndigit 7\non\nepoch 21\n.\nThis particular property is illustrated clearly in the Fashion-MNIST example below.\n  The model confuses sandals, sneakers and ankle boots, as data points form a triangular shape in the softmax layer.\nThis\nlinear projection\nclearly shows model’s confusion among\nsandals\n,\nsneakers\n, and\nankle boots\n.\n  Similarly,\nthis projection\nshows the true three-way confusion about\npullovers\n,\ncoats\n, and\nshirts\n.\n  (The\nshirts\nare also get confused with\nt-shirts/tops\n. )\n  Both projections are found by direct manipulations.\nExamples falling between classes indicate that the model has trouble distinguishing the two, such as sandals vs. sneakers, and sneakers vs. ankle boot classes. \n  Note, however, that this does not happen as much for sandals vs. ankle boots: not many examples fall between these two classes. \n  Moreover, most data points are projected close to the edge of the triangle. \n  This tells us that most confusions happen between two out of the three classes, they are really two-way confusions.\n\n  Within the same dataset, we can also see pullovers, coats and shirts filling a triangular\nplane\n.\n  This is different from the sandal-sneaker-ankle-boot case, as examples not only fall on the boundary of a triangle, but also in its interior: a true three-way confusion. \n\n  Similarly, in the CIFAR-10 dataset we can see confusion between dogs and cats, airplanes and ships.\n  The mixing pattern in CIFAR-10 is not as clear as in fashion-MNIST, because many more examples are misclassified.\nThis\nlinear projection\nclearly shows model’s confusion between\ncats\nand\ndogs\n.\n  Similarly,\nthis projection\nshows the confusion about\nairplanes\nand\nships\n.\n  Both projections are found by direct manipulations.\nThe Grand Tour\nIn the previous section, we took advantage of the fact that we knew which classes to visualize.\n  That meant it was easy to design linear projections for the particular tasks at hand.\n  But what if we don’t know ahead of time which projection to choose from, because we don’t quite know what to look for?\n  Principal Component Analysis (PCA) is the quintessential linear dimensionality reduction method,\n  choosing to project the data so as to preserve the most variance possible. \n  However, the distribution of data in softmax layers often has similar variance along many axis directions, because each axis concentrates a similar number of examples around the class vector.\nWe are assuming a class-balanced training dataset. Nevertheless, if the training dataset is not balanced, PCA will prefer dimensions with more examples, which might not be help much either.\nAs a result, even though PCA projections are interpretable and consistent through training epochs, the first two principal components of softmax activations are not substantially better than the third.\n  So which of them should we choose?\n  Instead of PCA, we propose to visualize this data by smoothly animating random projections, using a technique called the Grand Tour\n.\nStarting with a random velocity, it smoothly rotates data points around the origin in high dimensional space, and then projects it down to 2D for display. \nHere are some examples of how Grand Tour acts on some (low-dimensional) objects:\nOn a square, the Grand Tour rotates it with a constant angular velocity.\nOn a cube, the Grand Tour rotates it in 3D, and its 2D projection let us see every facet of the cube.\nOn a 4D cube (a\ntesseract\n), the rotation happens in 4D and the 2D view shows every possible projection.\nGrand tours of a square, a cube and a tesseract\nThe Grand Tour of the Softmax Layer\nWe first look at the Grand Tour of the softmax layer. \n  The softmax layer is relatively easy to understand because its axes have strong semantics. As we described earlier, the\ni\ni\ni\n-th axis corresponds to network’s\nconfidence\nabout predicting that the given input belongs to the\ni\ni\ni\n-th class.\nThe Grand Tour of softmax layer in the last (99\nth\n) epoch, with\nMNIST\n,\nfashion-MNIST\nor\nCIFAR-10\ndataset.\nThe Grand Tour of the softmax layer lets us qualitatively assess the performance of our model.\n  In the particular case of this article, since we used comparable architectures for three datasets, this also allows us to gauge the relative difficulty of classifying each dataset. \n  We can see that data points are most confidently classified for the MNIST dataset, where the digits are close to one of the ten corners of the softmax space. For Fashion-MNIST or CIFAR-10, the separation is not as clean, and more points appear\ninside\nthe volume.\nThe Grand Tour of Training Dynamics\nLinear projection methods naturally give a formulation that is independent of the input points, allowing us to keep the projection fixed while the\n  data changes.\n  To recap our working example, we trained each of the neural networks for 99 epochs and recorded the entire history of neuron activations on a subset of training and testing examples. We can use the Grand Tour, then, to visualize the actual training process of these networks.\nIn the beginning when the neural networks are randomly initialized, all examples are placed around the center of the softmax space, with equal weights to each class. \n  Through training, examples move to class vectors in the softmax space. The Grand Tour also lets us\n  compare visualizations of the training and testing data, giving us a qualitative assessment of over-fitting. \n  In the MNIST dataset, the trajectory of testing images through training is consistent with the training set. \n  Data points went directly toward the corner of its true class and all classes are stabilized after about 50 epochs.\n  On the other hand, in CIFAR-10 there is an\ninconsistency\nbetween the training and testing sets. Images from the testing set keep oscillating while most images from training converges to the corresponding class corner. \n  In epoch 99, we can clearly see a difference in distribution between these two sets.\n  This signals that the model overfits the training set and thus does not generalize well to the testing set.\nWith\nthis view of CIFAR-10\n, \n  the color of points are more mixed in testing (right) than training (left) set, showing an over-fitting in the training process.\n  Compare\nCIFAR-10\nwith\nMNIST\nor\nfashion-MNIST\n, \n  where there is less difference between training and testing sets.\nThe Grand Tour of Layer Dynamics\nGiven the presented techniques of the Grand Tour and direct manipulations on the axes, we can in theory visualize and manipulate any intermediate layer of a neural network by itself.  Nevertheless, this is not a very satisfying approach, for two reasons:\nIn the same way that we’ve kept the projection fixed as the training data changed, we would like to “keep the projection fixed”, as the data moves through the layers in the neural network. However, this is not straightforward. For example, different layers in a neural network have different dimensions. How do we connect rotations of one layer to rotations of the other?\nThe class “axis handles” in the softmax layer convenient, but that’s only practical when the dimensionality of the layer is relatively small.\n      With hundreds of dimensions, for example, there would be too many axis handles to naturally interact with.\n      In addition, hidden layers do not have as clear semantics as the softmax layer, so manipulating them would not be as intuitive.\nTo address the first problem, we will need to pay closer attention to the way in which layers transform the data that they are given.  \n  To see how a linear transformation can be visualized in a particularly ineffective way, consider the following (very simple) weights (represented by a matrix\nA\nA\nA\n) which take a 2-dimensional hidden layer\nk\nk\nk\nand produce activations in another 2-dimensional layer\nk\n+\n1\nk+1\nk\n+\n1\n. The weights simply negate two activations in 2D:\nA\n=\n[\n−\n1\n,\n0\n0\n,\n−\n1\n]\nA = \\begin{bmatrix}\n    -1, 0 \\\\\n    0, -1\n    \\end{bmatrix}\nA\n=\n[\n−\n1\n,\n0\n0\n,\n−\n1\n​\n]\nImagine that we wish to visualize the behavior of network as the data moves from layer to layer. One way to interpolate the source\nx\n0\nx_0\nx\n0\n​\nand destination\nx\n1\n=\nA\n(\nx\n0\n)\n=\n−\nx\n0\nx_1 = A(x_0) = -x_0\nx\n1\n​\n=\nA\n(\nx\n0\n​\n)\n=\n−\nx\n0\n​\nof this action\nA\nA\nA\nis by a simple linear interpolation\nx\nt\n=\n(\n1\n−\nt\n)\n⋅\nx\n0\n+\nt\n⋅\nx\n1\n=\n(\n1\n−\n2\nt\n)\n⋅\nx\n0\nx_t = (1-t) \\cdot x_0 + t \\cdot x_1 = (1-2t) \\cdot x_0\nx\nt\n​\n=\n(\n1\n−\nt\n)\n⋅\nx\n0\n​\n+\nt\n⋅\nx\n1\n​\n=\n(\n1\n−\n2\nt\n)\n⋅\nx\n0\n​\nfor\nt\n∈\n[\n0\n,\n1\n]\n.\nt \\in [0,1].\nt\n∈\n[\n0\n,\n1\n]\n.\nEffectively, this strategy reuses the linear projection coefficients from one layer to the next. This is a natural thought, since they have the same dimension.\n  However, notice the following: the transformation given by A is a simple rotation of the data. Every linear transformation of the layer\nk\n+\n1\nk+1\nk\n+\n1\ncould be encoded simply as a linear transformation of the layer\nk\nk\nk\n, if only that transformation operated on the negative values of the entries.\n  In addition, since the Grand Tour has a rotation itself built-in, for every configuration that gives a certain picture of the layer\nk\nk\nk\n, there exists a\ndifferent\nconfiguration that would yield the same picture for layer\nk\n+\n1\nk+1\nk\n+\n1\n, by taking the action of\nA\nA\nA\ninto account.\n  In effect, the naive interpolation fails the principle of data-visual correspondence: a simple change in data (negation in 2D/180 degree rotation) results in a drastic change in visualization (all points cross the origin).\nThis observation points to a more general strategy: when designing a visualization, we should be as explicit as possible about which parts of the input (or process) we seek to capture in our visualizations.\n  We should seek to explicitly articulate what are purely representational artifacts that we should discard, and what are the real features a visualization we should\ndistill\nfrom the representation.\n  Here, we claim that rotational factors in linear transformations of neural networks are significantly less important than other factors such as scalings and nonlinearities.\n  As we will show, the Grand Tour is particularly attractive in this case because it is can be made to be invariant to rotations in data.\n  As a result, the rotational components in the linear transformations of a neural network will be explicitly made invisible.\nConcretely, we achieve this by taking advantage of a central theorem of linear algebra. \n  The\nSingular Value Decomposition\n(SVD) theorem shows that\nany\nlinear transformation can be decomposed into a sequence of very simple operations: a rotation, a scaling, and another rotation\n. \n\n  Applying a matrix\nA\nA\nA\nto a vector\nx\nx\nx\nis then equivalent to applying those simple operations:\nx\nA\n=\nx\nU\nΣ\nV\nT\nx A = x U \\Sigma V^T\nx\nA\n=\nx\nU\nΣ\nV\nT\n.\n  But remember that the Grand Tour works by rotating the dataset and then projecting it to 2D.\n  Combined, these two facts mean that as far as the Grand Tour is concerned, visualizing a vector\nx\nx\nx\nis the same as visualizing\nx\nU\nx U\nx\nU\n, and visualizing a vector\nx\nU\nΣ\nV\nT\nx U \\Sigma V^T\nx\nU\nΣ\nV\nT\nis the same as visualizing\nx\nU\nΣ\nx U \\Sigma\nx\nU\nΣ\n. \n  This means that any linear transformation seen by the Grand Tour is equivalent to the transition between\nx\nU\nx U\nx\nU\nand\nx\nU\nΣ\nx U \\Sigma\nx\nU\nΣ\n- a simple (coordinate-wise) scaling. \n  This is explicitly saying that any linear operation (whose matrix is represented in standard bases) is a scaling operation with appropriately chosen orthonormal bases on both sides.\n  So the Grand Tour provides a natural, elegant and computationally efficient way to\nalign\nvisualizations of activations separated by fully-connected (linear) layers.\nConvolutional layers are also linear. One can instantly see that by forming the linear transformations between flattened feature maps, or by taking the circulant structure of convolutional layers directly into account\n(For the following portion, we reduce the number of data points to 500 and epochs to 50, in order to reduce the amount of data transmitted in a web-based demonstration.)\n  With the linear algebra structure at hand, now we are able to trace behaviors and patterns from the softmax back to previous layers.\n  In fashion-MNIST, for example, we observe a separation of shoes (sandals, sneakers and ankle boots as a group) from all other classes in the softmax layer. \n  Tracing it back to earlier layers, we can see that this separation happened as early as\nlayer 5\n:\nWith layers aligned, it is easy to see the early separation of shoes from\nthis view\n.\nThe Grand Tour of Adversarial Dynamics\nAs a final application scenario, we show how the Grand Tour can also elucidate the behavior of adversarial examples\nas they are processed by a neural network.\n  For this illustration, we use the MNIST dataset, and we adversarially add perturbations to 89 digit 8s to fool the network into thinking they are 0s.\n  Previously, we either animated the training dynamics or the layer dynamics.\n  We fix a well-trained neural network, and visualize the training process of adversarial examples, since they are often themselves generated by an optimization process. Here, we used the Fast Gradient Sign method.\nAgain, because the Grand Tour is a linear method, the change in the positions of the adversarial examples over time can be faithfully attributed to changes in how the neural network perceives the images, rather than potential artifacts of the visualization.\n  Let us examine how adversarial examples evolved to fool the network:\nFrom\nthis view of softmax\n, we can see how\nadversarial examples\nevolved from\n8s\ninto\n0s\n.\n  In the corresponding\npre-softmax\nhowever, these adversarial examples stop around the decision boundary of two classes. \n  Show data as\nimages\nto see the actual images generated in each step, or\ndots\ncolored by labels.\nThrough this adversarial training, the network eventually claims, with high confidence, that the inputs given are all 0s.\n  If we stay in the softmax layer and slide though the adversarial training steps in the plot, we can see adversarial examples move from a high score for class 8 to a high score for class 0.\n  Although all adversarial examples are classified as the target class (digit 0s) eventually, some of them detoured somewhere close to the centroid of the space (around the 25th epoch) and then moved towards the target. \n  Comparing the actual images of the two groups, we see those that those “detouring” images tend to be noisier.\nMore interesting, however, is what happens in the intermediate layers.\n  In pre-softmax, for example, we see that these\nfake 0s\nbehave differently from the\ngenuine 0s\n: they live closer to the decision boundary of two classes and form a plane by themselves.\nDiscussion\nLimitations of the Grand Tour\nEarly on, we compared several state-of-the-art dimensionality reduction techniques with the Grand Tour, showing that non-linear methods do not have as many desirable properties as the Grand Tour for understanding the behavior of neural networks. \n  However, the state-of-the-art non-linear methods come with their own strength. \n  Whenever geometry is concerned, like the case of understanding multi-way confusions in the softmax layer, linear methods are more interpretable because they preserve certain geometrical structures of data in the projection. \n  When topology is the main focus, such as when we want to cluster the data or we need dimensionality reduction for downstream models that are less sensitive to geometry, we might choose non-linear methods such as UMAP or t-SNE for they have more freedom in projecting the data, and will generally make better use of the fewer dimensions available.\nThe Power of Animation and Direct Manipulation\nWhen comparing linear projections with non-linear dimensionality reductions, we used small multiples to contrast training epochs and dimensionality reduction methods.\n  The Grand Tour, on the other hand, uses a single animated view.\n  When comparing small multiples and animations, there is no general consensus on which one is better than the other in the literature, aside. \n  from specific settings such as dynamic graph drawing\n, or concerns about incomparable contents\nbetween small multiples and animated plots.\n  Regardless of these concerns, in our scenarios, the use of animation comes naturally from the direct manipulation and the existence of a continuum of rotations for the Grand Tour to operate in.\nNon-sequential Models\nIn our work we have used models that are purely “sequential”, in the sense that the layers can be put in numerical ordering, and that the activations for\n  the\nn\n+\n1\nn+1\nn\n+\n1\n-th layer are a function exclusively of the activations at the\nn\nn\nn\n-th layer. \n  In recent DNN architectures, however, it is common to have non-sequential parts such as highway\nbranches or dedicated branches for different tasks\n. \n  With our technique, one can visualize neuron activations on each such branch, but additional research is required to incorporate multiple branches directly.\nScaling to Larger Models\nModern architectures are also wide. Especially when convolutional layers are concerned, one could run into issues with scalability if we see such layers as a large sparse matrix acting on flattened multi-channel images.\n  For the sake of simplicity, in this article we brute-forced the computation of the alignment of such convolutional layers by writing out their explicit matrix representation. \n  However, the singular value decomposition of multi-channel 2D convolutions can be computed efficiently\n, which can be then be directly used for alignment, as we described above.\nTechnical Details\nThis section presents the technical details necessary to implement the direct manipulation of axis handles and data points, as well as how to implement the projection consistency technique for layer transitions.\nNotation\nIn this section, our notational convention is that data points are represented as row vectors.\n  An entire dataset is laid out as a matrix, where each row is a data point, and each column represents a different feature/dimension.\n  As a result, when a linear transformation is applied to the data, the row vectors (and the data matrix overall) are left-multiplied by the transformation matrix.\n  This has a side benefit that when applying matrix multiplications in a chain, the formula reads from left to right and aligns with a commutative diagram.\n  For example, when a data matrix\nX\nX\nX\nis multiplied by a matrix\nM\nM\nM\nto generate\nY\nY\nY\n, in formula we write\nX\nM\n=\nY\nXM = Y\nX\nM\n=\nY\n, the letters have the same order in diagram:\nX\n↦\nM\nY\nX \n    \\overset{M}{\\mapsto}\n    Y\nX\n↦\nM\n​\nY\nFurthermore, if the SVD of\nM\nM\nM\nis\nM\n=\nU\nΣ\nV\nT\nM = U \\Sigma V^{T}\nM\n=\nU\nΣ\nV\nT\n, we have\nX\nU\nΣ\nV\nT\n=\nY\nX U \\Sigma V^{T} = Y\nX\nU\nΣ\nV\nT\n=\nY\n, and the diagram\nX\n↦\nU\n↦\nΣ\n↦\nV\nT\nY\nX \n    \\overset{U}{\\mapsto} \n    \\overset{\\Sigma}{\\mapsto} \n    \\overset{V^T}{\\mapsto} Y\nX\n↦\nU\n​\n↦\nΣ\n​\n↦\nV\nT\n​\nY\nnicely aligns with the formula.\nDirect Manipulation\nThe direct manipulations we presented earlier provide explicit control over the possible projections for the data points.\n  We provide two modes: directly manipulating class axes (the “axis mode”), or directly manipulating a group of data points through their centroid (the “data point mode”).\n  Based on the dimensionality and axis semantics, as discussed in\nLayer Dynamics\n, we may prefer one mode than the other.\n  \n  We will see that the axis mode is a special case of data point mode, because we can view an axis handle as a particular “fictitious” point in the dataset.\n  Because of its simplicity, we will first introduce the axis mode.\nThe Axis Mode\nThe implied semantics of direct manipulation is that when a user drags an UI element (in this case, an axis handle), they are signaling to the system that they wished that the corresponding\n    data point had been projected to the location where the UI element was dropped, rather than where it was dragged from.\n    In our case the overall projection is a rotation (originally determined by the Grand Tour), and an arbitrary user manipulation might not necessarily generate a new projection that is also a rotation. Our goal, then, is to find a new rotation which satisfies the user request and is close to the previous state of the Grand Tour projection, so that the resulting state satisfies the user request.\n\n  In a nutshell, when user drags the\ni\nt\nh\ni^{th}\ni\nt\nh\naxis handle by\n(\nd\nx\n,\nd\ny\n)\n(dx, dy)\n(\nd\nx\n,\nd\ny\n)\n, we add them to the first two entries of the\ni\nt\nh\ni^{th}\ni\nt\nh\nrow of the Grand Tour matrix, and then perform\nGram-Schmidt orthonormalization\non the rows of the new matrix.\nRows have to be reordered such that the\ni\nt\nh\ni^{th}\ni\nt\nh\nrow is considered first in the Gram-Schmidt procedure.\nBefore we see in detail why this works well, let us formalize the process of the Grand Tour on a standard basis vector\ne\ni\ne_i\ne\ni\n​\n. \n  As shown in the diagram below,\ne\ni\ne_i\ne\ni\n​\ngoes through an orthogonal Grand Tour matrix\nG\nT\nGT\nG\nT\nto produce a rotated version of itself,\ne\ni\n~\n\\tilde{e_i}\ne\ni\n​\n~\n​\n. \n  Then,\nπ\n2\n\\pi_2\nπ\n2\n​\nis a function that keeps only the first two entries of\ne\ni\n~\n\\tilde{e_i}\ne\ni\n​\n~\n​\nand gives the 2D coordinate of the handle to be shown in the plot,\n(\nx\ni\n,\ny\ni\n)\n(x_i, y_i)\n(\nx\ni\n​\n,\ny\ni\n​\n)\n.\ne\ni\n↦\nG\nT\ne\ni\n~\n↦\nπ\n2\n(\nx\ni\n,\ny\ni\n)\ne_i \\overset{GT}{\\mapsto} \\tilde{e_i} \\overset{\\pi_2}{\\mapsto} (x_i, y_i)\ne\ni\n​\n↦\nG\nT\n​\ne\ni\n​\n~\n​\n↦\nπ\n2\n​\n​\n(\nx\ni\n​\n,\ny\ni\n​\n)\nWhen user drags an axis handle on the screen canvas, they induce a delta change\nΔ\n=\n(\nd\nx\n,\nd\ny\n)\n\\Delta = (dx, dy)\nΔ\n=\n(\nd\nx\n,\nd\ny\n)\non the\nx\ny\nxy\nx\ny\n-plane. \n  The coordinate of the handle becomes:\n(\nx\ni\n(\nn\ne\nw\n)\n,\ny\ni\n(\nn\ne\nw\n)\n)\n:\n=\n(\nx\ni\n+\nd\nx\n,\ny\ni\n+\nd\ny\n)\n(x_i^{(new)}, y_i^{(new)}) := (x_i+dx, y_i+dy)\n(\nx\ni\n(\nn\ne\nw\n)\n​\n,\ny\ni\n(\nn\ne\nw\n)\n​\n)\n:\n=\n(\nx\ni\n​\n+\nd\nx\n,\ny\ni\n​\n+\nd\ny\n)\nNote that\nx\ni\nx_i\nx\ni\n​\nand\ny\ni\ny_i\ny\ni\n​\nare the first two coordinates of the axis handle in high dimensions after the Grand Tour rotation, so a delta change on\n(\nx\ni\n,\ny\ni\n)\n(x_i, y_i)\n(\nx\ni\n​\n,\ny\ni\n​\n)\ninduces a delta change\nΔ\n~\n:\n=\n(\nd\nx\n,\nd\ny\n,\n0\n,\n0\n,\n⋯\n)\n\\tilde{\\Delta} := (dx, dy, 0, 0, \\cdots)\nΔ\n~\n:\n=\n(\nd\nx\n,\nd\ny\n,\n0\n,\n0\n,\n⋯\n)\non\ne\ni\n~\n\\tilde{e_i}\ne\ni\n​\n~\n​\n:\ne\ni\n~\n↦\nΔ\n~\ne\ni\n~\n+\nΔ\n~\n\\tilde{e_i} \\overset{\\tilde{\\Delta}}{\\mapsto} \\tilde{e_i} + \\tilde{\\Delta}\ne\ni\n​\n~\n​\n↦\nΔ\n~\n​\ne\ni\n​\n~\n​\n+\nΔ\n~\nTo find a nearby Grand Tour rotation that respects this change, first note that\ne\ni\n~\n\\tilde{e_i}\ne\ni\n​\n~\n​\nis exactly the\ni\nt\nh\ni^{th}\ni\nt\nh\nrow of orthogonal Grand Tour matrix\nG\nT\nGT\nG\nT\nRecall that the convention is that vectors are in row form and linear transformations are matrices that are multiplied on the right.\n    So\ne\ni\ne_i\ne\ni\n​\nis a row vector whose\ni\ni\ni\n-th entry is\n1\n1\n1\n(and\n0\n0\n0\ns elsewhere) and\ne\ni\n~\n:\n=\ne\ni\n⋅\nG\nT\n\\tilde{e_i} := e_i \\cdot GT\ne\ni\n​\n~\n​\n:\n=\ne\ni\n​\n⋅\nG\nT\nis the\ni\ni\ni\n-th row of\nG\nT\nGT\nG\nT\n. \n  Naturally, we want the new matrix to be the original\nG\nT\nGT\nG\nT\nwith its\ni\nt\nh\ni^{th}\ni\nt\nh\nrow replaced by\ne\ni\n~\n+\nΔ\n~\n\\tilde{e_i}+\\tilde{\\Delta}\ne\ni\n​\n~\n​\n+\nΔ\n~\n, i.e. we should add\nd\nx\ndx\nd\nx\nand\nd\ny\ndy\nd\ny\nto the\n(\ni\n,\n1\n)\n(i,1)\n(\ni\n,\n1\n)\n-th entry and\n(\ni\n,\n2\n)\n(i,2)\n(\ni\n,\n2\n)\n-th entry of\nG\nT\nGT\nG\nT\nrespectively:\nG\nT\n~\n←\nG\nT\n\\widetilde{GT} \\leftarrow GT\nG\nT\n←\nG\nT\nG\nT\n~\ni\n,\n1\n←\nG\nT\ni\n,\n1\n+\nd\nx\n\\widetilde{GT}_{i,1} \\leftarrow GT_{i,1} + dx\nG\nT\ni\n,\n1\n​\n←\nG\nT\ni\n,\n1\n​\n+\nd\nx\nG\nT\n~\ni\n,\n2\n←\nG\nT\ni\n,\n2\n+\nd\ny\n\\widetilde{GT}_{i,2} \\leftarrow GT_{i,2} + dy\nG\nT\ni\n,\n2\n​\n←\nG\nT\ni\n,\n2\n​\n+\nd\ny\nHowever,\nG\nT\n~\n\\widetilde{GT}\nG\nT\nis not orthogonal for arbitrary\n(\nd\nx\n,\nd\ny\n)\n(dx, dy)\n(\nd\nx\n,\nd\ny\n)\n.\n  In order to find an approximation to\nG\nT\n~\n\\widetilde{GT}\nG\nT\nthat is orthogonal, we apply\nGram-Schmidt orthonormalization\non the rows of\nG\nT\n~\n\\widetilde{GT}\nG\nT\n, with the\ni\nt\nh\ni^{th}\ni\nt\nh\nrow considered first in the Gram-Schmidt process:\nG\nT\n(\nn\ne\nw\n)\n:\n=\nGramSchmidt\n(\nG\nT\n~\n)\nGT^{(new)} := \\textsf{GramSchmidt}(\\widetilde{GT})\nG\nT\n(\nn\ne\nw\n)\n:\n=\nGramSchmidt\n(\nG\nT\n)\nNote that the\ni\nt\nh\ni^{th}\ni\nt\nh\nrow is normalized to a unit vector during the Gram-Schmidt, so the resulting position of the handle is\ne\ni\n~\n(\nn\ne\nw\n)\n=\nnormalize\n(\ne\ni\n~\n+\nΔ\n~\n)\n\\tilde{e_i}^{(new)} = \\textsf{normalize}(\\tilde{e_i} + \\tilde{\\Delta})\ne\ni\n​\n~\n​\n(\nn\ne\nw\n)\n=\nnormalize\n(\ne\ni\n​\n~\n​\n+\nΔ\n~\n)\nwhich may not be exactly the same as\ne\ni\n~\n+\nΔ\n~\n\\tilde{e_i}+\\tilde{\\Delta}\ne\ni\n​\n~\n​\n+\nΔ\n~\n, as the following figure shows\nHowever, for any\nΔ\n~\n\\tilde{\\Delta}\nΔ\n~\n, the norm of the difference is bounded above by\n∣\n∣\nΔ\n~\n∣\n∣\n||\\tilde{\\Delta}||\n∣\n∣\nΔ\n~\n∣\n∣\n, as the following figure proves.\n.\nThe Data Point Mode\nWe now explain how we directly manipulate data points. \n  Technically speaking, this method only considers one point at a time.\n  For a group of points, we compute their centroid and directly manipulate this single point with this method.\n  Thinking more carefully about the process in axis mode gives us a way to drag any single point.\n  Recall that in axis mode, we added user’s manipulation\nΔ\n~\n:\n=\n(\nd\nx\n,\nd\ny\n,\n0\n,\n0\n,\n⋯\n)\n\\tilde{\\Delta} := (dx, dy, 0, 0, \\cdots)\nΔ\n~\n:\n=\n(\nd\nx\n,\nd\ny\n,\n0\n,\n0\n,\n⋯\n)\nto the position of the\ni\nt\nh\ni^{th}\ni\nt\nh\naxis handle\ne\ni\n~\n\\tilde{e_i}\ne\ni\n​\n~\n​\n.\n  This induces a delta change in the\ni\nt\nh\ni^{th}\ni\nt\nh\nrow of the Grand Tour matrix\nG\nT\nGT\nG\nT\n.\n  Next, as the first step in Gram-Schmidt, we normalized this row:\nG\nT\ni\n(\nn\ne\nw\n)\n:\n=\nnormalize\n(\nG\nT\n~\ni\n)\n=\nnormalize\n(\ne\ni\n~\n+\nΔ\n~\n)\nGT_i^{(new)} := \\textsf{normalize}(\\widetilde{GT}_i) = \\textsf{normalize}(\\tilde{e_i} + \\tilde{\\Delta})\nG\nT\ni\n(\nn\ne\nw\n)\n​\n:\n=\nnormalize\n(\nG\nT\ni\n​\n)\n=\nnormalize\n(\ne\ni\n​\n~\n​\n+\nΔ\n~\n)\nThese two steps make the axis handle move from\ne\ni\n~\n\\tilde{e_i}\ne\ni\n​\n~\n​\nto\ne\ni\n~\n(\nn\ne\nw\n)\n:\n=\nnormalize\n(\ne\ni\n~\n+\nΔ\n~\n)\n\\tilde{e_i}^{(new)} := \\textsf{normalize}(\\tilde{e_i}+\\tilde{\\Delta})\ne\ni\n​\n~\n​\n(\nn\ne\nw\n)\n:\n=\nnormalize\n(\ne\ni\n​\n~\n​\n+\nΔ\n~\n)\n.\nLooking at the geometry of this movement, the “add-delta-then-normalize” on\ne\ni\n~\n\\tilde{e_i}\ne\ni\n​\n~\n​\nis equivalent to a\nrotation\nfrom\ne\ni\n~\n\\tilde{e_i}\ne\ni\n​\n~\n​\ntowards\ne\ni\n~\n(\nn\ne\nw\n)\n\\tilde{e_i}^{(new)}\ne\ni\n​\n~\n​\n(\nn\ne\nw\n)\n, illustrated in the figure below. \n  This geometric interpretation can be directly generalized to any arbitrary data point.\nThe figure shows the case in 3D, but in higher dimensional space it is essentially the same, since the two vectors\ne\ni\n~\n\\tilde{e_i}\ne\ni\n​\n~\n​\nand\ne\ni\n~\n+\nΔ\n~\n\\tilde{e_i}+\\tilde{\\Delta}\ne\ni\n​\n~\n​\n+\nΔ\n~\nonly span a 2-subspace.\n  Now we have a nice geometric intuition about direct manipulation: dragging a point induces a\nsimple rotation\nSimple rotations\nare rotations with only one\nplane of rotation\n.\nin high dimensional space.\n  This intuition is precisely how we implemented our direct manipulation on arbitrary data points, which we will specify as below.\nGeneralizing this observation from axis handle to arbitrary data point, we want to find the rotation that moves the centroid of a selected subset of data points\nc\n~\n\\tilde{c}\nc\n~\nto\nc\n~\n(\nn\ne\nw\n)\n:\n=\n(\nc\n~\n+\nΔ\n~\n)\n⋅\n∣\n∣\nc\n~\n∣\n∣\n/\n∣\n∣\nc\n~\n+\nΔ\n~\n∣\n∣\n\\tilde{c}^{(new)} := (\\tilde{c} + \\tilde{\\Delta}) \\cdot ||\\tilde{c}|| / ||\\tilde{c} + \\tilde{\\Delta}||\nc\n~\n(\nn\ne\nw\n)\n:\n=\n(\nc\n~\n+\nΔ\n~\n)\n⋅\n∣\n∣\nc\n~\n∣\n∣\n/\n∣\n∣\nc\n~\n+\nΔ\n~\n∣\n∣\nFirst, the angle of rotation can be found by their cosine similarity:\nθ\n=\narccos\n(\n⟨\nc\n~\n,\nc\n~\n(\nn\ne\nw\n)\n⟩\n∣\n∣\nc\n~\n∣\n∣\n⋅\n∣\n∣\nc\n~\n(\nn\ne\nw\n)\n∣\n∣\n)\n\\theta = \\textrm{arccos}(\n    \\frac{\\langle \\tilde{c}, \\tilde{c}^{(new)} \\rangle}{||\\tilde{c}|| \\cdot ||\\tilde{c}^{(new)}||}\n  )\nθ\n=\narccos\n(\n∣\n∣\nc\n~\n∣\n∣\n⋅\n∣\n∣\nc\n~\n(\nn\ne\nw\n)\n∣\n∣\n⟨\nc\n~\n,\nc\n~\n(\nn\ne\nw\n)\n⟩\n​\n)\nNext, to find the matrix form of the rotation, we need a convenient basis.\n  Let\nQ\nQ\nQ\nbe a change of (orthonormal) basis matrix in which the first two rows form the 2-subspace\nspan\n(\nc\n~\n,\nc\n~\n(\nn\ne\nw\n)\n)\n\\textrm{span}(\\tilde{c}, \\tilde{c}^{(new)})\nspan\n(\nc\n~\n,\nc\n~\n(\nn\ne\nw\n)\n)\n.\n  For example, we can let its first row to be\nnormalize\n(\nc\n~\n)\n\\textsf{normalize}(\\tilde{c})\nnormalize\n(\nc\n~\n)\n, second row to be its orthonormal complement\nnormalize\n(\nc\n~\n⊥\n(\nn\ne\nw\n)\n)\n\\textsf{normalize}(\\tilde{c}^{(new)}_{\\perp})\nnormalize\n(\nc\n~\n⊥\n(\nn\ne\nw\n)\n​\n)\nin\nspan\n(\nc\n~\n,\nc\n~\n(\nn\ne\nw\n)\n)\n\\textrm{span}(\\tilde{c}, \\tilde{c}^{(new)})\nspan\n(\nc\n~\n,\nc\n~\n(\nn\ne\nw\n)\n)\n, and the remaining rows complete the whole space:\nc\n~\n⊥\n(\nn\ne\nw\n)\n:\n=\nc\n~\n−\n∣\n∣\nc\n~\n∣\n∣\n⋅\nc\no\ns\nθ\nc\n~\n(\nn\ne\nw\n)\n∣\n∣\nc\n~\n(\nn\ne\nw\n)\n∣\n∣\n\\tilde{c}^{(new)}_{\\perp} \n  := \\tilde{c} - ||\\tilde{c}|| \\cdot cos \\theta \\frac{\\tilde{c}^{(new)}}{||\\tilde{c}^{(new)}||}\nc\n~\n⊥\n(\nn\ne\nw\n)\n​\n:\n=\nc\n~\n−\n∣\n∣\nc\n~\n∣\n∣\n⋅\nc\no\ns\nθ\n∣\n∣\nc\n~\n(\nn\ne\nw\n)\n∣\n∣\nc\n~\n(\nn\ne\nw\n)\n​\nQ\n:\n=\n[\n⋯\nnormalize\n(\nc\n~\n)\n⋯\n⋯\nnormalize\n(\nc\n~\n⊥\n(\nn\ne\nw\n)\n)\n⋯\nP\n]\nQ :=\n    \\begin{bmatrix}\n    \\cdots \\textsf{normalize}(\\tilde{c}) \\cdots \\\\\n    \\cdots \\textsf{normalize}(\\tilde{c}^{(new)}_{\\perp}) \\cdots \\\\\n    P\n    \\end{bmatrix}\nQ\n:\n=\n⎣\n⎡\n​\n⋯\nnormalize\n(\nc\n~\n)\n⋯\n⋯\nnormalize\n(\nc\n~\n⊥\n(\nn\ne\nw\n)\n​\n)\n⋯\nP\n​\n⎦\n⎤\n​\nwhere\nP\nP\nP\ncompletes the remaining space.\n\n  Making use of\nQ\nQ\nQ\n, we can find the matrix that rotates the plane\nspan\n(\nc\n~\n,\nc\n~\n(\nn\ne\nw\n)\n)\n\\textrm{span}(\\tilde{c}, \\tilde{c}^{(new)})\nspan\n(\nc\n~\n,\nc\n~\n(\nn\ne\nw\n)\n)\nby the angle\nθ\n\\theta\nθ\n:\nρ\n=\nQ\nT\n[\ncos\nθ\nsin\nθ\n0\n0\n⋯\n−\nsin\nθ\ncos\nθ\n0\n0\n⋯\n0\n0\n⋮\n⋮\nI\n]\nQ\n=\n:\nQ\nT\nR\n1\n,\n2\n(\nθ\n)\nQ\n\\rho = Q^T\n    \\begin{bmatrix}\n    \\cos \\theta& \\sin \\theta&  0&  0& \\cdots\\\\\n    -\\sin \\theta& \\cos \\theta&  0&  0& \\cdots\\\\\n    0& 0&  \\\\ \n    \\vdots& \\vdots& & I& \\\\\n    \\end{bmatrix}\n    Q\n    =: Q^T R_{1,2}(\\theta) Q\nρ\n=\nQ\nT\n⎣\n⎢\n⎢\n⎢\n⎢\n⎡\n​\ncos\nθ\n−\nsin\nθ\n0\n⋮\n​\nsin\nθ\ncos\nθ\n0\n⋮\n​\n0\n0\n​\n0\n0\nI\n​\n⋯\n⋯\n​\n⎦\n⎥\n⎥\n⎥\n⎥\n⎤\n​\nQ\n=\n:\nQ\nT\nR\n1\n,\n2\n​\n(\nθ\n)\nQ\nThe new Grand Tour matrix is the matrix product of the original\nG\nT\nGT\nG\nT\nand\nρ\n\\rho\nρ\n:\nG\nT\n(\nn\ne\nw\n)\n:\n=\nG\nT\n⋅\nρ\nGT^{(new)} := GT \\cdot \\rho\nG\nT\n(\nn\ne\nw\n)\n:\n=\nG\nT\n⋅\nρ\nNow we should be able to see the connection between axis mode and data point mode.\n  In data point mode, finding\nQ\nQ\nQ\ncan be done by Gram-Schmidt: Let the first basis be\nc\n~\n\\tilde{c}\nc\n~\n, find the orthogonal component of\nc\n~\n(\nn\ne\nw\n)\n\\tilde{c}^{(new)}\nc\n~\n(\nn\ne\nw\n)\nin\nspan\n(\nc\n~\n,\nc\n~\n(\nn\ne\nw\n)\n)\n\\textrm{span}(\\tilde{c}, \\tilde{c}^{(new)})\nspan\n(\nc\n~\n,\nc\n~\n(\nn\ne\nw\n)\n)\n, repeatedly take a random vector, find its orthogonal component to the span of the current basis vectors and add it to the basis set. \n  In axis mode, the\ni\nt\nh\ni^{th}\ni\nt\nh\n-row-first Gram-Schmidt does the rotation and change of basis in one step.\nLayer Transitions\nReLU Layers\nWhen the\nl\nt\nh\nl^{th}\nl\nt\nh\nlayer is a ReLU function, the output activation is\nX\nl\n=\nR\ne\nL\nU\n(\nX\nl\n−\n1\n)\nX^{l} = ReLU(X^{l-1})\nX\nl\n=\nR\ne\nL\nU\n(\nX\nl\n−\n1\n)\n. Since ReLU does not change the dimensionality and the function is taken coordinate wise, we can animate the transition by a simple linear interpolation: for a time parameter\nt\n∈\n[\n0\n,\n1\n]\nt \\in [0,1]\nt\n∈\n[\n0\n,\n1\n]\n,\nX\n(\nl\n−\n1\n)\n→\nl\n(\nt\n)\n:\n=\n(\n1\n−\nt\n)\nX\nl\n−\n1\n+\nt\nX\nl\nX^{(l-1) \\to l}(t) := (1-t) X^{l-1} + t X^{l}\nX\n(\nl\n−\n1\n)\n→\nl\n(\nt\n)\n:\n=\n(\n1\n−\nt\n)\nX\nl\n−\n1\n+\nt\nX\nl\nLinear Layers\nTransitions between linear layers can seem complicated, but as we will show, this comes from choosing mismatching bases on either side of the transition. \n  If\nX\nl\n=\nX\nl\n−\n1\nM\nX^{l} = X^{l-1} M\nX\nl\n=\nX\nl\n−\n1\nM\nwhere\nM\n∈\nR\nm\n×\nn\nM \\in \\mathbb{R}^{m \\times n}\nM\n∈\nR\nm\n×\nn\nis the matrix of a linear transformation, then it has a singular value decomposition (SVD):\nM\n=\nU\nΣ\nV\nT\nM = U \\Sigma V^T\nM\n=\nU\nΣ\nV\nT\nwhere\nU\n∈\nR\nm\n×\nm\nU \\in \\mathbb{R}^{m \\times m}\nU\n∈\nR\nm\n×\nm\nand\nV\nT\n∈\nR\nn\n×\nn\nV^T \\in \\mathbb{R}^{n \\times n}\nV\nT\n∈\nR\nn\n×\nn\nare orthogonal,\nΣ\n∈\nR\nm\n×\nn\n\\Sigma \\in \\mathbb{R}^{m \\times n}\nΣ\n∈\nR\nm\n×\nn\nis diagonal.\n  For arbitrary\nU\nU\nU\nand\nV\nT\nV^T\nV\nT\n, the transformation on\nX\nl\n−\n1\nX^{l-1}\nX\nl\n−\n1\nis a composition of a rotation (\nU\nU\nU\n), scaling (\nΣ\n\\Sigma\nΣ\n) and another rotation (\nV\nT\nV^T\nV\nT\n), which can look complicated. \n  However, consider the problem of relating the Grand Tour view of layer\nX\nl\nX^{l}\nX\nl\nto that of layer\nX\nl\n+\n1\nX^{l+1}\nX\nl\n+\n1\n. The Grand Tour has a single parameter that represents the current rotation of the dataset. Since our goal is to keep the transition consistent, we notice that\nU\nU\nU\nand\nV\nT\nV^T\nV\nT\nhave essentially no significance - they are just rotations to the view that can be exactly “canceled” by changing the rotation parameter of the Grand Tour in either layer.\n  Hence, instead of showing\nM\nM\nM\n, we seek for the transition to animate only the effect of\nΣ\n\\Sigma\nΣ\n.\nΣ\n\\Sigma\nΣ\nis a coordinate-wise scaling, so we can animate it similar to the ReLU after the proper change of basis.\n  Given\nX\nl\n=\nX\nl\n−\n1\nU\nΣ\nV\nT\nX^{l} = X^{l-1} U \\Sigma V^T\nX\nl\n=\nX\nl\n−\n1\nU\nΣ\nV\nT\n, we have\n(\nX\nl\nV\n)\n=\n(\nX\nl\n−\n1\nU\n)\nΣ\n(X^{l}V) = (X^{l-1}U)\\Sigma\n(\nX\nl\nV\n)\n=\n(\nX\nl\n−\n1\nU\n)\nΣ\nFor a time parameter\nt\n∈\n[\n0\n,\n1\n]\nt \\in [0,1]\nt\n∈\n[\n0\n,\n1\n]\n,\nX\n(\nl\n−\n1\n)\n→\nl\n(\nt\n)\n:\n=\n(\n1\n−\nt\n)\n(\nX\nl\n−\n1\nU\n)\n+\nt\n(\nX\nl\nV\n)\n=\n(\n1\n−\nt\n)\n(\nX\nl\n−\n1\nU\n)\n+\nt\n(\nX\nl\n−\n1\nU\nΣ\n)\nX^{(l-1) \\to l}(t) := (1-t) (X^{l-1}U) + t (X^{l}V) = (1-t) (X^{l-1}U) + t (X^{l-1} U \\Sigma)\nX\n(\nl\n−\n1\n)\n→\nl\n(\nt\n)\n:\n=\n(\n1\n−\nt\n)\n(\nX\nl\n−\n1\nU\n)\n+\nt\n(\nX\nl\nV\n)\n=\n(\n1\n−\nt\n)\n(\nX\nl\n−\n1\nU\n)\n+\nt\n(\nX\nl\n−\n1\nU\nΣ\n)\nConvolutional Layers\nConvolutional layers can be represented as special linear layers.\n  With a change of representation, we can animate a convolutional layer like the previous section.\n  For 2D convolutions this change of representation involves flattening the input and output, and repeating the kernel pattern in a sparse matrix\nM\n∈\nR\nm\n×\nn\nM \\in \\mathbb{R}^{m \\times n}\nM\n∈\nR\nm\n×\nn\n, where\nm\nm\nm\nand\nn\nn\nn\nare the dimensionalities of the input and output respectively.\n  This change of representation is only practical for a small dimensionality (e.g. up to 1000), since we need to solve SVD for linear layers.\n  However, the singular value decomposition of multi-channel 2D convolutions can be computed efficiently\n, which can be then be directly used for alignment.\nMax-pooling Layers\nAnimating max-pooling layers is nontrivial because max-pooling is neither linear\nA max-pooling layer is piece-wise linear\nor coordinate-wise.\n  We replace it by average-pooling and scaling by the ratio of the average to the max.\n  We compute the matrix form of average-pooling and use its SVD to align the view before and after this layer. \n  Functionally, our operations have equivalent results to max-pooling, but this introduces\n  unexpected artifacts. For example, the max-pooling version of the vector\n[\n0\n.\n9\n,\n0\n.\n9\n,\n0\n.\n9\n,\n1\n.\n0\n]\n[0.9, 0.9, 0.9, 1.0]\n[\n0\n.\n9\n,\n0\n.\n9\n,\n0\n.\n9\n,\n1\n.\n0\n]\nshould “give no credit” to the\n0\n.\n9\n0.9\n0\n.\n9\nentries; our implementation, however, will\n  attribute about 25% of the result in the downstream layer to each those coordinates.\nConclusion\nAs powerful as t-SNE and UMAP are, they often fail to offer the correspondences we need, and such correspondences can come, surprisingly, from relatively simple methods like the Grand Tour. The Grand Tour method we presented is particularly useful when direct manipulation from the user is available or desirable.\n  We believe that it might be possible to design methods that highlight the best of both worlds, using non-linear dimensionality reduction to create intermediate, relatively low-dimensional representations of the activation layers, and using the Grand Tour and direct manipulation to compute the final projection.\nAcknowledgments\nThe utility code for WebGL under js/lib/webgl_utils/ are adapted from Angel’s computer graphics book supplementary\nhere\n.\nDiscussion and Review\nReview 1 - Anonymous\nReview 2 - Anonymous\nReview 3 - Anonymous\nReferences\nThe grand tour: a tool for viewing multidimensional data\n[link]\nAsimov, D., 1985. SIAM journal on scientific and statistical computing, Vol 6(1), pp. 128--143. SIAM.\nVisualizing data using t-SNE\n[PDF]\nMaaten, L.v.d. and Hinton, G., 2008. Journal of machine learning research, Vol 9(Nov), pp. 2579--2605.\nUmap: Uniform manifold approximation and projection for dimension reduction\n[PDF]\nMcInnes, L. and Healy, J., 2018. arXiv preprint arXiv:1802.03426.\nIntriguing properties of neural networks\nSzegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. and Fergus, R., 2013. arXiv preprint arXiv:1312.6199.\nImageNet Large Scale Visual Recognition Challenge\nRussakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C. and Fei-Fei, L., 2015. International Journal of Computer Vision (IJCV), Vol 115(3), pp. 211-252.\nDOI: 10.1007/s11263-015-0816-y\nThe mythos of model interpretability\nLipton, Z.C., 2016. arXiv preprint arXiv:1606.03490.\nVisualizing dataflow graphs of deep learning models in tensorflow\nWongsuphasawat, K., Smilkov, D., Wexler, J., Wilson, J., Mane, D., Fritz, D., Krishnan, D., Viegas, F.B. and Wattenberg, M., 2018. IEEE transactions on visualization and computer graphics, Vol 24(1), pp. 1--12. IEEE.\nAn algebraic process for visualization design\nKindlmann, G. and Scheidegger, C., 2014. IEEE transactions on visualization and computer graphics, Vol 20(12), pp. 2181--2190. IEEE.\nFeature visualization\n[link]\nOlah, C., Mordvintsev, A. and Schubert, L., 2017. Distill, Vol 2(11), pp. e7.\nMNIST handwritten digit database\n[link]\nLeCun, Y. and Cortes, C., 2010.\nFashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms\n[link]\nXiao, H., Rasul, K. and Vollgraf, R., 2017.\nLearning multiple layers of features from tiny images\n[HTML]\nKrizhevsky, A., Hinton, G. and others,, 2009.\nRectified linear units improve restricted boltzmann machines\n[PDF]\nNair, V. and Hinton, G.E., 2010. Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807--814.\nVisualizing time-dependent data using dynamic t-SNE\n[PDF]\nRauber, P.E., Falcao, A.X. and Telea, A.C., 2016. Proc. EuroVis Short Papers, Vol 2(5).\nHow to use t-sne effectively\n[link]\nWattenberg, M., Viegas, F. and Johnson, I., 2016. Distill, Vol 1(10), pp. e2.\nWe Recommend a Singular Value Decomposition\n[link]\nAustin, D., 2009.\nThe singular values of convolutional layers\n[PDF]\nSedghi, H., Gupta, V. and Long, P.M., 2018. arXiv preprint arXiv:1805.10408.\nExplaining and harnessing adversarial examples\nGoodfellow, I.J., Shlens, J. and Szegedy, C., 2014. arXiv preprint arXiv:1412.6572.\nAnimation, small multiples, and the effect of mental map preservation in dynamic graphs\nArchambault, D., Purchase, H. and Pinaud, B., 2010. IEEE Transactions on Visualization and Computer Graphics, Vol 17(4), pp. 539--552. IEEE.\nAnimation: can it facilitate?\nTversky, B., Morrison, J.B. and Betrancourt, M., 2002. International journal of human-computer studies, Vol 57(4), pp. 247--262. Elsevier.\nHighway networks\n[PDF]\nSrivastava, R.K., Greff, K. and Schmidhuber, J., 2015. arXiv preprint arXiv:1505.00387.\nGoing deeper with convolutions\n[PDF]\nSzegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V. and Rabinovich, A., 2015. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1--9.\nUpdates and Corrections\nIf you see mistakes or want to suggest changes, please\ncreate an issue on GitHub\n.\nReuse\nDiagrams and text are licensed under Creative Commons Attribution\nCC-BY 4.0\nwith the\nsource available on GitHub\n, unless noted otherwise. The figures that have been reused from other sources don’t fall under this license and can be recognized by a note in their caption: “Figure from …”.\nCitation\nFor attribution in academic contexts, please cite this work as\nLi, et al., \"Visualizing Neural Networks with the Grand Tour\", Distill, 2020.\nBibTeX citation\n@article{li2020visualizing,\n  author = {Li, Mingwei and Zhao, Zhenge and Scheidegger, Carlos},\n  title = {Visualizing Neural Networks with the Grand Tour},\n  journal = {Distill},\n  year = {2020},\n  note = {https://distill.pub/2020/grand-tour},\n  doi = {10.23915/distill.00025}\n}\nDistill\nis dedicated to clear explanations of machine learning\nAbout\nSubmit\nPrize\nArchive\nRSS\nGitHub\nTwitter\nISSN 2476-0757",
  "metadata": {
    "title": "Visualizing Neural Networks with the Grand Tour",
    "description": null,
    "keywords": null,
    "author": null,
    "og:type": "article",
    "og:title": "Visualizing Neural Networks with the Grand Tour",
    "og:description": "By focusing on linear dimensionality reduction, we show how to visualize many dynamic phenomena in neural networks.",
    "og:url": "https://distill.pub/2020/grand-tour",
    "og:image": "https://distill.pub/2020/grand-tour/thumbnail.jpg",
    "og:locale": "en_US",
    "og:site_name": "Distill",
    "twitter:card": "summary_large_image",
    "twitter:title": "Visualizing Neural Networks with the Grand Tour",
    "twitter:description": "By focusing on linear dimensionality reduction, we show how to visualize many dynamic phenomena in neural networks.",
    "twitter:url": "https://distill.pub/2020/grand-tour",
    "twitter:image": "https://distill.pub/2020/grand-tour/thumbnail.jpg",
    "twitter:image:width": "560",
    "twitter:image:height": "295",
    "article:published": "2020-03-16",
    "article:created": "2020-03-16",
    "article:modified": "2020-05-27T16:57:31.000Z",
    "article:author": "Carlos Scheidegger"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/02c83ff36532/code/body-000.txt",
      "chars": 77,
      "language": "text"
    },
    {
      "file": "attachments/02c83ff36532/code/body-001.txt",
      "chars": 278,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://distill.pub/2020/grand-tour/figs/mnist.png",
      "bytes": 70938,
      "reused": false,
      "local_path": "attachments/02c83ff36532/images/body-bce80faf40e5243a5005.png",
      "src": "https://distill.pub/2020/grand-tour/figs/mnist.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/grand-tour/figs/fashion-mnist.png",
      "bytes": 470008,
      "reused": false,
      "local_path": "attachments/02c83ff36532/images/body-91cb10ab2284ce676b58.png",
      "src": "https://distill.pub/2020/grand-tour/figs/fashion-mnist.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/grand-tour/figs/cifar-10.png",
      "bytes": 982409,
      "reused": false,
      "local_path": "attachments/02c83ff36532/images/body-7f1f0efc6285ea8349d1.png",
      "src": "https://distill.pub/2020/grand-tour/figs/cifar-10.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/grand-tour/figs/conv.gif",
      "bytes": 400761,
      "reused": false,
      "local_path": "attachments/02c83ff36532/images/body-592b69d20c7a99ccdd0d.gif",
      "src": "https://distill.pub/2020/grand-tour/figs/conv.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/grand-tour/figs/relu.png",
      "bytes": 24843,
      "reused": false,
      "local_path": "attachments/02c83ff36532/images/body-6e23d170ebf8033d182e.png",
      "src": "https://distill.pub/2020/grand-tour/figs/relu.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/grand-tour/figs/softmax.png",
      "bytes": 122211,
      "reused": false,
      "local_path": "attachments/02c83ff36532/images/body-a1c6c499457f98fc8c35.png",
      "src": "https://distill.pub/2020/grand-tour/figs/softmax.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/grand-tour/figs/sigmoid.png",
      "bytes": 13372,
      "reused": false,
      "local_path": "attachments/02c83ff36532/images/body-73aaf94fa6ec945c327e.png",
      "src": "https://distill.pub/2020/grand-tour/figs/sigmoid.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/grand-tour/figs/maxpool.gif",
      "bytes": 250480,
      "reused": false,
      "local_path": "attachments/02c83ff36532/images/body-91589e5678dceb6a699a.gif",
      "src": "https://distill.pub/2020/grand-tour/figs/maxpool.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/grand-tour/figs/direct-manipulation-rotation-2d-proof.png",
      "bytes": 84246,
      "reused": false,
      "local_path": "attachments/02c83ff36532/images/body-089816015d168ed96884.png",
      "src": "https://distill.pub/2020/grand-tour/figs/direct-manipulation-rotation-2d-proof.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/grand-tour/figs/direct-manipulation-rotation-2d.png",
      "bytes": 103504,
      "reused": false,
      "local_path": "attachments/02c83ff36532/images/body-a723ac98b20350f46307.png",
      "src": "https://distill.pub/2020/grand-tour/figs/direct-manipulation-rotation-2d.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/grand-tour/figs/direct-manipulation-rotation-3d.png",
      "bytes": 143120,
      "reused": false,
      "local_path": "attachments/02c83ff36532/images/body-da5354a5cfcdacd1c102.png",
      "src": "https://distill.pub/2020/grand-tour/figs/direct-manipulation-rotation-3d.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/grand-tour/figs/direct-manipulation-rotation-2d-perp.png",
      "bytes": 132932,
      "reused": false,
      "local_path": "attachments/02c83ff36532/images/body-25cf63bb621664be3188.png",
      "src": "https://distill.pub/2020/grand-tour/figs/direct-manipulation-rotation-2d-perp.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "http://hdc.cs.arizona.edu/~mwli/",
      "label": "Mingwei Li",
      "type": "a"
    },
    {
      "href": "https://www.cs.arizona.edu/",
      "label": "University of Arizona",
      "type": "a"
    },
    {
      "href": "https://zhengezhao.wordpress.com/",
      "label": "Zhenge Zhao",
      "type": "a"
    },
    {
      "href": "https://cscheid.net/",
      "label": "Carlos Scheidegger",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.23915/distill.00025",
      "label": "10.23915/distill.00025",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/File:MnistExamples.png",
      "label": "https://en.wikipedia.org/wiki/File:MnistExamples.png",
      "type": "a"
    },
    {
      "href": "https://towardsdatascience.com/multi-label-classification-and-class-activation-map-on-fashion-mnist-1454f09f5925",
      "label": "https://towardsdatascience.com/multi-label-classification-and-class-activation-map-on-fashion-mnist-1454f09f5925",
      "type": "a"
    },
    {
      "href": "https://www.cs.toronto.edu/~kriz/cifar.html",
      "label": "https://www.cs.toronto.edu/~kriz/cifar.html",
      "type": "a"
    },
    {
      "href": "https://towardsdatascience.com/gentle-dive-into-math-behind-convolutional-neural-networks-79a07dd44cf9",
      "label": "https://towardsdatascience.com/gentle-dive-into-math-behind-convolutional-neural-networks-79a07dd44cf9",
      "type": "a"
    },
    {
      "href": "https://github.com/vdumoulin/conv_arithmetic",
      "label": "Convolution arithmetic",
      "type": "a"
    },
    {
      "href": "https://pytorch.org/docs/stable/nn.html",
      "label": "https://pytorch.org/docs/stable/nn.html#relu",
      "type": "a"
    },
    {
      "href": "https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/",
      "label": "https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Sigmoid_function",
      "label": "https://en.wikipedia.org/wiki/Sigmoid_function",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Gram%E2%80%93Schmidt_process",
      "label": "Gram-Schmidt orthonormalization",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Rotations_in_4-dimensional_Euclidean_space",
      "label": "Simple rotations",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Plane_of_rotation",
      "label": "plane of rotation",
      "type": "a"
    },
    {
      "href": "https://www.cs.unm.edu/~angel/BOOK/INTERACTIVE_COMPUTER_GRAPHICS/SEVENTH_EDITION/",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--grand-tour/issues/6",
      "label": "Review 1 - Anonymous",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--grand-tour/issues/7",
      "label": "Review 2 - Anonymous",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--grand-tour/issues/8",
      "label": "Review 3 - Anonymous",
      "type": "a"
    },
    {
      "href": "https://epubs.siam.org/doi/pdf/10.1137/0906011",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1802.03426.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.1007/s11263-015-0816-y",
      "label": "DOI: 10.1007/s11263-015-0816-y",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2017/feature-visualization/",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "http://yann.lecun.com/exdb/mnist/",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "https://github.com/zalandoresearch/fashion-mnist",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "https://www.cs.toronto.edu/~hinton/absps/reluICML.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://www.cs.rug.nl/~alext/PAPERS/EuroVis16/paper.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2016/misread-tsne/",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "http://www.ams.org/publicoutreach/feature-column/fcarc-svd",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1805.10408.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1505.00387.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://www.cs.unc.edu/~wliu/papers/GoogLeNet.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--grand-tour/issues/new",
      "label": "create an issue on GitHub",
      "type": "a"
    },
    {
      "href": "https://creativecommons.org/licenses/by/4.0/",
      "label": "CC-BY 4.0",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--grand-tour",
      "label": "source available on GitHub",
      "type": "a"
    },
    {
      "href": "https://distill.pub/",
      "label": "Distill",
      "type": "a"
    },
    {
      "href": "https://distill.pub/about/",
      "label": "About",
      "type": "a"
    },
    {
      "href": "https://distill.pub/journal/",
      "label": "Submit",
      "type": "a"
    },
    {
      "href": "https://distill.pub/prize/",
      "label": "Prize",
      "type": "a"
    },
    {
      "href": "https://distill.pub/archive/",
      "label": "Archive",
      "type": "a"
    },
    {
      "href": "https://distill.pub/rss.xml",
      "label": "RSS",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub",
      "label": "GitHub",
      "type": "a"
    },
    {
      "href": "https://twitter.com/distillpub",
      "label": "Twitter",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:54:14Z",
  "source_url": "https://distill.pub/2020/grand-tour/",
  "papers": [
    {
      "status": "ok",
      "url": "http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf",
      "bytes": 3716712,
      "reused": true,
      "local_path": "papers/repair/d7c9562cd8ea8254cacc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1802.03426.pdf",
      "bytes": 7880321,
      "reused": false,
      "local_path": "papers/repair/d1bab6db0ca18260ba72.pdf"
    },
    {
      "status": "ok",
      "url": "https://www.cs.toronto.edu/~hinton/absps/reluICML.pdf",
      "bytes": 668384,
      "reused": false,
      "local_path": "papers/repair/d985f50a081982918a9e.pdf"
    },
    {
      "status": "http_404",
      "url": "http://www.cs.rug.nl/~alext/PAPERS/EuroVis16/paper.pdf",
      "retried_at": "2026-10-20T17:39:26Z"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1805.10408.pdf",
      "bytes": 1087693,
      "reused": false,
      "local_path": "papers/repair/f9ddd7afb78ac8f609d0.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1505.00387.pdf",
      "bytes": 927485,
      "reused": false,
      "local_path": "papers/repair/4af23ed4e2c519bec394.pdf"
    },
    {
      "status": "http_404",
      "url": "https://www.cs.unc.edu/~wliu/papers/GoogLeNet.pdf",
      "retried_at": "2026-10-20T17:39:19Z"
    }
  ],
  "actual_video_files": [],
  "body_chars": 56128,
  "expected_body_images": 12,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "http_404",
  "raw_html": "attachments/02c83ff36532/pages/source.html",
  "local_html": "attachments/02c83ff36532/pages/article.html",
  "body_base_path": "attachments/02c83ff36532/pages/",
  "body_markdown_path": "attachments/02c83ff36532/pages/article.md"
}