{
  "source_file": "0930",
  "source_platform": "技术平台",
  "author": "Distill",
  "title_listed": "Understanding RL Vision",
  "keywords": "强化学习,视觉,理解",
  "domain": "机器学习",
  "content_type": "图解教程",
  "is_paid_flag": "否",
  "url": "https://distill.pub/2020/understanding-rl-vision/",
  "publish_date": "2020-08",
  "corpus_stage": "强化学习/对齐",
  "notes": "理解强化学习视觉",
  "id": "181224c20509",
  "collected_at": "2026-10-02T17:54:45",
  "title": "Understanding RL Vision",
  "body_markdown": "Distill\nUnderstanding RL Vision\nWith diverse environments, we can analyze, diagnose and edit deep reinforcement learning models using attribution.\nObservation\n(video game still)\nPositive attribution\n(good news)\nNegative attribution\n(bad news)\nAttribution from a hidden layer to the value function, showing what features of the observation (left) are used to predict success (middle) and failure (right). Applying dimensionality reduction (NMF) yields features that detect various in-game objects.\nCoin\nEnemy\nBuzzsaw\nAuthors\nAffiliations\nJacob Hilton\nOpenAI\nNick Cammarata\nOpenAI\nShan Carter\nObservable\nGabriel Goh\nOpenAI\nChris Olah\nOpenAI\nPublished\nNov. 17, 2020\nDOI\n10.23915/distill.00029\nIn this article, we apply interpretability techniques to a reinforcement learning (RL) model trained to play the video game CoinRun\n. Using attribution\ncombined with dimensionality reduction as in\n, we build an interface for exploring the objects detected by the model, and how they influence its value function and policy. We leverage this interface in several ways.\nDissecting failure\n.\nWe perform a step-by-step analysis of the agent’s behavior in cases where it failed to achieve the maximum reward, allowing us to understand what went wrong, and why. For example, one case of failure was caused by an obstacle being temporarily obscured from view.\nHallucinations\n.\nWe find situations when the model “hallucinated” a feature not present in the observation, thereby explaining inaccuracies in the model’s value function. These were brief enough that they did not affect the agent’s behavior.\nModel editing\n.\nWe hand-edit the weights of the model to blind the agent to certain hazards, without otherwise changing the agent’s behavior. We verify the effects of these edits by checking which hazards cause the new agents to fail. Such editing is only made possible by our previous analysis, and thus provides a quantitative validation of this analysis.\nOur results depend on levels in CoinRun being procedurally-generated, leading us to formulate a\ndiversity hypothesis\nfor interpretability. If it is correct, then we can expect RL models to become more interpretable as the environments they are trained on become more diverse. We provide evidence for our hypothesis by measuring the relationship between interpretability and generalization.\nFinally, we provide a thorough\ninvestigation\nof several interpretability techniques in the context of RL vision, and pose a number of\nquestions\nfor further research.\nOur CoinRun model\nCoinRun is a side-scrolling platformer in which the agent must dodge enemies and other traps and collect the coin at the end of the level.\nOur trained model playing CoinRun.\nLeft\n: full resolution.\nRight\n: 64x64 RGB observations given to the model.\nCoinRun is procedurally-generated, meaning that each new level encountered by the agent is randomly generated from scratch. This incentivizes the model to learn how to spot the different kinds of objects in the game, since it cannot get away with simply memorizing a small number of specific trajectories\n.\nWe use the original version of CoinRun\n, not the version from Procgen Benchmark\n, which is slightly different. To play CoinRun yourself, please follow the instructions\nhere\n.\nHere are some examples of the objects used, along with walls and floors, to generate CoinRun levels.\nFull resolution\nModel resolution\nThe agent, in mid air (left) and about to jump (right). The agent also appears in beige, blue and green.\nCoins, which have to be collected.\nStationary buzzsaw obstacles, which must be dodged.\nEnemies, which must be dodged, moving left and right. There are several alternative sprites, all with white trails.\nBoxes, which the agent can both move past and land on top of.\nLava at the bottom of a chasm.\nThe velocity info painted into the top left of each observation, indicating the agent’s horizontal and vertical velocities.\nPainting in the velocity info allows the model to infer the agent’s motion from a single frame. The shade of the left square indicates the agent’s horizontal velocity (black for left at full speed, white for right at full speed), and the shade of the right square indicates the agent’s vertical velocity (black for down at full speed, white for up at full speed). In this example, the agent is moving forward and about to land (and is thus moving right and down).\nThere are 9 actions available to the agent in CoinRun:\n←\n→\nLeft and right change the agent’s horizontal velocity. They still work while the agent is in mid-air, but have less of an effect.\n↓\nDown cancels a jump if used immediately after up, and steps the agent down from boxes.\n↑\n↖\n↗\nUp causes the agent to jump after the next non-up action. Diagonal directions have the same effect as both component directions combined.\nA\nB\nC\nA, B and C do nothing.\nThe original version of CoinRun only has 1 “do nothing” action, but our version ended up with 3 when “A” and “B” actions were added to be used in other games. For consistency, we have relabeled the original “do nothing” action as “C”.\nWe trained a convolutional neural network on CoinRun for around 2 billion timesteps, using PPO\n, an actor-critic algorithm.\nWe used the standard PPO hyperparameters for CoinRun\n, except that we used twice as many copies of the environment per worker and twice and many workers. The effect of these changes was to increase the effective batch size, which seemed to be necessary to reach the same performance with our smaller architecture.\nThe architecture of our network is described in\nAppendix C\n. We used a non-recurrent network, to avoid any need to visualize multiple frames at once. Thus our model observes a single downsampled 64x64 image, and outputs a value function (an estimate of the total future time-discounted reward) and a policy (a probability distribution over the actions, from which the next action is sampled).\nobservation\nCNN\nvalue function\nlogits\nsoftmax\npolicy\nSchematic of a typical non-recurrent convolutional actor-critic model, such as ours.\nSince the only available reward is a fixed bonus for collecting the coin, the value function estimates the time-discounted\nWe use a discount rate of 0.999 per timestep.\nprobability that the agent will successfully complete the level.\nModel analysis\nHaving trained a strong RL agent, we were curious to see what it had learned. Following\n, we developed an interface for examining trajectories of the agent playing the game. This incorporates attribution from a hidden layer that recognizes objects, which serves to highlight objects that positively or negatively influence a particular network output. By applying dimensionality reduction, we obtain attribution vectors whose components correspond to different types of object, which we indicate using different colors.\nHere is our interface for a typical trajectory, with the value function as the network output. It reveals the model using obstacles, coins, enemies and more to compute the value function.\nDissecting failure\nOur fully-trained model fails to complete around 1 in every 200 levels. We explored a few of these failures using our interface, and found that we were usually able to understand why they occurred.\nThe failure often boils down to the fact that the model has no memory, and must therefore choose its action based only on the current observation. It is also common for some unlucky sampling of actions from the agent’s policy to be partly responsible.\nHere are some cherry-picked examples of failures, carefully analyzed step-by-step.\nBuzzsaw obstacle obscured by enemy\nStepping down to avoid jumping\nLanding platform moving off-screen\nThe agent moves too far to the right while in mid-air as a result of a buzzsaw obstacle being temporarily hidden from view by a moving enemy. The buzzsaw comes back into view, but too late to avoid a collision.\nThe agent presses down in a bid to delay a jump. This causes the agent to inadvertently step down from a box and onto an enemy.\nThe agent fails to move far enough to the right while in mid-air, as a result of the platform where it was intending to land moving below the field of view.\nPrev\nStart\n►\nNext\nEnd\nHallucinations\nWe searched for errors in the model using generalized advantage estimation (GAE)\n,\nWe use the same GAE hyperparameters as in training, namely\nγ\n=\n0\n.\n9\n9\n9\n\\gamma=0.999\nγ\n=\n0\n.\n9\n9\n9\nand\nλ\n=\n0\n.\n9\n5\n\\lambda=0.95\nλ\n=\n0\n.\n9\n5\n.\nwhich measures how successful each action turned out relative to the agent’s expectations. An unusually high or low GAE indicates that either something unexpected occurred, or the agent’s expectations were miscalibrated. Filtering for such timesteps can therefore find problems with the value function or policy.\nUsing our interface, we found a couple of cases in which the model “hallucinated” a feature not present in the observation, causing the value function to spike.\nCoin hallucination\nBuzzsaw hallucination\nAt one point the value function spiked upwards from 95% to 98% for a single timestep. This was due to a curved yellow-brown shape in the background, which happened to appear next to a wall, being mistaken for a coin.\nAt another point the value function spiked downwards from 94% to 85% for a single timestep. This was due to the agent, colored in gray-blue and crouching against a mottled background, being mistaken for a buzzsaw obstacle. An actual buzzsaw was also present in the observation, but the main effect was from the misjudged agent, as shown by the larger red circle around the agent (hover over the first legend item to isolate).\nModel editing\nOur analysis so far has been mostly qualitative. To quantitatively validate our analysis, we hand-edited the model to make the agent blind to certain features identified by our interface: buzzsaw obstacles in one case, and left-moving enemies in another. Our method for this can be thought of as a primitive form of\ncircuit\n-editing\n, and we explain it in detail in\nAppendix A\n.\nWe evaluated each edit by measuring the percentage of levels that the new agent failed to complete, broken down by the object that the agent collided with to cause the failure. Our results show that our edits were successful and targeted, with no statistically measurable effects on the agent’s other abilities.\nThe data for this plot are as follows.\nPercentage of levels failed due to: buzzsaw obstacle / enemy moving left / enemy moving right / multiple or other:\n- Original model: 0.37% / 0.16% / 0.12% / 0.08%\n- Buzzsaw obstacle blindness: 12.76% / 0.16% / 0.08% / 0.05%\n- Enemy moving left blindness: 0.36% / 4.69% / 0.97% / 0.07%\nEach model was tested on 10,000 levels.\nOriginal model\nBuzzsaw obstacle\nblindness\nEnemy moving left\nblindness\n0%\n2%\n4%\n6%\n8%\n10%\n12%\nPercentage of levels failed\nFailure rate by cause\nCauses\nBuzzsaw obstacle\nEnemy moving left\nEnemy moving right\nMultiple or other\nResults of testing each model on 10,000 levels. Note that moving enemies can change direction.\nWe did not manage to achieve complete blindness, however: the buzzsaw-edited model still performed significantly better than the original model did when we made the buzzsaws completely invisible.\nOur results on the version of the game with invisible buzzsaws are as follows.\nPercentage of levels failed due to: buzzsaw obstacle / enemy moving left / enemy moving right / multiple or other:\nOriginal model, invisible buzzsaws: 32.20% / 0.05% / 0.05% / 0.05%\nWe tested the model on 10,000 levels.\nWe experimented briefly with iterating the editing procedure, but were not able to achieve more than around 50% buzzsaw blindness by this metric without affecting the model’s other abilities.\nThis implies that the model has other ways of detecting buzzsaws than the feature identified by our interface.\nHere are the original and edited models playing some cherry-picked levels.\nLevel 1\nLevel 2\nLevel 3\n►\n►\n►\nOriginal model.\nBuzzsaw obstacle blindness.\nEnemy moving left blindness.\nThe diversity hypothesis\nAll of the above analysis uses the same hidden layer of our network, the third of five convolutional layers, since it was much harder to find interpretable features at other layers. Interestingly, the level of abstraction at which this layer operates – finding the locations of various in-game objects – is exactly the level at which CoinRun levels are randomized using procedural generation. Furthermore, we found that training on many randomized levels was essential for us to be able to find any interpretable features at all.\nThis led us to suspect that the diversity introduced by CoinRun’s randomization is linked to the formation of interpretable features. We call this the\ndiversity hypothesis\n:\nInterpretable features tend to arise (at a given level of abstraction) if and only if the training distribution is diverse enough (at that level of abstraction).\nOur explanation for this hypothesis is as follows. For the forward implication (“only if”), we only expect features to be interpretable if they are general enough, and when the training distribution is not diverse enough, models have no incentive to develop features that generalize instead of overfitting. For the reverse implication (“if”), we do not expect it to hold in a strict sense: diversity on its own is not enough to guarantee the development of interpretable features, since they must also be relevant to the task. Rather, our intention with the reverse implication is to hypothesize that it holds very often in practice, as a result of generalization being bottlenecked by diversity.\nIn CoinRun, procedural generation is used to incentivize the model to learn skills that generalize to unseen levels\n. However, only the\nlayout\nof each level is randomized, and correspondingly, we were only able to find interpretable features at the level of abstraction of objects. At a lower level, there are only a handful of visual patterns in the game, and the low-level features of our model seem to consist mostly of memorized color configurations used for picking these out. Similarly, the game’s high-level dynamics follow a few simple rules, and accordingly the high-level features of our model seem to involve mixtures of combinations of objects that are hard to decipher. To explore the other convolutional layers, see the interface\nhere\n.\nInterpretability and generalization\nTo test our hypothesis, we made the training distribution less diverse, by training the agent on a fixed set of 100 levels. This dramatically reduced our ability to interpret the model’s features. Here we display an interface for the new model, generated in the same way as the one\nabove\n. The smoothly increasing value function suggests that the model has memorized the number of timesteps until the end of the level, and the features it uses for this focus on irrelevant background objects. Similar overfitting occurs for other video games with a limited number of levels\n.\nWe attempted to quantify this effect by varying the number of levels used to train the agent, and evaluating the 8 features identified by our interface on how interpretable they were.\nThe interfaces used for this evaluation can be found\nhere\n.\nFeatures were scored based on how consistently they focused on the same objects, and whether the value function attribution made sense – for example, background objects should not be relevant. This process was subjective and noisy, but that may be unavoidable. We also measured the generalization ability of each model, by testing the agent on unseen levels\n.\nThe data for this plot are as follows.\n- Number of training levels: 100 / 300 / 1000 / 3,000 / 10,000 / 30,000 / 100,000\n- Percentage of levels completed (train, run 1): 99.96% / 99.82% / 99.67% / 99.65% / 99.47% / 99.55% / 99.57%\n- Percentage of levels completed (train, run 2): 99.97% / 99.86% / 99.70% / 99.46% / 99.39% / 99.50% / 99.37%\n- Percentage of levels completed (test, run 1): 61.81% / 66.95% / 74.93% / 89.87% / 97.53% / 98.66% / 99.25%\n- Percentage of levels completed (test, run 2): 64.13% / 67.64% / 73.46% / 90.36% / 97.44% / 98.89% / 99.35%\n- Percentage of features interpretable (researcher 1, run 1): 52.5% / 22.5% / 11.25% / 45% / 90% / 75% / 91.25%\n- Percentage of features interpretable (researcher 2, run 1): 8.75% / 8.75% / 10% / 26.25% / 56.25% / 90% / 70%\n- Percentage of features interpretable (researcher 1, run 2): 15% / 13.75% / 15% / 23.75% / 53.75% / 90% / 96.25%\n- Percentage of features interpretable (researcher 2, run 2): 3.75% / 6.25% / 21.25% / 45% / 72.5% / 83.75% / 77.5%\nPercentages of levels completed are estimated by sampling 10,000 levels with replacement.\n10\n2\n10\n3\n10\n4\n10\n5\n60%\n70%\n80%\n90%\n100%\nPercentage of levels completed\nGeneralization\nTrain\nTest\n10\n2\n10\n3\n10\n4\n10\n5\n0%\n20%\n40%\n60%\n80%\n100%\nPercentage of features interpretable\nInterpretable features\n0.0\n0.2\n0.4\n0.6\n0.8\n1.0\nNumber of training levels\n0.0\n0.2\n0.4\n0.6\n0.8\n1.0\nComparison of models trained on different numbers of levels. Two models were trained for each number of levels, and two researchers independently evaluated how interpretable the features of each model were, without being shown the number of levels.\nOur methodology had some flaws. Firstly, the researchers were not completely blind to the number of levels: for example, it is possible to infer something about the number of levels from the smoothness of graphs of the value function, since with fewer levels the model is better able to memorize the number of timesteps until the end of the level. Secondly, since evaluations are somewhat tedious, we stopped them once we thought the trend had become clear, introducing some selection bias. Therefore these results should be considered primarily illustrative.\nEach model was tested on 10,000 train and 10,000 test levels sampled with replacement. Shaded areas in the left plot show the range of values over both models, though these are mostly too narrow to be visible. Error bars in the right plot show ±1 population standard deviation over all four model–researcher pairs.\nOur results illustrate how diversity may lead to interpretable features via generalization, lending support to the diversity hypothesis. Nevertheless, we still consider the hypothesis to be highly unproven.\nFeature visualization\nFeature visualization\nanswers questions about what certain parts of a network are looking for by generating examples. This can be done by applying gradient descent to the input image, starting from random noise, with the objective of activating a particular neuron or group of neurons. While this method works well for an image classifier trained on ImageNet\n, for our CoinRun model it yields only featureless clouds of color. Only for the first layer, which computes simple convolutions of the input, does the method produce comparable visualizations for the two models.\nImageNet\nCoinRun\nFirst layer\nIntermediate layer\nComparison of gradient-based feature visualization for CNNs trained on ImageNet (GoogLeNet\n) and on CoinRun (architecture described\nbelow\n). Each image was chosen to activate a neuron in the center, with the 3 images corresponding to the first 3 channels. Jittering was applied between optimization steps of up to 2 pixels for the first layer, and up to 8 pixels for the intermediate layer (mixed4a for ImageNet,\n2b\nfor CoinRun).\nGradient-based feature visualization has previously been shown to struggle with RL models trained on Atari games\n. To try to get it to work for CoinRun, we varied the method in a number of ways. Nothing we tried had any noticeable effect on the quality of the visualizations.\nTransformation robustness.\nThis is the method of stochastically jittering, rotating and scaling the image between optimization steps, to search for examples that are robust to these transformations\n. We tried both increasing and decreasing the size of the jittering. Rotating and scaling are less appropriate for CoinRun, since the observations themselves are not invariant to these transformations.\nPenalizing extremal colors.\nBy an “extremal” color we mean one of the 8 colors with maximal or minimal RGB values (black, white, red, green, blue, yellow, cyan and magenta).\nNoticing that our visualizations tend to use extremal colors towards the middle, we tried including in the visualization objective an L2 penalty of various strengths on the activations of the first layer, which successfully reduced the size of the extremally-colored region but did not otherwise help.\nAlternative objectives.\nWe tried using an alternative optimization objective\n, such as the caricature objective.\nThe caricature objective is to maximize the dot product between the activations of the input image and the activations of a reference image. Caricatures are often an especially easy type of feature visualization to make work, and helpful for getting a first glance into what features a model has. They are demonstrated in\nthis notebook\n. A more detailed manuscript by its authors\nis forthcoming.\nWe also tried using dimensionality reduction, as described\nbelow\n, to choose non-axis-aligned directions in activation space to maximize.\nLow-level visual diversity.\nIn an attempt to broaden the distribution of images seen by the model, we retrained it on a version of the game with procedurally-generated sprites. We additionally tried adding noise to the images, both independent per-pixel noise and spatially-correlated noise. Finally, we experimented briefly with adversarial training\n, though we did not pursue this line of inquiry very far.\nAs shown\nbelow\n, we were able to use dataset examples to identify a number of channels that pick out human-interpretable features. It is therefore striking how resistant gradient-based methods were to our efforts. We believe that this is because solving CoinRun does not ultimately require much visual ability. Even with our modifications, it is possible to solve the game using simple visual shortcuts, such as picking out certain small configurations of pixels. These shortcuts work well on the narrow distribution of images on which the model is trained, but behave unpredictably in the full space of images in which gradient-based optimization takes place.\nOur analysis here provides further insight into the\ndiversity hypothesis\n. In support of the hypothesis, we have examples of features that are hard to interpret in the absence of diversity. But there is also evidence that the hypothesis may need to be refined. Firstly, it seems to be a lack of diversity at a low level of abstraction that harms our ability to interpret features at all levels of abstraction, which could be due to the fact that gradient-based feature visualization needs to back-propagate through earlier layers. Secondly, the failure of our efforts to increase low-level visual diversity suggests that diversity may need to be assessed in the context of the requirements of the task.\nDataset example-based feature visualization\nAs an alternative to gradient-based feature visualization, we use dataset examples. This idea has a long history, and can be thought of as a heavily-regularized form of feature visualization\n. In more detail, we sample a few thousand observations infrequently from the agent playing the game, and pass them through the model. We then apply a dimensionality reduction method known as non-negative matrix factorization (NMF) to the activation channels\n.\nMore precisely, we find a non-negative approximate low-rank factorization of the matrix obtained by flattening the spatial dimensions of the activations into the batch dimension. This matrix has one row per observation\nper spatial position\nand one column per channel: thus the dimensionality reduction does not use spatial information.\nFor each of the resulting channels (which correspond to weighted combinations of the original channels), we choose the observations and spatial positions with the strongest activation (with a limited number of examples per position, for diversity), and display a patch from the observation at that position.\nShort left-facing wall\nVelocity info or left edge of screen\nLong left-facing wall\nLeft end of platform\nRight end of platform\nBuzzsaw obstacle or platform\nCoin\nTop/right edge of screen\nLeft end of platform\nStep\nAgent or enemy moving right\nLeft edge of box\nRight end of platform\nBuzzsaw obstacle\nTop left corner of box\nLeft end of platform or bottom/right of screen?\nDataset example-based feature visualizations for 16 NMF directions of layer\n2b\nof our CoinRun model. The grey-white checkerboard represents the edge of the screen. The labels are hand-composed.\nUnlike gradient-based feature visualization, this method finds some meaning to the different directions in activation space. However, it may still fail to provide a complete picture for each direction, since it only shows a limited number of dataset examples, and with limited context.\nSpatially-aware feature visualization\nCoinRun observations differ from natural images in that they are much less spatially invariant. For example, the agent always appears in the center, and the agent’s velocity is always encoded in the top left. As a result, some features detect unrelated things at different spatial positions, such as reading the agent’s velocity in the top left while detecting an unrelated object elsewhere. To account for this, we developed a spatially-aware version of dataset example-based feature visualization, in which we fix each spatial position in turn, and choose the observation with the strongest activation at that position (with a limited number of reuses of the same observation, for diversity). This creates a spatial correspondence between visualizations and observations.\nHere is such a visualization for a feature that responds strongly to coins. The white squares in the top left show that the feature also responds strongly to the horizontal velocity info when it is white, corresponding to the agent moving right at full speed.\nSpatially-aware dataset example-based feature visualization for the coin-detecting NMF direction of layer\n2b\n. Transparency (revealing the diagonally-striped background) indicates a weak response, so the left half of the visualization is mostly transparent because coins never appear in the left half of observations.\nAttribution\nAttribution\nanswers questions about the relationships between neurons. It is most commonly used to see how the input to a network affects a particular output – for example, in RL\n– but it can also be applied to the activations of hidden layers\n. Although there are many approaches to attribution we could have used, we chose the method of integrated gradients\n. We explain in\nAppendix B\nhow we applied this method a hidden layer, and how positive value function attribution can be thought of as “good news” and negative value function attribution can as “bad news”.\nDimensionality reduction for attribution\nWe showed\nabove\nthat a dimensionality reduction method known as non-negative matrix factorization (NMF) could be applied to the channels of activations to produce meaningful directions in activation space\n. We found that it is even more effective to apply NMF not to activations, but to\nvalue function attributions\nAs before, we obtain the NMF directions by sampling a few thousand observations infrequently from the agent playing the game, computing the attributions, flattening the spatial dimensions into the batch dimension, and applying NMF.\n(working around the fact that NMF can only be applied to non-negative matrices\nOur workaround is to separate out the positive and negative parts of the attributions and concatenate them along the batch dimension. We could also have concatenated them along the channel dimension.\n). Both methods tend to produce NMF directions that are close to one-hot, and so can be thought of as picking out the most relevant channels. However, when reducing to a small number of dimensions, using attributions usually picks out more salient features, because attribution takes into account not just\nwhat neurons respond to\nbut also\nwhether their response matters\n.\nFollowing\n, after applying NMF to attributions, we visualize them by assigning a different color to each of the resulting channels. We overlay these visualizations over the observation\nand contextualize each channel using feature visualization\n, making use of\ndataset example-based feature visualization\n. This gives a basic version of our interface, which allows us to see the effect of the main features at different spatial positions.\nObservation\nPositive attribution\n(good news)\nNegative attribution\n(bad news)\nLegend\n(hover to isolate)\nBuzzsaw\nobstacle\nCoin\nEnemy\nmoving\nleft\nAgent\nor enemy\nmoving right\nValue function attribution for a cherry-picked observation using layer\n2b\nof our CoinRun model, reduced to 4 channels using attribution-based NMF. The dataset example-based feature visualizations of these directions reveal more salient features than the visualizations of the first 4 activation-based NMF directions from the preceding section.\nFor the\nfull version\nof our interface, we simply repeat this for an entire trajectory of the agent playing the game. We also incorporate video controls, a timeline view of compressed observations\n, and additional information, such as model outputs and sampled actions. Together these allow the trajectory to be easily explored and understood.\nAttribution discussion\nAttributions for our CoinRun model have some interesting properties that would be unusual for an ImageNet model.\nSparsity.\nAttribution tends to be concentrated in a very small number of spatial positions and (post-NMF) channels. For example, in the figure above, the top 10 position–channel pairs account for more than 80% of the total absolute attribution. This might be explained by our\nearlier\nhypothesis that the model identifies objects by picking out certain small configurations of pixels. Because of this sparsity, we smooth out attribution over nearby spatial positions for the full version of our interface, so that the amount of visual space taken up can be used to judge attribution strength. This trades off some spatial precision for more precision with magnitudes.\nUnexpected sign.\nValue function attribution usually has the sign one would expect: positive for coins, negative for enemies, and so on. However, this is sometimes not the case. For example, in the figure above, the red channel that detects buzzsaw obstacles has both positive and negative attribution in two neighboring spatial positions towards the left. Our best guess is that this phenomenon is a result of statistical\ncollinearity\n, caused by certain correlations in the procedural level generation together with the agent’s behavior. These could be visual, such as correlations between nearby pixels, or more abstract, such as both coins and long walls appearing at the end of every level. As a toy example, supposing the value function ought to increase by 2% when the end of the level becomes visible, the model could either increase the value function by 1% for coins and 1% for long walls, or by 3% for coins and −1% for long walls, and the effect would be similar.\nOutlier frames.\nWhen an unusual event causes the network to output extreme values, attribution can behave especially strangely. For example, in the\nbuzzsaw hallucination\nframe, most features have a significant amount of both positive and negative attribution. We do not have a good explanation for this, but perhaps features are interacting in more complicated ways than usual. Moreover, in these cases there is often a significant component of the attribution lying outside the space spanned by the NMF directions, which we display as an additional “residual” feature. This could be because each frame is weighted equally when computing NMF, so outlier frames have little influence over the NMF directions.\nThese considerations suggest that some care may be required when interpreting attributions.\nQuestions for further research\nThe\ndiversity hypothesis\nValidity.\nDoes the diversity hypothesis hold in other contexts, both within and outside of reinforcement learning?\nRelationship to generalization.\nWhat is the three-way relationship between diversity, interpretable features and generalization? Do non-interpretable features indicate that a model will fail to generalize in certain ways? Generalization refers implicitly to an underlying distribution – how should this distribution be chosen?\nFor example, to measure generalization for CoinRun models trained on a limited number of levels, we used the distribution over all possible procedurally-generated levels. However, to formalize the sense in which CoinRun is not diverse in its visual patterns or dynamics rules, one would need a distribution over levels from a wider class of games.\nCaveats.\nHow are interpretable features affected by other factors, such as the choice of task or algorithm, and how do these interact with diversity? Speculatively, do big enough models obtain interpretable features via the double descent phenomenon\n, even in the absence of diversity?\nQuantification.\nCan we quantitatively predict how much diversity is needed for interpretable features, perhaps using generalization metrics? Can we be precise about what is meant by an “interpretable feature” and a “level of abstraction”?\nInterpretability in the absence of diversity\nPervasiveness of non-diverse features.\nDo “non-diverse features”, by which we mean the hard-to-interpret features that tend to arise in the absence of diversity, remain when diversity is present? Is there a connection between these non-diverse features and the “non-robust features” that have been posited to explain adversarial examples\n?\nCoping with non-diverse levels of abstraction.\nAre there levels of abstraction at which even broad distributions like ImageNet remain non-diverse, and how can we best interpret models at these levels of abstraction?\nGradient-based feature visualization.\nWhy does gradient-based feature visualization\nbreak down\nin the absence of diversity, and can it be made to work using transformation robustness, regularization, data augmentation, adversarial training, or other techniques? What property of the optimization leads to the clouds of\nextremal colors\n?\nTrustworthiness of dataset examples and attribution.\nHow reliable and trustworthy can we make very heavily-regularized versions of feature visualization, such as those based on\ndataset examples\n?\nHeavily-regularized feature visualization may be untrustworthy by failing to separate the things causing certain behavior from the things that merely correlate with those causes\n.\nWhat explains the\nstrange behavior\nof attribution, and how trustworthy is it?\nInterpretability in the RL framework\nNon-visual and abstract features.\nWhat are the best methods for interpreting models with non-visual inputs? Even vision models may also have interpretable abstract features, such as relationships between objects or anticipated events: will any method of generating examples be enough to understand these, or do we need an entirely new approach? For models with memory, how can we interpret their hidden states\n?\nImproving reliability.\nHow can we best identify, understand and correct rare\nfailures\nand\nother errors\nin RL models? Can we actually improve models by\nmodel editing\n, rather than merely degrading them?\nModifying training.\nIn what ways can we train RL models to make them more interpretable without a significant performance cost, such as by altering architectures or adding auxiliary predictive losses?\nLeveraging the environment.\nHow can we enrich interfaces using RL-specific data, such as trajectories of agent–environment interaction, state distributions, and advantage estimates? What are the benefits of incorporating user–environment interaction, such as for exploring counterfactuals?\nWhat we would like to see from further research and why\nWe are motivated to study interpretability for RL for two reasons.\nTo be able to interpret RL models.\nRL can be applied to an enormous variety of tasks, and seems likely to be a part of increasingly influential AI systems. It is therefore important to be able to scrutinize RL models and to understand how they might fail. This may also benefit RL research through an improved understanding of the pitfalls of different algorithms and environments.\nAs a testbed for interpretability techniques.\nRL models pose a number of distinctive challenges for interpretability techniques. In particular, environments like CoinRun straddle the boundary between memorization and generalization, making them useful for studying the\ndiversity hypothesis\nand related ideas.\nWe think that large neural networks are currently the most likely type of model to be used in highly capable and influential AI systems in the future. Contrary to the traditional perception of neural networks as black boxes, we think that there is a fighting chance that we will be able to clearly and thoroughly understand the behavior even of very large networks. We are therefore most excited by neural network interpretability research that scores highly according to the following criteria.\nScalability.\nThe takeaways of the research should have some chance of scaling to harder problems and larger networks. If the techniques themselves do not scale, they should at least reveal some relevant insight that might.\nTrustworthiness.\nExplanations should be faithful to the model. Even if they do not tell the full story, they should at least not be biased in some fatal way (such as by using an approval-based objective that leads to bad explanations that sound good, or by depending on another model that badly distorts information).\nExhaustiveness.\nThis may turn out to be impossible at scale, but we should strive for techniques that explain every essential feature of our models. If there are theoretical limits to exhaustiveness, we should try to understand these.\nLow cost.\nOur techniques should not be significantly more computationally expensive than training the model. We hope that we will not need to train models differently for them to be interpretable, but if we do, we should try to minimize both the computational expense and any performance cost, so that interpretable models are not disincentivized from being used in practice.\nOur proposed questions reflect this perspective. One of the reasons we emphasize diversity relates to exhaustiveness. If “non-diverse features” remain when diversity is present, then our current techniques are not exhaustive and could end up missing important features of more capable models. Developing tools to understand non-diverse features may shed light on whether this is likely to be a problem.\nWe think there may be significant mileage in simply applying existing interpretability techniques, with attention to detail, to more models. Indeed, this was the mindset with which we initially approached this project. If the diversity hypothesis is correct, then this may become easier as we train our models to perform more complex tasks. Like early biologists encountering a new species, there may be a lot we can glean from taking a magnifying glass to the creatures in front of us.\nSupplementary material\nCode.\nUtilities for computing feature visualization, attribution and dimensionality reduction for our models can be found in\nlucid.scratch.rl_util\n, a submodule of\nLucid\n. We demonstrate these in a\nnotebook\n.\nModel weights.\nThe weights of our model are available for download, along with those of a number of other models, including the models trained on different numbers of levels, the edited models, and models trained on all 16 of the Procgen Benchmark\ngames. These are indexed\nhere\n.\nMore interfaces.\nWe generated an expanded version of our interface for every convolutional layer in our model, which can be found\nhere\n. We also generated similar interfaces for each of our other models, which are indexed\nhere\n.\nInterface code.\nThe code used to generate the expanded version of our interface can be found\nhere\n.\nAppendix A: Model editing method\nHere we explain our method for\nediting the model\nto make the agent blind to certain features.\nThe features in our interface correspond to directions in activation space obtained by applying\nattribution-based NMF\nto layer\n2b\nof our model. To blind the agent to a feature, we edit the weights to make them project out the corresponding NMF direction.\nMore precisely, let\nv\n\\mathbf v\nv\nbe the NMF direction corresponding to the feature we wish to blind the model to. This is a vector of length\nc\nc\nc\n, the number of channels in activation space. Using this we construct the\northogonal projection\nmatrix\nP\n:\n=\nI\n−\n1\n∥\nv\n∥\n2\nv\nv\nT\nP:=I-\\frac 1{\\|\\mathbf v\\|^2}\\mathbf v\\mathbf v^{\\mathsf T}\nP\n:\n=\nI\n−\n∥\nv\n∥\n2\n1\n​\nv\nv\nT\n, which projects out the direction of\nv\n\\mathbf v\nv\nfrom activation vectors. We then take the convolutional kernel of the following layer, which has shape\nheight\n×\nwidth\n×\nc\n×\nd\n\\text{height}\\times\\text{width}\\times c\\times d\nheight\n×\nwidth\n×\nc\n×\nd\n, where\nd\nd\nd\nis the number of output channels. Broadcasting across the height and width dimensions, we left-multiply each\nc\n×\nd\nc\\times d\nc\n×\nd\nmatrix in the kernel by\nP\nP\nP\n. The effect of the new kernel is to project out the direction of\nv\n\\mathbf v\nv\nfrom activations before applying the original kernel.\nAs it turned out, the NMF directions were close to one-hot, so this procedure is approximately equivalent to zeroing out the slice of the kernel corresponding to a particular in-channel.\nAppendix B: Integrated gradients for a hidden layer\nHere we explain the application of integrated gradients\nto a hidden layer for the purpose of\nattribution\n. This method can be applied to any of the network’s outputs, but we focus here on the value function. Recall that this is the model’s estimate of the time-discounted probability that the agent will successfully complete the level.\nobservation\nspace\nR\nactivation\nspace\nV\nA\nF\nThe\ndiagram\ndefining\nF\nF\nF\n, whose gradient we take.\nLet\nV\n:\nR\n6\n4\n×\n6\n4\n×\n3\n→\nR\nV:\\mathbb R^{64\\times 64\\times 3}\\to\\mathbb R\nV\n:\nR\n6\n4\n×\n6\n4\n×\n3\n→\nR\nbe the value function computed by our network, which accepts a 64x64 RGB observation. Given any layer in the network, we may write\nV\nV\nV\nas\nV\n(\nx\n)\n=\nF\n(\nA\n(\nx\n)\n)\nV\\left(\\mathbf x\\right)=F\\left(\\mathbf A\\left(\\mathbf x\\right)\\right)\nV\n(\nx\n)\n=\nF\n(\nA\n(\nx\n)\n)\n, where\nA\n\\mathbf A\nA\ncomputes the layer’s activations. Given an observation\nx\n\\mathbf x\nx\n, a simple method of attribution is to compute\n∇\na\nF\n(\na\n)\n⊙\na\n\\nabla_{\\mathbf a}F\\left(\\mathbf a\\right)\\odot\\mathbf a\n∇\na\n​\nF\n(\na\n)\n⊙\na\n, where\na\n=\nA\n(\nx\n)\n\\mathbf a=\\mathbf A\\left(\\mathbf x\\right)\na\n=\nA\n(\nx\n)\nand\n⊙\n\\odot\n⊙\ndenotes the pointwise product. This tells us the sensitivity of the value function to each activation, multiplied by the strength of that activation. However, it uses the sensitivity of the value function at the activation itself, which does not account for the fact that this sensitivity may change as the activation is increased from zero.\nViewing\nF\nF\nF\nas the height of a surface, represented here using the background gradient, the integrated gradient of\nF\nF\nF\nmeasures the elevation gained while traveling in each direction, and sums to the total elevation gain\n.\nTo account for this, the integrated gradients method instead chooses a path\nP\n\\mathcal P\nP\nin activation space from some starting point\na\n0\n\\mathbf a_0\na\n0\n​\nto the ending point\na\n1\n:\n=\nA\n(\nx\n)\n\\mathbf a_1:=\\mathbf A\\left(\\mathbf x\\right)\na\n1\n​\n:\n=\nA\n(\nx\n)\n. We then compute the integrated gradient of\nF\nF\nF\nalong\nP\n\\mathcal P\nP\n, which is defined as the path integral\n∫\nP\n∇\na\nF\n(\na\n)\n⊙\nd\na\n.\n\\int_{\\mathcal P}\\nabla_{\\mathbf a}F\\left(\\mathbf a\\right)\\odot\\mathrm d\\mathbf a.\n∫\nP\n​\n∇\na\n​\nF\n(\na\n)\n⊙\nd\na\n.\nNote the use of the pointwise product rather than the usual dot product here, which makes the integral vector-valued. By the\nfundamental theorem of calculus for line integrals\n, when the components of the vector produced by this integral are summed, the result depends only on the endpoints\na\n0\n\\mathbf a_0\na\n0\n​\nand\na\n1\n\\mathbf a_1\na\n1\n​\n, equaling\nF\n(\na\n1\n)\n−\nF\n(\na\n0\n)\nF\\left(\\mathbf a_1\\right)-F\\left(\\mathbf a_0\\right)\nF\n(\na\n1\n​\n)\n−\nF\n(\na\n0\n​\n)\n. Thus the components of this vector provide a true decomposition of this difference, “attributing” it across the activations.\nFor our purposes, we take\nP\n\\mathcal P\nP\nto be the straight line from\n0\n\\mathbf 0\n0\nto\nA\n(\nx\n)\n\\mathbf A\\left(\\mathbf x\\right)\nA\n(\nx\n)\n.\nIn theory, we could choose any point in activation space as the starting point of our path, but in practice,\n0\n\\mathbf 0\n0\ntends to be a good baseline against which to compare other activations, with\nF\n(\n0\n)\nF\\left(\\mathbf 0\\right)\nF\n(\n0\n)\nbeing on the same order as the average value function. Sundararajan, Taly and Yan\ndiscuss the choice of this baseline in more depth.\nIn other words, given an observation\nx\n\\mathbf x\nx\n, we define the value function attribution as\nIn practice, we numerically approximate the integral by evaluating the integrand at\nα\n=\n0\n.\n1\n,\n0\n.\n2\n,\n…\n,\n1\n\\alpha=0.1,0.2,\\ldots,1\nα\n=\n0\n.\n1\n,\n0\n.\n2\n,\n…\n,\n1\n.\n∫\nα\n=\n0\n1\n∇\na\nF\n(\nα\nA\n(\nx\n)\n)\nd\nα\n⊙\nA\n(\nx\n)\n.\n\\int_{\\alpha=0}^1\\nabla_{\\mathbf a}F\\left(\\alpha\\mathbf A\\left(\\mathbf x\\right)\\right)\\mathrm d\\alpha\\odot\\mathbf A\\left(\\mathbf x\\right).\n∫\nα\n=\n0\n1\n​\n∇\na\n​\nF\n(\nα\nA\n(\nx\n)\n)\nd\nα\n⊙\nA\n(\nx\n)\n.\nThis has the same dimensions as\nA\n(\nx\n)\n\\mathbf A\\left(\\mathbf x\\right)\nA\n(\nx\n)\n, and its components sum to\nV\n(\nx\n)\n−\nF\n(\n0\n)\nV\\left(\\mathbf x\\right)-F\\left(\\mathbf 0\\right)\nV\n(\nx\n)\n−\nF\n(\n0\n)\n. So for a convolutional layer, this method allows us to attribute the value function (in excess of the baseline\nF\n(\n0\n)\nF\\left(\\mathbf 0\\right)\nF\n(\n0\n)\n) across the horizontal, vertical and channel dimensions of activation space. Positive value function attribution can be thought of as “good news”, components that cause the agent to think it is more likely to collect the coin at the end of the level. Similarly, negative value function attribution can be thought of as “bad news”.\nAppendix C: Architecture\nOur architecture consists of the following layers in the order given, together with ReLU activations for all except the final layer.\n7x7 convolutional layer with 16 channels (layer 1a)\n2x2 L2 pooling layer\n5x5 convolutional layer with 32 channels (layer 2a)\n5x5 convolutional layer with 32 channels (layer 2b)\n2x2 L2 pooling layer\n5x5 convolutional layer with 32 channels (layer 3a)\n2x2 L2 pooling layer\n5x5 convolutional layer with 32 channels (layer 4a)\n2x2 L2 pooling layer\n256-unit dense layer\n512-unit dense layer\n10-unit dense layer (1 unit for the value function, 9 units for the policy logits)\nWe designed this architecture by starting with the architecture from IMPALA\n, and making the following modifications in an attempt to aid interpretability without noticeably sacrificing performance.\nWe used fewer convolutional layers and more dense layers, to allow for more non-visual processing.\nWe removed the residual connections, so that the flow of information passes through every layer.\nWe made the pool size equal to the pool stride, to avoid gradient gridding.\nWe used L2 pooling instead of max pooling, for more continuous gradients.\nThe choice that seemed to make the most difference was using 5 rather than 12 convolutional layers, resulting in the object-identifying features (which were the most interpretable, as discussed\nabove\n) being concentrated in a single layer (layer 2b), rather than being spread over multiple layers and mixed in with less interpretable features.\nAcknowledgments\nWe would like to thank our reviewers Jonathan Uesato, Joel Lehman and one anonymous reviewer for their detailed and thoughtful feedback. We would also like to thank Karl Cobbe, Daniel Filan, Sam Greydanus, Christopher Hesse, Jacob Jackson, Michael Littman, Ben Millwood, Konstantinos Mitsopoulos, Mira Murati, Jorge Orbay, Alex Ray, Ludwig Schubert, John Schulman, Ilya Sutskever, Nevan Wichers, Liang Zhang and Daniel Ziegler for research discussions, feedback, follow-up work, help and support that have greatly benefited this project.\nAuthor Contributions\nJacob Hilton\nwas the primary contributor.\nNick Cammarata\ndeveloped the model editing methodology and suggested applying it to CoinRun models.\nShan Carter\n(while working at OpenAI) advised on interface design throughout the project, and worked on many of the diagrams in the article.\nGabriel Goh\nprovided evaluations of feature interpretability for the section\nInterpretability and generalization\n.\nChris Olah\nguided the direction of the project, performing initial exploratory research on the models, coming up with many of the research ideas, and helping to construct the article’s narrative.\nDiscussion and Review\nReview 1 - Anonymous\nReview 2 - Jonathan Uesato\nReview 3 - Joel Lehman\nReferences\nQuantifying generalization in reinforcement learning\n[link]\nCobbe, K., Klimov, O., Hesse, C., Kim, T. and Schulman, J., 2018. arXiv preprint arXiv:1812.02341.\nDeep inside convolutional networks: Visualising image classification models and saliency maps\n[PDF]\nSimonyan, K., Vedaldi, A. and Zisserman, A., 2013. arXiv preprint arXiv:1312.6034.\nVisualizing and understanding convolutional networks\n[PDF]\nZeiler, M.D. and Fergus, R., 2014. European conference on computer vision, pp. 818--833.\nStriving for simplicity: The all convolutional net\n[PDF]\nSpringenberg, J.T., Dosovitskiy, A., Brox, T. and Riedmiller, M., 2014. arXiv preprint arXiv:1412.6806.\nGrad-CAM: Visual explanations from deep networks via gradient-based localization\n[PDF]\nSelvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D. and Batra, D., 2017. Proceedings of the IEEE International Conference on Computer Vision, pp. 618--626.\nInterpretable explanations of black boxes by meaningful perturbation\n[PDF]\nFong, R.C. and Vedaldi, A., 2017. Proceedings of the IEEE International Conference on Computer Vision, pp. 3429--3437.\nPatternNet and PatternLRP--Improving the interpretability of neural networks\n[PDF]\nKindermans, P., Schutt, K.T., Alber, M., Muller, K. and Dahne, S., 2017. stat, Vol 1050, pp. 16.\nThe (un)reliability of saliency methods\n[PDF]\nKindermans, P., Hooker, S., Adebayo, J., Alber, M., Schutt, K.T., Dahne, S., Erhan, D. and Kim, B., 2019. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 267--280. Springer.\nAxiomatic attribution for deep networks\n[PDF]\nSundararajan, M., Taly, A. and Yan, Q., 2017. Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 3319--3328.\nThe Building Blocks of Interpretability\nOlah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K. and Mordvintsev, A., 2018. Distill.\nDOI: 10.23915/distill.00010\nLeveraging Procedural Generation to Benchmark Reinforcement Learning\n[link]\nCobbe, K., Hesse, C., Hilton, J. and Schulman, J., 2019.\nProximal policy optimization algorithms\n[link]\nSchulman, J., Wolski, F., Dhariwal, P., Radford, A. and Klimov, O., 2017. arXiv preprint arXiv:1707.06347.\nHigh-dimensional continuous control using generalized advantage estimation\n[PDF]\nSchulman, J., Moritz, P., Levine, S., Jordan, M. and Abbeel, P., 2015. arXiv preprint arXiv:1506.02438.\nThread: Circuits\nCammarata, N., Carter, S., Goh, G., Olah, C., Petrov, M. and Schubert, L., 2020. Distill.\nDOI: 10.23915/distill.00024\nGeneral Video Game AI: A multi-track framework for evaluating agents, games and content generation algorithms\n[link]\nPerez-Liebana, D., Liu, J., Khalifa, A., Gaina, R.D., Togelius, J. and Lucas, S.M., 2018. arXiv preprint arXiv:1802.10363.\nObstacle Tower: A Generalization Challenge in Vision, Control, and Planning\n[PDF]\nJuliani, A., Khalifa, A., Berges, V., Harper, J., Henry, H., Crespi, A., Togelius, J. and Lange, D., 2019. arXiv preprint arXiv:1902.01378.\nObservational Overfitting in Reinforcement Learning\n[PDF]\nSong, X., Jiang, Y., Du, Y. and Neyshabur, B., 2019. arXiv preprint arXiv:1912.02975.\nFeature Visualization\nOlah, C., Mordvintsev, A. and Schubert, L., 2017. Distill.\nDOI: 10.23915/distill.00007\nVisualizing higher-layer features of a deep network\n[PDF]\nErhan, D., Bengio, Y., Courville, A. and Vincent, P., 2009. University of Montreal, Vol 1341(3), pp. 1.\nDeep neural networks are easily fooled: High confidence predictions for unrecognizable images\n[PDF]\nNguyen, A., Yosinski, J. and Clune, J., 2015. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427--436.\nInceptionism: Going deeper into neural networks\n[HTML]\nMordvintsev, A., Olah, C. and Tyka, M., 2015. Google Research Blog.\nPlug & play generative networks: Conditional iterative generation of images in latent space\n[PDF]\nNguyen, A., Clune, J., Bengio, Y., Dosovitskiy, A. and Yosinski, J., 2017. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4467--4477.\nImagenet: A large-scale hierarchical image database\n[PDF]\nDeng, J., Dong, W., Socher, R., Li, L., Li, K. and Fei-Fei, L., 2009. Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248--255.\nDOI: 10.1109/cvprw.2009.5206848\nGoing deeper with convolutions\n[PDF]\nSzegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V. and Rabinovich, A., 2015. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1--9.\nAn Atari model zoo for analyzing, visualizing, and comparing deep reinforcement learning agents\n[PDF]\nSuch, F.P., Madhavan, V., Liu, R., Wang, R., Castro, P.S., Li, Y., Schubert, L., Bellemare, M., Clune, J. and Lehman, J., 2018. arXiv preprint arXiv:1812.07069.\nFinding and Visualizing Weaknesses of Deep Reinforcement Learning Agents\n[PDF]\nRupprecht, C., Ibrahim, C. and Pal, C.J., 2019. arXiv preprint arXiv:1904.01318.\nCaricatures\nCammerata, N., Olah, C. and Satyanarayan, A., unpublished. Distill draft. Author list not yet finalized.\nTowards deep learning models resistant to adversarial attacks\nMadry, A., Makelov, A., Schmidt, L., Tsipras, D. and Vladu, A., 2017. arXiv preprint arXiv:1706.06083.\nIntriguing properties of neural networks\n[PDF]\nSzegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. and Fergus, R., 2013. arXiv preprint arXiv:1312.6199.\nVisualizing and understanding Atari agents\n[PDF]\nGreydanus, S., Koul, A., Dodge, J. and Fern, A., 2017. arXiv preprint arXiv:1711.00138.\nExplain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution\n[link]\nPuri, N., Verma, S., Gupta, P., Kayastha, D., Deshmukh, S., Krishnamurthy, B. and Singh, S., 2019. International Conference on Learning Representations.\nVideo Interface: Assuming Multiple Perspectives on a Video Exposes Hidden Structure\n[link]\nOchshorn, R.M., 2017.\nReconciling modern machine-learning practice and the classical bias--variance trade-off\n[PDF]\nBelkin, M., Hsu, D., Ma, S. and Mandal, S., 2019. Proceedings of the National Academy of Sciences, Vol 116(32), pp. 15849--15854. National Acad Sciences.\nAdversarial examples are not bugs, they are features\n[link]\nIlyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B. and Madry, A., 2019. arXiv preprint arXiv:1905.02175.\nA Discussion of 'Adversarial Examples Are Not Bugs, They Are Features'\nEngstrom, L., Gilmer, J., Goh, G., Hendrycks, D., Ilyas, A., Madry, A., Nakano, R., Nakkiran, P., Santurkar, S., Tran, B., Tsipras, D. and Wallace, E., 2019. Distill.\nDOI: 10.23915/distill.00019\nHuman-level performance in 3D multiplayer games with population-based reinforcement learning\n[link]\nJaderberg, M., Czarnecki, W.M., Dunning, I., Marris, L., Lever, G., Castaneda, A.G., Beattie, C., Rabinowitz, N.C., Morcos, A.S., Ruderman, A. and others,, 2019. Science, Vol 364(6443), pp. 859--865. American Association for the Advancement of Science.\nSolving Rubik's Cube with a Robot Hand\n[link]\nAkkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R. and others,, 2019. arXiv preprint arXiv:1910.07113.\nDota 2 with Large Scale Deep Reinforcement Learning\n[link]\nBerner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C. and others,, 2019. arXiv preprint arXiv:1912.06680.\nDoes Attribution Make Sense?\nOlah, C. and Satyanarayan, A., unpublished. Distill draft. Author list not yet finalized.\nIMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures\n[PDF]\nEspeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I. and others,, 2018. arXiv preprint arXiv:1802.01561.\nUpdates and Corrections\nIf you see mistakes or want to suggest changes, please\ncreate an issue on GitHub\n.\nReuse\nDiagrams and text are licensed under Creative Commons Attribution\nCC-BY 4.0\nwith the\nsource available on GitHub\n, unless noted otherwise. The figures that have been reused from other sources don’t fall under this license and can be recognized by a note in their caption: “Figure from …”.\nCitation\nFor attribution in academic contexts, please cite this work as\nHilton, et al., \"Understanding RL Vision\", Distill, 2020.\nBibTeX citation\n@article{hilton2020understanding,\n  author = {Hilton, Jacob and Cammarata, Nick and Carter, Shan and Goh, Gabriel and Olah, Chris},\n  title = {Understanding RL Vision},\n  journal = {Distill},\n  year = {2020},\n  note = {https://distill.pub/2020/understanding-rl-vision},\n  doi = {10.23915/distill.00029}\n}\nDistill\nis dedicated to clear explanations of machine learning\nAbout\nSubmit\nPrize\nArchive\nRSS\nGitHub\nTwitter\nISSN 2476-0757",
  "metadata": {
    "title": "Understanding RL Vision",
    "description": null,
    "keywords": null,
    "author": null,
    "og:type": "article",
    "og:title": "Understanding RL Vision",
    "og:description": "With diverse environments, we can analyze, diagnose and edit deep reinforcement learning models using attribution.",
    "og:url": "https://distill.pub/2020/understanding-rl-vision",
    "og:image": "https://distill.pub/2020/understanding-rl-vision/thumbnail.jpg",
    "og:locale": "en_US",
    "og:site_name": "Distill",
    "twitter:card": "summary_large_image",
    "twitter:title": "Understanding RL Vision",
    "twitter:description": "With diverse environments, we can analyze, diagnose and edit deep reinforcement learning models using attribution.",
    "twitter:url": "https://distill.pub/2020/understanding-rl-vision",
    "twitter:image": "https://distill.pub/2020/understanding-rl-vision/thumbnail.jpg",
    "twitter:image:width": "560",
    "twitter:image:height": "295",
    "article:published": "2020-11-17",
    "article:created": "2020-11-17",
    "article:modified": "2022-03-25T04:51:11.000Z",
    "article:author": "Chris Olah"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/181224c20509/code/body-000.txt",
      "chars": 57,
      "language": "text"
    },
    {
      "file": "attachments/181224c20509/code/body-001.txt",
      "chars": 304,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/hero/coin.png",
      "bytes": 1807,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-eadb3c2f5efe05ea5371.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/hero/coin.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/hero/enemy.png",
      "bytes": 2299,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-d9dca283da1654448eeb.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/hero/enemy.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/hero/saw.png",
      "bytes": 1582,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-d3dd78464a532c4fff9f.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/hero/saw.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/agent_0_human.png",
      "bytes": 2404,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-7ff794847d71116b8992.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/agent_0_human.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/agent_1_human.png",
      "bytes": 2687,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-73cad9d3b0352d54906a.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/agent_1_human.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/coin_1_human.png",
      "bytes": 2507,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-a28a15af6b3e55e551f2.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/coin_1_human.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/saw_1_human.png",
      "bytes": 2755,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-235a7ea1bd21594eb80d.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/saw_1_human.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/enemy_0_human.png",
      "bytes": 3524,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-99e7ae4fd68e91f2e57c.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/enemy_0_human.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/enemy_1_human.png",
      "bytes": 2989,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-999b95a6e82cb94b50ea.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/enemy_1_human.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/box_0_human.png",
      "bytes": 1967,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-7446e8b4b0958fabf26d.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/box_0_human.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/box_1_human.png",
      "bytes": 3327,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-8cd5728201f6f569bc06.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/box_1_human.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/lava_human.png",
      "bytes": 3948,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-95e88539b5a390f4c408.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/lava_human.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/velocity_info_human.png",
      "bytes": 3055,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-fdc61282f79426783981.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/velocity_info_human.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/agent_0_model.png",
      "bytes": 591,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-d345fe2aa41bc251e3b7.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/agent_0_model.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/agent_1_model.png",
      "bytes": 566,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-cdc4f791cabd602edd84.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/agent_1_model.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/coin_1_model.png",
      "bytes": 613,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-90160e4c3f4542a94ad2.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/coin_1_model.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/saw_1_model.png",
      "bytes": 595,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-790d91a718f8e08a7ed2.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/saw_1_model.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/enemy_0_model.png",
      "bytes": 615,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-9db57c0f940ab8606b83.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/enemy_0_model.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/enemy_1_model.png",
      "bytes": 625,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-e3ad8e858d77b3c473c8.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/enemy_1_model.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/box_0_model.png",
      "bytes": 618,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-90d7344e2a467a405500.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/box_0_model.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/box_1_model.png",
      "bytes": 846,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-3a72c21b0203f92409f0.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/box_1_model.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/lava_model.png",
      "bytes": 958,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-c12057cf6a2a2ceb026c.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/lava_model.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/velocity_info_model.png",
      "bytes": 602,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-d0850886c794139c7c12.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/coinrun/velocity_info_model.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/imagenet_first_0.png",
      "bytes": 19028,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-f3ac89c699f539beba0e.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/imagenet_first_0.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/imagenet_first_1.png",
      "bytes": 18091,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-ea58b490adcf34949143.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/imagenet_first_1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/imagenet_first_2.png",
      "bytes": 19729,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-361f4e9a620390e3734e.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/imagenet_first_2.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/coinrun_first_0.png",
      "bytes": 19073,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-282f1860edd49fd8cbb5.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/coinrun_first_0.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/coinrun_first_1.png",
      "bytes": 19876,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-77ff395fbf3e4b33d1b8.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/coinrun_first_1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/coinrun_first_2.png",
      "bytes": 19459,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-a9443394445dbcbf6487.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/coinrun_first_2.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/imagenet_intermediate_0.png",
      "bytes": 40761,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-2c35c618c95468906c3a.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/imagenet_intermediate_0.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/imagenet_intermediate_1.png",
      "bytes": 41042,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-f1f80d72cb5a5d57abc7.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/imagenet_intermediate_1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/imagenet_intermediate_2.png",
      "bytes": 41679,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-17c73352b0a2663083fe.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/imagenet_intermediate_2.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/coinrun_intermediate_0.png",
      "bytes": 22749,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-54b671e351dfd85b36d5.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/coinrun_intermediate_0.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/coinrun_intermediate_1.png",
      "bytes": 22466,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-afaeaa751943fdab6ccb.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/coinrun_intermediate_1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/coinrun_intermediate_2.png",
      "bytes": 22690,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-04bdd6b394840ee0d776.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_traditional/coinrun_intermediate_2.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_0.png",
      "bytes": 3822,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-f5a097163cf50e0b51fc.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_0.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_1.png",
      "bytes": 2516,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-6ea12e143dbb681c8bc4.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_2.png",
      "bytes": 3215,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-6e99312fcf9bfb20ed4f.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_2.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_3.png",
      "bytes": 4041,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-20d818369f579ee359d4.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_3.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_4.png",
      "bytes": 4345,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-7feb63e865403a97a849.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_4.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_5.png",
      "bytes": 4600,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-2d2990cbdae937f4b5c6.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_5.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_6.png",
      "bytes": 3973,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-857b4dd4c596c522684b.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_6.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_7.png",
      "bytes": 2780,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-de3fa791bfaa987da752.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_7.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_8.png",
      "bytes": 3967,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-e2f6c045e96f15f49ad0.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_8.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_9.png",
      "bytes": 4270,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-f7b97a4b3a3ef6ca6c05.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_9.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_10.png",
      "bytes": 5353,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-cc41b36fb888e8ee58ff.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_10.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_11.png",
      "bytes": 3613,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-0d42eeb454cd78857a0a.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_11.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_12.png",
      "bytes": 4018,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-3580c1337a952db39011.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_12.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_13.png",
      "bytes": 3850,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-3d7c7e038c8375a29354.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_13.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_14.png",
      "bytes": 3602,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-6b36911272a64b1267a5.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_14.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_15.png",
      "bytes": 3996,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-312bfd9b6b7eb838ddee.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_dataset/layer_2b_feature_15.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_spatial.png",
      "bytes": 148340,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-3b0e03d50e0c5bf43d05.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/feature_vis_spatial.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/colab.svg",
      "bytes": 1990,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-10a4d9d26733f805bd21.svg",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/colab.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://distill.pub/2020/understanding-rl-vision/images/integrated_gradient.png",
      "bytes": 20786,
      "reused": false,
      "local_path": "attachments/181224c20509/images/body-8502b1db7c1285d9ba57.png",
      "src": "https://distill.pub/2020/understanding-rl-vision/images/integrated_gradient.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://distill.pub/2020/understanding-rl-vision/videos/coinrun.mp4",
      "src": "https://distill.pub/2020/understanding-rl-vision/videos/coinrun.mp4",
      "status": "invalid_or_wrong_resolution",
      "kind": "direct_video",
      "bytes": 8243090,
      "reused": true,
      "content_type": "video/mp4",
      "valid": true,
      "height": 512,
      "width": 1088,
      "duration": "60.000000",
      "local_path": "attachments/181224c20509/videos/repair-66182550e27020813ba7.mp4",
      "title": "",
      "retried_at": "2026-10-20T17:48:17Z"
    },
    {
      "url": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/original_episode_0.mp4",
      "src": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/original_episode_0.mp4",
      "status": "invalid_or_wrong_resolution",
      "kind": "direct_video",
      "bytes": 272563,
      "reused": true,
      "content_type": "video/mp4",
      "valid": true,
      "height": 512,
      "width": 512,
      "duration": "5.666667",
      "local_path": "attachments/181224c20509/videos/repair-0dbfcfa9fc828c843427.mp4",
      "title": "",
      "retried_at": "2026-10-20T17:48:17Z"
    },
    {
      "url": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/original_episode_1.mp4",
      "src": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/original_episode_1.mp4",
      "status": "invalid_or_wrong_resolution",
      "kind": "direct_video",
      "bytes": 711927,
      "reused": true,
      "content_type": "video/mp4",
      "valid": true,
      "height": 512,
      "width": 512,
      "duration": "3.533333",
      "local_path": "attachments/181224c20509/videos/repair-dba1f09c71b8c4d456aa.mp4",
      "title": "",
      "retried_at": "2026-10-20T17:48:17Z"
    },
    {
      "url": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/original_episode_2.mp4",
      "src": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/original_episode_2.mp4",
      "status": "invalid_or_wrong_resolution",
      "kind": "direct_video",
      "bytes": 251630,
      "reused": true,
      "content_type": "video/mp4",
      "valid": true,
      "height": 512,
      "width": 512,
      "duration": "4.733333",
      "local_path": "attachments/181224c20509/videos/repair-221a7e49967a38e863ed.mp4",
      "title": "",
      "retried_at": "2026-10-20T17:48:17Z"
    },
    {
      "url": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/saw_episode_0.mp4",
      "src": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/saw_episode_0.mp4",
      "status": "invalid_or_wrong_resolution",
      "kind": "direct_video",
      "bytes": 127958,
      "reused": true,
      "content_type": "video/mp4",
      "valid": true,
      "height": 512,
      "width": 512,
      "duration": "2.400000",
      "local_path": "attachments/181224c20509/videos/repair-b12784961992c1e940b7.mp4",
      "title": "",
      "retried_at": "2026-10-20T17:48:17Z"
    },
    {
      "url": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/saw_episode_1.mp4",
      "src": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/saw_episode_1.mp4",
      "status": "invalid_or_wrong_resolution",
      "kind": "direct_video",
      "bytes": 465104,
      "reused": true,
      "content_type": "video/mp4",
      "valid": true,
      "height": 512,
      "width": 512,
      "duration": "2.400000",
      "local_path": "attachments/181224c20509/videos/repair-06ef69234f189776bf82.mp4",
      "title": "",
      "retried_at": "2026-10-20T17:48:17Z"
    },
    {
      "url": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/saw_episode_2.mp4",
      "src": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/saw_episode_2.mp4",
      "status": "invalid_or_wrong_resolution",
      "kind": "direct_video",
      "bytes": 134477,
      "reused": true,
      "content_type": "video/mp4",
      "valid": true,
      "height": 512,
      "width": 512,
      "duration": "2.466667",
      "local_path": "attachments/181224c20509/videos/repair-538c7290a62d65682dd4.mp4",
      "title": "",
      "retried_at": "2026-10-20T17:48:18Z"
    },
    {
      "url": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/enemy_episode_0.mp4",
      "src": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/enemy_episode_0.mp4",
      "status": "invalid_or_wrong_resolution",
      "kind": "direct_video",
      "bytes": 122079,
      "reused": true,
      "content_type": "video/mp4",
      "valid": true,
      "height": 512,
      "width": 512,
      "duration": "2.266667",
      "local_path": "attachments/181224c20509/videos/repair-dadd50be9aa2fdc95a39.mp4",
      "title": "",
      "retried_at": "2026-10-20T17:48:18Z"
    },
    {
      "url": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/enemy_episode_1.mp4",
      "src": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/enemy_episode_1.mp4",
      "status": "invalid_or_wrong_resolution",
      "kind": "direct_video",
      "bytes": 287115,
      "reused": true,
      "content_type": "video/mp4",
      "valid": true,
      "height": 512,
      "width": 512,
      "duration": "1.533333",
      "local_path": "attachments/181224c20509/videos/repair-f3aea81959c4e3c4ad86.mp4",
      "title": "",
      "retried_at": "2026-10-20T17:48:18Z"
    },
    {
      "url": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/enemy_episode_2.mp4",
      "src": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/enemy_episode_2.mp4",
      "status": "invalid_or_wrong_resolution",
      "kind": "direct_video",
      "bytes": 225830,
      "reused": true,
      "content_type": "video/mp4",
      "valid": true,
      "height": 512,
      "width": 512,
      "duration": "4.000000",
      "local_path": "attachments/181224c20509/videos/repair-27a5ad6b51251dfd7955.mp4",
      "title": "",
      "retried_at": "2026-10-20T17:48:18Z"
    }
  ],
  "links": [
    {
      "href": "https://distill.pub/",
      "label": "Distill",
      "type": "a"
    },
    {
      "href": "https://www.jacobh.co.uk/",
      "label": "Jacob Hilton",
      "type": "a"
    },
    {
      "href": "https://openai.com/",
      "label": "OpenAI",
      "type": "a"
    },
    {
      "href": "http://nickcammarata.com/",
      "label": "Nick Cammarata",
      "type": "a"
    },
    {
      "href": "http://shancarter.com/",
      "label": "Shan Carter",
      "type": "a"
    },
    {
      "href": "http://observablehq.com/",
      "label": "Observable",
      "type": "a"
    },
    {
      "href": "http://gabgoh.github.io/",
      "label": "Gabriel Goh",
      "type": "a"
    },
    {
      "href": "https://colah.github.io/",
      "label": "Chris Olah",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.23915/distill.00029",
      "label": "10.23915/distill.00029",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2020/understanding-rl-vision/videos/coinrun.mp4",
      "label": "",
      "type": "source"
    },
    {
      "href": "https://github.com/openai/coinrun",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2020/circuits/",
      "label": "circuit",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/original_episode_0.mp4",
      "label": "",
      "type": "source"
    },
    {
      "href": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/original_episode_1.mp4",
      "label": "",
      "type": "source"
    },
    {
      "href": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/original_episode_2.mp4",
      "label": "",
      "type": "source"
    },
    {
      "href": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/saw_episode_0.mp4",
      "label": "",
      "type": "source"
    },
    {
      "href": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/saw_episode_1.mp4",
      "label": "",
      "type": "source"
    },
    {
      "href": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/saw_episode_2.mp4",
      "label": "",
      "type": "source"
    },
    {
      "href": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/enemy_episode_0.mp4",
      "label": "",
      "type": "source"
    },
    {
      "href": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/enemy_episode_1.mp4",
      "label": "",
      "type": "source"
    },
    {
      "href": "https://distill.pub/2020/understanding-rl-vision/videos/model_editing/enemy_episode_2.mp4",
      "label": "",
      "type": "source"
    },
    {
      "href": "https://openaipublic.blob.core.windows.net/rl-clarity/attribution/demo/interface.html",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://openaipublic.blob.core.windows.net/rl-clarity/attribution/finite_levels/index.html",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://distill.pub/2017/feature-visualization/",
      "label": "Feature visualization",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/github/tensorflow/lucid/blob/master/notebooks/misc/feature_inversion_caricatures.ipynb",
      "label": "this notebook",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Multicollinearity",
      "label": "collinearity",
      "type": "a"
    },
    {
      "href": "https://github.com/tensorflow/lucid",
      "label": "Lucid",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/github/tensorflow/lucid/blob/master/notebooks/misc/rl_util.ipynb",
      "label": "notebook",
      "type": "a"
    },
    {
      "href": "https://openaipublic.blob.core.windows.net/rl-clarity/attribution/models/index.html",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://openaipublic.blob.core.windows.net/rl-clarity/attribution/index.html",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/understanding-rl-vision",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Projection_(linear_algebra)",
      "label": "orthogonal projection",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Commutative_diagram",
      "label": "diagram",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Gradient_theorem",
      "label": "fundamental theorem of calculus for line integrals",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--understanding-rl-vision/issues/6",
      "label": "Review 1 - Anonymous",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--understanding-rl-vision/issues/7",
      "label": "Review 2 - Jonathan Uesato",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--understanding-rl-vision/issues/8",
      "label": "Review 3 - Joel Lehman",
      "type": "a"
    },
    {
      "href": "https://openai.com/blog/quantifying-generalization-in-reinforcement-learning/",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1312.6034.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1311.2901.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1412.6806.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://openaccess.thecvf.com/content_ICCV_2017/papers/Selvaraju_Grad-CAM_Visual_Explanations_ICCV_2017_paper.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://openaccess.thecvf.com/content_ICCV_2017/papers/Fong_Interpretable_Explanations_of_ICCV_2017_paper.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1705.05598.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1711.00867.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1703.01365.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.23915/distill.00010",
      "label": "DOI: 10.23915/distill.00010",
      "type": "a"
    },
    {
      "href": "https://openai.com/blog/procgen-benchmark/",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "https://openai.com/blog/openai-baselines-ppo/",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1506.02438.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.23915/distill.00024",
      "label": "DOI: 10.23915/distill.00024",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1802.10363",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1902.01378.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1912.02975.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.23915/distill.00007",
      "label": "DOI: 10.23915/distill.00007",
      "type": "a"
    },
    {
      "href": "https://www.researchgate.net/profile/Aaron_Courville/publication/265022827_Visualizing_Higher-Layer_Features_of_a_Deep_Network/links/53ff82b00cf24c81027da530.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Nguyen_Deep_Neural_Networks_2015_CVPR_paper.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html",
      "label": "[HTML]",
      "type": "a"
    },
    {
      "href": "http://openaccess.thecvf.com/content_cvpr_2017/papers/Nguyen_Plug__Play_CVPR_2017_paper.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://www.image-net.org/papers/imagenet_cvpr09.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.1109/cvprw.2009.5206848",
      "label": "DOI: 10.1109/cvprw.2009.5206848",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1409.4842.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1812.07069.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1904.01318.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1312.6199.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1711.00138.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://nikaashpuri.github.io/sarfa-saliency/",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "https://rmozone.com/snapshots/2017/10/rmo-at-google/",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "http://www.cs.columbia.edu/~djhsu/papers/biasvariance-pnas.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://gradientscience.org/adv/",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.23915/distill.00019",
      "label": "DOI: 10.23915/distill.00019",
      "type": "a"
    },
    {
      "href": "https://deepmind.com/blog/article/capture-the-flag-science",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "https://openai.com/blog/solving-rubiks-cube/",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "https://openai.com/projects/five/",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1802.01561.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--understanding-rl-vision/issues/new",
      "label": "create an issue on GitHub",
      "type": "a"
    },
    {
      "href": "https://creativecommons.org/licenses/by/4.0/",
      "label": "CC-BY 4.0",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--understanding-rl-vision",
      "label": "source available on GitHub",
      "type": "a"
    },
    {
      "href": "https://distill.pub/about/",
      "label": "About",
      "type": "a"
    },
    {
      "href": "https://distill.pub/journal/",
      "label": "Submit",
      "type": "a"
    },
    {
      "href": "https://distill.pub/prize/",
      "label": "Prize",
      "type": "a"
    },
    {
      "href": "https://distill.pub/archive/",
      "label": "Archive",
      "type": "a"
    },
    {
      "href": "https://distill.pub/rss.xml",
      "label": "RSS",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub",
      "label": "GitHub",
      "type": "a"
    },
    {
      "href": "https://twitter.com/distillpub",
      "label": "Twitter",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:54:15Z",
  "source_url": "https://distill.pub/2020/understanding-rl-vision/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1312.6034.pdf",
      "bytes": 2262025,
      "reused": true,
      "local_path": "papers/repair/5c375055da17ead61604.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1311.2901.pdf",
      "bytes": 36235279,
      "reused": true,
      "local_path": "papers/repair/9d149e860294518aabab.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1412.6806.pdf",
      "bytes": 4229020,
      "reused": true,
      "local_path": "papers/repair/0072116e42866e51c960.pdf"
    },
    {
      "status": "ok",
      "url": "http://openaccess.thecvf.com/content_ICCV_2017/papers/Selvaraju_Grad-CAM_Visual_Explanations_ICCV_2017_paper.pdf",
      "bytes": 1496588,
      "reused": false,
      "local_path": "papers/repair/6b28420118cf4b344cbb.pdf"
    },
    {
      "status": "ok",
      "url": "http://openaccess.thecvf.com/content_ICCV_2017/papers/Fong_Interpretable_Explanations_of_ICCV_2017_paper.pdf",
      "bytes": 1065108,
      "reused": false,
      "local_path": "papers/repair/4992198197354acb0f36.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1705.05598.pdf",
      "bytes": 9055859,
      "reused": true,
      "local_path": "papers/repair/9d15dade9336792a7c96.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1711.00867.pdf",
      "bytes": 1080140,
      "reused": true,
      "local_path": "papers/repair/66c6d0970d3a379276f5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1703.01365.pdf",
      "bytes": 3383448,
      "reused": true,
      "local_path": "papers/repair/a5d2fcec94dfa07d83c9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1506.02438.pdf",
      "bytes": 1798117,
      "reused": true,
      "local_path": "papers/repair/be9d8b8466e33236e35e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1802.10363",
      "bytes": 856842,
      "reused": false,
      "local_path": "papers/repair/102a31dfbfdafd312809.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1902.01378.pdf",
      "bytes": 2872628,
      "reused": false,
      "local_path": "papers/repair/604df4fa4c89683af72c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1912.02975.pdf",
      "bytes": 11828188,
      "reused": false,
      "local_path": "papers/repair/c61b78b24a9da13bd212.pdf"
    },
    {
      "status": "http_403",
      "url": "https://www.researchgate.net/profile/Aaron_Courville/publication/265022827_Visualizing_Higher-Layer_Features_of_a_Deep_Network/links/53ff82b00cf24c81027da530.pdf",
      "retried_at": "2026-10-20T17:48:17Z"
    },
    {
      "status": "ok",
      "url": "https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Nguyen_Deep_Neural_Networks_2015_CVPR_paper.pdf",
      "bytes": 8380845,
      "reused": false,
      "local_path": "papers/repair/fd11125f23ed7e1a8447.pdf"
    },
    {
      "status": "ok",
      "url": "http://openaccess.thecvf.com/content_cvpr_2017/papers/Nguyen_Plug__Play_CVPR_2017_paper.pdf",
      "bytes": 1508526,
      "reused": false,
      "local_path": "papers/repair/bed39c4ea023c3ce79eb.pdf"
    },
    {
      "status": "http_404",
      "url": "http://www.image-net.org/papers/imagenet_cvpr09.pdf",
      "retried_at": "2026-10-20T17:48:17Z"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1409.4842.pdf",
      "bytes": 1214160,
      "reused": true,
      "local_path": "papers/repair/2d17a8dd5e8a9936f8ef.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1812.07069.pdf",
      "bytes": 5667599,
      "reused": false,
      "local_path": "papers/repair/5ba6b858996b87e57bc1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.01318.pdf",
      "bytes": 2267807,
      "reused": false,
      "local_path": "papers/repair/86fac4528bb076af2830.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1312.6199.pdf",
      "bytes": 6565323,
      "reused": true,
      "local_path": "papers/repair/67aa78d89bf9969f80c2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1711.00138.pdf",
      "bytes": 2657346,
      "reused": false,
      "local_path": "papers/repair/b3b635b1dcbfb2ec0012.pdf"
    },
    {
      "status": "ok",
      "url": "http://www.cs.columbia.edu/~djhsu/papers/biasvariance-pnas.pdf",
      "bytes": 1179365,
      "reused": false,
      "local_path": "papers/repair/287a1ced83136499c0d3.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1802.01561.pdf",
      "bytes": 5601159,
      "reused": false,
      "local_path": "papers/repair/371d8c99b87c07ea8d08.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 57510,
  "expected_body_images": 54,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "http_403；http_404；invalid_or_wrong_resolution",
  "raw_html": "attachments/181224c20509/pages/source.html",
  "local_html": "attachments/181224c20509/pages/article.html",
  "body_base_path": "attachments/181224c20509/pages/",
  "body_markdown_path": "attachments/181224c20509/pages/article.md"
}