{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Lilian Weng / Lil'Log",
  "title_listed": "Neural Architecture Search",
  "keywords": "NAS,架构搜索,自动ML",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://lilianweng.github.io/posts/2020-08-06-nas/",
  "publish_date": "2020-08",
  "corpus_stage": "预训练/SFT",
  "notes": "神经架构搜索综述",
  "id": "475f6d363978",
  "collected_at": "2026-10-02T14:13:15",
  "title": "Neural Architecture Search | Lil'Log",
  "body_markdown": "Although most popular and successful model architectures are designed by human experts, it doesn’t mean we have explored the entire network architecture space and settled down with the best option. We would have a better chance to find the optimal solution if we adopt a systematic and automatic way of learning high-performance model architectures.\nAutomatically learning and evolving network topologies is not a new idea (\nStanley & Miikkulainen, 2002\n). In recent years, the pioneering work by\nZoph & Le 2017\nand\nBaker et al. 2017\nhas attracted a lot of attention into the field of Neural Architecture Search (NAS), leading to many interesting ideas for better, faster and more cost-efficient NAS methods.\nAs I started looking into NAS, I found this nice survey very helpful by\nElsken, et al 2019\n. They characterize NAS as a system with three major components, which is clean & concise, and also commonly adopted in other NAS papers.\nSearch space\n: The NAS search space defines a set of operations (e.g. convolution, fully-connected, pooling) and how operations can be connected to form valid network architectures. The design of search space usually involves human expertise, as well as unavoidably human biases.\nSearch algorithm\n: A NAS search algorithm samples a population of network architecture candidates. It receives the child model performance metrics as rewards (e.g. high accuracy, low latency) and optimizes to generate high-performance architecture candidates.\nEvaluation strategy\n: We need to measure, estimate, or predict the performance of a large number of proposed child models in order to obtain feedback for the search algorithm to learn. The process of candidate evaluation could be very expensive and many new methods have been proposed to save time or computation resources.\nThree main components of Neural Architecture Search (NAS) models.  (Image source:\nElsken, et al. 2019\nwith customized annotation in red)\nSearch Space\n#\nThe NAS search space defines a set of basic network operations and how operations can be connected to construct valid network architectures.\nSequential Layer-wise Operations\n#\nThe most naive way to design the search space for neural network architectures is to depict network topologies, either CNN or RNN, with a list of\nsequential layer-wise operations\n, as seen in the early work of\nZoph & Le 2017\n&\nBaker et al. 2017\n. The serialization of network representation requires a decent amount of expert knowledge, since each operation is associated with different layer-specific parameters and such associations need to be hardcoded. For example, after predicting a\nconv\nop, the model should output kernel size, stride size, etc; or after predicting an\nFC\nop, we need to see the number of units as the next prediction.\n(Top) A sequential representation of CNN. (Bottom) A sequential representation of the tree structure of a recurrent cell. (Image source:\nZoph & Le 2017\n)\nTo make sure the generated architecture is valid, additional rules might be needed (\nZoph & Le 2017\n):\nIf a layer is not connected to any input layer then it is used as the input layer;\nAt the final layer, take all layer outputs that have not been connected and concatenate them;\nIf one layer has many input layers, then all input layers are concatenated in the depth dimension;\nIf input layers to be concatenated have different sizes, we pad the small layers with zeros so that the concatenated layers have the same sizes.\nThe skip connection can be predicted as well, using an\nattention\n-style mechanism. At layer $i$ , an anchor point is added with $i−1$ content-based sigmoids to indicate which of the previous layers to be connected. Each sigmoid takes as input the hidden states of the current node $h_i$ and $i-1$ previous nodes $h_j, j=1, \\dots, i-1$ .\n$$\nP(\\text{Layer j is an input to layer i}) = \\text{sigmoid}(v^\\top \\tanh(\\mathbf{W}_\\text{prev} h_j + \\mathbf{W}_\\text{curr} h_i))\n$$\nThe sequential search space has a lot of representation power, but it is very large and consumes a ton of computation resources to exhaustively cover the search space. In the experiments by\nZoph & Le 2017\n, they were running 800 GPUs in parallel for 28 days and\nBaker et al. 2017\nrestricted the search space to contain at most 2\nFC\nlayers.\nCell-based Representation\n#\nInspired by the design of using repeated modules in successful vision model architectures (e.g. Inception, ResNet), the\nNASNet search space\n(\nZoph et al. 2018\n) defines the architecture of a conv net as the same cell getting repeated multiple times and each cell contains several operations predicted by the NAS algorithm. A well-designed cell module enables transferability between datasets. It is also easy to scale down or up the model size by adjusting the number of cell repeats.\nPrecisely, the NASNet search space learns two types of cells for network construction:\nNormal Cell\n: The input and output feature maps have the same dimension.\nReduction Cell\n: The output feature map has its width and height reduced by half.\nThe NASNet search space constrains the architecture as a repeated stack of cells. The cell architecture is optimized via NAS algorithms. (Image source:\nZoph et al. 2018\n)\nThe predictions for each cell are grouped into $B$ blocks ($B=5$ in the NASNet paper), where each block has 5 prediction steps made by 5 distinct softmax classifiers corresponding to discrete choices of the elements of a block. Note that the NASNet search space does not have residual connections between cells and the model only learns skip connections on their own within blocks.\n(a) Each cell consists of $B$ blocks and each block is predicted by 5 discrete decisions. (b) An concrete example of what operations can be chosen in each decision step.\nDuring the experiments, they discovered that a modified version of\nDropPath\n, named\nScheduledDropPath\n, significantly improves the final performance of NASNet experiments. DropPath stochastically drops out paths (i.e. edges with operations attached in NASNet) with a fixed probability. ScheduledDropPath is DropPath with a linearly increasing probability of path dropping during training time.\nElsken, et al (2019)\npoint out three major advantages of the NASNet search space:\nThe search space size is reduced drastically;\nThe\nmotif\n-based architecture can be more easily transferred to different datasets.\nIt demonstrates a strong proof of a useful design pattern of repeatedly stacking modules in architecture engineering. For example, we can build strong models by stacking residual blocks in CNN or stacking multi-headed attention blocks in Transformer.\nHierarchical Structure\n#\nTo take advantage of already discovered well-designed network\nmotifs\n, the NAS search space can be constrained as a hierarchical structure, as in\nHierarchical NAS\n(\nHNAS\n; (\nLiu et al 2017\n)). It starts with a small set of primitives, including individual operations like convolution operation, pooling, identity, etc. Then small sub-graphs (or “motifs”) that consist of primitive operations are recursively used to form higher-level computation graphs.\nA computation motif at level $\\ell=1, \\dots, L$ can be represented by $(G^{(\\ell)}, \\mathcal{O}^{(\\ell)})$, where:\n$\\mathcal{O}^{(\\ell)}$ is a set of operations, $\\mathcal{O}^{(\\ell)} = \\{ o^{(\\ell)}_1, o^{(\\ell)}_2, \\dots \\}$\n$G^{(\\ell)}$ is an adjacency matrix, where the entry $G_{ij}=k$ indicates that operation $o^{(\\ell)}_k$ is placed between node $i$ and $j$. The node indices follow\ntopological ordering\nin DAG, where the index $1$ is the source and the maximal index is the sink node.\n(Top) Three level-1 primitive operations are composed into a level-2 motif. (Bottom) Three level-2 motifs are plugged into a base network structure and assembled into a level-3 motif. (Image source:\nLiu et al 2017\n)\nTo build a network according to the hierarchical structure, we start from the lowest level $\\ell=1$ and recursively define the $m$-th motif operation at level $\\ell$ as\n$$\no^{(\\ell)}_m = \\text{assemble}\\Big( G_m^{(\\ell)}, \\mathcal{O}^{(\\ell-1)} \\Big)\n$$\nA hierarchical representation becomes $\\Big( \\big\\{ \\{ G_m^{(\\ell)} \\}_{m=1}^{M_\\ell} \\big\\}_{\\ell=2}^L, \\mathcal{O}^{(1)} \\Big), \\forall \\ell=2, \\dots, L$, where $\\mathcal{O}^{(1)}$ contains a set of primitive operations.\nThe $\\text{assemble}()$ process is equivalent to sequentially compute the feature map of node $i$ by aggregating all the feature maps of its predecessor node $j$ following the topological ordering:\n$$\nx_i = \\text{merge} \\big[ \\{ o^{(\\ell)}_{G^{(\\ell)}_{ij}}(x_j) \\}_{j < i} \\big], i = 2, \\dots, \\vert G^{(\\ell)} \\vert\n$$\nwhere $\\text{merge}[]$ is implemented as depth-wise concatenation in the\npaper\n.\nSame as NASNet, experiments in\nLiu et al (2017)\nfocused on discovering good cell architecture within a predefined “macro” structure with repeated modules. They showed that the power of simple search methods (e.g. random search or evolutionary algorithms) can be substantially enhanced using well-designed search spaces.\nCai et al (2018b)\npropose a tree-structure search space using path-level network transformation. Each node in a tree structure defines an\nallocation\nscheme for splitting inputs for child nodes and a\nmerge\nscheme for combining results from child nodes. The path-level network transformation allows replacing a single layer with a multi-branch motif if its corresponding merge scheme is add or concat.\nAn illustration of transforming a single layer to a tree-structured motif via path-level transformation operations. (Image source:\nCai et al. 2018b\n)\nMemory-bank Representation\n#\nA memory-bank representation of feed-forward networks is proposed by\nBrock et al. (2017)\nin\nSMASH\n. Instead of a graph of operations, they view a neural network as a system with multiple memory blocks which can read and write. Each layer operation is designed to: (1) read from a subset of memory blocks; (2) computes results; finally (3) write the results into another subset of blocks. For example, in a sequential model, a single memory block would get read and overwritten consistently.\nMemory-bank representation of several popular network architecture blocks. (Image source:\nBrock et al. 2017\n)\nSearch Algorithms\n#\nNAS search algorithms sample a population of child networks. It receives the child models’ performance metrics as rewards and learns to generate high-performance architecture candidates. You may a lot in common with the field of hyperparameter search.\nRandom Search\n#\nRandom search is the most naive baseline. It samples a valid architecture candidate from the search space\nat random\nand no learning model is involved. Random search has proved to be quite useful in hyperparameter search (\nBergstra & Bengio 2012\n). With a well-designed search space, random search could be a very challenging baseline to beat.\nReinforcement Learning\n#\nThe initial design of\nNAS\n(\nZoph & Le 2017\n) involves a RL-based controller for proposing child model architectures for evaluation. The controller is implemented as a RNN, outputting a variable-length sequence of tokens used for configuring a network architecture.\nA high level overview of NAS, containing a RNN controller and a pipeline for evaluating child models. (Image source:\nZoph & Le 2017\n)\nThe controller is trained as a\nRL task\nusing\nREINFORCE\n.\nAction space\n: The action space is a list of tokens for defining a child network predicted by the controller (See more in the above\nsection\n). The controller outputs\naction\n, $a_{1:T}$, where $T$ is the total number of tokens.\nReward\n: The accuracy of a child network that can be achieved at convergence is the reward for training the controller, $R$.\nLoss\n: NAS optimizes the controller parameters $\\theta$ with a REINFORCE loss. We want to maximize the expected reward (high accuracy) with the gradient as follows. The nice thing here with policy gradient is that it works even when the reward is non-differentiable.\n$$\n\\nabla_{\\theta} J(\\theta) = \\sum_{t=1}^T \\mathbb{E}[\\nabla_{\\theta} \\log P(a_t \\vert a_{1:(t-1)}; \\theta) R ]\n$$\nMetaQNN\n(\nBaker et al. 2017\n) trains an agent to sequentially choose CNN layers using\nQ-learning\nwith an\n$\\epsilon$-greedy\nexploration strategy and experience replay. The reward is the validation accuracy as well.\n$$\nQ^{(t+1)}(s_t,  a_t) = (1 - \\alpha)Q^{(t)}(s_t, a_t) + \\alpha (R_t + \\gamma \\max_{a \\in \\mathcal{A}} Q^{(t)}(s_{t+1},  a'))\n$$\nwhere a state $s_t$ is a tuple of layer operation and related parameters. An action $a$ determines the connectivity between operations. The Q-value is proportional to how confident we are in two connected operations leading to high accuracy.\nOverview of MetaQNN - designing CNN models with Q-Learning. (Image source:\nBaker et al. 2017\n)\nEvolutionary Algorithms\n#\nNEAT\n(short for\nNeuroEvolution of Augmenting Topologies\n) is an approach for evolving neural network topologies with\ngenetic algorithm (GA)\n, proposed by\nStanley & Miikkulainen\nin 2002. NEAT evolves both connection weights and network topology together. Each gene encodes the full information for configuring a network, including node weights and edges. The population grows by applying mutation of both weights and connections, as well as crossover between two parent genes. For more in neuroevolution, please refer to the in-depth\nsurvey\nby Stanley et al. (2019).\nMutations in the NEAT algorithm. (Image source: Fig 3 & 4 in\nStanley & Miikkulainen, 2002\n)\nReal et al. (2018)\nadopt the evolutionary algorithms (EA) as a way to search for high-performance network architectures, named\nAmoebaNet\n. They apply the\ntournament selection\nmethod, which at each iteration picks a best candidate out of a random set of samples and places its mutated offspring back into the population. When the tournament size is $1$, it is equivalent to random selection.\nAmoebaNet modified the tournament selection to favor\nyounger\ngenotypes and always discard the oldest models within each cycle. Such an approach, named\naging evolution\n, allows AmoebaNet to cover and explore more search space, rather than to narrow down on good performance models too early.\nPrecisely, in every cycle of the tournament selection with aging regularization (See Figure 11):\nSample $S$ models from the population and the one with highest accuracy is chosen as\nparent\n.\nA\nchild\nmodel is produced by mutating\nparent\n.\nThen the child model is trained, evaluated and added back into the population.\nThe oldest model is removed from the population.\nThe algorithm of aging evolution. (Image source:\nReal et al. 2018\n)\nTwo types of mutations are applied:\nHidden state mutation\n: randomly chooses a pairwise combination and rewires a random end such that there is no loop in the graph.\nOperation mutation\n: randomly replaces an existing operation with a random one.\nTwo types of mutations in AmoebaNet. (Image source:\nReal et al. 2018\n)\nIn their experiments, EA and RL work equally well in terms of the final validation accuracy, but EA has better anytime performance and is able to find smaller models. Here using EA in NAS is still expensive in terms of computation, as each experiment took 7 days with 450 GPUs.\nHNAS\n(\nLiu et al 2017\n) also employs the evolutionary algorithms (the original tournament selection) as their search strategy. In the\nhierarchical structure\nsearch space, each edge is an operation. Thus genotype mutation in their experiments is applied by replacing a random edge with a different operation. The replacement set includes an\nnone\nop, so it can alter, remove and add an edge. The initial set of genotypes is created by applying a large number of random mutations on “trivial” motifs (all identity mappings).\nProgressive Decision Process\n#\nConstructing a model architecture is a sequential process. Every additional operator or layer brings extra complexity. If we guide the search model to start the investigation from simple models and gradually evolve to more complex architectures, it is like to introduce\n“curriculum”\ninto the search model’s learning process.\nProgressive NAS\n(\nPNAS\n;\nLiu, et al 2018\n) frames the problem of NAS as a progressive procedure for searching models of increasing complexity. Instead of RL or EA, PNAS adopts a Sequential Model-based Bayesian Optimization (SMBO) as the search strategy. PNAS works similar to A* search, as it searches for models from simple to hard while simultaneously learning a surrogate function to guide the search.\nA* search algorithm\n(“best-first search”) is a popular algorithm for path finding. The problem is framed as finding a path of smallest cost from a specific starting node to a given target node in a weighted graph. At each iteration, A* finds a path to extend by minimizing: $f(n)=g(n)+h(n)$, where $n$ is the next node, $g(n)$ is the cost from start to $n$, and $h(n)$ is the heuristic function that estimates the minimum cost of going from node $n$ to the goal.\nPNAS uses the\nNASNet\nsearch space. Each block is specified as a 5-element tuple and PNAS only considers the element-wise addition as the step 5 combination operator, no concatenation. Differently, instead of setting the number of blocks $B$ at a fixed number, PNAS starts with $B=1$, a model with only one block in a cell, and gradually increases $B$.\nThe performance on a validation set is used as feedback to train a\nsurrogate\nmodel for\npredicting\nthe performance of novel architectures. With this predictor, we can thus decide which models should be prioritized to be evaluated next. Since the performance predictor should be able to handle various-sized inputs, accuracy, and sample-efficient, they ended up using an RNN model.\nThe algorithm of Progressive NAS. (Image source:\nLiu, et al 2018\n)\nGradient descent\n#\nUsing gradient descent to update the architecture search model requires an effort to make the process of choosing discrete operations differentiable. These approaches usually combine the learning of both architecture parameters and network weights together into one model. See more in the\nsection\non the\n“one-shot”\napproach.\nEvaluation Strategy\n#\nWe need to measure, estimate or predict the performance of every child model in order to obtain feedback for optimizing the search algorithm. The process of candidate evaluation could be very expensive and many new evaluation methods have been proposed to save time or computation. When evaluating a child model, we mostly care about its performance measured as accuracy on a validation set. Recent work has started looking into other factors of a model, such as model size and latency, as certain devices may have limitations on memory or demand fast response time.\nTraining from Scratch\n#\nThe most naive approach is to train every child network independently from scratch until\nconvergence\nand then measure its accuracy on a validation set (\nZoph & Le 2017\n). It provides solid performance numbers, but one complete train-converge-evaluate loop only generates a single data sample for training the RL controller (let alone RL is known to be sample-inefficient in general). Thus it is very expensive in terms of computation consumption.\nProxy Task Performance\n#\nThere are several approaches for using a proxy task performance as the performance estimator of a child network, which is generally cheaper and faster to calculate:\nTrain on a smaller dataset.\nTrain for fewer epochs.\nTrain and evaluate a down-scaled model in the search stage. For example, once a cell structure is learned, we can play with the number of cell repeats or scale up the number of filters (\nZoph et al. 2018\n).\nPredict the learning curve.\nBaker et al (2018)\nmodel the prediction of validation accuracies as a time-series regression problem. The features for the regression model ($\\nu$-support vector machine regressions; $\\nu$-SVR) include the early sequences of accuracy per epoch, architecture parameters, and hyperparameters.\nParameter Sharing\n#\nInstead of training every child model independently from scratch. You may ask, ok, what if we fabricate dependency between them and find a way to reuse weights? Some researchers succeeded to make such approaches work.\nInspired by\nNet2net\ntransformation,\nCai et al (2017)\nproposed\nEfficient Architecture Search\n(\nEAS\n). EAS sets up an RL agent, known as a meta-controller, to predict function-preserving network transformation so as to grow the network depth or layer width. Because the network is growing incrementally, the weights of previously validated networks can be\nreused\nfor further exploration. With inherited weights, newly constructed networks only need some light-weighted training.\nA meta-controller learns to generate\nnetwork transformation actions\ngiven the current network architecture, which is specified with a variable-length string. In order to handle architecture configuration of a variable length, the meta-controller is implemented as a bi-directional recurrent network. Multiple actor networks output different transformation decisions:\nNet2WiderNet\noperation allows to replace a layer with a wider layer, meaning more units for fully-connected layers, or more filters for convolutional layers, while preserving the functionality.\nNet2DeeperNet\noperation allows to insert a new layer that is initialized as adding an identity mapping between two layers so as to preserve the functionality.\nOverview of the RL based meta-controller in Efficient Architecture Search (NAS). After encoding the architecture configuration, it outputs net2net transformation actions through two separate actor networks. (Image source:\nCai et al 2017\n)\nWith similar motivation,\nEfficient NAS\n(\nENAS\n;\nPham et al. 2018\n) speeds up NAS (i.e. 1000x less) by aggressively sharing parameters among child models. The core motivation behind ENAS is the observation that all of the sampled architecture graphs can be viewed as\nsub-graphs\nof a larger\nsupergraph\n. All the child networks are sharing weights of this supergraph.\n(Left) The graph represents the entire search space for a 4-node recurrent cell, but only connections in red are active. (Middle) An example of how the left active sub-graph can be translated into a child model architecture. (Right) The network parameters produced by an RNN controller for the architecture in the middle. (Image source:\nPham et al. 2018\n)\nENAS alternates between training the shared model weights $\\omega$ and training the controller $\\theta$:\nThe parameters of the controller LSTM $\\theta$ are trained with\nREINFORCE\n, where the reward $R(\\mathbf{m}, \\omega)$ is computed on the validation set.\nThe shared parameters of the child models $\\omega$ are trained with standard supervised learning loss. Note that different operators associated with the same node in the supergraph would have their own distinct parameters.\nPrediction-Based\n#\nA routine child model evaluation loop is to update model weights via standard gradient descent. SMASH (\nBrock et al. 2017\n) proposes a different and interesting idea:\nCan we predict the model weights directly based on the network architecture parameters?\nThey employ a\nHyperNet\n(\nHa et al 2016\n) to directly generate the weights of a model conditioned on an encoding of its architecture configuration. Then the model with HyperNet-generated weights is validated directly. Note that we don’t need extra training for every child model but we do need to train the HyperNet.\nThe algorithm of SMASH. (Image source:\nBrock et al. 2017\n)\nThe correlation between model performance with SMASH-generated weights and true validation errors suggests that predicted weights can be used for model comparison, to some extent. We do need a HyperNet of large enough capacity, as the correlation would be corrupted if the HyperNet model is too small compared to the child model size.\nThe algorithm of SMASH. (Image source:\nBrock et al. 2017\n)\nSMASH can be viewed as another way to implement the idea of\nparameter sharing\n. One problem of SMASH as pointed out by\nPham et al. (2018)\nis: The usage of HyperNet restricts the weights of SMASH child models to a\nlow-rank space\n, because weights are generated via tensor products. In comparison,\nENAS\nhas no such restrictions.\nOne-Shot Approach: Search + Evaluation\n#\nRunning search & evaluation independently for a large population of child models is expensive. We have seen promising approaches like\nBrock et al. (2017)\nor\nPham et al. (2018)\n, where training a single model is enough for emulating any child model in the search space.\nThe\none-shot\narchitecture search extends the idea of weight sharing and further combines the learning of architecture generation together with weight parameters. The following approaches all treat child architectures as different sub-graphs of a supergraph with shared weights between common edges in the supergraph.\nBender et al (2018)\nconstruct a single large over-parameterized network, known as the\nOne-Shot model\n, such that it contains every possible operation in the search space. With\nScheduledDropPath\n(the dropout rate is increased over time, which is $r^{1/k}$ at the end of training, where $0 < r < 1$ is a hyperparam and $k$ is the number of incoming paths) and some carefully designed tricks (e.g. ghost batch normalization, L2 regularization only on the active architecture), the training of such a giant model can be stabilized enough and used for evaluating any child model sampled from the supergraph.\nThe architecture of the One-Shot model in\nBender et al 2018\n. Each cell has $N$ choice blocks and each choice block can select up to 2 operations. Solid edges are used in every architecture, where dash lines are optional. (Image source:\nBender et al 2018\n)\nOnce the one-shot model is trained, it is used for evaluating the performance of many different architectures sampled at random by zeroing out or removing some operations. This sampling process can be replaced by RL or evolution.\nThey observed that the difference between the accuracy measured with the one-shot model and the accuracy of the same architecture after a small fine-tuning could be very large. Their hypothesis is that the one-shot model automatically learns to focus on the\nmost useful\noperations in the network and comes to\nrely on\nthese operations when they are available. Thus zeroing out useful operations lead to big reduction in model accuracy, while removing less important components only causes a small impact — Therefore, we see a larger variance in scores when using the one-shot model for evaluation.\nA stratified sample of models with different one-shot model accuracy versus their true validation accuracy as stand-alone models. (Image source:\nBender et al 2018\n)\nClearly designing such a search graph is not a trivial task, but it demonstrates a strong potential with the one-shot approach. It works well with only gradient descent and no additional algorithm like RL or EA is a must.\nSome believe that one main cause for inefficiency in NAS is to treat the architecture search as a\nblack-box optimization\nand thus we fall into methods like RL, evolution, SMBO, etc. If we shift to rely on standard gradient descent, we could potentially make the search process more effectively. As a result,\nLiu et al (2019)\npropose\nDifferentiable Architecture Search\n(\nDARTS\n). DARTS introduces a continuous relaxation on each path in the search supergraph, making it possible to jointly train architecture parameters and weights via gradient descent.\nLet’s use the directed acyclic graph (DAG) representation here. A cell is a DAG consisting of a topologically ordered sequence of $N$ nodes. Each node has a latent representation $x_i$ to be learned. Each edge $(i, j)$ is tied to some operation $o^{(i,j)} \\in \\mathcal{O}$ that transforms $x_j$ to compose $x_i$:\n$$\nx_i = \\sum_{j < i} o^{(i,j)}(x_j)\n$$\nTo make the search space continuous, DARTS relaxes the categorical choice of a particular operation as a softmax over all the operations and the task of architecture search is reduced to learn a set of mixing probabilities $\\alpha = \\{ \\alpha^{(i,j)} \\}$.\n$$\n\\bar{o}^{(i,j)}(x) = \\sum_{o\\in\\mathcal{O}} \\frac{\\exp(\\alpha_{ij}^o)}{\\sum_{o'\\in\\mathcal{O}} \\exp(\\alpha^{o'}_{ij})} o(x)\n$$\nwhere $\\alpha_{ij}$ is a vector of dimension $\\vert \\mathcal{O} \\vert$, containing weights between nodes $i$ and $j$ over different operations.\nThe bilevel optimization exists as we want to optimize both the network weights $w$ and the architecture representation $\\alpha$:\n$$\n\\begin{aligned}\n\\min_\\alpha & \\mathcal{L}_\\text{validate} (w^*(\\alpha), \\alpha) \\\\\n\\text{s.t.} & w^*(\\alpha) = \\arg\\min_w \\mathcal{L}_\\text{train} (w, \\alpha)\n\\end{aligned}\n$$\nAt step $k$, given the current architecture parameters $\\alpha_{k−1}$, we first optimize weights $w_k$ by moving $w_{k−1}$ in the direction of minimizing the training loss $\\mathcal{L}_\\text{train}(w_{k−1}, \\alpha_{k−1})$ with a learning rate $\\xi$. Next, while keeping the newly updated weights $w_k$ fixed, we update the mixing probabilities so as to minimize the validation loss\nafter a single step of gradient descent w.r.t. the weights\n:\n$$\nJ_\\alpha = \\mathcal{L}_\\text{val}(w_k - \\xi \\nabla_w \\mathcal{L}_\\text{train}(w_k, \\alpha_{k-1}), \\alpha_{k-1})\n$$\nThe motivation here is that we want to find an architecture with a low validation loss when its weights are optimized by gradient descent and the one-step unrolled weights serve as the\nsurrogate\nfor $w^∗(\\alpha)$.\nSide note: Earlier we have seen similar formulation in\nMAML\nwhere the two-step optimization happens between task losses and the meta-learner update, as well as framing\nDomain Randomization\nas a bilevel optimization for better transfer in the real environment.\nAn illustration of how DARTS applies continuous relaxation on edges in DAG supergraph and identifies the final model. (Image source:\nLiu et al 2019\n)\n$$\n\\begin{aligned}\n\\text{Let }w'_k &= w_k - \\xi \\nabla_w \\mathcal{L}_\\text{train}(w_k, \\alpha_{k-1}) & \\\\\nJ_\\alpha &= \\mathcal{L}_\\text{val}(w_k - \\xi \\nabla_w \\mathcal{L}_\\text{train}(w_k, \\alpha_{k-1}), \\alpha_{k-1}) = \\mathcal{L}_\\text{val}(w'_k, \\alpha_{k-1}) & \\\\\n\\nabla_\\alpha J_\\alpha \n&= \\nabla_{\\alpha_{k-1}} \\mathcal{L}_\\text{val}(w'_k, \\alpha_{k-1}) \\nabla_\\alpha \\alpha_{k-1} + \\nabla_{w'_k} \\mathcal{L}_\\text{val}(w'_k, \\alpha_{k-1})\\nabla_\\alpha w'_k & \\\\& \\text{; multivariable chain rule}\\\\\n&= \\nabla_{\\alpha_{k-1}} \\mathcal{L}_\\text{val}(w'_k, \\alpha_{k-1}) + \\nabla_{w'_k} \\mathcal{L}_\\text{val}(w'_k, \\alpha_{k-1}) \\big( - \\xi \\color{red}{\\nabla^2_{\\alpha, w} \\mathcal{L}_\\text{train}(w_k, \\alpha_{k-1})} \\big) & \\\\\n&\\approx \\nabla_{\\alpha_{k-1}} \\mathcal{L}_\\text{val}(w'_k, \\alpha_{k-1}) - \\xi \\nabla_{w'_k} \\mathcal{L}_\\text{val}(w'_k, \\alpha_{k-1}) \\color{red}{\\frac{\\nabla_\\alpha \\mathcal{L}_\\text{train}(w_k^+, \\alpha_{k-1}) - \\nabla_\\alpha \\mathcal{L}_\\text{train}(w_k^-, \\alpha_{k-1}) }{2\\epsilon}} & \\\\\n& \\text{; apply numerical differentiation approximation}\n\\end{aligned}\n$$\nwhere the red part is using numerical differentiation approximation where $w_k^+ = w_k + \\epsilon \\nabla_{w’_k} \\mathcal{L}_\\text{val}(w’_k, \\alpha_{k-1})$ and $w_k^- = w_k - \\epsilon \\nabla_{w’_k} \\mathcal{L}_\\text{val}(w’_k, \\alpha_{k-1})$.\nThe algorithm overview of DARTS. (Image source:\nLiu et al 2019\n)\nAs another idea similar to DARTS, Stochastic NAS (\nXie et al., 2019\n) applies a continuous relaxation by employing the concrete distribution (CONCRETE = CONtinuous relaxations of disCRETE random variables;\nMaddison et al 2017\n) and reparametrization tricks. The goal is same as DARTS, to make the discrete distribution differentiable and thus enable optimization by gradient descent.\nDARTS is able to greatly reduce the cost of GPU hours. Their experiments for searching for CNN cells have $N=7$ and only took 1.5 days with a single GPU. However, it suffers from the high GPU memory consumption issue due to its continuous representation of network architecture. In order to fit the model into the memory of a single GPU, they picked a small $N$.\nTo constrain the GPU memory consumption,\nProxylessNAS\n(\nCai et al., 2019\n) views NAS as a path-level pruning process in DAG and binarizes the architecture parameters to force only one path to be active between two nodes at a time. The probabilities for an edge being either masked out or not are then learned by sampling a few binarized architectures and using\nBinaryConnect\n(\nCourbariaux et al., 2015\n) to update the corresponding probabilities. ProxylessNAS demonstrates a strong connection between NAS and model compression. By using path-level compression, it is able to save memory consumption by one order of magnitude.\nLet’s continue with the graph representation. In a DAG adjacency matrix $G$ where $G_{ij}$ represents an edge between node $i$ and $j$ and its value can be chosen from the set of $\\vert \\mathcal{O} \\vert$ candidate primitive operations, $\\mathcal{O} = \\{ o_1, \\dots \\}$. The One-Shot model, DARTS and ProxylessNAS all consider each edge as a mixture of operations, $m_\\mathcal{O}$, but with different tweaks.\nIn One-Shot, $m_\\mathcal{O}(x)$ is the sum of all the operations. In DARTS, it is a weighted sum where weights are softmax over a real-valued architecture weighting vector $\\alpha$ of length $\\vert \\mathcal{O} \\vert$. ProxylessNAS transforms the softmax probabilities of $\\alpha$ into a binary gate and uses the binary gate to keep only one operation active at a time.\n$$\n\\begin{aligned}\nm^\\text{one-shot}_\\mathcal{O}(x) &= \\sum_{i=1}^{\\vert \\mathcal{O} \\vert} o_i(x) \\\\\nm^\\text{DARTS}_\\mathcal{O}(x) &= \\sum_{i=1}^{\\vert \\mathcal{O} \\vert} p_i o_i(x) = \\sum_{i=1}^{\\vert \\mathcal{O} \\vert} \\frac{\\exp(\\alpha_i)}{\\sum_j \\exp(\\alpha_j)} o_i(x) \\\\\nm^\\text{binary}_\\mathcal{O}(x) &= \\sum_{i=1}^{\\vert \\mathcal{O} \\vert} g_i o_i(x) = \\begin{cases}\no_1(x) & \\text{with probability }p_1, \\\\\n\\dots &\\\\\no_{\\vert \\mathcal{O} \\vert}(x) & \\text{with probability }p_{\\vert \\mathcal{O} \\vert}\n\\end{cases} \\\\\n\\text{ where } g &= \\text{binarize}(p_1, \\dots, p_N) = \\begin{cases}\n[1, 0, \\dots, 0] & \\text{with probability }p_1, \\\\\n\\dots & \\\\\n[0, 0, \\dots, 1] & \\text{with probability }p_N. \\\\\n\\end{cases}\n\\end{aligned}\n$$\nProxylessNAS has two training steps running alternatively. (Image source:\nCai et al., 2019\n)\nProxylessNAS runs two training steps alternatively:\nWhen training weight parameters $w$, it freezes the architecture parameters $\\alpha$ and stochastically samples binary gates $g$ according to the above $m^\\text{binary}_\\mathcal{O}(x)$. The weight parameters can be updated with standard gradient descent.\nWhen training architecture parameters $\\alpha$, it freezes $w$, resets the binary gates and then updates $\\alpha$ on the validation set. Following the idea of\nBinaryConnect\n,  the gradient w.r.t. architecture parameters can be approximately estimated using $\\partial \\mathcal{L} / \\partial g_i$ in replacement for $\\partial \\mathcal{L} / \\partial p_i$:\n$$\n\\begin{aligned}\n\\frac{\\partial \\mathcal{L}}{\\partial \\alpha_i} \n&= \\sum_{j=1}^{\\vert \\mathcal{O} \\vert} \\frac{\\partial \\mathcal{L}}{\\partial p_j} \\frac{\\partial p_j}{\\partial \\alpha_i} \n\\approx \\sum_{j=1}^{\\vert \\mathcal{O} \\vert} \\frac{\\partial \\mathcal{L}}{\\partial g_j} \\frac{\\partial p_j}{\\partial \\alpha_i} \n= \\sum_{j=1}^{\\vert \\mathcal{O} \\vert} \\frac{\\partial \\mathcal{L}}{\\partial g_j} \\frac{\\partial \\frac{e^{\\alpha_j}}{\\sum_k e^{\\alpha_k}}}{\\partial \\alpha_i} \\\\\n&= \\sum_{j=1}^{\\vert \\mathcal{O} \\vert} \\frac{\\partial \\mathcal{L}}{\\partial g_j} \\frac{\\sum_k e^{\\alpha_k} (\\mathbf{1}_{i=j} e^{\\alpha_j}) - e^{\\alpha_j} e^{\\alpha_i} }{(\\sum_k e^{\\alpha_k})^2}\n= \\sum_{j=1}^{\\vert \\mathcal{O} \\vert} \\frac{\\partial \\mathcal{L}}{\\partial g_j} p_j (\\mathbf{1}_{i=j} -p_i)\n\\end{aligned}\n$$\nInstead of BinaryConnect, REINFORCE can also be used for parameter updates with the goal for maximizing the reward, while no RNN meta-controller is involved.\nComputing $\\partial \\mathcal{L} / \\partial g_i$ needs to calculate and store $o_i(x)$, which requires $\\vert \\mathcal{O} \\vert$ times GPU memory. To resolve this issue, they factorize the task of choosing one path out of $N$ into multiple binary selection tasks (Intuition: “if a path is the best choice, it should be better than any other path”). At every update step, only two paths are sampled while others are masked. These two selected paths are updated according to the above equation and then scaled properly so that other path weights are unchanged. After this process, one of the sampled paths is enhanced (path weight increases) and the other is attenuated (path weight decreases), while all other paths stay unaltered.\nBesides accuracy, ProxylessNAS also considers\nlatency\nas an important metric to optimize, as different devices might have very different requirements on inference time latency (e.g. GPU, CPU, mobile). To make latency differentiable, they model latency as a continuous function of the network dimensions. The expected latency of a mixed operation can be written as $\\mathbb{E}[\\text{latency}] = \\sum_j p_j F(o_j)$, where $F(.)$ is a latency prediction model:\nAdd a differentiable latency loss into the training of ProxylessNAS.  (Image source:\nCai et al., 2019\n)\nWhat’s the Future?\n#\nSo far we have seen many interesting new ideas on automating the network architecture engineering through neural architecture search and many have achieved very impressive performance. However, it is a bit hard to do inference on\nwhy\nsome architecture work well and how we can develop modules generalizable across tasks rather than being very dataset-specific.\nAs also noted in\nElsken, et al (2019)\n:\n“…, so far it provides little insights into why specific architectures work well and how similar the architectures derived in independent runs would be. Identifying common motifs, providing an understanding why those motifs are important for high performance, and investigating if these motifs generalize over different problems would be desirable.”\nIn the meantime, purely focusing on improvement over validation accuracy might not be enough (\nCai et al., 2019\n). Devices like mobile phones for daily usage in general have limited memory and computation power. While AI applications are on the way to affect our daily life, it is unavoidable to be more\ndevice-specific\n.\nAnother interesting investigation is to consider\nunlabelled dataset\nand\nself-supervised learning\nfor NAS. The size of labelled dataset is always limited and it is not easy to tell whether such a dataset has biases or big deviation from the real world data distribution.\nLiu et al (2020)\ndelve into the question\n“Can we find high-quality neural architecture without human-annotated labels?”\nand proposed a new setup called\nUnsupervised Neural Architecture Search\n(\nUnNAS\n). The quality of the architecture needs to be estimated in an unsupervised fashion during the search phase. The paper experimented with three unsupervised\npretext tasks\n: image rotation prediction, colorization, and solving the jigsaw puzzle.\nThey observed in a set of UnNAS experiments that:\nHigh rank correlation between supervised accuracy and pretext accuracy\non the same dataset\n. Typically the rank correlation is higher than 0.8, regardless of the dataset, the search space, and the pretext task.\nHigh rank correlation between supervised accuracy and pretext accuracy\nacross datasets\n.\nBetter pretext accuracy translates to better supervised accuracy.\nPerformance of UnNAS architecture is comparable to supervised counterparts, though not better yet.\nOne hypothesis is that the architecture quality is correlated with image statistics. Because CIFAR-10 and ImageNet are all on the natural images, they are comparable and the results are transferable. UnNAS could potentially enable a much larger amount of unlabelled data into the search phase which captures image statistics better.\nHyperparameter search is a long-standing topic in the ML community. And NAS automates architecture engineering. Gradually we are trying to automate processes in ML which usually demand a lot of human efforts. Taking even one more step further, is it possible to automatically discover ML algorithms?\nAutoML-Zero\n(\nReal et al 2020\n) investigates this idea. Using\naging evolutionary algorithms\n, AutoML-Zero automatically searches for whole ML algorithms using little restriction on the form with only simple mathematical operations as building blocks.\nIt learns three component functions. Each function only adopts very basic operations.\nSetup\n: initialize memory variables (weights).\nLearn\n: modify memory variables\nPredict\n: make a prediction from an input $x$.\nAlgorithm evaluation on one task (Image source:\nReal et al 2020\n)\nThree types of operations are considered when mutating a parent genotype:\nInsert a random instruction or remove an instruction at a random location in a component function;\nRandomize all the instructions in a component function;\nModify one of the arguments of an instruction by replacing it with a random choice (e.g. “swap the output address” or “change the value of a constant”)\nAn illustration of evolutionary progress on projected binary CIFAR-10 with example code. (Image source:\nReal et al 2020\n)\nAppendix: Summary of NAS Papers\n#\nModel name\nSearch space\nSearch algorithms\nChild model evaluation\nNEAT (2002)\n-\nEvolution (Genetic algorithm)\n-\nNAS (2017)\nSequential layer-wise ops\nRL (REINFORCE)\nTrain from scratch until convergence\nMetaQNN (2017)\nSequential layer-wise ops\nRL (Q-learning with $\\epsilon$-greedy)\nTrain for 20 epochs\nHNAS (2017)\nHierarchical structure\nEvolution (Tournament selection)\nTrain for a fixed number of iterations\nNASNet (2018)\nCell-based\nRL (PPO)\nTrain for 20 epochs\nAmoebaNet (2018)\nNASNet search space\nEvolution (Tournament selection with aging regularization)\nTrain for 25 epochs\nEAS (2018a)\nNetwork transformation\nRL (REINFORCE)\n2-stage training\nPNAS (2018)\nReduced version of NASNet search space\nSMBO; Progressive search for architectures of increasing complexity\nTrain for 20 epochs\nENAS (2018)\nBoth sequential and cell-based search space\nRL (REINFORCE)\nTrain one model with shared weights\nSMASH (2017)\nMemory-bank representation\nRandom search\nHyperNet predicts weights of evaluated architectures.\nOne-Shot (2018)\nAn over-parameterized one-shot model\nRandom search (zero out some paths at random)\nTrain the one-shot model\nDARTS (2019)\nNASNet search space\nGradient descent (Softmax weights over operations)\nProxylessNAS (2019)\nTree structure architecture\nGradient descent (BinaryConnect) or REINFORCE\nSNAS (2019)\nNASNet search space\nGradient descent (concrete distribution)\nCitation\n#\nCited as:\nWeng, Lilian. (Aug 2020). Neural architecture search. Lil’Log. https://lilianweng.github.io/posts/2020-08-06-nas/.\nOr\n@article{weng2020nas,\n  title   = \"Neural Architecture Search\",\n  author  = \"Weng, Lilian\",\n  journal = \"lilianweng.github.io\",\n  year    = \"2020\",\n  month   = \"Aug\",\n  url     = \"https://lilianweng.github.io/posts/2020-08-06-nas/\"\n}\nReference\n#\n[1] Thomas Elsken, Jan Hendrik Metzen, Frank Hutter.\n“Neural Architecture Search: A Survey”\nJMLR 20 (2019) 1-21.\n[2] Kenneth O. Stanley, et al.\n“Designing neural networks through neuroevolution”\nNature Machine Intelligence volume 1, pages 24–35 (2019).\n[3] Kenneth O. Stanley & Risto Miikkulainen.\n“Evolving Neural Networks through Augmenting Topologies”\nEvolutionary Computation 10(2): 99-127 (2002).\n[4] Barret Zoph, Quoc V. Le.\n“Neural architecture search with reinforcement learning”\nICLR 2017.\n[5] Bowen Baker, et al.\n“Designing Neural Network Architectures using Reinforcement Learning”\nICLR 2017.\n[6] Bowen Baker, et al.\n“Accelerating neural architecture search using performance prediction”\nICLR Workshop 2018.\n[7] Barret Zoph, et al.\n“Learning transferable architectures for scalable image recognition”\nCVPR 2018.\n[8] Hanxiao Liu, et al.\n“Hierarchical representations for efficient architecture search.”\nICLR 2018.\n[9] Esteban Real, et al.\n“Regularized Evolution for Image Classifier Architecture Search”\narXiv:1802.01548 (2018).\n[10] Han Cai, et al. [“Efficient architecture search by network transformation”] AAAI 2018a.\n[11] Han Cai, et al.\n“Path-Level Network Transformation for Efficient Architecture Search”\nICML 2018b.\n[12] Han Cai, Ligeng Zhu & Song Han.\n“ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware”\nICLR 2019.\n[13] Chenxi Liu, et al.\n“Progressive neural architecture search”\nECCV 2018.\n[14] Hieu Pham, et al.\n“Efficient neural architecture search via parameter sharing”\nICML 2018.\n[15] Andrew Brock, et al.\n“SMASH: One-shot model architecture search through hypernetworks.”\nICLR 2018.\n[16] Gabriel Bender, et al.\n“Understanding and simplifying one-shot architecture search.”\nICML 2018.\n[17] Hanxiao Liu, Karen Simonyan, Yiming Yang.\n“DARTS: Differentiable Architecture Search”\nICLR 2019.\n[18] Sirui Xie, Hehui Zheng, Chunxiao Liu, Liang Lin.\n“SNAS: Stochastic Neural Architecture Search”\nICLR 2019.\n[19] Chenxi Liu et al.\n“Are Labels Necessary for Neural Architecture Search?”\nECCV 2020.\n[20] Esteban Real, et al.\n“AutoML-Zero: Evolving Machine Learning Algorithms From Scratch”\nICML 2020.",
  "metadata": {
    "title": "Neural Architecture Search | Lil'Log",
    "description": "Although most popular and successful model architectures are designed by human experts, it doesn’t mean we have explored the entire network architecture space and settled down with the best option. We would have a better chance to find the optimal solution if we adopt a systematic and automatic way of learning high-performance model architectures.",
    "keywords": "architecture, reinforcement-learning, evolution, auto-ML",
    "author": "Lilian Weng",
    "og:title": "Neural Architecture Search",
    "og:description": "Although most popular and successful model architectures are designed by human experts, it doesn’t mean we have explored the entire network architecture space and settled down with the best option. We would have a better chance to find the optimal solution if we adopt a systematic and automatic way of learning high-performance model architectures.",
    "og:type": "article",
    "og:url": "https://lilianweng.github.io/posts/2020-08-06-nas/",
    "twitter:card": "summary",
    "twitter:title": "Neural Architecture Search",
    "twitter:description": "Although most popular and successful model architectures are designed by human experts, it doesn’t mean we have explored the entire network architecture space and settled down with the best option. We would have a better chance to find the optimal solution if we adopt a systematic and automatic way of learning high-performance model architectures.",
    "article:section": "posts",
    "article:published_time": "2020-08-06T00:00:00+00:00",
    "article:modified_time": "2020-08-06T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/475f6d363978/code/body-000.txt",
      "chars": 234,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/NAS-high-level.png",
      "bytes": 176077,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-22a9e6438726d336ffe1.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/NAS-high-level.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/NAS-search-space.png",
      "bytes": 852559,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-a036ff1cba89bd8ca177.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/NAS-search-space.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/NASNet-search-space.png",
      "bytes": 189886,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-18722c930cf5c497d7b4.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/NASNet-search-space.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/cell-prediction-steps.png",
      "bytes": 234856,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-68fbdff3da3a7b692d3a.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/cell-prediction-steps.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/hierarchical-NAS-search-space.png",
      "bytes": 405820,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-41e54ee7ec562394ded4.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/hierarchical-NAS-search-space.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/path-level-network-transformations.png",
      "bytes": 147376,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-d6536618f786cd88a98d.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/path-level-network-transformations.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/NAS-memory-bank-view-representation.png",
      "bytes": 82152,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-e6b375e7ba5a5bff8ba1.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/NAS-memory-bank-view-representation.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/NAS.png",
      "bytes": 157138,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-f21c72c925e425e2c413.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/NAS.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/MetaQNN.png",
      "bytes": 227348,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-20cc8fbcc63711039176.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/MetaQNN.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/NEAT-mutations.png",
      "bytes": 307054,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-844cb0884d62492c8431.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/NEAT-mutations.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/aging-evolution-algorithm.png",
      "bytes": 380120,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-2beee1783718ce1e1233.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/aging-evolution-algorithm.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/AmoebaNet-mutations.png",
      "bytes": 136228,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-841034ba340426dcc42f.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/AmoebaNet-mutations.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/progressive-NAS-algorithm.png",
      "bytes": 403536,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-608e700d91e7d97d213c.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/progressive-NAS-algorithm.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/EAS-meta-controller.png",
      "bytes": 359761,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-740a0454d4525705f3c3.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/EAS-meta-controller.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/ENAS-example.png",
      "bytes": 100940,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-c04d9fc97a4006715831.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/ENAS-example.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/SMASH-algorithm.png",
      "bytes": 138999,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-ebfcd764d9e484649bd7.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/SMASH-algorithm.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/SMASH-error-correlation.png",
      "bytes": 266480,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-ea47ad327c3e525d7680.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/SMASH-error-correlation.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/one-shot-model-architecture.png",
      "bytes": 247091,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-4bfe6aecad730ed414c3.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/one-shot-model-architecture.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/one-shot-model-accuracy-correlation.png",
      "bytes": 128766,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-dc1c1909a9278b076020.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/one-shot-model-accuracy-correlation.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/DARTS-illustration.png",
      "bytes": 352267,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-f40298ff01af8fc9e4dc.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/DARTS-illustration.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/DARTS-algorithm.png",
      "bytes": 127977,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-2453003154b5219dd230.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/DARTS-algorithm.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/proxylessNAS-training.png",
      "bytes": 89862,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-94f6d5d4f164d466c4fb.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/proxylessNAS-training.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/proxylessNAS-latency.png",
      "bytes": 144054,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-827f8fb6a1866bbd0478.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/proxylessNAS-latency.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/AutoML-zero-evaluation.png",
      "bytes": 191620,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-f86a63f93973014347b7.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/AutoML-zero-evaluation.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2020-08-06-nas/AutoML-zero-progress.png",
      "bytes": 582897,
      "reused": false,
      "local_path": "attachments/475f6d363978/images/body-52ae3de20af990c921e3.png",
      "src": "https://lilianweng.github.io/posts/2020-08-06-nas/AutoML-zero-progress.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf",
      "label": "Stanley & Miikkulainen, 2002",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1611.01578",
      "label": "Zoph & Le 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1611.02167",
      "label": "Baker et al. 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1808.05377",
      "label": "Elsken, et al 2019",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-06-24-attention/",
      "label": "attention",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1707.07012",
      "label": "Zoph et al. 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1605.07648",
      "label": "DropPath",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Network_motif",
      "label": "motif",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1711.00436",
      "label": "Liu et al 2017",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Topological_sorting",
      "label": "topological ordering",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1806.02639",
      "label": "Cai et al (2018b)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1708.05344",
      "label": "Brock et al. (2017)",
      "type": "a"
    },
    {
      "href": "http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf",
      "label": "Bergstra & Bengio 2012",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/",
      "label": "REINFORCE",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/",
      "label": "Q-learning",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-01-23-multi-armed-bandit/",
      "label": "$\\epsilon$-greedy",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Genetic_algorithm",
      "label": "genetic algorithm (GA)",
      "type": "a"
    },
    {
      "href": "https://www.nature.com/articles/s42256-018-0006-z",
      "label": "survey",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1802.01548",
      "label": "Real et al. (2018)",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Tournament_selection",
      "label": "tournament selection",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2020-08-06-nas/aging-evolutionary-algorithms",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2020-01-29-curriculum-rl/",
      "label": "“curriculum”",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1712.00559",
      "label": "Liu, et al 2018",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/A*_search_algorithm",
      "label": "A* search algorithm",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1705.10823",
      "label": "Baker et al (2018)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1511.05641",
      "label": "Net2net",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1707.04873",
      "label": "Cai et al (2017)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1802.03268",
      "label": "Pham et al. 2018",
      "type": "a"
    },
    {
      "href": "https://blog.otoro.net/2016/09/28/hyper-networks/",
      "label": "HyperNet",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1609.09106",
      "label": "Ha et al 2016",
      "type": "a"
    },
    {
      "href": "http://proceedings.mlr.press/v80/bender18a/bender18a.pdf",
      "label": "Bender et al (2018)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1806.09055",
      "label": "Liu et al (2019)",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/",
      "label": "MAML",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2019-05-05-domain-randomization/",
      "label": "Domain Randomization",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1812.09926",
      "label": "Xie et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1611.00712",
      "label": "Maddison et al 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1812.00332",
      "label": "Cai et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1511.00363",
      "label": "Courbariaux et al., 2015",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2019-11-10-self-supervised/",
      "label": "self-supervised learning",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2003.12056",
      "label": "Liu et al (2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2003.03384",
      "label": "Real et al 2020",
      "type": "a"
    },
    {
      "href": "http://proceedings.mlr.press/v80/bender18a.html",
      "label": "One-Shot (2018)",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:31Z",
  "source_url": "https://lilianweng.github.io/posts/2020-08-06-nas/",
  "papers": [
    {
      "status": "ok",
      "url": "http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf",
      "bytes": 455683,
      "reused": false,
      "local_path": "papers/repair/805499a4b371ca736e62.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1611.01578",
      "bytes": 733204,
      "reused": false,
      "local_path": "papers/repair/a111656177a4c8876b48.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1611.02167",
      "bytes": 2766423,
      "reused": false,
      "local_path": "papers/repair/760094c3d304751da217.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1808.05377",
      "bytes": 513920,
      "reused": false,
      "local_path": "papers/repair/97142d15c75389ec1783.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1707.07012",
      "bytes": 8223935,
      "reused": false,
      "local_path": "papers/repair/263d4aa0ef18300727e9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1605.07648",
      "bytes": 301203,
      "reused": false,
      "local_path": "papers/repair/3c2d2df24de01e01fbd2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1711.00436",
      "bytes": 625718,
      "reused": false,
      "local_path": "papers/repair/eb6cf4cb96b16ab6e4ae.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1806.02639",
      "bytes": 4382572,
      "reused": false,
      "local_path": "papers/repair/680796c56d03bb0724c3.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1708.05344",
      "bytes": 3204919,
      "reused": false,
      "local_path": "papers/repair/e88c3d4c693981e49fce.pdf"
    },
    {
      "status": "ok",
      "url": "http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf",
      "bytes": 728274,
      "reused": false,
      "local_path": "papers/repair/8a459dcd765c190d80b7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1802.01548",
      "bytes": 719617,
      "reused": false,
      "local_path": "papers/repair/65805c891aedf369ff37.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1712.00559",
      "bytes": 1201540,
      "reused": false,
      "local_path": "papers/repair/609e6dc18e9ec4f12c8e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1705.10823",
      "bytes": 696064,
      "reused": false,
      "local_path": "papers/repair/0fe98a08286c39292e71.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1511.05641",
      "bytes": 514031,
      "reused": false,
      "local_path": "papers/repair/92e6dc487fb58180dcc6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1707.04873",
      "bytes": 1963936,
      "reused": false,
      "local_path": "papers/repair/ed8c0c59198f186976bc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1802.03268",
      "bytes": 437408,
      "reused": false,
      "local_path": "papers/repair/385bdb4f3e6008bec0cf.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1609.09106",
      "bytes": 3125291,
      "reused": false,
      "local_path": "papers/repair/9a50e3e16a9ac5be3fd1.pdf"
    },
    {
      "status": "ok",
      "url": "http://proceedings.mlr.press/v80/bender18a/bender18a.pdf",
      "bytes": 608254,
      "reused": false,
      "local_path": "papers/repair/3a859646e6c029ea003e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1806.09055",
      "bytes": 636787,
      "reused": false,
      "local_path": "papers/repair/40e69415044afc05d599.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1812.09926",
      "bytes": 711882,
      "reused": false,
      "local_path": "papers/repair/35befec5995034ad4bfd.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1611.00712",
      "bytes": 858710,
      "reused": false,
      "local_path": "papers/repair/27fa0ebf79e9f6aa3132.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1812.00332",
      "bytes": 2325253,
      "reused": false,
      "local_path": "papers/repair/61f9a157fe986680a3a9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1511.00363",
      "bytes": 1297364,
      "reused": false,
      "local_path": "papers/repair/1bc5305341828713b8ed.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2003.12056",
      "bytes": 1315976,
      "reused": false,
      "local_path": "papers/repair/05573f22fe71d2abefb7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2003.03384",
      "bytes": 1781766,
      "reused": false,
      "local_path": "papers/repair/daabc7af10551ec97ee1.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 44902,
  "expected_body_images": 25,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/475f6d363978/pages/source.html",
  "local_html": "attachments/475f6d363978/pages/article.html",
  "body_base_path": "attachments/475f6d363978/pages/",
  "body_markdown_path": "attachments/475f6d363978/pages/article.md"
}