{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Lilian Weng / Lil'Log",
  "title_listed": "Meta-Learning: Learning to Learn Fast",
  "keywords": "元学习,小样本,MAML",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/",
  "publish_date": "2018-11",
  "corpus_stage": "预训练/SFT",
  "notes": "元学习综述",
  "id": "a8dc6bf17c71",
  "collected_at": "2026-10-02T14:13:37",
  "title": "Meta-Learning: Learning to Learn Fast | Lil'Log",
  "body_markdown": "[Updated on 2019-10-01: thanks to Tianhao, we have this post translated in\nChinese\n!]\nA good machine learning model often requires training with a large number of samples. Humans, in contrast, learn new concepts and skills much faster and more efficiently. Kids who have seen cats and birds only a few times can quickly tell them apart. People who know how to ride a bike are likely to discover the way to ride a motorcycle fast with little or even no demonstration. Is it possible to design a machine learning model with similar properties — learning new concepts and skills fast with a few training examples? That’s essentially what\nmeta-learning\naims to solve.\nWe expect a good meta-learning model capable of well adapting or generalizing to new tasks and new environments that have never been encountered during training time. The adaptation process, essentially a mini learning session, happens during test but with a limited exposure to the new task configurations. Eventually, the adapted model can complete new tasks. This is why meta-learning is also known as\nlearning to learn\n.\nThe tasks can be any well-defined family of machine learning problems: supervised learning, reinforcement learning, etc. For example, here are a couple concrete meta-learning tasks:\nA classifier trained on non-cat images can tell whether a given image contains a cat after seeing a handful of cat pictures.\nA game bot is able to quickly master a new game.\nA mini robot completes the desired task on an uphill surface during test even through it was only trained in a flat surface environment.\nDefine the Meta-Learning Problem\n#\nIn this post, we focus on the case when each desired task is a supervised learning problem like image classification. There is a lot of interesting literature on meta-learning with reinforcement learning problems (aka “Meta Reinforcement Learning”), but we would not cover them here.\nA Simple View\n#\nA good meta-learning model should be trained over a variety of learning tasks and optimized for the best performance on a distribution of tasks, including potentially unseen tasks. Each task is associated with a dataset $\\mathcal{D}$, containing both feature vectors and true labels. The optimal model parameters are:\n$$\n\\theta^* = \\arg\\min_\\theta \\mathbb{E}_{\\mathcal{D}\\sim p(\\mathcal{D})} [\\mathcal{L}_\\theta(\\mathcal{D})]\n$$\nIt looks very similar to a normal learning task, but\none dataset\nis considered as\none data sample\n.\nFew-shot classification\nis an instantiation of meta-learning in the field of supervised learning. The dataset $\\mathcal{D}$ is often split into two parts, a support set $S$ for learning and a prediction set $B$ for training or testing, $\\mathcal{D}=\\langle S, B\\rangle$. Often we consider a\nK-shot N-class classification\ntask: the support set contains K labelled examples for each of N classes.\nAn example of 4-shot 2-class image classification. (Image thumbnails are from\nPinterest\n)\nTraining in the Same Way as Testing\n#\nA dataset $\\mathcal{D}$ contains pairs of feature vectors and labels, $\\mathcal{D} = \\{(\\mathbf{x}_i, y_i)\\}$ and each label belongs to a known label set $\\mathcal{L}^\\text{label}$.  Let’s say, our classifier $f_\\theta$ with parameter $\\theta$ outputs a probability of a data point belonging to the class $y$ given the feature vector $\\mathbf{x}$, $P_\\theta(y\\vert\\mathbf{x})$.\nThe optimal parameters should maximize the probability of true labels across multiple training batches $B \\subset \\mathcal{D}$:\n$$\n\\begin{aligned}\n\\theta^* &= {\\arg\\max}_{\\theta} \\mathbb{E}_{(\\mathbf{x}, y)\\in \\mathcal{D}}[P_\\theta(y \\vert \\mathbf{x})] &\\\\\n\\theta^* &= {\\arg\\max}_{\\theta} \\mathbb{E}_{B\\subset \\mathcal{D}}[\\sum_{(\\mathbf{x}, y)\\in B}P_\\theta(y \\vert \\mathbf{x})] & \\scriptstyle{\\text{; trained with mini-batches.}}\n\\end{aligned}\n$$\nIn few-shot classification, the goal is to reduce the prediction error on data samples with unknown labels given a small support set for “fast learning” (think of how “fine-tuning” works). To make the training process mimics what happens during inference, we would like to “fake” datasets with a subset of labels to avoid exposing all the labels to the model and modify the optimization procedure accordingly to encourage fast learning:\nSample a subset of labels, $L\\subset\\mathcal{L}^\\text{label}$.\nSample a support set $S^L \\subset \\mathcal{D}$ and a training batch $B^L \\subset \\mathcal{D}$. Both of them only contain data points with labels belonging to the sampled label set $L$, $y \\in L, \\forall (x, y) \\in S^L, B^L$.\nThe support set is part of the model input.\nThe final optimization uses the mini-batch $B^L$ to compute the loss and update the model parameters through backpropagation, in the same way as how we use it in the supervised learning.\nYou may consider each pair of sampled dataset $(S^L, B^L)$ as one data point. The model is trained such that it can generalize to other datasets.  Symbols in red are added for meta-learning in addition to the supervised learning objective.\n$$\n\\theta = \\arg\\max_\\theta \\color{red}{E_{L\\subset\\mathcal{L}}[} E_{\\color{red}{S^L \\subset\\mathcal{D}, }B^L \\subset\\mathcal{D}} [\\sum_{(x, y)\\in B^L} P_\\theta(x, y\\color{red}{, S^L})] \\color{red}{]}\n$$\nThe idea is to some extent similar to using a pre-trained model in image classification (ImageNet) or language modeling (big text corpora) when only a limited set of task-specific data samples are available. Meta-learning takes this idea one step further, rather than fine-tuning according to one down-steam task, it optimizes the model to be good at many, if not all.\nLearner and Meta-Learner\n#\nAnother popular view of meta-learning decomposes the model update into two stages:\nA classifier $f_\\theta$ is the “learner” model, trained for operating a given task;\nIn the meantime, a optimizer $g_\\phi$ learns how to update the learner model’s parameters via the support set $S$, $\\theta’ = g_\\phi(\\theta, S)$.\nThen in final optimization step, we need to update both $\\theta$ and $\\phi$ to maximize:\n$$\n\\mathbb{E}_{L\\subset\\mathcal{L}}[ \\mathbb{E}_{S^L \\subset\\mathcal{D}, B^L \\subset\\mathcal{D}} [\\sum_{(\\mathbf{x}, y)\\in B^L} P_{g_\\phi(\\theta, S^L)}(y \\vert \\mathbf{x})]]\n$$\nCommon Approaches\n#\nThere are three common approaches to meta-learning: metric-based, model-based, and optimization-based. Oriol Vinyals has a nice summary in his\ntalk\nat meta-learning symposium @ NIPS 2018:\n| ————- | ————- | ————- | ————- |\nModel-based\nMetric-based\nOptimization-based\nKey idea\nRNN; memory\nMetric learning\nGradient descent\nHow $P_\\theta(y \\vert \\mathbf{x})$ is modeled?\n$f_\\theta(\\mathbf{x}, S)$\n$\\sum_{(\\mathbf{x}_i, y_i) \\in S} k_\\theta(\\mathbf{x}, \\mathbf{x}_i)y_i$ (*)\n$P_{g_\\phi(\\theta, S^L)}(y \\vert \\mathbf{x})$\n(*) $k_\\theta$ is a kernel function measuring the similarity between $\\mathbf{x}_i$ and $\\mathbf{x}$.\nNext we are gonna review classic models in each approach.\nMetric-Based\n#\nThe core idea in metric-based meta-learning is similar to nearest neighbors algorithms (i.e.,\nk-NN\nclassificer and\nk-means\nclustering) and\nkernel density estimation\n. The predicted probability over a set of known labels $y$ is a weighted sum of labels of support set samples. The weight is generated by a kernel function $k_\\theta$, measuring the similarity between two data samples.\n$$\nP_\\theta(y \\vert \\mathbf{x}, S) = \\sum_{(\\mathbf{x}_i, y_i) \\in S} k_\\theta(\\mathbf{x}, \\mathbf{x}_i)y_i \n$$\nTo learn a good kernel is crucial to the success of a metric-based meta-learning model.\nMetric learning\nis well aligned with this intention, as it aims to learn a metric or distance function over objects. The notion of a good metric is problem-dependent. It should represent the relationship between inputs in the task space and facilitate problem solving.\nAll the models introduced below learn embedding vectors of input data explicitly and use them to design proper kernel functions.\nConvolutional Siamese Neural Network\n#\nThe\nSiamese Neural Network\nis composed of two twin networks and their outputs are jointly trained on top with a function to learn the relationship between pairs of input data samples. The twin networks are identical, sharing the same weights and network parameters. In other words, both refer to the same embedding network that learns an efficient embedding to reveal relationship between pairs of data points.\nKoch, Zemel & Salakhutdinov (2015)\nproposed a method to use the siamese neural network to do one-shot image classification. First, the siamese network is trained for a verification task for telling whether two input images are in the same class. It outputs the probability of two images belonging to the same class. Then, during test time, the siamese network processes all the image pairs between a test image and every image in the support set. The final prediction is the class of the support image with the highest probability.\nThe architecture of convolutional siamese neural network for few-show image classification.\nFirst, convolutional siamese network learns to encode two images into feature vectors via a embedding function $f_\\theta$ which contains a couple of convolutional layers.\nThe L1-distance between two embeddings is $\\vert f_\\theta(\\mathbf{x}_i) - f_\\theta(\\mathbf{x}_j) \\vert$.\nThe distance is converted to a probability $p$ by a linear feedforward layer and sigmoid. It is the probability of whether two images are drawn from the same class.\nIntuitively the loss is cross entropy because the label is binary.\n$$\n\\begin{aligned}\np(\\mathbf{x}_i, \\mathbf{x}_j) &= \\sigma(\\mathbf{W}\\vert f_\\theta(\\mathbf{x}_i) - f_\\theta(\\mathbf{x}_j) \\vert) \\\\\n\\mathcal{L}(B) &= \\sum_{(\\mathbf{x}_i, \\mathbf{x}_j, y_i, y_j)\\in B} \\mathbf{1}_{y_i=y_j}\\log p(\\mathbf{x}_i, \\mathbf{x}_j) + (1-\\mathbf{1}_{y_i=y_j})\\log (1-p(\\mathbf{x}_i, \\mathbf{x}_j))\n\\end{aligned}\n$$\nImages in the training batch $B$ can be augmented with distortion. Of course, you can replace the L1 distance with other distance metric, L2, cosine, etc. Just make sure they are differential and then everything else works the same.\nGiven a support set $S$ and a test image $\\mathbf{x}$, the final predicted class is:\n$$\n\\hat{c}_S(\\mathbf{x}) = c(\\arg\\max_{\\mathbf{x}_i \\in S} P(\\mathbf{x}, \\mathbf{x}_i))\n$$\nwhere $c(\\mathbf{x})$ is the class label of an image $\\mathbf{x}$ and $\\hat{c}(.)$ is the predicted label.\nThe assumption is that the learned embedding can be generalized to be useful for measuring the distance between images of unknown categories. This is the same assumption behind transfer learning via the adoption of a pre-trained model; for example, the convolutional features learned in the model pre-trained with ImageNet are expected to help other image tasks. However, the benefit of a pre-trained model decreases when the new task diverges from the original task that the model was trained on.\nMatching Networks\n#\nThe task of\nMatching Networks\n(\nVinyals et al., 2016\n) is to learn a classifier $c_S$ for any given (small) support set $S=\\{x_i, y_i\\}_{i=1}^k$ (\nk-shot\nclassification). This classifier defines a probability distribution over output labels $y$ given a test example $\\mathbf{x}$. Similar to other metric-based models, the classifier output is defined as a sum of labels of support samples weighted by attention kernel $a(\\mathbf{x}, \\mathbf{x}_i)$ - which should be proportional to the similarity between $\\mathbf{x}$ and $\\mathbf{x}_i$.\nThe architecture of Matching Networks. (Image source:\noriginal paper\n)\n$$\nc_S(\\mathbf{x}) = P(y \\vert \\mathbf{x}, S) = \\sum_{i=1}^k a(\\mathbf{x}, \\mathbf{x}_i) y_i\n\\text{, where }S=\\{(\\mathbf{x}_i, y_i)\\}_{i=1}^k\n$$\nThe attention kernel depends on two embedding functions, $f$ and $g$, for encoding the test sample and the support set samples respectively. The attention weight between two data points is the cosine similarity, $\\text{cosine}(.)$, between their embedding vectors, normalized by softmax:\n$$\na(\\mathbf{x}, \\mathbf{x}_i) = \\frac{\\exp(\\text{cosine}(f(\\mathbf{x}), g(\\mathbf{x}_i))}{\\sum_{j=1}^k\\exp(\\text{cosine}(f(\\mathbf{x}), g(\\mathbf{x}_j))}\n$$\nSimple Embedding\n#\nIn the simple version, an embedding function is a neural network with a single data sample as input. Potentially we can set $f=g$.\nFull Context Embeddings\n#\nThe embedding vectors are critical inputs for building a good classifier. Taking a single data point as input might not be enough to efficiently gauge the entire feature space. Therefore, the Matching Network model further proposed to enhance the embedding functions by taking as input the whole support set $S$ in addition to the original input, so that the learned embedding can be adjusted based on the relationship with other support samples.\n$g_\\theta(\\mathbf{x}_i, S)$ uses a bidirectional LSTM to encode $\\mathbf{x}_i$ in the context of the entire support set $S$.\n$f_\\theta(\\mathbf{x}, S)$ encodes the test sample $\\mathbf{x}$ visa an LSTM with read attention over the support set $S$.\nFirst the test sample goes through a simple neural network, such as a CNN, to extract basic features, $f’(\\mathbf{x})$.\nThen an LSTM is trained with a read attention vector over the support set as part of the hidden state:\n$$\n  \\begin{aligned}\n  \\hat{\\mathbf{h}}_t, \\mathbf{c}_t &= \\text{LSTM}(f'(\\mathbf{x}), [\\mathbf{h}_{t-1}, \\mathbf{r}_{t-1}], \\mathbf{c}_{t-1}) \\\\\n  \\mathbf{h}_t &= \\hat{\\mathbf{h}}_t + f'(\\mathbf{x}) \\\\\n  \\mathbf{r}_{t-1} &= \\sum_{i=1}^k a(\\mathbf{h}_{t-1}, g(\\mathbf{x}_i)) g(\\mathbf{x}_i) \\\\\n  a(\\mathbf{h}_{t-1}, g(\\mathbf{x}_i)) &= \\text{softmax}(\\mathbf{h}_{t-1}^\\top g(\\mathbf{x}_i)) = \\frac{\\exp(\\mathbf{h}_{t-1}^\\top g(\\mathbf{x}_i))}{\\sum_{j=1}^k \\exp(\\mathbf{h}_{t-1}^\\top g(\\mathbf{x}_j))}\n  \\end{aligned}\n  $$\nEventually $f(\\mathbf{x}, S)=\\mathbf{h}_K$ if we do K steps of “read”.\nThis embedding method is called “Full Contextual Embeddings (FCE)”. Interestingly it does help improve the performance on a hard task (few-shot classification on mini ImageNet), but makes no difference on a simple task (Omniglot).\nThe training process in Matching Networks is designed to match inference at test time, see the details in the earlier\nsection\n. It is worthy of mentioning that the Matching Networks paper refined the idea that training and testing conditions should match.\n$$\n\\theta^* = \\arg\\max_\\theta \\mathbb{E}_{L\\subset\\mathcal{L}}[ \\mathbb{E}_{S^L \\subset\\mathcal{D}, B^L \\subset\\mathcal{D}} [\\sum_{(\\mathbf{x}, y)\\in B^L} P_\\theta(y\\vert\\mathbf{x}, S^L)]]\n$$\nRelation Network\n#\nRelation Network (RN)\n(\nSung et al., 2018\n) is similar to\nsiamese network\nbut with a few differences:\nThe relationship is not captured by a simple L1 distance in the feature space, but predicted by a CNN classifier $g_\\phi$. The relation score between a pair of inputs, $\\mathbf{x}_i$ and $\\mathbf{x}_j$, is $r_{ij} = g_\\phi([\\mathbf{x}_i, \\mathbf{x}_j])$ where $[.,.]$ is concatenation.\nThe objective function is MSE loss instead of cross-entropy, because conceptually RN focuses more on predicting relation scores which is more like regression, rather than binary classification, $\\mathcal{L}(B) = \\sum_{(\\mathbf{x}_i, \\mathbf{x}_j, y_i, y_j)\\in B} (r_{ij} - \\mathbf{1}_{y_i=y_j})^2$.\nRelation Network architecture for a 5-way 1-shot problem with one query example. (Image source:\noriginal paper\n)\n(Note: There is another\nRelation Network\nfor relational reasoning, proposed by DeepMind. Don’t get confused.)\nPrototypical Networks\n#\nPrototypical Networks\n(\nSnell, Swersky & Zemel, 2017\n) use an embedding function $f_\\theta$ to encode each input into a $M$-dimensional feature vector. A\nprototype\nfeature vector is defined for every class $c \\in \\mathcal{C}$, as the mean vector of the embedded support data samples in this class.\n$$\n\\mathbf{v}_c = \\frac{1}{|S_c|} \\sum_{(\\mathbf{x}_i, y_i) \\in S_c} f_\\theta(\\mathbf{x}_i)\n$$\nPrototypical networks in the few-shot and zero-shot scenarios. (Image source:\noriginal paper\n)\nThe distribution over classes for a given test input $\\mathbf{x}$ is a softmax over the inverse of distances between the test data embedding and prototype vectors.\n$$\nP(y=c\\vert\\mathbf{x})=\\text{softmax}(-d_\\varphi(f_\\theta(\\mathbf{x}), \\mathbf{v}_c)) = \\frac{\\exp(-d_\\varphi(f_\\theta(\\mathbf{x}), \\mathbf{v}_c))}{\\sum_{c' \\in \\mathcal{C}}\\exp(-d_\\varphi(f_\\theta(\\mathbf{x}), \\mathbf{v}_{c'}))}\n$$\nwhere $d_\\varphi$ can be any distance function as long as $\\varphi$ is differentiable. In the paper, they used the squared euclidean distance.\nThe loss function is the negative log-likelihood: $\\mathcal{L}(\\theta) = -\\log P_\\theta(y=c\\vert\\mathbf{x})$.\nModel-Based\n#\nModel-based meta-learning models make no assumption on the form of $P_\\theta(y\\vert\\mathbf{x})$. Rather it depends on a model designed specifically for fast learning — a model that updates its parameters rapidly with a few training steps. This rapid parameter update can be achieved by its internal architecture or controlled by another meta-learner model.\nMemory-Augmented Neural Networks\n#\nA family of model architectures use external memory storage to facilitate the learning process of neural networks, including\nNeural Turing Machines\nand\nMemory Networks\n. With an explicit storage buffer, it is easier for the network to rapidly incorporate new information and not to forget in the future. Such a model is known as\nMANN\n, short for “\nMemory-Augmented Neural Network\n”.  Note that recurrent neural networks with only\ninternal memory\nsuch as vanilla RNN or LSTM are not MANNs.\nBecause MANN is expected to encode new information fast and thus to adapt to new tasks after only a few samples, it fits well for meta-learning. Taking the Neural Turing Machine (NTM) as the base model,\nSantoro et al. (2016)\nproposed a set of modifications on the training setup and the memory retrieval mechanisms (or “addressing mechanisms”, deciding how to assign attention weights to memory vectors). Please go through\nthe NTM section\nin my other post first if you are not familiar with this matter before reading forward.\nAs a quick recap, NTM couples a controller neural network with external memory storage. The controller learns to read and write memory rows by soft attention, while the memory serves as a knowledge repository. The attention weights are generated by its addressing mechanism: content-based + location based.\nThe architecture of Neural Turing Machine (NTM). The memory at time t, $\\mathbf{M}\\_t$ is a matrix of size $N \\times M$, containing N vector rows and each has M dimensions.\nMANN for Meta-Learning\n#\nTo use MANN for meta-learning tasks, we need to train it in a way that the memory can encode and capture information of new tasks fast and, in the meantime, any stored representation is easily and stably accessible.\nThe training described in\nSantoro et al., 2016\nhappens in an interesting way so that the memory is forced to hold information for longer until the appropriate labels are presented later. In each training episode, the truth label $y_t$ is presented with\none step offset\n, $(\\mathbf{x}_{t+1}, y_t)$: it is the true label for the input at the previous time step t, but presented as part of the input at time step t+1.\nTask setup in MANN for meta-learning (Image source:\noriginal paper\n).\nIn this way, MANN is motivated to memorize the information of a new dataset, because the memory has to hold the current input until the label is present later and then retrieve the old information to make a prediction accordingly.\nNext let us see how the memory is updated for efficient information retrieval and storage.\nAddressing Mechanism for Meta-Learning\n#\nAside from the training process, a new pure content-based addressing mechanism is utilized to make the model better suitable for meta-learning.\n» How to read from memory?\nThe read attention is constructed purely based on the content similarity.\nFirst, a key feature vector $\\mathbf{k}_t$ is produced at the time step t by the controller as a function of the input $\\mathbf{x}$. Similar to NTM, a read weighting vector $\\mathbf{w}_t^r$ of N elements is computed as the cosine similarity between the key vector and every memory vector row, normalized by softmax. The read vector $\\mathbf{r}_t$ is a sum of memory records weighted by such weightings:\n$$\n\\mathbf{r}_i = \\sum_{i=1}^N w_t^r(i)\\mathbf{M}_t(i)\n\\text{, where } w_t^r(i) = \\text{softmax}(\\frac{\\mathbf{k}_t \\cdot \\mathbf{M}_t(i)}{\\|\\mathbf{k}_t\\| \\cdot \\|\\mathbf{M}_t(i)\\|})\n$$\nwhere $M_t$ is the memory matrix at time t and $M_t(i)$ is the i-th row in this matrix.\n» How to write into memory?\nThe addressing mechanism for writing newly received information into memory operates a lot like the\ncache replacement\npolicy. The\nLeast Recently Used Access (LRUA)\nwriter is designed for MANN to better work in the scenario of meta-learning. A LRUA write head prefers to write new content to either the\nleast used\nmemory location or the\nmost recently used\nmemory location.\nRarely used locations: so that we can preserve frequently used information (see\nLFU\n);\nThe last used location: the motivation is that once a piece of information is retrieved once, it probably won’t be called again for a while (see\nMRU\n).\nThere are many cache replacement algorithms and each of them could potentially replace the design here with better performance in different use cases. Furthermore, it would be a good idea to learn the memory usage pattern and addressing strategies rather than arbitrarily set it.\nThe preference of LRUA is carried out in a way that everything is differentiable:\nThe usage weight $\\mathbf{w}^u_t$ at time t is a sum of current read and write vectors, in addition to the decayed last usage weight, $\\gamma \\mathbf{w}^u_{t-1}$, where $\\gamma$ is a decay factor.\nThe write vector is an interpolation between the previous read weight (prefer “the last used location”) and the previous least-used weight (prefer “rarely used location”). The interpolation parameter is the sigmoid of a hyperparameter $\\alpha$.\nThe least-used weight $\\mathbf{w}^{lu}$ is scaled according to usage weights $\\mathbf{w}_t^u$, in which any dimension remains at 1 if smaller than the n-th smallest element in the vector and 0 otherwise.\n$$\n\\begin{aligned}\n\\mathbf{w}_t^u &= \\gamma \\mathbf{w}_{t-1}^u + \\mathbf{w}_t^r + \\mathbf{w}_t^w \\\\\n\\mathbf{w}_t^r &= \\text{softmax}(\\text{cosine}(\\mathbf{k}_t, \\mathbf{M}_t(i))) \\\\\n\\mathbf{w}_t^w &= \\sigma(\\alpha)\\mathbf{w}_{t-1}^r + (1-\\sigma(\\alpha))\\mathbf{w}^{lu}_{t-1}\\\\\n\\mathbf{w}_t^{lu} &= \\mathbf{1}_{w_t^u(i) \\leq m(\\mathbf{w}_t^u, n)}\n\\text{, where }m(\\mathbf{w}_t^u, n)\\text{ is the }n\\text{-th smallest element in vector }\\mathbf{w}_t^u\\text{.}\n\\end{aligned}\n$$\nFinally, after the least used memory location, indicated by $\\mathbf{w}_t^{lu}$, is set to zero, every memory row is updated:\n$$\n\\mathbf{M}_t(i) = \\mathbf{M}_{t-1}(i) + w_t^w(i)\\mathbf{k}_t, \\forall i\n$$\nMeta Networks\n#\nMeta Networks\n(\nMunkhdalai & Yu, 2017\n), short for\nMetaNet\n, is a meta-learning model with architecture and training process designed for\nrapid\ngeneralization across tasks.\nFast Weights\n#\nThe rapid generalization of MetaNet relies on “fast weights”. There are a handful of papers on this topic, but I haven’t read all of them in detail and I failed to find a very concrete definition, only a vague agreement on the concept. Normally weights in the neural networks are updated by stochastic gradient descent in an objective function and this process is known to be slow. One faster way to learn is to utilize one neural network to predict the parameters of another neural network and the generated weights are called\nfast weights\n. In comparison, the ordinary SGD-based weights are named\nslow weights\n.\nIn MetaNet, loss gradients are used as\nmeta information\nto populate models that learn fast weights. Slow and fast weights are combined to make predictions in neural networks.\nCombining slow and fast weights in a MLP. $\\bigoplus$ is element-wise sum. (Image source:\noriginal paper\n).\nModel Components\n#\nDisclaimer: Below you will find my annotations are different from those in the paper. imo, the paper is poorly written, but the idea is still interesting. So I’m presenting the idea in my own language.\nKey components of MetaNet are:\nAn embedding function $f_\\theta$, parameterized by $\\theta$, encodes raw inputs into feature vectors. Similar to\nSiamese Neural Network\n, these embeddings are trained to be useful for telling whether two inputs are of the same class (verification task).\nA base learner model $g_\\phi$, parameterized by weights $\\phi$, completes the actual learning task.\nIf we stop here, it looks just like\nRelation Network\n. MetaNet, in addition, explicitly models the fast weights of both functions and then aggregates them back into the model (See Fig. 8).\nTherefore we need additional two functions to output fast weights for $f$ and $g$ respectively.\n$F_w$: a LSTM parameterized by $w$ for learning fast weights $\\theta^+$ of the embedding function $f$. It takes as input gradients of $f$’s embedding loss for verification task.\n$G_v$: a neural network parameterized by $v$ learning fast weights $\\phi^+$ for the base learner $g$ from its loss gradients. In MetaNet, the learner’s loss gradients are viewed as the\nmeta information\nof the task.\nOk, now let’s see how meta networks are trained. The training data contains multiple pairs of datasets: a support set $S=\\{\\mathbf{x}’_i, y’_i\\}_{i=1}^K$ and a test set  $U=\\{\\mathbf{x}_i, y_i\\}_{i=1}^L$. Recall that we have four networks and four sets of model parameters to learn, $(\\theta, \\phi, w, v)$.\nFig.9. The MetaNet architecture.\nTraining Process\n#\nSample a random pair of inputs at each time step t from the support set $S$, $(\\mathbf{x}’_i, y’_i)$ and $(\\mathbf{x}’_j, y_j)$. Let $\\mathbf{x}_{(t,1)}=\\mathbf{x}’_i$ and $\\mathbf{x}_{(t,2)}=\\mathbf{x}’_j$.\nfor $t = 1, \\dots, K$:\na. Compute a loss for representation learning; i.e., cross entropy for the verification task:\n$\\mathcal{L}^\\text{emb}_t = \\mathbf{1}_{y’_i=y’_j} \\log P_t + (1 - \\mathbf{1}_{y’_i=y’_j})\\log(1 - P_t)\\text{, where }P_t = \\sigma(\\mathbf{W}\\vert f_\\theta(\\mathbf{x}_{(t,1)}) - f_\\theta(\\mathbf{x}_{(t,2)})\\vert)$\nCompute the task-level fast weights:\n$\\theta^+ = F_w(\\nabla_\\theta \\mathcal{L}^\\text{emb}_1, \\dots, \\mathcal{L}^\\text{emb}_T)$\nNext go through examples in the support set $S$ and compute the example-level fast weights. Meanwhile, update the memory with learned representations.\nfor $i=1, \\dots, K$:\na. The base learner outputs a probability distribution: $P(\\hat{y}_i \\vert \\mathbf{x}_i) = g_\\phi(\\mathbf{x}_i)$ and the loss can be cross-entropy or MSE: $\\mathcal{L}^\\text{task}_i = y’_i \\log g_\\phi(\\mathbf{x}’_i) + (1- y’_i) \\log (1 - g_\\phi(\\mathbf{x}’_i))$\nb. Extract meta information (loss gradients) of the task and compute the example-level fast weights:\n$\\phi_i^+ = G_v(\\nabla_\\phi\\mathcal{L}^\\text{task}_i)$\nThen store $\\phi^+_i$ into $i$-th location of the “value” memory $\\mathbf{M}$.\nd. Encode the support sample into a task-specific input representation using both slow and fast weights: $r’_i = f_{\\theta, \\theta^+}(\\mathbf{x}’_i)$\nThen store $r’_i$ into $i$-th location of the “key” memory $\\mathbf{R}$.\nFinally it is the time to construct the training loss using the test set $U=\\{\\mathbf{x}_i, y_i\\}_{i=1}^L$.\nStarts with $\\mathcal{L}_\\text{train}=0$:\nfor $j=1, \\dots, L$:\na. Encode the test sample into a task-specific input representation:\n$r_j = f_{\\theta, \\theta^+}(\\mathbf{x}_j)$\nb. The fast weights are computed by attending to representations of support set samples in memory $\\mathbf{R}$. The attention function is of your choice. Here MetaNet uses cosine similarity:\n$$\n \\begin{aligned}\n a_j &= \\text{cosine}(\\mathbf{R}, r_j) = [\\frac{r'_1\\cdot r_j}{\\|r'_1\\|\\cdot\\|r_j\\|}, \\dots, \\frac{r'_N\\cdot r_j}{\\|r'_N\\|\\cdot\\|r_j\\|}]\\\\\n \\phi^+_j &= \\text{softmax}(a_j)^\\top \\mathbf{M}\n \\end{aligned}\n $$\nc. Update the training loss: $\\mathcal{L}_\\text{train} \\leftarrow \\mathcal{L}_\\text{train} + \\mathcal{L}^\\text{task}(g_{\\phi, \\phi^+}(\\mathbf{x}_i), y_i) $\nUpdate all the parameters $(\\theta, \\phi, w, v)$ using $\\mathcal{L}_\\text{train}$.\nOptimization-Based\n#\nDeep learning models learn through backpropagation of gradients. However, the gradient-based optimization is neither designed to cope with a small number of training samples, nor to converge within a small number of optimization steps. Is there a way to adjust the optimization algorithm so that the model can be good at learning with a few examples? This is what optimization-based approach meta-learning algorithms intend for.\nLSTM Meta-Learner\n#\nThe optimization algorithm can be explicitly modeled.\nRavi & Larochelle (2017)\ndid so and named it “meta-learner”, while the original model for handling the task is called “learner”. The goal of the meta-learner is to efficiently update the learner’s parameters using a small support set so that the learner can adapt to the new task quickly.\nLet’s denote the learner model as $M_\\theta$ parameterized by $\\theta$, the meta-learner as $R_\\Theta$ with parameters $\\Theta$, and the loss function $\\mathcal{L}$.\nWhy LSTM?\n#\nThe meta-learner is modeled as a LSTM, because:\nThere is similarity between the gradient-based update in backpropagation and the cell-state update in LSTM.\nKnowing a history of gradients benefits the gradient update; think about how\nmomentum\nworks.\nThe update for the learner’s parameters at time step t with a learning rate $\\alpha_t$ is:\n$$\n\\theta_t = \\theta_{t-1} - \\alpha_t \\nabla_{\\theta_{t-1}}\\mathcal{L}_t\n$$\nIt has the same form as the cell state update in LSTM, if we set forget gate $f_t=1$, input gate $i_t = \\alpha_t$, cell state $c_t = \\theta_t$, and new cell state $\\tilde{c}_t = -\\nabla_{\\theta_{t-1}}\\mathcal{L}_t$:\n$$\n\\begin{aligned}\nc_t &= f_t \\odot c_{t-1} + i_t \\odot \\tilde{c}_t\\\\\n    &= \\theta_{t-1} - \\alpha_t\\nabla_{\\theta_{t-1}}\\mathcal{L}_t\n\\end{aligned}\n$$\nWhile fixing $f_t=1$ and $i_t=\\alpha_t$ might not be the optimal, both of them can be learnable and adaptable to different datasets.\n$$\n\\begin{aligned}\nf_t &= \\sigma(\\mathbf{W}_f \\cdot [\\nabla_{\\theta_{t-1}}\\mathcal{L}_t, \\mathcal{L}_t, \\theta_{t-1}, f_{t-1}] + \\mathbf{b}_f) & \\scriptstyle{\\text{; how much to forget the old value of parameters.}}\\\\\ni_t &= \\sigma(\\mathbf{W}_i \\cdot [\\nabla_{\\theta_{t-1}}\\mathcal{L}_t, \\mathcal{L}_t, \\theta_{t-1}, i_{t-1}] + \\mathbf{b}_i) & \\scriptstyle{\\text{; corresponding to the learning rate at time step t.}}\\\\\n\\tilde{\\theta}_t &= -\\nabla_{\\theta_{t-1}}\\mathcal{L}_t &\\\\\n\\theta_t &= f_t \\odot \\theta_{t-1} + i_t \\odot \\tilde{\\theta}_t &\\\\\n\\end{aligned}\n$$\nModel Setup\n#\nHow the learner $M\\_\\theta$ and the meta-learner $R\\_\\Theta$ are trained. (Image source:\noriginal paper\nwith more annotations)\nThe training process mimics what happens during test, since it has been proved to be beneficial in\nMatching Networks\n. During each training epoch, we first sample a dataset $\\mathcal{D} = (\\mathcal{D}_\\text{train}, \\mathcal{D}_\\text{test}) \\in \\hat{\\mathcal{D}}_\\text{meta-train}$ and then sample mini-batches out of $\\mathcal{D}_\\text{train}$ to update $\\theta$ for $T$ rounds. The final state of the learner parameter $\\theta_T$ is used to train the meta-learner on the test data $\\mathcal{D}_\\text{test}$.\nTwo implementation details to pay extra attention to:\nHow to compress the parameter space in LSTM meta-learner? As the meta-learner is modeling parameters of another neural network, it would have hundreds of thousands of variables to learn. Following the\nidea\nof sharing parameters across coordinates,\nTo simplify the training process, the meta-learner assumes that the loss $\\mathcal{L}_t$ and the gradient $\\nabla_{\\theta_{t-1}} \\mathcal{L}_t$ are independent.\nMAML\n#\nMAML\n, short for\nModel-Agnostic Meta-Learning\n(\nFinn, et al. 2017\n) is a fairly general optimization algorithm, compatible with any model that learns through gradient descent.\nLet’s say our model is $f_\\theta$ with parameters $\\theta$. Given a task $\\tau_i$ and its associated dataset $(\\mathcal{D}^{(i)}_\\text{train}, \\mathcal{D}^{(i)}_\\text{test})$, we can update the model parameters by one or more gradient descent steps (the following example only contains one step):\n$$\n\\theta'_i = \\theta - \\alpha \\nabla_\\theta\\mathcal{L}^{(0)}_{\\tau_i}(f_\\theta)\n$$\nwhere $\\mathcal{L}^{(0)}$ is the loss computed using the mini data batch with id (0).\nDiagram of MAML. (Image source:\noriginal paper\n)\nWell, the above formula only optimizes for one task. To achieve a good generalization across a variety of tasks, we would like to find the optimal $\\theta^*$ so that the task-specific fine-tuning is more efficient. Now, we sample a new data batch with id (1) for updating the meta-objective. The loss, denoted as $\\mathcal{L}^{(1)}$, depends on the mini batch (1). The superscripts in $\\mathcal{L}^{(0)}$ and $\\mathcal{L}^{(1)}$ only indicate different data batches, and they refer to the same loss objective for the same task.\n$$\n\\begin{aligned}\n\\theta^* \n&= \\arg\\min_\\theta \\sum_{\\tau_i \\sim p(\\tau)} \\mathcal{L}_{\\tau_i}^{(1)} (f_{\\theta'_i}) = \\arg\\min_\\theta \\sum_{\\tau_i \\sim p(\\tau)} \\mathcal{L}_{\\tau_i}^{(1)} (f_{\\theta - \\alpha\\nabla_\\theta \\mathcal{L}_{\\tau_i}^{(0)}(f_\\theta)}) & \\\\\n\\theta &\\leftarrow \\theta - \\beta \\nabla_{\\theta} \\sum_{\\tau_i \\sim p(\\tau)} \\mathcal{L}_{\\tau_i}^{(1)} (f_{\\theta - \\alpha\\nabla_\\theta \\mathcal{L}_{\\tau_i}^{(0)}(f_\\theta)}) & \\scriptstyle{\\text{; updating rule}}\n\\end{aligned}\n$$\nThe general form of MAML algorithm. (Image source:\noriginal paper\n)\nFirst-Order MAML\n#\nThe meta-optimization step above relies on second derivatives. To make the computation less expensive, a modified version of MAML omits second derivatives, resulting in a simplified and cheaper implementation, known as\nFirst-Order MAML (FOMAML)\n.\nLet’s consider the case of performing $k$ inner gradient steps, $k\\geq1$. Starting with the initial model parameter $\\theta_\\text{meta}$:\n$$\n\\begin{aligned}\n\\theta_0 &= \\theta_\\text{meta}\\\\\n\\theta_1 &= \\theta_0 - \\alpha\\nabla_\\theta\\mathcal{L}^{(0)}(\\theta_0)\\\\\n\\theta_2 &= \\theta_1 - \\alpha\\nabla_\\theta\\mathcal{L}^{(0)}(\\theta_1)\\\\\n&\\dots\\\\\n\\theta_k &= \\theta_{k-1} - \\alpha\\nabla_\\theta\\mathcal{L}^{(0)}(\\theta_{k-1})\n\\end{aligned}\n$$\nThen in the outer loop, we sample a new data batch for updating the meta-objective.\n$$\n\\begin{aligned}\n\\theta_\\text{meta} &\\leftarrow \\theta_\\text{meta} - \\beta g_\\text{MAML} & \\scriptstyle{\\text{; update for meta-objective}} \\\\[2mm]\n\\text{where } g_\\text{MAML}\n&= \\nabla_{\\theta} \\mathcal{L}^{(1)}(\\theta_k) &\\\\[2mm]\n&= \\nabla_{\\theta_k} \\mathcal{L}^{(1)}(\\theta_k) \\cdot (\\nabla_{\\theta_{k-1}} \\theta_k) \\dots (\\nabla_{\\theta_0} \\theta_1) \\cdot (\\nabla_{\\theta} \\theta_0) & \\scriptstyle{\\text{; following the chain rule}} \\\\\n&= \\nabla_{\\theta_k} \\mathcal{L}^{(1)}(\\theta_k) \\cdot \\Big( \\prod_{i=1}^k \\nabla_{\\theta_{i-1}} \\theta_i \\Big) \\cdot I &  \\\\\n&= \\nabla_{\\theta_k} \\mathcal{L}^{(1)}(\\theta_k) \\cdot \\prod_{i=1}^k \\nabla_{\\theta_{i-1}} (\\theta_{i-1} - \\alpha\\nabla_\\theta\\mathcal{L}^{(0)}(\\theta_{i-1})) &  \\\\\n&= \\nabla_{\\theta_k} \\mathcal{L}^{(1)}(\\theta_k) \\cdot \\prod_{i=1}^k (I - \\alpha\\nabla_{\\theta_{i-1}}(\\nabla_\\theta\\mathcal{L}^{(0)}(\\theta_{i-1}))) &\n\\end{aligned}\n$$\nThe MAML gradient is:\n$$\ng_\\text{MAML} = \\nabla_{\\theta_k} \\mathcal{L}^{(1)}(\\theta_k) \\cdot \\prod_{i=1}^k (I - \\alpha \\color{red}{\\nabla_{\\theta_{i-1}}(\\nabla_\\theta\\mathcal{L}^{(0)}(\\theta_{i-1}))})\n$$\nThe First-Order MAML ignores the second derivative part in red. It is simplified as follows, equivalent to the derivative of the last inner gradient update result.\n$$\ng_\\text{FOMAML} = \\nabla_{\\theta_k} \\mathcal{L}^{(1)}(\\theta_k)\n$$\nReptile\n#\nReptile\n(\nNichol, Achiam & Schulman, 2018\n) is a remarkably simple meta-learning optimization algorithm. It is similar to MAML in many ways, given that both rely on meta-optimization through gradient descent and both are model-agnostic.\nThe Reptile works by repeatedly:\nsampling a task,\ntraining on it by multiple gradient descent steps,\nand then moving the model weights towards the new parameters.\nSee the algorithm below:\n$\\text{SGD}(\\mathcal{L}_{\\tau_i}, \\theta, k)$ performs stochastic gradient update for k steps on the loss $\\mathcal{L}_{\\tau_i}$ starting with initial parameter $\\theta$ and returns the final parameter vector. The batch version samples multiple tasks instead of one within each iteration. The reptile gradient is defined as $(\\theta - W)/\\alpha$, where $\\alpha$ is the stepsize used by the SGD operation.\nThe batched version of Reptile algorithm. (Image source:\noriginal paper\n)\nAt a glance, the algorithm looks a lot like an ordinary SGD. However, because the task-specific optimization can take more than one step. it eventually makes $\\text{SGD}(\\mathbb{E}\n_\\tau[\\mathcal{L}_{\\tau}], \\theta, k)$ diverge from $\\mathbb{E}_\\tau [\\text{SGD}(\\mathcal{L}_{\\tau}, \\theta, k)]$ when k > 1.\nThe Optimization Assumption\n#\nAssuming that a task $\\tau \\sim p(\\tau)$ has a manifold of optimal network configuration, $\\mathcal{W}_{\\tau}^*$. The model $f_\\theta$ achieves the best performance for task $\\tau$ when $\\theta$ lays on the surface of $\\mathcal{W}_{\\tau}^*$. To find a solution that is good across tasks, we would like to find a parameter close to all the optimal manifolds of all tasks:\n$$\n\\theta^* = \\arg\\min_\\theta \\mathbb{E}_{\\tau \\sim p(\\tau)} [\\frac{1}{2} \\text{dist}(\\theta, \\mathcal{W}_\\tau^*)^2]\n$$\nThe Reptile algorithm updates the parameter alternatively to be closer to the optimal manifolds of different tasks. (Image source:\noriginal paper\n)\nLet’s use the L2 distance as $\\text{dist}(.)$ and the distance between a point $\\theta$ and a set $\\mathcal{W}_\\tau^*$ equals to the distance between $\\theta$ and a point $W_{\\tau}^*(\\theta)$ on the manifold that is closest to $\\theta$:\n$$\n\\text{dist}(\\theta, \\mathcal{W}_{\\tau}^*) = \\text{dist}(\\theta, W_{\\tau}^*(\\theta)) \\text{, where }W_{\\tau}^*(\\theta) = \\arg\\min_{W\\in\\mathcal{W}_{\\tau}^*} \\text{dist}(\\theta, W)\n$$\nThe gradient of the squared euclidean distance is:\n$$\n\\begin{aligned}\n\\nabla_\\theta[\\frac{1}{2}\\text{dist}(\\theta, \\mathcal{W}_{\\tau_i}^*)^2]\n&= \\nabla_\\theta[\\frac{1}{2}\\text{dist}(\\theta, W_{\\tau_i}^*(\\theta))^2] & \\\\\n&= \\nabla_\\theta[\\frac{1}{2}(\\theta - W_{\\tau_i}^*(\\theta))^2] & \\\\\n&= \\theta - W_{\\tau_i}^*(\\theta) & \\scriptstyle{\\text{; See notes.}}\n\\end{aligned}\n$$\nNotes: According to the Reptile paper,\n“the gradient of the squared euclidean distance between a point $\\Theta$ and a set $S$ is the vector $2(\\Theta − p)$, where p is the closest point in $S$ to $\\Theta$”\n. Technically the closest point in $S$ is also a function of $\\Theta$, but I’m not sure why the gradient does not need to worry about the derivative of $p$. (Please feel free to leave me a comment or send me an email about this if you have ideas.)\nThus the update rule for one stochastic gradient step is:\n$$\n\\theta = \\theta - \\alpha \\nabla_\\theta[\\frac{1}{2} \\text{dist}(\\theta, \\mathcal{W}_{\\tau_i}^*)^2] = \\theta - \\alpha(\\theta - W_{\\tau_i}^*(\\theta)) = (1-\\alpha)\\theta + \\alpha W_{\\tau_i}^*(\\theta)\n$$\nThe closest point on the optimal task manifold $W_{\\tau_i}^*(\\theta)$ cannot be computed exactly, but Reptile approximates it using $\\text{SGD}(\\mathcal{L}_\\tau, \\theta, k)$.\nReptile vs FOMAML\n#\nTo demonstrate the deeper connection between Reptile and MAML, let’s expand the update formula with an example performing two gradient steps, k=2 in $\\text{SGD}(.)$. Same as defined\nabove\n, $\\mathcal{L}^{(0)}$ and $\\mathcal{L}^{(1)}$ are losses using different mini-batches of data. For ease of reading, we adopt two simplified annotations: $g^{(i)}_j = \\nabla_{\\theta} \\mathcal{L}^{(i)}(\\theta_j)$ and $H^{(i)}_j = \\nabla^2_{\\theta} \\mathcal{L}^{(i)}(\\theta_j)$.\n$$\n\\begin{aligned}\n\\theta_0 &= \\theta_\\text{meta}\\\\\n\\theta_1 &= \\theta_0 - \\alpha\\nabla_\\theta\\mathcal{L}^{(0)}(\\theta_0)= \\theta_0 - \\alpha g^{(0)}_0 \\\\\n\\theta_2 &= \\theta_1 - \\alpha\\nabla_\\theta\\mathcal{L}^{(1)}(\\theta_1) = \\theta_0 - \\alpha g^{(0)}_0 - \\alpha g^{(1)}_1\n\\end{aligned}\n$$\nAccording to the\nearly section\n, the gradient of FOMAML is the last inner gradient update result. Therefore, when k=1:\n$$\n\\begin{aligned}\ng_\\text{FOMAML} &= \\nabla_{\\theta_1} \\mathcal{L}^{(1)}(\\theta_1) = g^{(1)}_1 \\\\\ng_\\text{MAML} &= \\nabla_{\\theta_1} \\mathcal{L}^{(1)}(\\theta_1) \\cdot (I - \\alpha\\nabla^2_{\\theta} \\mathcal{L}^{(0)}(\\theta_0)) = g^{(1)}_1 - \\alpha H^{(0)}_0 g^{(1)}_1\n\\end{aligned}\n$$\nThe Reptile gradient is defined as:\n$$\ng_\\text{Reptile} = (\\theta_0 - \\theta_2) / \\alpha = g^{(0)}_0 + g^{(1)}_1\n$$\nUp to now we have:\nReptile versus FOMAML in one loop of meta-optimization. (Image source:\nslides\non Reptile by Yoonho Lee.)\n$$\n\\begin{aligned}\ng_\\text{FOMAML} &= g^{(1)}_1 \\\\\ng_\\text{MAML} &= g^{(1)}_1 - \\alpha H^{(0)}_0 g^{(1)}_1 \\\\\ng_\\text{Reptile} &= g^{(0)}_0 + g^{(1)}_1\n\\end{aligned}\n$$\nNext let’s try further expand $g^{(1)}_1$ using\nTaylor expansion\n. Recall that Taylor expansion of a function $f(x)$ that is differentiable at a number $a$ is:\n$$\nf(x) = f(a) + \\frac{f'(a)}{1!}(x-a) + \\frac{f''(a)}{2!}(x-a)^2 + \\dots = \\sum_{i=0}^\\infty \\frac{f^{(i)}(a)}{i!}(x-a)^i\n$$\nWe can consider $\\nabla_{\\theta}\\mathcal{L}^{(1)}(.)$ as a function and $\\theta_0$ as a value point. The Taylor expansion of $g_1^{(1)}$ at the value point $\\theta_0$ is:\n$$\n\\begin{aligned}\ng_1^{(1)} &= \\nabla_{\\theta}\\mathcal{L}^{(1)}(\\theta_1) \\\\\n&= \\nabla_{\\theta}\\mathcal{L}^{(1)}(\\theta_0) + \\nabla^2_\\theta\\mathcal{L}^{(1)}(\\theta_0)(\\theta_1 - \\theta_0) + \\frac{1}{2}\\nabla^3_\\theta\\mathcal{L}^{(1)}(\\theta_0)(\\theta_1 - \\theta_0)^2 + \\dots & \\\\\n&= g_0^{(1)} - \\alpha H^{(1)}_0 g_0^{(0)} + \\frac{\\alpha^2}{2}\\nabla^3_\\theta\\mathcal{L}^{(1)}(\\theta_0) (g_0^{(0)})^2 + \\dots & \\scriptstyle{\\text{; because }\\theta_1-\\theta_0=-\\alpha g_0^{(0)}} \\\\\n&= g_0^{(1)} - \\alpha H^{(1)}_0 g_0^{(0)} + O(\\alpha^2)\n\\end{aligned}\n$$\nPlug in the expanded form of $g_1^{(1)}$ into the MAML gradients with one step inner gradient update:\n$$\n\\begin{aligned}\ng_\\text{FOMAML} &= g^{(1)}_1 = g_0^{(1)} - \\alpha H^{(1)}_0 g_0^{(0)} + O(\\alpha^2)\\\\\ng_\\text{MAML} &= g^{(1)}_1 - \\alpha H^{(0)}_0 g^{(1)}_1 \\\\\n&= g_0^{(1)} - \\alpha H^{(1)}_0 g_0^{(0)} + O(\\alpha^2) - \\alpha H^{(0)}_0 (g_0^{(1)} - \\alpha H^{(1)}_0 g_0^{(0)} + O(\\alpha^2))\\\\\n&= g_0^{(1)} - \\alpha H^{(1)}_0 g_0^{(0)} - \\alpha H^{(0)}_0 g_0^{(1)} + \\alpha^2 \\alpha H^{(0)}_0 H^{(1)}_0 g_0^{(0)} + O(\\alpha^2)\\\\\n&= g_0^{(1)} - \\alpha H^{(1)}_0 g_0^{(0)} - \\alpha H^{(0)}_0 g_0^{(1)} + O(\\alpha^2)\n\\end{aligned}\n$$\nThe Reptile gradient becomes:\n$$\n\\begin{aligned}\ng_\\text{Reptile} \n&= g^{(0)}_0 + g^{(1)}_1 \\\\\n&= g^{(0)}_0 + g_0^{(1)} - \\alpha H^{(1)}_0 g_0^{(0)} + O(\\alpha^2)\n\\end{aligned}\n$$\nSo far we have the formula of three types of gradients:\n$$\n\\begin{aligned}\ng_\\text{FOMAML} &= g_0^{(1)} - \\alpha H^{(1)}_0 g_0^{(0)} + O(\\alpha^2)\\\\\ng_\\text{MAML} &= g_0^{(1)} - \\alpha H^{(1)}_0 g_0^{(0)} - \\alpha H^{(0)}_0 g_0^{(1)} + O(\\alpha^2)\\\\\ng_\\text{Reptile}  &= g^{(0)}_0 + g_0^{(1)} - \\alpha H^{(1)}_0 g_0^{(0)} + O(\\alpha^2)\n\\end{aligned}\n$$\nDuring training, we often average over multiple data batches. In our example, the mini batches (0) and (1) are interchangeable since both are drawn at random. The expectation $\\mathbb{E}_{\\tau,0,1}$ is averaged over two data batches, ids (0) and (1), for task $\\tau$.\nLet,\n$A = \\mathbb{E}_{\\tau,0,1} [g_0^{(0)}] = \\mathbb{E}_{\\tau,0,1} [g_0^{(1)}]$; it is the average gradient of task loss. We expect to improve the model parameter to achieve better task performance by following this direction pointed by $A$.\n$B = \\mathbb{E}_{\\tau,0,1} [H^{(1)}_0 g_0^{(0)}] = \\frac{1}{2}\\mathbb{E}_{\\tau,0,1} [H^{(1)}_0 g_0^{(0)} + H^{(0)}_0 g_0^{(1)}] = \\frac{1}{2}\\mathbb{E}_{\\tau,0,1} [\\nabla_\\theta(g^{(0)}_0 g_0^{(1)})]$; it is the direction (gradient) that increases the inner product of gradients of two different mini batches for the same task. We expect to improve the model parameter to achieve better generalization over different data by following this direction pointed by $B$.\nTo conclude, both MAML and Reptile aim to optimize for the same goal, better task performance (guided by A) and better generalization (guided by B), when the gradient update is approximated by first three leading terms.\n$$\n\\begin{aligned}\n\\mathbb{E}_{\\tau,1,2}[g_\\text{FOMAML}] &= A - \\alpha B + O(\\alpha^2)\\\\\n\\mathbb{E}_{\\tau,1,2}[g_\\text{MAML}] &= A - 2\\alpha B + O(\\alpha^2)\\\\\n\\mathbb{E}_{\\tau,1,2}[g_\\text{Reptile}]  &= 2A - \\alpha B + O(\\alpha^2)\n\\end{aligned}\n$$\nIt is not clear to me whether the ignored term $O(\\alpha^2)$ might play a big impact on the parameter learning. But given that FOMAML is able to obtain a similar performance as the full version of MAML, it might be safe to say higher-level derivatives would not be critical during gradient descent update.\nCited as:\n@article{weng2018metalearning,\n  title   = \"Meta-Learning: Learning to Learn Fast\",\n  author  = \"Weng, Lilian\",\n  journal = \"lilianweng.github.io\",\n  year    = \"2018\",\n  url     = \"https://lilianweng.github.io/posts/2018-11-30-meta-learning/\"\n}\nReference\n#\n[1] Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum.\n“Human-level concept learning through probabilistic program induction.”\nScience 350.6266 (2015): 1332-1338.\n[2] Oriol Vinyals’ talk on\n“Model vs Optimization Meta Learning”\n[3] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov.\n“Siamese neural networks for one-shot image recognition.”\nICML Deep Learning Workshop. 2015.\n[4] Oriol Vinyals, et al.\n“Matching networks for one shot learning.”\nNIPS. 2016.\n[5] Flood Sung, et al.\n“Learning to compare: Relation network for few-shot learning.”\nCVPR. 2018.\n[6] Jake Snell, Kevin Swersky, and Richard Zemel.\n“Prototypical Networks for Few-shot Learning.”\nCVPR. 2018.\n[7] Adam Santoro, et al.\n“Meta-learning with memory-augmented neural networks.”\nICML. 2016.\n[8] Alex Graves, Greg Wayne, and Ivo Danihelka.\n“Neural turing machines.”\narXiv preprint arXiv:1410.5401 (2014).\n[9] Tsendsuren Munkhdalai and Hong Yu.\n“Meta Networks.”\nICML. 2017.\n[10] Sachin Ravi and Hugo Larochelle.\n“Optimization as a Model for Few-Shot Learning.”\nICLR. 2017.\n[11] Chelsea Finn’s BAIR blog on\n“Learning to Learn”\n.\n[12] Chelsea Finn, Pieter Abbeel, and Sergey Levine.\n“Model-agnostic meta-learning for fast adaptation of deep networks.”\nICML 2017.\n[13] Alex Nichol, Joshua Achiam, John Schulman.\n“On First-Order Meta-Learning Algorithms.”\narXiv preprint arXiv:1803.02999 (2018).\n[14]\nSlides on Reptile\nby Yoonho Lee.",
  "metadata": {
    "title": "Meta-Learning: Learning to Learn Fast | Lil'Log",
    "description": "[Updated on 2019-10-01: thanks to Tianhao, we have this post translated in Chinese!]",
    "keywords": "meta-learning, long-read",
    "author": "Lilian Weng",
    "og:title": "Meta-Learning: Learning to Learn Fast",
    "og:description": "[Updated on 2019-10-01: thanks to Tianhao, we have this post translated in Chinese!]",
    "og:type": "article",
    "og:url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/",
    "twitter:card": "summary",
    "twitter:title": "Meta-Learning: Learning to Learn Fast",
    "twitter:description": "[Updated on 2019-10-01: thanks to Tianhao, we have this post translated in Chinese!]",
    "article:section": "posts",
    "article:published_time": "2018-11-30T00:00:00+00:00",
    "article:modified_time": "2018-11-30T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/a8dc6bf17c71/code/body-000.txt",
      "chars": 245,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/few-shot-classification.png",
      "bytes": 1548239,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-23d602a28b9d9dda6fe1.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/few-shot-classification.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/siamese-conv-net.png",
      "bytes": 665187,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-11419d5ec12fba3fb7f9.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/siamese-conv-net.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/matching-networks.png",
      "bytes": 730525,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-34f37f1d8bf391fe9f42.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/matching-networks.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/relation-network.png",
      "bytes": 808103,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-851555a6cfa04864e872.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/relation-network.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/prototypical-networks.png",
      "bytes": 184486,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-1195100b0c6841a237dd.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/prototypical-networks.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/NTM.png",
      "bytes": 194344,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-c47eeb17ebb22f866942.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/NTM.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/mann-meta-learning.png",
      "bytes": 273207,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-9ae2425c1e62b530a443.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/mann-meta-learning.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/combine-slow-fast-weights.png",
      "bytes": 174582,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-cb826c25071414ad89e4.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/combine-slow-fast-weights.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/meta-network.png",
      "bytes": 405772,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-0167fe1ca07386abf1d2.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/meta-network.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/lstm-meta-learner.png",
      "bytes": 1245251,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-ae1778af670dfaca75e8.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/lstm-meta-learner.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/train-meta-learner.png",
      "bytes": 397702,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-9baa7ed8723c2cae889a.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/train-meta-learner.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/maml.png",
      "bytes": 102014,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-f1b0701dad5bcf83ef5a.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/maml.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/maml-algo.png",
      "bytes": 195622,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-7f2ae4aa3936d2a466d8.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/maml-algo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/reptile-algo.png",
      "bytes": 286187,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-d947284bd0373951e25d.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/reptile-algo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/reptile-optim.png",
      "bytes": 124913,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-830017a377b95ff33b98.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/reptile-optim.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/reptile_vs_FOMAML.png",
      "bytes": 1295521,
      "reused": false,
      "local_path": "attachments/a8dc6bf17c71/images/body-d0137fe96ca3220fcf1f.png",
      "src": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/reptile_vs_FOMAML.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://wei-tianhao.github.io/blog/2019/09/17/meta-learning.html",
      "label": "Chinese",
      "type": "a"
    },
    {
      "href": "https://www.cs.cmu.edu/~rsalakhu/papers/LakeEtAl2015Science.pdf",
      "label": "learning to learn",
      "type": "a"
    },
    {
      "href": "https://www.pinterest.com/",
      "label": "Pinterest",
      "type": "a"
    },
    {
      "href": "http://metalearning-symposium.ml/files/vinyals.pdf",
      "label": "talk",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm",
      "label": "k-NN",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/K-means_clustering",
      "label": "k-means",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Kernel_density_estimation",
      "label": "kernel density estimation",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Similarity_learning",
      "label": "Metric learning",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/769-signature-verification-using-a-siamese-time-delay-neural-network.pdf",
      "label": "Siamese Neural Network",
      "type": "a"
    },
    {
      "href": "http://www.cs.toronto.edu/~rsalakhu/papers/oneshot1.pdf",
      "label": "Koch, Zemel & Salakhutdinov (2015)",
      "type": "a"
    },
    {
      "href": "http://papers.nips.cc/paper/6385-matching-networks-for-one-shot-learning.pdf",
      "label": "Vinyals et al., 2016",
      "type": "a"
    },
    {
      "href": "http://openaccess.thecvf.com/content_cvpr_2018/papers_backup/Sung_Learning_to_Compare_CVPR_2018_paper.pdf",
      "label": "Sung et al., 2018",
      "type": "a"
    },
    {
      "href": "https://deepmind.com/blog/neural-approach-relational-reasoning/",
      "label": "Relation Network",
      "type": "a"
    },
    {
      "href": "http://papers.nips.cc/paper/6996-prototypical-networks-for-few-shot-learning.pdf",
      "label": "Snell, Swersky & Zemel, 2017",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-06-24-attention/",
      "label": "Neural Turing Machines",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1410.3916",
      "label": "Memory Networks",
      "type": "a"
    },
    {
      "href": "http://proceedings.mlr.press/v48/santoro16.pdf",
      "label": "Santoro et al. (2016)",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Cache_replacement_policies",
      "label": "cache replacement",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Least_frequently_used",
      "label": "LFU",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1703.00837",
      "label": "Munkhdalai & Yu, 2017",
      "type": "a"
    },
    {
      "href": "https://openreview.net/pdf?id=rJY0-Kcll",
      "label": "Ravi & Larochelle (2017)",
      "type": "a"
    },
    {
      "href": "http://ruder.io/optimizing-gradient-descent/index.html",
      "label": "momentum",
      "type": "a"
    },
    {
      "href": "https://openreview.net/pdf?id=rJY0-Kcll)",
      "label": "original paper",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1606.04474",
      "label": "idea",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1703.03400",
      "label": "Finn, et al. 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1803.02999",
      "label": "Nichol, Achiam & Schulman, 2018",
      "type": "a"
    },
    {
      "href": "https://www.slideshare.net/YoonhoLee4/on-firstorder-metalearning-algorithms",
      "label": "slides",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Taylor_series",
      "label": "Taylor expansion",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1410.5401",
      "label": "“Neural turing machines.”",
      "type": "a"
    },
    {
      "href": "https://bair.berkeley.edu/blog/2017/07/18/learning-to-learn/",
      "label": "“Learning to Learn”",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:39Z",
  "source_url": "https://lilianweng.github.io/posts/2018-11-30-meta-learning/",
  "papers": [
    {
      "status": "ok",
      "url": "https://www.cs.cmu.edu/~rsalakhu/papers/LakeEtAl2015Science.pdf",
      "bytes": 1826353,
      "reused": false,
      "local_path": "papers/repair/da4e56c26698557aa61c.pdf"
    },
    {
      "status": "download_error",
      "url": "http://metalearning-symposium.ml/files/vinyals.pdf",
      "reason": "ConnectionError: HTTPConnectionPool(host='metalearning-symposium.ml', port=80): Max retries exceeded with url: /files/vinyals.pdf (Caused by NameResolutionError(\"HTTPConnection(host='metalearning-s",
      "retried_at": "2026-10-20T22:24:44Z"
    },
    {
      "status": "http_404",
      "url": "https://papers.nips.cc/paper/769-signature-verification-using-a-siamese-time-delay-neural-network.pdf",
      "retried_at": "2026-10-20T18:04:05Z"
    },
    {
      "status": "ok",
      "url": "http://www.cs.toronto.edu/~rsalakhu/papers/oneshot1.pdf",
      "bytes": 766340,
      "reused": false,
      "local_path": "papers/repair/fde088adb2e69f52cd16.pdf"
    },
    {
      "status": "ok",
      "url": "http://papers.nips.cc/paper/6385-matching-networks-for-one-shot-learning.pdf",
      "bytes": 2890623,
      "reused": false,
      "local_path": "papers/repair/66a471c52cb43cdc0b56.pdf"
    },
    {
      "status": "ok",
      "url": "http://openaccess.thecvf.com/content_cvpr_2018/papers_backup/Sung_Learning_to_Compare_CVPR_2018_paper.pdf",
      "bytes": 798122,
      "reused": false,
      "local_path": "papers/repair/1dce78ce43bce934d7c8.pdf"
    },
    {
      "status": "ok",
      "url": "http://papers.nips.cc/paper/6996-prototypical-networks-for-few-shot-learning.pdf",
      "bytes": 536399,
      "reused": false,
      "local_path": "papers/repair/33350b7fc96a8da3fa63.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1410.3916",
      "bytes": 215900,
      "reused": false,
      "local_path": "papers/repair/e6bb14affb9d1f13f660.pdf"
    },
    {
      "status": "ok",
      "url": "http://proceedings.mlr.press/v48/santoro16.pdf",
      "bytes": 1648270,
      "reused": false,
      "local_path": "papers/repair/b9ba58828decfbd2ad1e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1703.00837",
      "bytes": 455366,
      "reused": false,
      "local_path": "papers/repair/b15e9ef4f202cbfada2c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1606.04474",
      "bytes": 2702892,
      "reused": false,
      "local_path": "papers/repair/76aa0aeb7717cd2e1f8f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1703.03400",
      "bytes": 4064212,
      "reused": true,
      "local_path": "papers/repair/745c6af0eff74ad13278.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1803.02999",
      "bytes": 1145571,
      "reused": false,
      "local_path": "papers/repair/7f84fa715e690339a4bf.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1410.5401",
      "bytes": 1357690,
      "reused": true,
      "local_path": "papers/repair/968a0ef3e0f98e3e8588.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 46235,
  "expected_body_images": 16,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "download_error；http_404",
  "raw_html": "attachments/a8dc6bf17c71/pages/source.html",
  "local_html": "attachments/a8dc6bf17c71/pages/article.html",
  "body_base_path": "attachments/a8dc6bf17c71/pages/",
  "body_markdown_path": "attachments/a8dc6bf17c71/pages/article.md"
}