{
  "source_file": "0930",
  "source_platform": "技术平台",
  "author": "Distill",
  "title_listed": "Why Momentum Really Works",
  "keywords": "动量,优化,梯度下降",
  "domain": "机器学习",
  "content_type": "图解教程",
  "is_paid_flag": "否",
  "url": "https://distill.pub/2017/momentum/",
  "publish_date": "2017-04",
  "corpus_stage": "预训练/SFT",
  "notes": "动量优化为何有效",
  "id": "d6f0f4773deb",
  "collected_at": "2026-10-02T17:54:36",
  "title": "Why Momentum Really Works",
  "body_markdown": "Distill\nAbout\nPrize\nSubmit\n⋆\n\\star\n⋆\n+\n+\n+\n−\n-\n−\n=\n=\n=\nα\n\\alpha\nα\nλ\n\\lambda\nλ\nβ\n\\beta\nβ\nR\nR\nR\nα\n=\n\\alpha=\nα\n=\nβ\n=\n\\beta=\nβ\n=\nβ\n=\n0\n\\beta = 0\nβ\n=\n0\nβ\n=\n1\n\\beta=1\nβ\n=\n1\nα\n=\n1\n/\nλ\ni\n\\alpha = 1/\\lambda_i\nα\n=\n1\n/\nλ\n​\ni\n​\n​\nm\no\nd\ne\nl\n\\text{model}\nmodel\n0\np\n1\n0 p_1\n0\np\n​\n1\n​\n​\n0\np\n¯\n1\n0 \\bar{p}_1\n0\n​\np\n​\n¯\n​\n​\n​\n1\n​\n​\n2\nβ\n2\\sqrt{\\beta}\n2\n√\n​\nβ\n​\n​\n​\nλ\ni\n\\lambda_i\nλ\n​\ni\n​\n​\nλ\ni\n=\n0\n\\lambda_i = 0\nλ\n​\ni\n​\n​\n=\n0\nα\n>\n1\n/\nλ\ni\n\\alpha > 1/\\lambda_i\nα\n>\n1\n/\nλ\n​\ni\n​\n​\nmax\n{\n∣\nσ\n1\n∣\n,\n∣\nσ\n2\n∣\n}\n>\n1\n\\max\\{|\\sigma_1|,|\\sigma_2|\\} > 1\nmax\n{\n∣\nσ\n​\n1\n​\n​\n∣\n,\n∣\nσ\n​\n2\n​\n​\n∣\n}\n>\n1\nx\ni\nk\n−\nx\ni\n∗\nx_i^k - x_i^*\nx\n​\ni\n​\nk\n​\n​\n−\nx\n​\ni\n​\n∗\n​\n​\nξ\ni\n\\xi_i\nξ\n​\ni\n​\n​\nβ\n=\n(\n1\n−\nα\nλ\ni\n)\n2\n\\beta = (1 - \\sqrt{\\alpha \\lambda_i})^2\nβ\n=\n(\n1\n−\n√\n​\nα\nλ\n​\ni\n​\n​\n​\n​\n​\n)\n​\n2\n​\n​\nWhy Momentum Really Works\nStep-size α = 0.02\nMomentum β = 0.99\nWe often think of Momentum as a means of dampening oscillations and speeding up the iterations, leading to faster convergence. But it has other interesting behavior. It allows a larger range of step-sizes to be used, and creates its own oscillations. What is going on?\nGabriel Goh\nUC Davis\nApril. 4\n2017\nCitation:\nGoh, 2017\nHere’s a popular story about momentum\n[1, 2, 3]\n: gradient descent is a man walking down a hill. He follows the steepest path downwards; his progress is slow, but steady. Momentum is a heavy ball rolling down the same hill. The added inertia acts both as a smoother and an accelerator, dampening oscillations and causing us to barrel through narrow valleys, small humps and local minima.\nThis standard story isn’t wrong, but it fails to explain many important behaviors of momentum. In fact, momentum can be understood far more precisely if we study it on the right model.\nOne nice model is the convex quadratic. This model is rich enough to reproduce momentum’s local dynamics in real problems, and yet simple enough to be understood in closed form. This balance gives us powerful traction for understanding this algorithm.\nWe begin with gradient descent. The algorithm has many virtues, but speed is not one of them. It is simple — when optimizing a smooth function\nf\nf\nf\n, we make a small step in the gradient\nw\nk\n+\n1\n=\nw\nk\n−\nα\n∇\nf\n(\nw\nk\n)\n.\nw^{k+1} = w^k-\\alpha\\nabla f(w^k).\nw\n​\nk\n+\n1\n​\n​\n=\nw\n​\nk\n​\n​\n−\nα\n∇\nf\n(\nw\n​\nk\n​\n​\n)\n.\nFor a step-size small enough, gradient descent makes a monotonic improvement at every iteration. It always converges, albeit to a local minimum. And under a few weak curvature conditions it can even get there at an exponential rate.\nBut the exponential decrease, though appealing in theory, can often be infuriatingly small. Things often begin quite well — with an impressive, almost immediate decrease in the loss. But as the iterations progress, things start to slow down. You start to get a nagging feeling you’re not making as much progress as you should be. What has gone wrong?\nThe problem could be the optimizer’s old nemesis, pathological curvature. Pathological curvature is, simply put, regions of\nf\nf\nf\nwhich aren’t scaled properly. The landscapes are often described as valleys, trenches, canals and ravines. The iterates either jump between valleys, or approach the optimum in small, timid steps. Progress along certain directions grind to a halt. In these unfortunate regions, gradient descent fumbles.\nMomentum proposes the following tweak to gradient descent. We give gradient descent a short-term memory:\nz\nk\n+\n1\n=\nβ\nz\nk\n+\n∇\nf\n(\nw\nk\n)\nw\nk\n+\n1\n=\nw\nk\n−\nα\nz\nk\n+\n1\n\\begin{aligned}\n  z^{k+1}&=\\beta z^{k}+\\nabla f(w^{k})\\\\[0.4em]\n  w^{k+1}&=w^{k}-\\alpha z^{k+1}\n  \\end{aligned}\n​\nz\n​\nk\n+\n1\n​\n​\n​\nw\n​\nk\n+\n1\n​\n​\n​\n​\n​\n=\nβ\nz\n​\nk\n​\n​\n+\n∇\nf\n(\nw\n​\nk\n​\n​\n)\n​\n=\nw\n​\nk\n​\n​\n−\nα\nz\n​\nk\n+\n1\n​\n​\n​\n​\nThe change is innocent, and costs almost nothing. When\nβ\n=\n0\n\\beta = 0\nβ\n=\n0\n, we recover gradient descent. But for\nβ\n=\n0\n.\n9\n9\n\\beta = 0.99\nβ\n=\n0\n.\n9\n9\n(sometimes\n0\n.\n9\n9\n9\n0.999\n0\n.\n9\n9\n9\n, if things are really bad), this appears to be the boost we need. Our iterations regain that speed and boldness it lost, speeding to the optimum with a renewed energy.\nOptimizers call this minor miracle “acceleration”.\nThe new algorithm may seem at first glance like a cheap hack. A simple trick to get around gradient descent’s more aberrant behavior — a smoother for oscillations between steep canyons. But the truth, if anything, is the other way round. It is gradient descent which is the hack. First, momentum gives up to a quadratic speedup on many functions.\n1\nThis is no small matter — this is similar to the speedup you get from the Fast Fourier Transform, Quicksort, and Grover’s Algorithm. When the universe gives you quadratic speedups, you should start to pay attention.\nBut there’s more. A lower bound, courtesy of Nesterov\n[5]\n, states that momentum is, in a certain very narrow and technical sense, optimal. Now, this doesn’t mean it is the best algorithm for all functions in all circumstances. But it does satisfy some curiously beautiful mathematical properties which scratch a very human itch for perfection and closure. But more on that later. Let’s say this for now — momentum is an algorithm for the book.\nFirst Steps: Gradient Descent\nWe begin by studying gradient descent on the simplest model possible which isn’t trivial — the convex quadratic,\nf\n(\nw\n)\n=\n1\n2\nw\nT\nA\nw\n−\nb\nT\nw\n,\nw\n∈\nR\nn\n.\nf(w) = \\tfrac{1}{2}w^TAw - b^Tw, \\qquad w \\in \\mathbf{R}^n.\nf\n(\nw\n)\n=\n​\n2\n​\n​\n1\n​\n​\nw\n​\nT\n​\n​\nA\nw\n−\nb\n​\nT\n​\n​\nw\n,\nw\n∈\nR\n​\nn\n​\n​\n.\nAssume\nA\nA\nA\nis symmetric and invertible, then the optimal solution\nw\n⋆\nw^{\\star}\nw\n​\n⋆\n​\n​\noccurs at\nw\n⋆\n=\nA\n−\n1\nb\n.\nw^{\\star} = A^{-1}b.\nw\n​\n⋆\n​\n​\n=\nA\n​\n−\n1\n​\n​\nb\n.\nSimple as this model may be, it is rich enough to approximate many functions (think of\nA\nA\nA\nas your favorite model of curvature — the Hessian, Fisher Information Matrix\n[6]\n, etc) and captures all the key features of pathological curvature. And more importantly, we can write an exact closed formula for gradient descent on this function.\nThis is how it goes. Since\n∇\nf\n(\nw\n)\n=\nA\nw\n−\nb\n\\nabla f(w)=Aw - b\n∇\nf\n(\nw\n)\n=\nA\nw\n−\nb\n, the iterates are\nw\nk\n+\n1\n=\nw\nk\n−\nα\n(\nA\nw\nk\n−\nb\n)\n.\nw^{k+1}=w^{k}- \\alpha (Aw^{k} - b).\nw\n​\nk\n+\n1\n​\n​\n=\nw\n​\nk\n​\n​\n−\nα\n(\nA\nw\n​\nk\n​\n​\n−\nb\n)\n.\nHere’s the trick. There is a very natural space to view gradient descent where all the dimensions act independently — the eigenvectors of\nA\nA\nA\n.\nEvery symmetric matrix\nA\nA\nA\nhas an eigenvalue decomposition\nA\n=\nQ\nd\ni\na\ng\n(\nλ\n1\n,\n…\n,\nλ\nn\n)\nQ\nT\n,\nQ\n=\n[\nq\n1\n,\n…\n,\nq\nn\n]\n,\nA=Q\\ \\text{diag}(\\lambda_{1},\\ldots,\\lambda_{n})\\ Q^{T},\\qquad Q = [q_1,\\ldots,q_n],\nA\n=\nQ\ndiag\n(\nλ\n​\n1\n​\n​\n,\n…\n,\nλ\n​\nn\n​\n​\n)\nQ\n​\nT\n​\n​\n,\nQ\n=\n[\nq\n​\n1\n​\n​\n,\n…\n,\nq\n​\nn\n​\n​\n]\n,\nand, as per convention, we will assume that the\nλ\ni\n\\lambda_i\nλ\n​\ni\n​\n​\n’s are sorted, from smallest\nλ\n1\n\\lambda_1\nλ\n​\n1\n​\n​\nto biggest\nλ\nn\n\\lambda_n\nλ\n​\nn\n​\n​\n. If we perform a change of basis,\nx\nk\n=\nQ\nT\n(\nw\nk\n−\nw\n⋆\n)\nx^{k} = Q^T(w^{k} - w^\\star)\nx\n​\nk\n​\n​\n=\nQ\n​\nT\n​\n​\n(\nw\n​\nk\n​\n​\n−\nw\n​\n⋆\n​\n​\n)\n, the iterations break apart, becoming:\nx\ni\nk\n+\n1\n=\nx\ni\nk\n−\nα\nλ\ni\nx\ni\nk\n=\n(\n1\n−\nα\nλ\ni\n)\nx\ni\nk\n=\n(\n1\n−\nα\nλ\ni\n)\nk\n+\n1\nx\ni\n0\n\\begin{aligned}\n  x_{i}^{k+1} & =x_{i}^{k}-\\alpha \\lambda_ix_{i}^{k} \\\\[0.4em]\n   &= (1-\\alpha\\lambda_i)x^k_i=(1-\\alpha \\lambda_i)^{k+1}x^0_i\n  \\end{aligned}\n​\nx\n​\ni\n​\nk\n+\n1\n​\n​\n​\n​\n​\n​\n=\nx\n​\ni\n​\nk\n​\n​\n−\nα\nλ\n​\ni\n​\n​\nx\n​\ni\n​\nk\n​\n​\n​\n=\n(\n1\n−\nα\nλ\n​\ni\n​\n​\n)\nx\n​\ni\n​\nk\n​\n​\n=\n(\n1\n−\nα\nλ\n​\ni\n​\n​\n)\n​\nk\n+\n1\n​\n​\nx\n​\ni\n​\n0\n​\n​\n​\n​\nMoving back to our original space\nw\nw\nw\n, we can see that\nw\nk\n−\nw\n⋆\n=\nQ\nx\nk\n=\n∑\ni\nn\nx\ni\n0\n(\n1\n−\nα\nλ\ni\n)\nk\nq\ni\nw^k - w^\\star = Qx^k=\\sum_i^n x^0_i(1-\\alpha\\lambda_i)^k q_i\nw\n​\nk\n​\n​\n−\nw\n​\n⋆\n​\n​\n=\nQ\nx\n​\nk\n​\n​\n=\n​\ni\n​\n∑\n​\nn\n​\n​\nx\n​\ni\n​\n0\n​\n​\n(\n1\n−\nα\nλ\n​\ni\n​\n​\n)\n​\nk\n​\n​\nq\n​\ni\n​\n​\nand there we have it — gradient descent in closed form.\nDecomposing the Error\nThe above equation admits a simple interpretation. Each element of\nx\n0\nx^0\nx\n​\n0\n​\n​\nis the component of the error in the initial guess in the\nQ\nQ\nQ\n-basis. There are\nn\nn\nn\nsuch errors, and each of these errors follows its own, solitary path to the minimum, decreasing exponentially with a compounding rate of\n1\n−\nα\nλ\ni\n1-\\alpha\\lambda_i\n1\n−\nα\nλ\n​\ni\n​\n​\n. The closer that number is to\n1\n1\n1\n, the slower it converges.\nFor most step-sizes, the eigenvectors with largest eigenvalues converge the fastest. This triggers an explosion of progress in the first few iterations, before things slow down as the smaller eigenvectors’ struggles are revealed. By writing the contributions of each eigenspace’s error to the loss\nf\n(\nw\nk\n)\n−\nf\n(\nw\n⋆\n)\n=\n∑\n(\n1\n−\nα\nλ\ni\n)\n2\nk\nλ\ni\n[\nx\ni\n0\n]\n2\nf(w^{k})-f(w^{\\star})=\\sum(1-\\alpha\\lambda_{i})^{2k}\\lambda_{i}[x_{i}^{0}]^2\nf\n(\nw\n​\nk\n​\n​\n)\n−\nf\n(\nw\n​\n⋆\n​\n​\n)\n=\n∑\n(\n1\n−\nα\nλ\n​\ni\n​\n​\n)\n​\n2\nk\n​\n​\nλ\n​\ni\n​\n​\n[\nx\n​\ni\n​\n0\n​\n​\n]\n​\n2\n​\n​\nwe can visualize the contributions of each error component to the loss.\nOptimization can be seen as combination of several component problems, shown here as\n1\n2\n3 with eigenvalues\nλ\n1\n=\n0\n.\n0\n1\n\\lambda_1=0.01\nλ\n​\n1\n​\n​\n=\n0\n.\n0\n1\n,\nλ\n2\n=\n0\n.\n1\n\\lambda_2=0.1\nλ\n​\n2\n​\n​\n=\n0\n.\n1\n, and\nλ\n3\n=\n1\n\\lambda_3=1\nλ\n​\n3\n​\n​\n=\n1\nrespectively.\nStep-size\nOptimal Step-size\nChoosing A Step-size\nThe above analysis gives us immediate guidance as to how to set a step-size\nα\n\\alpha\nα\n. In order to converge, each\n∣\n1\n−\nα\nλ\ni\n∣\n|1-\\alpha \\lambda_i|\n∣\n1\n−\nα\nλ\n​\ni\n​\n​\n∣\nmust be strictly less than 1. All workable step-sizes, therefore, fall in the interval\n0\n<\nα\nλ\ni\n<\n2\n.\n0<\\alpha\\lambda_i<2.\n0\n<\nα\nλ\n​\ni\n​\n​\n<\n2\n.\nThe overall convergence rate is determined by the slowest error component, which must be either\nλ\n1\n\\lambda_1\nλ\n​\n1\n​\n​\nor\nλ\nn\n\\lambda_n\nλ\n​\nn\n​\n​\n:\nr\na\nt\ne\n(\nα\n)\n=\nmax\ni\n∣\n1\n−\nα\nλ\ni\n∣\n=\nmax\n{\n∣\n1\n−\nα\nλ\n1\n∣\n,\n∣\n1\n−\nα\nλ\nn\n∣\n}\n\\begin{aligned}\\text{rate}(\\alpha) & ~=~ \\max_{i}\\left|1-\\alpha\\lambda_{i}\\right|\\\\[0.9em] & ~=~ \\max\\left\\{|1-\\alpha\\lambda_{1}|,~ |1-\\alpha\\lambda_{n}|\\right\\} \\end{aligned}\n​\nrate\n(\nα\n)\n​\n​\n​\n​\n=\n​\ni\n​\nmax\n​\n​\n∣\n1\n−\nα\nλ\n​\ni\n​\n​\n∣\n​\n=\nmax\n{\n∣\n1\n−\nα\nλ\n​\n1\n​\n​\n∣\n,\n∣\n1\n−\nα\nλ\n​\nn\n​\n​\n∣\n}\n​\n​\nThis overall rate is minimized when the rates for\nλ\n1\n\\lambda_1\nλ\n​\n1\n​\n​\nand\nλ\nn\n\\lambda_n\nλ\n​\nn\n​\n​\nare the same — this mirrors our informal observation in the previous section that the optimal step-size causes the first and last eigenvectors to converge at the same rate. If we work this through we get:\no\np\nt\ni\nm\na\nl\nα\n=\na\nr\ng\nm\ni\nn\nα\nr\na\nt\ne\n(\nα\n)\n=\n2\nλ\n1\n+\nλ\nn\no\np\nt\ni\nm\na\nl\nr\na\nt\ne\n=\nmin\nα\nr\na\nt\ne\n(\nα\n)\n=\nλ\nn\n/\nλ\n1\n−\n1\nλ\nn\n/\nλ\n1\n+\n1\n\\begin{aligned}\n  \\text{optimal }\\alpha ~=~{\\mathop{\\text{argmin}}\\limits_\\alpha} ~\\text{rate}(\\alpha) & ~=~\\frac{2}{\\lambda_{1}+\\lambda_{n}}\\\\[1.4em]\n  \\text{optimal rate} ~=~{\\min_\\alpha} ~\\text{rate}(\\alpha) & ~=~\\frac{\\lambda_{n}/\\lambda_{1}-1}{\\lambda_{n}/\\lambda_{1}+1}\n  \\end{aligned}\n​\noptimal\nα\n=\n​\nα\n​\nargmin\n​\n​\nrate\n(\nα\n)\n​\noptimal rate\n=\n​\nα\n​\nmin\n​\n​\nrate\n(\nα\n)\n​\n​\n​\n=\n​\nλ\n​\n1\n​\n​\n+\nλ\n​\nn\n​\n​\n​\n​\n2\n​\n​\n​\n=\n​\nλ\n​\nn\n​\n​\n/\nλ\n​\n1\n​\n​\n+\n1\n​\n​\nλ\n​\nn\n​\n​\n/\nλ\n​\n1\n​\n​\n−\n1\n​\n​\n​\n​\nNotice the ratio\nλ\nn\n/\nλ\n1\n\\lambda_n/\\lambda_1\nλ\n​\nn\n​\n​\n/\nλ\n​\n1\n​\n​\ndetermines the convergence rate of the problem. In fact, this ratio appears often enough that we give it a name, and a symbol — the condition number.\nc\no\nn\nd\ni\nt\ni\no\nn\nn\nu\nm\nb\ne\nr\n:\n=\nκ\n:\n=\nλ\nn\nλ\n1\n\\text{condition number} := \\kappa :=\\frac{\\lambda_n}{\\lambda_1}\ncondition number\n:\n=\nκ\n:\n=\n​\nλ\n​\n1\n​\n​\n​\n​\nλ\n​\nn\n​\n​\n​\n​\nThe condition number means many things. It is a measure of how close to singular a matrix is. It is a measure of how robust\nA\n−\n1\nb\nA^{-1}b\nA\n​\n−\n1\n​\n​\nb\nis to perturbations in\nb\nb\nb\n. And, in this context, the condition number gives us a measure of how poorly gradient descent will perform. A ratio of\nκ\n=\n1\n\\kappa = 1\nκ\n=\n1\nis ideal, giving convergence in one step (of course, the function is trivial). Unfortunately the larger the ratio, the slower gradient descent will be. The condition number is therefore a direct measure of pathological curvature.\nExample: Polynomial Regression\nThe above analysis reveals an insight: all errors are not made equal. Indeed, there are different kinds of errors,\nn\nn\nn\nto be exact, one for each of the eigenvectors of\nA\nA\nA\n. And gradient descent is better at correcting some kinds of errors than others. But what do the eigenvectors of\nA\nA\nA\nmean? Surprisingly, in many applications they admit a very concrete interpretation.\nLets see how this plays out in polynomial regression. Given 1D data,\nξ\ni\n\\xi_i\nξ\n​\ni\n​\n​\n, our problem is to fit the model\nm\no\nd\ne\nl\n(\nξ\n)\n=\nw\n1\np\n1\n(\nξ\n)\n+\n⋯\n+\nw\nn\np\nn\n(\nξ\n)\np\ni\n=\nξ\n↦\nξ\ni\n−\n1\n\\text{model}(\\xi)=w_{1}p_{1}(\\xi)+\\cdots+w_{n}p_{n}(\\xi)\\qquad p_{i}=\\xi\\mapsto\\xi^{i-1}\nmodel\n(\nξ\n)\n=\nw\n​\n1\n​\n​\np\n​\n1\n​\n​\n(\nξ\n)\n+\n⋯\n+\nw\n​\nn\n​\n​\np\n​\nn\n​\n​\n(\nξ\n)\np\n​\ni\n​\n​\n=\nξ\n↦\nξ\n​\ni\n−\n1\n​\n​\nto our observations,\nd\ni\nd_i\nd\n​\ni\n​\n​\n. This model, though nonlinear in the input\nξ\n\\xi\nξ\n, is linear in the weights, and therefore we can write the model as a linear combination of monomials, like:\nBecause of the linearity, we can fit this model to our data\nξ\ni\n\\xi_i\nξ\n​\ni\n​\n​\nusing linear regression on the model mismatch\nm\ni\nn\ni\nm\ni\nz\ne\nw\n1\n2\n∑\ni\n(\nm\no\nd\ne\nl\n(\nξ\ni\n)\n−\nd\ni\n)\n2\n=\n1\n2\n∥\nZ\nw\n−\nd\n∥\n2\n\\text{minimize}_w \\qquad\\tfrac{1}{2}\\sum_i (\\text{model}(\\xi_{i})-d_{i})^{2} ~~=~~ \\tfrac{1}{2}\\|Zw - d\\|^2\nminimize\n​\nw\n​\n​\n​\n2\n​\n​\n1\n​\n​\n​\ni\n​\n∑\n​\n​\n(\nmodel\n(\nξ\n​\ni\n​\n​\n)\n−\nd\n​\ni\n​\n​\n)\n​\n2\n​\n​\n=\n​\n2\n​\n​\n1\n​\n​\n∥\nZ\nw\n−\nd\n∥\n​\n2\n​\n​\nwhere\nZ\n=\n(\n1\nξ\n1\nξ\n1\n2\n…\nξ\n1\nn\n−\n1\n1\nξ\n2\nξ\n2\n2\n…\nξ\n2\nn\n−\n1\n⋮\n⋮\n⋮\n⋱\n⋮\n1\nξ\nm\nξ\nm\n2\n…\nξ\nm\nn\n−\n1\n)\n.\nZ=\\left(\\begin{array}{ccccc}\n  1 & \\xi_{1} & \\xi_{1}^{2} & \\ldots & \\xi_{1}^{n-1}\\\\\n  1 & \\xi_{2} & \\xi_{2}^{2} & \\ldots & \\xi_{2}^{n-1}\\\\\n  \\vdots & \\vdots & \\vdots & \\ddots & \\vdots\\\\\n  1 & \\xi_{m} & \\xi_{m}^{2} & \\ldots & \\xi_{m}^{n-1}\n  \\end{array}\\right).\nZ\n=\n​\n⎝\n​\n⎜\n​\n⎜\n​\n⎛\n​\n​\n​\n1\n​\n1\n​\n⋮\n​\n1\n​\n​\n​\nξ\n​\n1\n​\n​\n​\nξ\n​\n2\n​\n​\n​\n⋮\n​\nξ\n​\nm\n​\n​\n​\n​\n​\nξ\n​\n1\n​\n2\n​\n​\n​\nξ\n​\n2\n​\n2\n​\n​\n​\n⋮\n​\nξ\n​\nm\n​\n2\n​\n​\n​\n​\n​\n…\n​\n…\n​\n⋱\n​\n…\n​\n​\n​\nξ\n​\n1\n​\nn\n−\n1\n​\n​\n​\nξ\n​\n2\n​\nn\n−\n1\n​\n​\n​\n⋮\n​\nξ\n​\nm\n​\nn\n−\n1\n​\n​\n​\n​\n​\n⎠\n​\n⎟\n​\n⎟\n​\n⎞\n​\n​\n.\nThe path of convergence, as we know, is elucidated when we view the iterates in the space of\nQ\nQ\nQ\n(the eigenvectors of\nZ\nT\nZ\nZ^T Z\nZ\n​\nT\n​\n​\nZ\n). So let’s recast our regression problem in the basis of\nQ\nQ\nQ\n. First, we do a change of basis, by rotating\nw\nw\nw\ninto\nQ\nw\nQw\nQ\nw\n, and counter-rotating our feature maps\np\np\np\ninto eigenspace,\np\n¯\n\\bar{p}\n​\np\n​\n¯\n​\n​\n. We can now conceptualize the same regression as one over a different polynomial basis, with the model\nm\no\nd\ne\nl\n(\nξ\n)\n=\nx\n1\np\n¯\n1\n(\nξ\n)\n+\n⋯\n+\nx\nn\np\n¯\nn\n(\nξ\n)\np\n¯\ni\n=\n∑\nq\ni\nj\np\nj\n.\n\\text{model}(\\xi)~=~x_{1}\\bar{p}_{1}(\\xi)~+~\\cdots~+~x_{n}\\bar{p}_{n}(\\xi)\\qquad \\bar{p}_{i}=\\sum q_{ij}p_j.\nmodel\n(\nξ\n)\n=\nx\n​\n1\n​\n​\n​\np\n​\n¯\n​\n​\n​\n1\n​\n​\n(\nξ\n)\n+\n⋯\n+\nx\n​\nn\n​\n​\n​\np\n​\n¯\n​\n​\n​\nn\n​\n​\n(\nξ\n)\n​\np\n​\n¯\n​\n​\n​\ni\n​\n​\n=\n∑\nq\n​\ni\nj\n​\n​\np\n​\nj\n​\n​\n.\nThis model is identical to the old one. But these new features\np\n¯\n\\bar{p}\n​\np\n​\n¯\n​\n​\n(which I call “eigenfeatures”) and weights have the pleasing property that each coordinate acts independently of the others. Now our optimization problem breaks down, really, into\nn\nn\nn\nsmall 1D optimization problems. And each coordinate can be optimized greedily and independently, one at a time in any order, to produce the final, global, optimum. The eigenfeatures are also much more informative:\nThe observations in the above diagram can be justified mathematically. From a statistical point of view, we would like a model which is, in some sense, robust to noise. Our model cannot possibly be meaningful if the slightest perturbation to the observations changes the entire model dramatically. And the eigenfeatures, the principal components of the data, give us exactly the decomposition we need to sort the features by its sensitivity to perturbations in\nd\ni\nd_i\nd\n​\ni\n​\n​\n’s. The most robust components appear in the front (with the largest eigenvalues), and the most sensitive components in the back (with the smallest eigenvalues).\nThis measure of robustness, by a rather convenient coincidence, is also a measure of how easily an eigenspace converges. And thus, the “pathological directions” — the eigenspaces which converge the slowest — are also those which are most sensitive to noise! So starting at a simple initial point like\n0\n0\n0\n(by a gross abuse of language, let’s think of this as a prior), we track the iterates till a desired level of complexity is reached. Let’s see how this plays out in gradient descent.\nThis effect is harnessed with the heuristic of early stopping : by stopping the optimization early, you can often get better generalizing results. Indeed, the effect of early stopping is very similar to that of more conventional methods of regularization, such as Tikhonov Regression. Both methods try to suppress the components of the smallest eigenvalues directly, though they employ different methods of spectral decay.\n2\nBut early stopping has a distinct advantage. Once the step-size is chosen, there are no regularization parameters to fiddle with. Indeed, in the course of a single optimization, we have the entire family of models, from underfitted to overfitted, at our disposal. This gift, it seems, doesn’t come at a price. A beautiful free lunch\n[7]\nindeed.\nThe Dynamics of Momentum\nLet’s turn our attention back to momentum. Recall that the momentum update is\nz\nk\n+\n1\n=\nβ\nz\nk\n+\n∇\nf\n(\nw\nk\n)\nw\nk\n+\n1\n=\nw\nk\n−\nα\nz\nk\n+\n1\n.\n\\begin{aligned}\n  z^{k+1}&=\\beta z^{k}+\\nabla f(w^{k})\\\\[0.4em]\n  w^{k+1}&=w^{k}-\\alpha z^{k+1}.\n  \\end{aligned}\n​\nz\n​\nk\n+\n1\n​\n​\n​\nw\n​\nk\n+\n1\n​\n​\n​\n​\n​\n=\nβ\nz\n​\nk\n​\n​\n+\n∇\nf\n(\nw\n​\nk\n​\n​\n)\n​\n=\nw\n​\nk\n​\n​\n−\nα\nz\n​\nk\n+\n1\n​\n​\n.\n​\n​\nSince\n∇\nf\n(\nw\nk\n)\n=\nA\nw\nk\n−\nb\n\\nabla f(w^k) = Aw^k - b\n∇\nf\n(\nw\n​\nk\n​\n​\n)\n=\nA\nw\n​\nk\n​\n​\n−\nb\n, the update on the quadratic is\nz\nk\n+\n1\n=\nβ\nz\nk\n+\n(\nA\nw\nk\n−\nb\n)\nw\nk\n+\n1\n=\nw\nk\n−\nα\nz\nk\n+\n1\n.\n\\begin{aligned}\n  z^{k+1}&=\\beta z^{k}+ (Aw^{k}-b)\\\\[0.4em]\n  w^{k+1}&=w^{k}-\\alpha z^{k+1}.\n  \\end{aligned}\n​\nz\n​\nk\n+\n1\n​\n​\n​\nw\n​\nk\n+\n1\n​\n​\n​\n​\n​\n=\nβ\nz\n​\nk\n​\n​\n+\n(\nA\nw\n​\nk\n​\n​\n−\nb\n)\n​\n=\nw\n​\nk\n​\n​\n−\nα\nz\n​\nk\n+\n1\n​\n​\n.\n​\n​\nFollowing\n[8]\n, we go through the same motions, with the change of basis\nx\nk\n=\nQ\n(\nw\nk\n−\nw\n⋆\n)\nx^{k} = Q(w^{k} - w^\\star)\nx\n​\nk\n​\n​\n=\nQ\n(\nw\n​\nk\n​\n​\n−\nw\n​\n⋆\n​\n​\n)\nand\ny\nk\n=\nQ\nz\nk\ny^{k} = Qz^{k}\ny\n​\nk\n​\n​\n=\nQ\nz\n​\nk\n​\n​\n, to yield the update rule\ny\ni\nk\n+\n1\n=\nβ\ny\ni\nk\n+\nλ\ni\nx\ni\nk\nx\ni\nk\n+\n1\n=\nx\ni\nk\n−\nα\ny\ni\nk\n+\n1\n.\n\\begin{aligned}\n  y_{i}^{k+1}&=\\beta y_{i}^{k}+\\lambda_{i}x_{i}^{k}\\\\[0.4em]\n  x_{i}^{k+1}&=x_{i}^{k}-\\alpha y_{i}^{k+1}.\n  \\end{aligned}\n​\ny\n​\ni\n​\nk\n+\n1\n​\n​\n​\nx\n​\ni\n​\nk\n+\n1\n​\n​\n​\n​\n​\n=\nβ\ny\n​\ni\n​\nk\n​\n​\n+\nλ\n​\ni\n​\n​\nx\n​\ni\n​\nk\n​\n​\n​\n=\nx\n​\ni\n​\nk\n​\n​\n−\nα\ny\n​\ni\n​\nk\n+\n1\n​\n​\n.\n​\n​\nin which each component acts independently of the other components (though\nx\ni\nk\nx^k_i\nx\n​\ni\n​\nk\n​\n​\nand\ny\ni\nk\ny^k_i\ny\n​\ni\n​\nk\n​\n​\nare coupled). This lets us rewrite our iterates as\n3\n(\ny\ni\nk\nx\ni\nk\n)\n=\nR\nk\n(\ny\ni\n0\nx\ni\n0\n)\nR\n=\n(\nβ\nλ\ni\n−\nα\nβ\n1\n−\nα\nλ\ni\n)\n.\n\\left(\\!\\!\\begin{array}{c}\n  y_{i}^{k}\\\\\n  x_{i}^{k}\n  \\end{array}\\!\\!\\right)=R^k\\left(\\!\\!\\begin{array}{c}\n  y_{i}^{0}\\\\\n  x_{i}^{0}\n  \\end{array}\\!\\!\\right)\n  \\qquad\n  R = \\left(\\!\\!\\begin{array}{cc}\n  \\beta & \\lambda_{i}\\\\\n  -\\alpha\\beta & 1-\\alpha\\lambda_{i}\n  \\end{array}\\!\\!\\right).\n(\n​\ny\n​\ni\n​\nk\n​\n​\n​\nx\n​\ni\n​\nk\n​\n​\n​\n​\n)\n=\nR\n​\nk\n​\n​\n(\n​\ny\n​\ni\n​\n0\n​\n​\n​\nx\n​\ni\n​\n0\n​\n​\n​\n​\n)\nR\n=\n(\n​\nβ\n​\n−\nα\nβ\n​\n​\n​\nλ\n​\ni\n​\n​\n​\n1\n−\nα\nλ\n​\ni\n​\n​\n​\n​\n)\n.\nThere are many ways of taking a matrix to the\nk\nt\nh\nk^{th}\nk\n​\nt\nh\n​\n​\npower. But for the\n2\n×\n2\n2 \\times 2\n2\n×\n2\ncase there is an elegant and little known formula\n[9]\nin terms of the eigenvalues of\nR\nR\nR\n,\nσ\n1\n\\sigma_1\nσ\n​\n1\n​\n​\nand\nσ\n2\n\\sigma_2\nσ\n​\n2\n​\n​\n.\nR\nk\n=\n{\nσ\n1\nk\nR\n1\n−\nσ\n2\nk\nR\n2\nσ\n1\n≠\nσ\n2\nσ\n1\nk\n(\nk\nR\n/\nσ\n1\n−\n(\nk\n−\n1\n)\nI\n)\nσ\n1\n=\nσ\n2\n,\nR\nj\n=\nR\n−\nσ\nj\nI\nσ\n1\n−\nσ\n2\n\\color{#AAA}{\\color{black}{R^{k}}=\\begin{cases}\n\\color{black}{\\sigma_{1}^{k}}R_{1}-\\color{black}{\\sigma_{2}^{k}}R_{2} & \\sigma_{1}\\neq\\sigma_{2}\\\\\n\\sigma_{1}^{k}(kR/\\sigma_1-(k-1)I) & \\sigma_{1}=\\sigma_{2}\n\\end{cases},\\qquad R_{j}=\\frac{R-\\sigma_{j}I}{\\sigma_{1}-\\sigma_{2}}}\nR\n​\nk\n​\n​\n=\n{\n​\nσ\n​\n1\n​\nk\n​\n​\nR\n​\n1\n​\n​\n−\nσ\n​\n2\n​\nk\n​\n​\nR\n​\n2\n​\n​\n​\nσ\n​\n1\n​\nk\n​\n​\n(\nk\nR\n/\nσ\n​\n1\n​\n​\n−\n(\nk\n−\n1\n)\nI\n)\n​\n​\n​\nσ\n​\n1\n​\n​\n≠\nσ\n​\n2\n​\n​\n​\nσ\n​\n1\n​\n​\n=\nσ\n​\n2\n​\n​\n​\n​\n,\nR\n​\nj\n​\n​\n=\n​\nσ\n​\n1\n​\n​\n−\nσ\n​\n2\n​\n​\n​\n​\nR\n−\nσ\n​\nj\n​\n​\nI\n​\n​\nThis formula is rather complicated, but the takeaway here is that it plays the exact same role the individual convergence rates,\n1\n−\nα\nλ\ni\n1-\\alpha\\lambda_i\n1\n−\nα\nλ\n​\ni\n​\n​\ndo in gradient descent. But instead of one geometric series, we have two coupled series, which may have real or complex values. The convergence rate is therefore the slowest of the two rates,\nmax\n{\n∣\nσ\n1\n∣\n,\n∣\nσ\n2\n∣\n}\n\\max\\{|\\sigma_{1}|,|\\sigma_{2}|\\}\nmax\n{\n∣\nσ\n​\n1\n​\n​\n∣\n,\n∣\nσ\n​\n2\n​\n​\n∣\n}\n4\n. By plotting this out, we see there are distinct regions of the parameter space which reveal a rich taxonomy of convergence behavior\n[10]\n:\nConvergence Rate\nA plot of\nmax\n{\n∣\nσ\n1\n∣\n,\n∣\nσ\n2\n∣\n}\n\\max\\{|\\sigma_1|, |\\sigma_2|\\}\nmax\n{\n∣\nσ\n​\n1\n​\n​\n∣\n,\n∣\nσ\n​\n2\n​\n​\n∣\n}\nreveals distinct regions, each with its own style of convergence.\nFor what values of\nα\n\\alpha\nα\nand\nβ\n\\beta\nβ\ndoes momentum converge? Since we need both\nσ\n1\n\\sigma_1\nσ\n​\n1\n​\n​\nand\nσ\n2\n\\sigma_2\nσ\n​\n2\n​\n​\nto converge, our convergence criterion is now\nmax\n{\n∣\nσ\n1\n∣\n,\n∣\nσ\n2\n∣\n}\n<\n1\n\\max\\{|\\sigma_{1}|,|\\sigma_{2}|\\} < 1\nmax\n{\n∣\nσ\n​\n1\n​\n​\n∣\n,\n∣\nσ\n​\n2\n​\n​\n∣\n}\n<\n1\n. The range of available step-sizes work out\n5\nto be\n0\n<\nα\nλ\ni\n<\n2\n+\n2\nβ\nf\no\nr\n0\n≤\nβ\n<\n1\n0<\\alpha\\lambda_{i}<2+2\\beta \\qquad \\text{for} \\qquad 0 \\leq \\beta < 1\n0\n<\nα\nλ\n​\ni\n​\n​\n<\n2\n+\n2\nβ\nfor\n0\n≤\nβ\n<\n1\nWe recover the previous result for gradient descent when\nβ\n=\n0\n\\beta = 0\nβ\n=\n0\n. But notice an immediate boon we get. Momentum allows us to crank up the step-size up by a factor of 2 before diverging.\nThe Critical Damping Coefficient\nThe true magic happens, however, when we find the sweet spot of\nα\n\\alpha\nα\nand\nβ\n\\beta\nβ\n. Let us try to first optimize over\nβ\n\\beta\nβ\n.\nMomentum admits an interesting physical interpretation when\nα\n\\alpha\nα\nis\n[11]\nsmall: it is a discretization of a damped harmonic oscillator. Consider a physical simulation operating in discrete time (like a video game).\ny\ni\nk\n+\n1\ny_{i}^{k+1}\ny\n​\ni\n​\nk\n+\n1\n​\n​\n=\n=\n=\n+\n+\n+\nλ\ni\nx\ni\nk\n\\lambda_{i}x_{i}^{k}\nλ\n​\ni\n​\n​\nx\n​\ni\n​\nk\n​\n​\nand perturbed by an external force field\nWe can think of\n−\ny\ni\nk\n-y_i^k\n−\ny\n​\ni\n​\nk\n​\n​\nas\nvelocity\nβ\ny\ni\nk\n\\beta y_{i}^{k}\nβ\ny\n​\ni\n​\nk\n​\n​\nwhich is dampened at each step\nx\ni\nk\n+\n1\nx_i^{k+1}\nx\n​\ni\n​\nk\n+\n1\n​\n​\n=\n=\n=\nx\ni\nk\n−\nα\ny\ni\nk\n+\n1\nx_i^k - \\alpha y_i^{k+1}\nx\n​\ni\n​\nk\n​\n​\n−\nα\ny\n​\ni\n​\nk\n+\n1\n​\n​\nAnd\nx\nx\nx\nis our particle’s\nposition\nwhich is moved at each step by a small amount in the direction of the velocity\ny\ni\nk\n+\n1\ny^{k+1}_i\ny\n​\ni\n​\nk\n+\n1\n​\n​\n.\nWe can break this equation apart to see how each component affects the dynamics of the system. Here we plot, for\n1\n5\n0\n150\n1\n5\n0\niterates, the particle’s velocity (the horizontal axis) against its position (the vertical axis), in a phase diagram.\nThis system is best imagined as a weight suspended on a spring. We pull the weight down by one unit, and we study the path it follows as it returns to equilibrium. In the analogy, the spring is the source of our external force\nλ\ni\nx\ni\nk\n\\lambda_ix^k_i\nλ\n​\ni\n​\n​\nx\n​\ni\n​\nk\n​\n​\n, and equilibrium is the state when both the position\nx\ni\nk\nx^k_i\nx\n​\ni\n​\nk\n​\n​\nand the speed\ny\ni\nk\ny^k_i\ny\n​\ni\n​\nk\n​\n​\nare 0. The choice of\nβ\n\\beta\nβ\ncrucially affects the rate of return to equilibrium.\nThe critical value of\nβ\n=\n(\n1\n−\nα\nλ\ni\n)\n2\n\\beta = (1 - \\sqrt{\\alpha \\lambda_i})^2\nβ\n=\n(\n1\n−\n√\n​\nα\nλ\n​\ni\n​\n​\n​\n​\n​\n)\n​\n2\n​\n​\ngives us a convergence rate (in eigenspace\ni\ni\ni\n) of\n1\n−\nα\nλ\ni\n.\n1 - \\sqrt{\\alpha\\lambda_i}.\n1\n−\n√\n​\nα\nλ\n​\ni\n​\n​\n​\n​\n​\n.\nA square root improvement over gradient descent,\n1\n−\nα\nλ\ni\n1-\\alpha\\lambda_i\n1\n−\nα\nλ\n​\ni\n​\n​\n! Alas, this only applies to the error in the\ni\nt\nh\ni^{th}\ni\n​\nt\nh\n​\n​\neigenspace, with\nα\n\\alpha\nα\nfixed.\nOptimal parameters\nTo get a global convergence rate, we must optimize over both\nα\n\\alpha\nα\nand\nβ\n\\beta\nβ\n. This is a more complicated affair,\n6\nbut they work out to be\nα\n=\n(\n2\nλ\n1\n+\nλ\nn\n)\n2\nβ\n=\n(\nλ\nn\n−\nλ\n1\nλ\nn\n+\nλ\n1\n)\n2\n\\alpha = \\left(\\frac{2}{\\sqrt{\\lambda_{1}}+\\sqrt{\\lambda_{n}}}\\right)^{2}  \\quad \\beta = \\left(\\frac{\\sqrt{\\lambda_{n}}-\\sqrt{\\lambda_{1}}}{\\sqrt{\\lambda_{n}}+\\sqrt{\\lambda_{1}}}\\right)^{2}\nα\n=\n(\n​\n√\n​\nλ\n​\n1\n​\n​\n​\n​\n​\n+\n√\n​\nλ\n​\nn\n​\n​\n​\n​\n​\n​\n​\n2\n​\n​\n)\n​\n2\n​\n​\nβ\n=\n(\n​\n√\n​\nλ\n​\nn\n​\n​\n​\n​\n​\n+\n√\n​\nλ\n​\n1\n​\n​\n​\n​\n​\n​\n​\n√\n​\nλ\n​\nn\n​\n​\n​\n​\n​\n−\n√\n​\nλ\n​\n1\n​\n​\n​\n​\n​\n​\n​\n)\n​\n2\n​\n​\nPlug this into the convergence rate, and you get\nκ\n−\n1\nκ\n+\n1\n\\frac{\\sqrt{\\kappa}-1}{\\sqrt{\\kappa}+1}\n​\n√\n​\nκ\n​\n​\n​\n+\n1\n​\n​\n√\n​\nκ\n​\n​\n​\n−\n1\n​\n​\nConvergence rate,\nMomentum\nκ\n−\n1\nκ\n+\n1\n\\frac{\\kappa-1}{\\kappa+1}\n​\nκ\n+\n1\n​\n​\nκ\n−\n1\n​\n​\nConvergence rate,\nGradient Descent\nWith barely a modicum of extra effort, we have essentially square rooted the condition number! These gains, in principle, require explicit knowledge of\nλ\n1\n\\lambda_1\nλ\n​\n1\n​\n​\nand\nλ\nn\n\\lambda_n\nλ\n​\nn\n​\n​\n. But the formulas reveal a simple guideline. When the problem’s conditioning is poor, the optimal\nα\n\\alpha\nα\nis approximately twice that of gradient descent, and the momentum term is close to\n1\n1\n1\n. So set\nβ\n\\beta\nβ\nas close to\n1\n1\n1\nas you can, and then find the highest\nα\n\\alpha\nα\nwhich still converges. Being at the knife’s edge of divergence, like in gradient descent, is a good place to be.\nWe can do the same decomposition here with momentum, with eigenvalues\nλ\n1\n=\n0\n.\n0\n1\n\\lambda_1=0.01\nλ\n​\n1\n​\n​\n=\n0\n.\n0\n1\n,\nλ\n2\n=\n0\n.\n1\n\\lambda_2=0.1\nλ\n​\n2\n​\n​\n=\n0\n.\n1\n, and\nλ\n3\n=\n1\n\\lambda_3=1\nλ\n​\n3\n​\n​\n=\n1\n. Though the decrease is no longer monotonic, but significantly faster.\nf\n(\nw\nk\n)\n−\nf\n(\nw\n⋆\n)\nf(w^k) - f(w^\\star)\nf\n(\nw\n​\nk\n​\n​\n)\n−\nf\n(\nw\n​\n⋆\n​\n​\n)\nNote that the optimal parameters do not necessarily imply the fastest convergence, though, only the fastest asymptotic convergence rate.\nStep-size α =\nMomentum β =\nWhile the loss function of gradient descent had a graceful, monotonic curve, optimization with momentum displays clear oscillations. These ripples are not restricted to quadratics, and occur in all kinds of functions in practice. They are not cause for alarm, but are an indication that extra tuning of the hyperparameters is required.\nExample: The Colorization Problem\nLet’s look at how momentum accelerates convergence with a concrete example. On a grid of pixels let\nG\nG\nG\nbe the graph with vertices as pixels,\nE\nE\nE\nbe the set of edges connecting each pixel to its four neighboring pixels, and\nD\nD\nD\nbe a small set of a few distinguished vertices. Consider the problem of minimizing\nm\ni\nn\ni\nm\ni\nz\ne\n\\text{minimize}\nminimize\n1\n2\n∑\ni\n∈\nD\n(\nw\ni\n−\n1\n)\n2\n\\qquad  \\frac{1}{2} \\sum_{i\\in D} (w_i - 1)^2\n​\n2\n​\n​\n1\n​\n​\n​\ni\n∈\nD\n​\n∑\n​\n​\n(\nw\n​\ni\n​\n​\n−\n1\n)\n​\n2\n​\n​\nThe\ncolorizer\npulls distinguished pixels towards 1\n+\n+\n+\n1\n2\n∑\ni\n,\nj\n∈\nE\n(\nw\ni\n−\nw\nj\n)\n2\n.\n\\frac{1}{2} \\sum_{i,j\\in E} (w_i - w_j)^2.\n​\n2\n​\n​\n1\n​\n​\n​\ni\n,\nj\n∈\nE\n​\n∑\n​\n​\n(\nw\n​\ni\n​\n​\n−\nw\n​\nj\n​\n​\n)\n​\n2\n​\n​\n.\nThe\nsmoother\nspreads out the color\nThe optimal solution to this problem is a vector of all\n1\n1\n1\n’s\n7\n. An inspection of the gradient iteration reveals why we take a long time to get there. The gradient step, for each component, is some form of weighted average of the current value and its neighbors:\nw\ni\nk\n+\n1\n=\nw\ni\nk\n−\nα\n∑\nj\n∈\nN\n(\nw\ni\nk\n−\nw\nj\nk\n)\n−\n{\nα\n(\nw\ni\nk\n−\n1\n)\ni\n∈\nD\n0\ni\n∉\nD\nw_{i}^{k+1}=w_{i}^{k}-\\alpha\\sum_{j\\in N}(w_{i}^{k}-w_{j}^{k})-\\begin{cases}\n\\alpha(w_{i}^{k}-1) & i\\in D\\\\\n0 & i\\notin D\n\\end{cases}\nw\n​\ni\n​\nk\n+\n1\n​\n​\n=\nw\n​\ni\n​\nk\n​\n​\n−\nα\n​\nj\n∈\nN\n​\n∑\n​\n​\n(\nw\n​\ni\n​\nk\n​\n​\n−\nw\n​\nj\n​\nk\n​\n​\n)\n−\n{\n​\nα\n(\nw\n​\ni\n​\nk\n​\n​\n−\n1\n)\n​\n0\n​\n​\n​\ni\n∈\nD\n​\ni\n∉\nD\n​\n​\nThis kind of local averaging is effective at smoothing out local variations in the pixels, but poor at taking advantage of global structure. The updates are akin to a drop of ink, diffusing through water. Movement towards equilibrium is made only through local corrections and so, left undisturbed, its march towards the solution is slow and laborious. Fortunately, momentum speeds things up significantly.\nThe eigenvectors of the colorization problem form a generalized Fourier basis for\nR\nn\nR^n\nR\n​\nn\n​\n​\n. The smallest eigenvalues have low frequencies, hence gradient descent corrects high frequency errors well but not low frequency ones.\nIn vectorized form, the colorization problem is\nm\ni\nn\ni\nm\ni\nz\ne\n\\text{minimize}\nminimize\nThe\nsmoother\n’s quadratic form is the\nGraph Laplacian\n1\n2\n∑\ni\n∈\nD\n(\nx\nT\ne\ni\ne\ni\nT\nx\n−\ne\ni\nT\nx\n)\n\\frac{1}{2}\\sum_{i\\in D}\\left(x^{T}e_{i}e_{i}^{T}x-e_{i}^{T}x\\right)\n​\n2\n​\n​\n1\n​\n​\n​\ni\n∈\nD\n​\n∑\n​\n​\n(\nx\n​\nT\n​\n​\ne\n​\ni\n​\n​\ne\n​\ni\n​\nT\n​\n​\nx\n−\ne\n​\ni\n​\nT\n​\n​\nx\n)\n+\n+\n+\n1\n2\nx\nT\nL\nG\nx\n\\frac{1}{2}x^{T}L_{G}x\n​\n2\n​\n​\n1\n​\n​\nx\n​\nT\n​\n​\nL\n​\nG\n​\n​\nx\nAnd the colorizer is a small low rank correction with a linear term.\ne\ni\ne_i\ne\n​\ni\n​\n​\nis the\ni\nt\nh\ni^{th}\ni\n​\nt\nh\n​\n​\nunit vector.\nThe Laplacian matrix,\nL\nG\nL_G\nL\n​\nG\n​\n​\n8\n, which dominates the behavior of the optimization problem, is a valuable bridge between linear algebra and graph theory. This is a rich field of study, but one fact is pertinent to our discussion here. The conditioning of\nL\nG\nL_G\nL\n​\nG\n​\n​\n, here defined as the ratio of the second eigenvector to the last (the first eigenvalue is always 0, with eigenvector equal to the matrix of all 1′s), is directly connected to the connectivity of the graph.\nSmall world graphs, like expanders and dense graphs, have excellent conditioning\nThe conditioning of grids improves with its dimensionality.\nAnd long, wiry graphs, like paths, condition poorly.\nThese observations carry through to the colorization problem, and the intuition behind it should be clear. Well connected graphs allow rapid diffusion of information through the edges, while graphs with poor connectivity do not. And this principle, taken to the extreme, furnishes a class of functions so hard to optimize they reveal the limits of first order optimization.\nThe Limits of Descent\nLet’s take a step back. We have, with a clever trick, improved the convergence of gradient descent by a quadratic factor with the introduction of a single auxiliary sequence. But is this the best we can do? Could we improve convergence even more with two sequences? Could one perhaps choose the\nα\n\\alpha\nα\n’s and\nβ\n\\beta\nβ\n’s intelligently and adaptively? It is tempting to ride this wave of optimism - to the cube root and beyond!\nUnfortunately, while improvements to the momentum algorithm do exist, they all run into a certain, critical, almost inescapable lower bound.\nAdventures in Algorithmic Space\nTo understand the limits of what we can do, we must first formally define the algorithmic space in which we are searching. Here’s one possible definition. The observation we will make is that both gradient descent and momentum can be “unrolled”. Indeed, since\nw\n1\n=\nw\n0\n−\nα\n∇\nf\n(\nw\n0\n)\nw\n2\n=\nw\n1\n−\nα\n∇\nf\n(\nw\n1\n)\n=\nw\n0\n−\nα\n∇\nf\n(\nw\n0\n)\n−\nα\n∇\nf\n(\nw\n1\n)\n⋮\nw\nk\n+\n1\n=\nw\n0\n−\nα\n∇\nf\n(\nw\n0\n)\n−\n⋯\n⋯\n−\nα\n∇\nf\n(\nw\nk\n)\n\\begin{array}{lll}\n    w^{1} & \\!= & \\!w^{0} ~-~ \\alpha\\nabla f(w^{0})\\\\[0.35em]\n    w^{2} & \\!= & \\!w^{1} ~-~ \\alpha\\nabla f(w^{1})\\\\[0.35em]\n          & \\!= & \\!w^{0} ~-~ \\alpha\\nabla f(w^{0}) ~-~ \\alpha\\nabla f(w^{1})\\\\[0.35em]\n     & ~ \\!\\vdots \\\\\n\n   w^{k+1} & \\!= & \\!w^{0} ~-~ \\alpha\\nabla f(w^{0}) ~-~~~~ \\cdots\\cdots ~~~~-~ \\alpha\\nabla f(w^{k})\n  \\end{array}\n​\nw\n​\n1\n​\n​\n​\nw\n​\n2\n​\n​\n​\n​\n​\nw\n​\nk\n+\n1\n​\n​\n​\n​\n​\n=\n​\n=\n​\n=\n​\n⋮\n​\n=\n​\n​\n​\nw\n​\n0\n​\n​\n−\nα\n∇\nf\n(\nw\n​\n0\n​\n​\n)\n​\nw\n​\n1\n​\n​\n−\nα\n∇\nf\n(\nw\n​\n1\n​\n​\n)\n​\nw\n​\n0\n​\n​\n−\nα\n∇\nf\n(\nw\n​\n0\n​\n​\n)\n−\nα\n∇\nf\n(\nw\n​\n1\n​\n​\n)\n​\nw\n​\n0\n​\n​\n−\nα\n∇\nf\n(\nw\n​\n0\n​\n​\n)\n−\n⋯\n⋯\n−\nα\n∇\nf\n(\nw\n​\nk\n​\n​\n)\n​\n​\nwe can write gradient descent as\nw\nk\n+\n1\n=\nw\n0\n−\nα\n∑\ni\nk\n∇\nf\n(\nw\ni\n)\n.\nw^{k+1} ~~=~~ w^{0} ~-~ \\alpha\\sum_i^k\\nabla f(w^{i}).\nw\n​\nk\n+\n1\n​\n​\n=\nw\n​\n0\n​\n​\n−\nα\n​\ni\n​\n∑\n​\nk\n​\n​\n∇\nf\n(\nw\n​\ni\n​\n​\n)\n.\nA similar trick can be done with momentum:\nw\nk\n+\n1\n=\nw\n0\n+\nα\n∑\ni\nk\n(\n1\n−\nβ\nk\n+\n1\n−\ni\n)\n1\n−\nβ\n∇\nf\n(\nw\ni\n)\n.\nw^{k+1} ~~=~~ w^{0} ~+~ \\alpha\\sum_i^k\\frac{(1-\\beta^{k+1-i})}{1-\\beta}\\nabla f(w^i).\nw\n​\nk\n+\n1\n​\n​\n=\nw\n​\n0\n​\n​\n+\nα\n​\ni\n​\n∑\n​\nk\n​\n​\n​\n1\n−\nβ\n​\n​\n(\n1\n−\nβ\n​\nk\n+\n1\n−\ni\n​\n​\n)\n​\n​\n∇\nf\n(\nw\n​\ni\n​\n​\n)\n.\nIn fact, all manner of first order algorithms, including the Conjugate Gradient algorithm, AdaMax, Averaged Gradient and more, can be written (though not quite so neatly) in this unrolled form. Therefore the class of algorithms for which\nw\nk\n+\n1\n=\nw\n0\n+\n∑\ni\nk\nγ\ni\nk\n∇\nf\n(\nw\ni\n)\nf\no\nr\ns\no\nm\ne\nγ\ni\nk\nw^{k+1} ~~=~~ w^{0} ~+~ \\sum_{i}^{k}\\gamma_{i}^{k}\\nabla f(w^{i}) \\qquad \\text{ for some } \\gamma_{i}^{k}\nw\n​\nk\n+\n1\n​\n​\n=\nw\n​\n0\n​\n​\n+\n​\ni\n​\n∑\n​\nk\n​\n​\nγ\n​\ni\n​\nk\n​\n​\n∇\nf\n(\nw\n​\ni\n​\n​\n)\nfor some\nγ\n​\ni\n​\nk\n​\n​\ncontains momentum, gradient descent and a whole bunch of other algorithms you might dream up. This is what is assumed in Assumption 2.1.4\n[5]\nof Nesterov. But let’s push this even further, and expand this class to allow different step-sizes for different directions.\nw\nk\n+\n1\n=\nw\n0\n+\n∑\ni\nk\nΓ\ni\nk\n∇\nf\n(\nw\ni\n)\nf\no\nr\ns\no\nm\ne\nd\ni\na\ng\no\nn\na\nl\nm\na\nt\nr\ni\nx\nΓ\ni\nk\n.\nw^{k+1} ~~=~~ w^{0} ~+~ \\sum_{i}^{k}\\Gamma_{i}^{k}\\nabla f(w^{i}) \\quad \\text{ for some diagonal matrix } \\Gamma_{i}^{k} .\nw\n​\nk\n+\n1\n​\n​\n=\nw\n​\n0\n​\n​\n+\n​\ni\n​\n∑\n​\nk\n​\n​\nΓ\n​\ni\n​\nk\n​\n​\n∇\nf\n(\nw\n​\ni\n​\n​\n)\nfor some diagonal matrix\nΓ\n​\ni\n​\nk\n​\n​\n.\nThis class of methods covers most of the popular algorithms for training neural networks, including ADAM and AdaGrad. We shall refer to this class of methods as “Linear First Order Methods”, and we will show a single function all these methods ultimately fail on.\nThe Resisting Oracle\nEarlier, when we talked about the colorizer problem, we observed that wiry graphs cause bad conditioning in our optimization problem. Taking this to its extreme, we can look at a graph consisting of a single path — a function so badly conditioned that Nesterov called a variant of it “the worst function in the world”. The function follows the same structure as the colorizer problem, and we shall call this the Convex Rosenbrock,\nf\nn\n(\nw\n)\nf^n(w)\nf\n​\nn\n​\n​\n(\nw\n)\n=\n=\n=\nwith a colorizer of one node\n1\n2\n(\nw\n1\n−\n1\n)\n2\n\\frac{1}{2}\\left(w_{1}-1\\right)^{2}\n​\n2\n​\n​\n1\n​\n​\n(\nw\n​\n1\n​\n​\n−\n1\n)\n​\n2\n​\n​\n+\n+\n+\n1\n2\n∑\ni\n=\n1\nn\n(\nw\ni\n−\nw\ni\n+\n1\n)\n2\n\\frac{1}{2}\\sum_{i=1}^{n}(w_{i}-w_{i+1})^{2}\n​\n2\n​\n​\n1\n​\n​\n​\ni\n=\n1\n​\n∑\n​\nn\n​\n​\n(\nw\n​\ni\n​\n​\n−\nw\n​\ni\n+\n1\n​\n​\n)\n​\n2\n​\n​\nstrong couplings of adjacent nodes in the path,\n+\n+\n+\n2\nκ\n−\n1\n∥\nw\n∥\n2\n.\n\\frac{2}{\\kappa-1}\\|w\\|^{2}.\n​\nκ\n−\n1\n​\n​\n2\n​\n​\n∥\nw\n∥\n​\n2\n​\n​\n.\nand a small regularization term.\nThe optimal solution of this problem is\nw\ni\n⋆\n=\n(\nκ\n−\n1\nκ\n+\n1\n)\ni\nw_{i}^{\\star}=\\left(\\frac{\\sqrt{\\kappa}-1}{\\sqrt{\\kappa}+1}\\right)^{i}\nw\n​\ni\n​\n⋆\n​\n​\n=\n(\n​\n√\n​\nκ\n​\n​\n​\n+\n1\n​\n​\n√\n​\nκ\n​\n​\n​\n−\n1\n​\n​\n)\n​\ni\n​\n​\nand the condition number of the problem\nf\nn\nf^n\nf\n​\nn\n​\n​\napproaches\nκ\n\\kappa\nκ\nas\nn\nn\nn\ngoes to infinity. Now observe the behavior of the momentum algorithm on this function, starting from\nw\n0\n=\n0\nw^0 = 0\nw\n​\n0\n​\n​\n=\n0\n.\nStep-size α =\nMomentum β =\nHere we see the first 50 iterates of momentum on the Convex Rosenbrock for\nn\n=\n2\n5\nn=25\nn\n=\n2\n5\n. The behavior here is similar to that of any Linear First Order Algorithm.\nThis triangle is a “dead zone” of our iterates. The iterates are always 0, no matter what the parameters.\nThe remaining expanding space is the “light cone” of our iterate’s influence. Momentum does very well here with the optimal parameters.\nError\nWeights\nThe observations made in the above diagram are true for any Linear First Order algorithm. Let us prove this. First observe that each component of the gradient depends only on the values directly before and after it:\n∇\nf\n(\nx\n)\ni\n=\n2\nw\ni\n−\nw\ni\n−\n1\n−\nw\ni\n+\n1\n+\n4\nκ\n−\n1\nw\ni\n,\ni\n≠\n1\n.\n\\nabla f(x)_{i}=2w_{i}-w_{i-1}-w_{i+1} +\\frac{4}{\\kappa-1} w_{i}, \\qquad i \\neq 1.\n∇\nf\n(\nx\n)\n​\ni\n​\n​\n=\n2\nw\n​\ni\n​\n​\n−\nw\n​\ni\n−\n1\n​\n​\n−\nw\n​\ni\n+\n1\n​\n​\n+\n​\nκ\n−\n1\n​\n​\n4\n​\n​\nw\n​\ni\n​\n​\n,\ni\n≠\n1\n.\nTherefore the fact we start at 0 guarantees that that component must remain stoically there till an element either before or after it turns nonzero. And therefore, by induction, for any linear first order algorithm,\nw\n0\n=\n[\n0\n,\n0\n,\n0\n,\n…\n0\n,\n0\n,\n…\n0\n]\nw\n1\n=\n[\nw\n1\n1\n,\n0\n,\n0\n,\n…\n0\n,\n0\n,\n…\n0\n]\nw\n2\n=\n[\nw\n1\n2\n,\nw\n2\n2\n,\n0\n,\n…\n0\n,\n0\n,\n…\n0\n]\n⋮\nw\nk\n=\n[\nw\n1\nk\n,\nw\n2\nk\n,\nw\n3\nk\n,\n…\nw\nk\nk\n,\n0\n,\n…\n0\n]\n.\n\\begin{array}{lllllllll}\n    w^{0} & = & [~~0, & 0, & 0, & \\ldots & 0, & 0, & \\ldots & 0~]\\\\[0.35em]\n    w^{1} & = & [~w_{1}^{1}, & 0, & 0, & \\ldots & 0, & 0, & \\ldots & 0~]\\\\[0.35em]\n    w^{2} & = & [~w_{1}^{2}, & w_{2}^{2}, & 0, & \\ldots & 0, & 0, & \\ldots & 0~]\\\\[0.35em]\n     & ~ \\vdots \\\\\n   w^{k} & = & [~w_{1}^{k}, & w_{2}^{k}, & w_{3}^{k}, & \\ldots &  w_{k}^{k}, & 0, & \\ldots & 0~].\\\\\n  \\end{array}\n​\nw\n​\n0\n​\n​\n​\nw\n​\n1\n​\n​\n​\nw\n​\n2\n​\n​\n​\n​\nw\n​\nk\n​\n​\n​\n​\n​\n​\n=\n​\n=\n​\n=\n​\n⋮\n​\n=\n​\n​\n​\n[\n0\n,\n​\n[\nw\n​\n1\n​\n1\n​\n​\n,\n​\n[\nw\n​\n1\n​\n2\n​\n​\n,\n​\n[\nw\n​\n1\n​\nk\n​\n​\n,\n​\n​\n​\n0\n,\n​\n0\n,\n​\nw\n​\n2\n​\n2\n​\n​\n,\n​\nw\n​\n2\n​\nk\n​\n​\n,\n​\n​\n​\n0\n,\n​\n0\n,\n​\n0\n,\n​\nw\n​\n3\n​\nk\n​\n​\n,\n​\n​\n​\n…\n​\n…\n​\n…\n​\n…\n​\n​\n​\n0\n,\n​\n0\n,\n​\n0\n,\n​\nw\n​\nk\n​\nk\n​\n​\n,\n​\n​\n​\n0\n,\n​\n0\n,\n​\n0\n,\n​\n0\n,\n​\n​\n​\n…\n​\n…\n​\n…\n​\n…\n​\n​\n​\n0\n]\n​\n0\n]\n​\n0\n]\n​\n0\n]\n.\n​\n​\nThink of this restriction as a “speed of light” of information transfer. Error signals will take at least\nk\nk\nk\nsteps to move from\nw\n0\nw_0\nw\n​\n0\n​\n​\nto\nw\nk\nw_k\nw\n​\nk\n​\n​\n. We can therefore sum up the errors which cannot have changed yet\n9\n:\n∥\nw\nk\n−\nw\n⋆\n∥\n∞\n≥\nmax\ni\n≥\nk\n+\n1\n{\n∣\nw\ni\n⋆\n∣\n}\n=\n(\nκ\n−\n1\nκ\n+\n1\n)\nk\n+\n1\n=\n(\nκ\n−\n1\nκ\n+\n1\n)\nk\n∥\nw\n0\n−\nw\n⋆\n∥\n∞\n.\n\\begin{aligned}\n\\|w^{k}-w^{\\star}\\|_{\\infty}&\\geq\\max_{i\\geq k+1}\\{|w_{i}^{\\star}|\\}\\\\[0.9em]&=\\left(\\frac{\\sqrt{\\kappa}-1}{\\sqrt{\\kappa}+1}\\right)^{k+1}\\\\[0.9em]&=\\left(\\frac{\\sqrt{\\kappa}-1}{\\sqrt{\\kappa}+1}\\right)^{k}\\|w^{0}-w^{\\star}\\|_{\\infty}.\n  \\end{aligned}\n​\n∥\nw\n​\nk\n​\n​\n−\nw\n​\n⋆\n​\n​\n∥\n​\n∞\n​\n​\n​\n​\n​\n​\n​\n≥\n​\ni\n≥\nk\n+\n1\n​\nmax\n​\n​\n{\n∣\nw\n​\ni\n​\n⋆\n​\n​\n∣\n}\n​\n=\n(\n​\n√\n​\nκ\n​\n​\n​\n+\n1\n​\n​\n√\n​\nκ\n​\n​\n​\n−\n1\n​\n​\n)\n​\nk\n+\n1\n​\n​\n​\n=\n(\n​\n√\n​\nκ\n​\n​\n​\n+\n1\n​\n​\n√\n​\nκ\n​\n​\n​\n−\n1\n​\n​\n)\n​\nk\n​\n​\n∥\nw\n​\n0\n​\n​\n−\nw\n​\n⋆\n​\n​\n∥\n​\n∞\n​\n​\n.\n​\n​\nAs\nn\nn\nn\ngets large, the condition number of\nf\nn\nf^n\nf\n​\nn\n​\n​\napproaches\nκ\n\\kappa\nκ\n. And the gap therefore closes; the convergence rate that momentum promises matches the best any linear first order algorithm can do. And we arrive at the disappointing conclusion that on this problem, we cannot do better.\nLike many such lower bounds, this result must not be taken literally, but spiritually. It, perhaps, gives a sense of closure and finality to our investigation. But this is not the final word on first order optimization. This lower bound does not preclude the possibility, for example, of reformulating the problem to change the condition number itself! There is still much room for speedups, if you understand the right places to look.\nMomentum with Stochastic Gradients\nThere is a final point worth addressing. All the discussion above assumes access to the true gradient — a luxury seldom afforded in modern machine learning. Computing the exact gradient requires a full pass over all the data, the cost of which can be prohibitively expensive. Instead, randomized approximations of the gradient, like minibatch sampling, are often used as a plug-in replacement of\n∇\nf\n(\nw\n)\n\\nabla f(w)\n∇\nf\n(\nw\n)\n. We can write the approximation in two parts,\n∇\nf\n(\nw\n)\n\\nabla f(w)\n∇\nf\n(\nw\n)\nthe true gradient\n+\n+\n+\ne\nr\nr\no\nr\n(\nw\n)\n.\n\\text{error}(w).\nerror\n(\nw\n)\n.\nand an approximation error.\nIf the estimator is unbiased e.g.\nE\n[\ne\nr\nr\no\nr\n(\nw\n)\n]\n=\n0\n\\mathbf{E}[\\text{error}(w)] = 0\nE\n[\nerror\n(\nw\n)\n]\n=\n0\nIt is helpful to think of our approximate gradient as the injection of a special kind of noise into our iteration. And using the machinery developed in the previous sections, we can deal with this extra term directly. On a quadratic, the error term cleaves cleanly into a separate term, where\n10\n(\ny\ni\nk\nx\ni\nk\n)\n\\left(\\begin{array}{c}\n  y_{i}^{k}\\\\\n  x_{i}^{k}\n  \\end{array}\\right)\n(\n​\ny\n​\ni\n​\nk\n​\n​\n​\nx\n​\ni\n​\nk\n​\n​\n​\n​\n)\nthe noisy iterates are a sum of\n=\n=\n=\nR\nk\n(\ny\ni\n0\nx\ni\n0\n)\nR^{k}\\left(\\begin{array}{c}\n  y_{i}^{0}\\\\\n  x_{i}^{0}\n  \\end{array}\\right)\nR\n​\nk\n​\n​\n(\n​\ny\n​\ni\n​\n0\n​\n​\n​\nx\n​\ni\n​\n0\n​\n​\n​\n​\n)\nthe noiseless, deterministic iterates and\n+\n+\n+\nϵ\ni\nk\n∑\nj\n=\n1\nk\nR\nk\n−\nj\n(\n1\n−\nα\n)\n\\epsilon^k_i \\sum_{j=1}^{k}R^{k-j}\\left(\\begin{array}{c}\n1\\\\\n-\\alpha\n\\end{array}\\right)\nϵ\n​\ni\n​\nk\n​\n​\n​\nj\n=\n1\n​\n∑\n​\nk\n​\n​\nR\n​\nk\n−\nj\n​\n​\n(\n​\n1\n​\n−\nα\n​\n​\n)\na decaying sum of the errors, where\nϵ\nk\n=\nQ\n⋅\ne\nr\nr\no\nr\n(\nw\nk\n)\n\\epsilon^k = Q \\cdot \\text{error}(w^k)\nϵ\n​\nk\n​\n​\n=\nQ\n⋅\nerror\n(\nw\n​\nk\n​\n​\n)\n.\nThe error term,\nϵ\nk\n\\epsilon^k\nϵ\n​\nk\n​\n​\n, with its dependence on the\nw\nk\nw^k\nw\n​\nk\n​\n​\n, is a fairly hairy object. Following\n[10]\n, we model this as independent 0-mean Gaussian noise. In this simplified model, the objective also breaks into two separable components, a sum of a deterministic error and a stochastic error\n11\n, visualized here.\nWe decompose the expected value of the objective value\nE\nf\n(\nw\n)\n−\nf\n(\nw\n⋆\n)\n\\mathbf{E} f(w) - f(w^\\star)\nE\nf\n(\nw\n)\n−\nf\n(\nw\n​\n⋆\n​\n​\n)\ninto a deterministic part\nand a stochastic part\n.\nE\nf\n(\nw\n)\n−\nf\n(\nw\n⋆\n)\n\\mathbf{E} f(w) - f(w^\\star)\nE\nf\n(\nw\n)\n−\nf\n(\nw\n​\n⋆\n​\n​\n)\nThe small black dots are a single run of stochastic gradient\nStep-size α =\nMomentum β =\nAs\n[1]\nobserves, the optimization has two phases. In the initial transient phase the magnitude of the noise is smaller than the magnitude of the gradient, and Momentum still makes good progress. In the second, stochastic phase, the noise overwhelms the gradient, and momentum is less effective.\nNote that there are a set of unfortunate tradeoffs which seem to pit the two components of error against each other. Lowering the step-size, for example, decreases the stochastic error, but also slows down the rate of convergence. And increasing momentum, contrary to popular belief, causes the errors to compound. Despite these undesirable properties, stochastic gradient descent with momentum has still been shown to have competitive performance on neural networks. As\n[1]\nhas observed, the transient phase seems to matter more than the fine-tuning phase in machine learning. And in fact, it has been recently suggested\n[12]\nthat this noise is a good thing — it acts as a implicit regularizer, which, like early stopping, prevents overfitting in the fine-tuning phase of optimization.\nOnwards and Downwards\nThe study of acceleration is seeing a small revival within the optimization community. If the ideas in this article excite you, you may wish to read\n[13]\n, which fully explores the idea of momentum as the discretization of a certain differential equation. But other, less physical, interpretations exist. There is an algebraic interpretation of momentum in terms of approximating polynomials\n[3, 14]\n. Geometric interpretations are emerging\n[15, 16]\n, connecting momentum to older methods, like the Ellipsoid method. And finally, there are interpretations relating momentum to duality\n[17]\n, perhaps providing a clue as how to accelerate second order methods and Quasi Newton (for a first step, see\n[18]\n). But like the proverbial blind men feeling an elephant, momentum seems like something bigger than the sum of its parts. One day, hopefully soon, the many perspectives will converge into a satisfying whole.\nAcknowledgments\nI am deeply indebted to the editorial contributions of Shan Carter and Chris Olah, without which this article would be greatly impoverished. Shan Carter provided complete redesigns of many of my original interactive widgets, a visual coherence for all the figures, and valuable optimizations to the page’s performance. Chris Olah provided impeccable editorial feedback at all levels of detail and abstraction - from the structure of the content, to the alignment of equations.\nI am also grateful to Michael Nielsen for providing the title of this article, which really tied the article together. Marcos Ginestra provided editorial input for the earliest drafts of this article, and spiritual encouragement when I needed it the most. And my gratitude extends to my reviewers, Matt Hoffman and Anonymous Reviewer B for their astute observations and criticism. I would like to thank Reviewer B, in particular, for pointing out two non-trivial errors in the original manuscript (discussion\nhere\n). The contour plotting library for the hero visualization is the joint work of Ben Frederickson, Jeff Heer and Mike Bostock.\nMany thanks to the numerous pull requests and issues filed on github. Thanks in particular, to Osemwaro Pedro for spotting an off by one error in one of the equations. And also to Dan Schmidt who did an editing pass over the whole project, correcting numerous typographical and grammatical errors.\nDiscussion and Review\nReviewer A - Matt Hoffman\nReviewer B - Anonymous\nDiscussion with User derifatives\nFootnotes\nIt is possible, however, to construct very specific counterexamples where momentum does not converge, even on convex functions. See\n[4]\nfor a counterexample.\nIn Tikhonov Regression we add a quadratic penalty to the regression, minimizing\nm\ni\nn\ni\nm\ni\nz\ne\n1\n2\n∥\nZ\nw\n−\nd\n∥\n2\n+\nη\n2\n∥\nw\n∥\n2\n=\n1\n2\nw\nT\n(\nZ\nT\nZ\n+\nη\nI\n)\nw\n−\n(\nZ\nd\n)\nT\nw\n\\text{minimize}\\qquad\\tfrac{1}{2}\\|Zw-d\\|^{2}+\\frac{\\eta}{2}\\|w\\|^{2}=\\tfrac{1}{2}w^{T}(Z^{T}Z+\\eta I)w-(Zd)^{T}w\nminimize\n​\n2\n​\n​\n1\n​\n​\n∥\nZ\nw\n−\nd\n∥\n​\n2\n​\n​\n+\n​\n2\n​\n​\nη\n​\n​\n∥\nw\n∥\n​\n2\n​\n​\n=\n​\n2\n​\n​\n1\n​\n​\nw\n​\nT\n​\n​\n(\nZ\n​\nT\n​\n​\nZ\n+\nη\nI\n)\nw\n−\n(\nZ\nd\n)\n​\nT\n​\n​\nw\nRecall that\nZ\nT\nZ\n=\nQ\nd\ni\na\ng\n(\nΛ\n1\n,\n…\n,\nΛ\nn\n)\nQ\nT\nZ^{T}Z=Q\\ \\text{diag}(\\Lambda_{1},\\ldots,\\Lambda_{n})\\ Q^T\nZ\n​\nT\n​\n​\nZ\n=\nQ\ndiag\n(\nΛ\n​\n1\n​\n​\n,\n…\n,\nΛ\n​\nn\n​\n​\n)\nQ\n​\nT\n​\n​\n. The solution to Tikhonov Regression is therefore\n(\nZ\nT\nZ\n+\nη\nI\n)\n−\n1\n(\nZ\nd\n)\n=\nQ\nd\ni\na\ng\n(\n1\nλ\n1\n+\nη\n,\n⋯\n,\n1\nλ\nn\n+\nη\n)\nQ\nT\n(\nZ\nd\n)\n(Z^{T}Z+\\eta I)^{-1}(Zd)=Q\\ \\text{diag}\\left(\\frac{1}{\\lambda_{1}+\\eta},\\cdots,\\frac{1}{\\lambda_{n}+\\eta}\\right)Q^T(Zd)\n(\nZ\n​\nT\n​\n​\nZ\n+\nη\nI\n)\n​\n−\n1\n​\n​\n(\nZ\nd\n)\n=\nQ\ndiag\n(\n​\nλ\n​\n1\n​\n​\n+\nη\n​\n​\n1\n​\n​\n,\n⋯\n,\n​\nλ\n​\nn\n​\n​\n+\nη\n​\n​\n1\n​\n​\n)\nQ\n​\nT\n​\n​\n(\nZ\nd\n)\nWe can think of regularization as a function which decays the largest eigenvalues, as follows:\nT\ni\nk\nh\no\nn\no\nv\nR\ne\ng\nu\nl\na\nr\ni\nz\ne\nd\nλ\ni\n=\n1\nλ\ni\n+\nη\n=\n1\nλ\ni\n(\n1\n−\n(\n1\n+\nλ\ni\n/\nη\n)\n−\n1\n)\n.\n\\text{Tikhonov Regularized } \\lambda_i = \\frac{1}{\\lambda_{i}+\\eta}=\\frac{1}{\\lambda_{i}}\\left(1-\\left(1+\\lambda_{i}/\\eta\\right)^{-1}\\right).\nTikhonov Regularized\nλ\n​\ni\n​\n​\n=\n​\nλ\n​\ni\n​\n​\n+\nη\n​\n​\n1\n​\n​\n=\n​\nλ\n​\ni\n​\n​\n​\n​\n1\n​\n​\n(\n1\n−\n(\n1\n+\nλ\n​\ni\n​\n​\n/\nη\n)\n​\n−\n1\n​\n​\n)\n.\nGradient descent can be seen as employing a similar decay, but with the decay rate\nG\nr\na\nd\ni\ne\nn\nt\nD\ne\ns\nc\ne\nn\nt\nR\ne\ng\nu\nl\na\nr\ni\nz\ne\nd\nλ\ni\n=\n1\nλ\ni\n(\n1\n−\n(\n1\n−\nα\nλ\ni\n)\nk\n)\n\\text{ Gradient Descent Regularized } \\lambda_i = \\frac{1}{\\lambda_i} \\left( 1-\\left(1-\\alpha\\lambda_{i}\\right)^{k} \\right)\nGradient Descent Regularized\nλ\n​\ni\n​\n​\n=\n​\nλ\n​\ni\n​\n​\n​\n​\n1\n​\n​\n(\n1\n−\n(\n1\n−\nα\nλ\n​\ni\n​\n​\n)\n​\nk\n​\n​\n)\ninstead. Note that this decay is dependent on the step-size.\nThis is true as we can write updates in matrix form as\n(\n1\n0\nα\n1\n)\n(\ny\ni\nk\n+\n1\nx\ni\nk\n+\n1\n)\n=\n(\nβ\nλ\ni\n0\n1\n)\n(\ny\ni\nk\nx\ni\nk\n)\n\\left(\\!\\!\\begin{array}{cc}\n  1 & 0\\\\\n  \\alpha & 1\n  \\end{array}\\!\\!\\right)\\Bigg(\\!\\!\\begin{array}{c}\n  y_{i}^{k+1}\\\\\n  x_{i}^{k+1}\n  \\end{array}\\!\\!\\Bigg)=\\left(\\!\\!\\begin{array}{cc}\n  \\beta & \\lambda_{i}\\\\\n  0 & 1\n  \\end{array}\\!\\!\\right)\\left(\\!\\!\\begin{array}{c}\n  y_{i}^{k}\\\\\n  x_{i}^{k}\n  \\end{array}\\!\\!\\right)\n(\n​\n1\n​\nα\n​\n​\n​\n0\n​\n1\n​\n​\n)\n(\n​\ny\n​\ni\n​\nk\n+\n1\n​\n​\n​\nx\n​\ni\n​\nk\n+\n1\n​\n​\n​\n​\n)\n=\n(\n​\nβ\n​\n0\n​\n​\n​\nλ\n​\ni\n​\n​\n​\n1\n​\n​\n)\n(\n​\ny\n​\ni\n​\nk\n​\n​\n​\nx\n​\ni\n​\nk\n​\n​\n​\n​\n)\nwhich implies, by inverting the matrix on the left,\n(\ny\ni\nk\n+\n1\nx\ni\nk\n+\n1\n)\n=\n(\nβ\nλ\ni\n−\nα\nβ\n1\n−\nα\nλ\ni\n)\n(\ny\ni\nk\nx\ni\nk\n)\n=\nR\nk\n+\n1\n(\nx\ni\n0\ny\ni\n0\n)\n\\Bigg(\\!\\!\\begin{array}{c}\n  y_{i}^{k+1}\\\\\n  x_{i}^{k+1}\n  \\end{array}\\!\\!\\Bigg)=\\left(\\!\\!\\begin{array}{cc}\n  \\beta & \\lambda_{i}\\\\\n  -\\alpha\\beta & 1-\\alpha\\lambda_{i}\n  \\end{array}\\!\\!\\right)\\left(\\!\\!\\begin{array}{c}\n  y_{i}^{k}\\\\\n  x_{i}^{k}\n  \\end{array}\\!\\!\\right)=R^{k+1}\\left(\\!\\!\\begin{array}{c}\n  x_{i}^{0}\\\\\n  y_{i}^{0}\n  \\end{array}\\!\\!\\right)\n(\n​\ny\n​\ni\n​\nk\n+\n1\n​\n​\n​\nx\n​\ni\n​\nk\n+\n1\n​\n​\n​\n​\n)\n=\n(\n​\nβ\n​\n−\nα\nβ\n​\n​\n​\nλ\n​\ni\n​\n​\n​\n1\n−\nα\nλ\n​\ni\n​\n​\n​\n​\n)\n(\n​\ny\n​\ni\n​\nk\n​\n​\n​\nx\n​\ni\n​\nk\n​\n​\n​\n​\n)\n=\nR\n​\nk\n+\n1\n​\n​\n(\n​\nx\n​\ni\n​\n0\n​\n​\n​\ny\n​\ni\n​\n0\n​\n​\n​\n​\n)\nWe can write out the convergence rates explicitly. The eigenvalues are\nσ\n1\n=\n1\n2\n(\n1\n−\nα\nλ\n+\nβ\n+\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n2\n−\n4\nβ\n)\nσ\n2\n=\n1\n2\n(\n1\n−\nα\nλ\n+\nβ\n−\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n2\n−\n4\nβ\n)\n\\begin{aligned}\n\\sigma_{1} & =\\frac{1}{2}\\left(1-\\alpha\\lambda+\\beta+\\sqrt{(-\\alpha\\lambda+\\beta+1)^{2}-4\\beta}\\right)\\\\[0.6em]\n\\sigma_{2} & =\\frac{1}{2}\\left(1-\\alpha\\lambda+\\beta-\\sqrt{(-\\alpha\\lambda+\\beta+1)^{2}-4\\beta}\\right)\n\\end{aligned}\n​\nσ\n​\n1\n​\n​\n​\nσ\n​\n2\n​\n​\n​\n​\n​\n=\n​\n2\n​\n​\n1\n​\n​\n(\n1\n−\nα\nλ\n+\nβ\n+\n√\n​\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n​\n2\n​\n​\n−\n4\nβ\n​\n​\n​\n)\n​\n=\n​\n2\n​\n​\n1\n​\n​\n(\n1\n−\nα\nλ\n+\nβ\n−\n√\n​\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n​\n2\n​\n​\n−\n4\nβ\n​\n​\n​\n)\n​\n​\nWhen the\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n2\n−\n4\nβ\n<\n0\n(-\\alpha\\lambda+\\beta+1)^{2}-4\\beta<0\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n​\n2\n​\n​\n−\n4\nβ\n<\n0\nis less than zero,\nthen the roots are complex and the convergence rate is\n∣\nσ\n1\n∣\n=\n∣\nσ\n2\n∣\n=\n(\n1\n−\nα\nλ\n+\nβ\n)\n2\n+\n∣\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n2\n−\n4\nβ\n∣\n=\n2\nβ\n\\begin{aligned}\n|\\sigma_{1}|=|\\sigma_{2}| & =\\sqrt{(1-\\alpha\\lambda+\\beta)^{2}+|(-\\alpha\\lambda+\\beta+1)^{2}-4\\beta|}=2\\sqrt{\\beta}\n\\end{aligned}\n​\n∣\nσ\n​\n1\n​\n​\n∣\n=\n∣\nσ\n​\n2\n​\n​\n∣\n​\n​\n​\n=\n√\n​\n(\n1\n−\nα\nλ\n+\nβ\n)\n​\n2\n​\n​\n+\n∣\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n​\n2\n​\n​\n−\n4\nβ\n∣\n​\n​\n​\n=\n2\n√\n​\nβ\n​\n​\n​\n​\n​\nWhich is, surprisingly, independent of the step-size or the eigenvalue\nα\nλ\n\\alpha\\lambda\nα\nλ\n. When the roots are real, the convergence rate is\nmax\n{\n∣\nσ\n1\n∣\n,\n∣\nσ\n2\n∣\n}\n=\n1\n2\nmax\n{\n∣\n1\n−\nα\nλ\ni\n+\nβ\n±\n(\n1\n−\nα\nλ\ni\n+\nβ\n)\n2\n−\n4\nβ\n∣\n}\n\\max\\{|\\sigma_{1}|,|\\sigma_{2}|\\}=\\tfrac{1}{2}\\max\\left\\{ |1-\\alpha\\lambda_{i}+\\beta\\pm\\sqrt{(1-\\alpha\\lambda_{i}+\\beta)^{2}-4\\beta}|\\right\\}\nmax\n{\n∣\nσ\n​\n1\n​\n​\n∣\n,\n∣\nσ\n​\n2\n​\n​\n∣\n}\n=\n​\n2\n​\n​\n1\n​\n​\nmax\n{\n∣\n1\n−\nα\nλ\n​\ni\n​\n​\n+\nβ\n±\n√\n​\n(\n1\n−\nα\nλ\n​\ni\n​\n​\n+\nβ\n)\n​\n2\n​\n​\n−\n4\nβ\n​\n​\n​\n∣\n}\nThis can be derived by reducing the inequalities for all 4 + 1 cases in the explicit form of the convergence rate above.\nWe must optimize over\nmin\nα\n,\nβ\nmax\n{\n∥\n(\nβ\nλ\ni\n−\nα\nβ\n1\n−\nα\nλ\ni\n)\n∥\n,\n…\n,\n∥\n(\nβ\nλ\nn\n−\nα\nβ\n1\n−\nα\nλ\nn\n)\n∥\n}\n.\n\\min_{\\alpha,\\beta}\\max\\left\\{ \\bigg\\| \\! \\left(\\begin{array}{cc}\n  \\beta & \\lambda_{i}\\\\\n  -\\alpha\\beta & 1-\\alpha\\lambda_{i}\n  \\end{array}\\right) \\! \\bigg\\|,\\ldots,\\bigg\\| \\! \\left(\\begin{array}{cc}\n  \\beta & \\lambda_{n}\\\\\n  -\\alpha\\beta & 1-\\alpha\\lambda_{n}\n  \\end{array}\\right)\\! \\bigg\\|\\right\\}.\n​\nα\n,\nβ\n​\nmin\n​\n​\nmax\n{\n​\n∥\n​\n∥\n​\n∥\n​\n∥\n​\n​\n(\n​\nβ\n​\n−\nα\nβ\n​\n​\n​\nλ\n​\ni\n​\n​\n​\n1\n−\nα\nλ\n​\ni\n​\n​\n​\n​\n)\n​\n∥\n​\n∥\n​\n∥\n​\n∥\n​\n​\n,\n…\n,\n​\n∥\n​\n∥\n​\n∥\n​\n∥\n​\n​\n(\n​\nβ\n​\n−\nα\nβ\n​\n​\n​\nλ\n​\nn\n​\n​\n​\n1\n−\nα\nλ\n​\nn\n​\n​\n​\n​\n)\n​\n∥\n​\n∥\n​\n∥\n​\n∥\n​\n​\n}\n.\n(\n∥\n⋅\n∥\n\\|\\cdot \\|\n∥\n⋅\n∥\nhere denotes the magnitude of the maximum eigenvalue), and occurs when the roots of the characteristic polynomial are repeated for the matrices corresponding to the extremal eigenvalues.\nThe above optimization problem is bounded from below by\n0\n0\n0\n, and vector of all\n1\n1\n1\n’s achieve this.\nThis can be written explicitly as\n[\nL\nG\n]\ni\nj\n=\n{\nd\ne\ng\nr\ne\ne\no\nf\nv\ne\nr\nt\ne\nx\ni\ni\n=\nj\n−\n1\ni\n≠\nj\n,\n(\ni\n,\nj\n)\no\nr\n(\nj\n,\ni\n)\n∈\nE\n0\no\nt\nh\ne\nr\nw\ni\ns\ne\n[L_{G}]_{ij}=\\begin{cases} \\text{degree of vertex }i & i=j\\\\ -1 & i\\neq j,(i,j)\\text{ or }(j,i)\\in E\\\\ 0 & \\text{otherwise} \\end{cases}\n[\nL\n​\nG\n​\n​\n]\n​\ni\nj\n​\n​\n=\n​\n⎩\n​\n⎪\n​\n⎨\n​\n⎪\n​\n⎧\n​\n​\n​\ndegree of vertex\ni\n​\n−\n1\n​\n0\n​\n​\n​\ni\n=\nj\n​\ni\n≠\nj\n,\n(\ni\n,\nj\n)\nor\n(\nj\n,\ni\n)\n∈\nE\n​\notherwise\n​\n​\nWe use the infinity norm to measure our error, similar results can be derived for the 1 and 2 norms.\nThe momentum iterations are\nz\nk\n+\n1\n=\nβ\nz\nk\n+\nA\nw\nk\n+\ne\nr\nr\no\nr\n(\nw\nk\n)\nw\nk\n+\n1\n=\nw\nk\n−\nα\nz\nk\n+\n1\n.\n\\begin{aligned}\n  z^{k+1}&=\\beta z^{k}+ A w^{k}  + \\text{error}(w^k) \\\\[0.4em]\n  w^{k+1}&=w^{k}-\\alpha z^{k+1}.\n  \\end{aligned}\n​\nz\n​\nk\n+\n1\n​\n​\n​\nw\n​\nk\n+\n1\n​\n​\n​\n​\n​\n=\nβ\nz\n​\nk\n​\n​\n+\nA\nw\n​\nk\n​\n​\n+\nerror\n(\nw\n​\nk\n​\n​\n)\n​\n=\nw\n​\nk\n​\n​\n−\nα\nz\n​\nk\n+\n1\n​\n​\n.\n​\n​\nwhich, after a change of variables, become\n(\n1\n0\nα\n1\n)\n(\ny\ni\nk\n+\n1\nx\ni\nk\n+\n1\n)\n=\n(\nβ\nλ\ni\n0\n1\n)\n(\ny\ni\nk\nx\ni\nk\n)\n+\n(\nϵ\ni\nk\n0\n)\n\\left(\\!\\!\\begin{array}{cc}\n1 & 0\\\\\n\\alpha & 1\n\\end{array}\\!\\!\\right)\\Bigg(\\!\\!\\begin{array}{c}\ny_{i}^{k+1}\\\\\nx_{i}^{k+1}\n\\end{array}\\!\\!\\Bigg)=\\left(\\!\\!\\begin{array}{cc}\n\\beta & \\lambda_{i}\\\\\n0 & 1\n\\end{array}\\!\\!\\right)\\left(\\!\\!\\begin{array}{c}\ny_{i}^{k}\\\\\nx_{i}^{k}\n\\end{array}\\!\\!\\right)+\\left(\\!\\!\\begin{array}{c}\n\\epsilon_{i}^{k}\\\\\n0\n\\end{array}\\!\\!\\right)\n(\n​\n1\n​\nα\n​\n​\n​\n0\n​\n1\n​\n​\n)\n(\n​\ny\n​\ni\n​\nk\n+\n1\n​\n​\n​\nx\n​\ni\n​\nk\n+\n1\n​\n​\n​\n​\n)\n=\n(\n​\nβ\n​\n0\n​\n​\n​\nλ\n​\ni\n​\n​\n​\n1\n​\n​\n)\n(\n​\ny\n​\ni\n​\nk\n​\n​\n​\nx\n​\ni\n​\nk\n​\n​\n​\n​\n)\n+\n(\n​\nϵ\n​\ni\n​\nk\n​\n​\n​\n0\n​\n​\n)\nInverting the\n2\n×\n2\n2 \\times 2\n2\n×\n2\nmatrix on the left, and applying the formula recursively yields the final solution.\nOn the 1D function\nf\n(\nx\n)\n=\nλ\n2\nx\n2\nf(x)=\\frac{\\lambda}{2}x^{2}\nf\n(\nx\n)\n=\n​\n2\n​\n​\nλ\n​\n​\nx\n​\n2\n​\n​\n, the objective value is\nE\nf\n(\nx\nk\n)\n=\nλ\n2\nE\n[\n(\nx\nk\n)\n2\n]\n=\nλ\n2\nE\n(\ne\n2\nT\nR\nk\n(\ny\n0\nx\n0\n)\n+\nϵ\nk\ne\n2\nT\n∑\ni\n=\n1\nk\nR\nk\n−\ni\n(\n1\n−\nα\n)\n)\n2\n=\nλ\n2\ne\n2\nT\nR\nk\n(\ny\n0\nx\n0\n)\n+\nλ\n2\nE\n(\nϵ\nk\ne\n2\nT\n∑\ni\n=\n1\nk\nR\nk\n−\ni\n(\n1\n−\nα\n)\n)\n2\n=\nλ\n2\ne\n2\nT\nR\nk\n(\ny\n0\nx\n0\n)\n+\nλ\n2\nE\n[\nϵ\nk\n]\n⋅\n∑\ni\n=\n1\nk\n(\ne\n2\nT\nR\nk\n−\ni\n(\n1\n−\nα\n)\n)\n2\n=\nλ\n2\ne\n2\nT\nR\nk\n(\ny\n0\nx\n0\n)\n+\nλ\nE\n[\nϵ\nk\n2\n⋅\n∑\ni\n=\n1\nk\nγ\ni\n2\n,\nγ\ni\n=\ne\n2\nT\nR\nk\n−\ni\n(\n1\n−\nα\n)\n\\begin{aligned}\n\\mathbf{E}f(x^{k})&=\\frac{\\lambda}{2}\\mathbf{E}[(x^{k})^{2}]\\\\&=\\frac{\\lambda}{2}\\mathbf{E}\\left(e_{2}^{T}R^{k}\\left(\\begin{array}{c}\ny^{0}\\\\\nx^{0}\n\\end{array}\\right)+\\epsilon^{k}e_{2}^{T}\\sum_{i=1}^{k}R^{k-i}\\left(\\begin{array}{c}\n1\\\\\n-\\alpha\n\\end{array}\\right)\\right)^{2}\\\\&=\\frac{\\lambda}{2}e_{2}^{T}R^{k}\\left(\\begin{array}{c}\ny^{0}\\\\\nx^{0}\n\\end{array}\\right)+\\frac{\\lambda}{2}\\mathbf{E}\\left(\\epsilon^{k}e_{2}^{T}\\sum_{i=1}^{k}R^{k-i}\\left(\\begin{array}{c}\n1\\\\\n-\\alpha\n\\end{array}\\right)\\right)^{2}\\\\&=\\frac{\\lambda}{2}e_{2}^{T}R^{k}\\left(\\begin{array}{c}\ny^{0}\\\\\nx^{0}\n\\end{array}\\right)+\\frac{\\lambda}{2}\\mathbf{E}[\\epsilon^{k}]\\,\\cdot\\,\\sum_{i=1}^{k}\\left(e_{2}^{T}R^{k-i}\\left(\\begin{array}{c}\n1\\\\\n-\\alpha\n\\end{array}\\right)\\right)^{2}\\\\&=\\frac{\\lambda}{2}e_{2}^{T}R^{k}\\left(\\begin{array}{c}\ny^{0}\\\\\nx^{0}\n\\end{array}\\right)+\\frac{\\lambda\\mathbf{E}[\\epsilon^{k}}{2}\\cdot\\sum_{i=1}^{k}\\gamma_{i}^{2}, \\qquad \\gamma_i = e_{2}^{T}R^{k-i}\\left(\\begin{array}{c}\n1\\\\\n-\\alpha\n\\end{array}\\right)\n\\end{aligned}\n​\nE\nf\n(\nx\n​\nk\n​\n​\n)\n​\n​\n​\n​\n​\n​\n​\n=\n​\n2\n​\n​\nλ\n​\n​\nE\n[\n(\nx\n​\nk\n​\n​\n)\n​\n2\n​\n​\n]\n​\n=\n​\n2\n​\n​\nλ\n​\n​\nE\n(\ne\n​\n2\n​\nT\n​\n​\nR\n​\nk\n​\n​\n(\n​\ny\n​\n0\n​\n​\n​\nx\n​\n0\n​\n​\n​\n​\n)\n+\nϵ\n​\nk\n​\n​\ne\n​\n2\n​\nT\n​\n​\n​\ni\n=\n1\n​\n∑\n​\nk\n​\n​\nR\n​\nk\n−\ni\n​\n​\n(\n​\n1\n​\n−\nα\n​\n​\n)\n)\n​\n2\n​\n​\n​\n=\n​\n2\n​\n​\nλ\n​\n​\ne\n​\n2\n​\nT\n​\n​\nR\n​\nk\n​\n​\n(\n​\ny\n​\n0\n​\n​\n​\nx\n​\n0\n​\n​\n​\n​\n)\n+\n​\n2\n​\n​\nλ\n​\n​\nE\n(\nϵ\n​\nk\n​\n​\ne\n​\n2\n​\nT\n​\n​\n​\ni\n=\n1\n​\n∑\n​\nk\n​\n​\nR\n​\nk\n−\ni\n​\n​\n(\n​\n1\n​\n−\nα\n​\n​\n)\n)\n​\n2\n​\n​\n​\n=\n​\n2\n​\n​\nλ\n​\n​\ne\n​\n2\n​\nT\n​\n​\nR\n​\nk\n​\n​\n(\n​\ny\n​\n0\n​\n​\n​\nx\n​\n0\n​\n​\n​\n​\n)\n+\n​\n2\n​\n​\nλ\n​\n​\nE\n[\nϵ\n​\nk\n​\n​\n]\n⋅\n​\ni\n=\n1\n​\n∑\n​\nk\n​\n​\n(\ne\n​\n2\n​\nT\n​\n​\nR\n​\nk\n−\ni\n​\n​\n(\n​\n1\n​\n−\nα\n​\n​\n)\n)\n​\n2\n​\n​\n​\n=\n​\n2\n​\n​\nλ\n​\n​\ne\n​\n2\n​\nT\n​\n​\nR\n​\nk\n​\n​\n(\n​\ny\n​\n0\n​\n​\n​\nx\n​\n0\n​\n​\n​\n​\n)\n+\n​\n2\n​\n​\nλ\nE\n[\nϵ\n​\nk\n​\n​\n​\n​\n⋅\n​\ni\n=\n1\n​\n∑\n​\nk\n​\n​\nγ\n​\ni\n​\n2\n​\n​\n,\nγ\n​\ni\n​\n​\n=\ne\n​\n2\n​\nT\n​\n​\nR\n​\nk\n−\ni\n​\n​\n(\n​\n1\n​\n−\nα\n​\n​\n)\n​\n​\nThe third inequality uses the fact that\nE\nϵ\nk\n=\n0\n\\mathbf{E} \\epsilon^k = 0\nE\nϵ\n​\nk\n​\n​\n=\n0\nand the fourth uses the fact they are uncorrelated.\nReferences\nOn the importance of initialization and momentum in deep learning.\n[PDF]\nSutskever, I., Martens, J., Dahl, G.E. and Hinton, G.E., 2013. ICML (3), Vol 28, pp. 1139—1147.\nSome methods of speeding up the convergence of iteration methods\n[PDF]\nPolyak, B.T., 1964. USSR Computational Mathematics and Mathematical Physics, Vol 4(5), pp. 1—17. Elsevier.\nDOI: 10.1016/0041-5553(64)90137-5\nTheory of gradient methods\nRutishauser, H., 1959. Refined iterative methods for computation of the solution and the eigenvalues of self-adjoint boundary value problems, pp. 24—49. Springer.\nDOI: 10.1007/978-3-0348-7224-9_2\nAnalysis and design of optimization algorithms via integral quadratic constraints\n[PDF]\nLessard, L., Recht, B. and Packard, A., 2016. SIAM Journal on Optimization, Vol 26(1), pp. 57—95. SIAM.\nIntroductory lectures on convex optimization: A basic course\nNesterov, Y., 2013. , Vol 87. Springer Science \\& Business Media.\nDOI: 10.1007/978-1-4419-8853-9\nNatural gradient works efficiently in learning\n[link]\nAmari, S., 1998. Neural computation, Vol 10(2), pp. 251—276. MIT Press.\nDOI: 10.1162/089976698300017746\nDeep Learning, NIPS′2015 Tutorial\n[PDF]\nHinton, G., Bengio, Y. and LeCun, Y., 2015.\nAdaptive restart for accelerated gradient schemes\n[PDF]\nO’Donoghue, B. and Candes, E., 2015. Foundations of computational mathematics, Vol 15(3), pp. 715—732. Springer.\nDOI: 10.1007/s10208-013-9150-3\nThe Nth Power of a 2x2 Matrix.\n[PDF]\nWilliams, K., 1992. Mathematics Magazine, Vol 65(5), pp. 336. MAA.\nDOI: 10.2307/2691246\nFrom Averaging to Acceleration, There is Only a Step-size.\n[PDF]\nFlammarion, N. and Bach, F.R., 2015. COLT, pp. 658—695.\nOn the momentum term in gradient descent learning algorithms\n[PDF]\nQian, N., 1999. Neural networks, Vol 12(1), pp. 145—151. Elsevier.\nDOI: 10.1016/s0893-6080(98)00116-6\nUnderstanding deep learning requires rethinking generalization\n[PDF]\nZhang, C., Bengio, S., Hardt, M., Recht, B. and Vinyals, O., 2016. arXiv preprint arXiv:1611.03530.\nA differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights\n[PDF]\nSu, W., Boyd, S. and Candes, E., 2014. Advances in Neural Information Processing Systems, pp. 2510—2518.\nThe Zen of Gradient Descent\n[HTML]\nHardt, M., 2013.\nA geometric alternative to Nesterov’s accelerated gradient descent\n[PDF]\nBubeck, S., Lee, Y.T. and Singh, M., 2015. arXiv preprint arXiv:1506.08187.\nAn optimal first order method based on optimal quadratic averaging\n[PDF]\nDrusvyatskiy, D., Fazel, M. and Roy, S., 2016. arXiv preprint arXiv:1604.06543.\nLinear coupling: An ultimate unification of gradient and mirror descent\n[PDF]\nAllen-Zhu, Z. and Orecchia, L., 2014. arXiv preprint arXiv:1407.1537.\nAccelerating the cubic regularization of Newton’s method on convex problems\n[PDF]\nNesterov, Y., 2008. Mathematical Programming, Vol 112(1), pp. 159—181. Springer.\nDOI: 10.1007/s10107-006-0089-x\nUpdates and Corrections\nView all changes\nto this article since it was first published. If you see a mistake or want to suggest a change, please\ncreate an issue on GitHub\n.\nCitations and Reuse\nDiagrams and text are licensed under Creative Commons Attribution\nCC-BY 2.0\n, unless noted otherwise, with the\nsource available on GitHub\n. The figures that have been reused from other sources don't fall under this license and can be recognized by a note in their caption: “Figure from …”.\nFor attribution in academic contexts, please cite this work as\nGoh, \"Why Momentum Really Works\", Distill, 2017. http://doi.org/10.23915/distill.00006\nBibTeX citation\n@article{goh2017why,\n  author = {Goh, Gabriel},\n  title = {Why Momentum Really Works},\n  journal = {Distill},\n  year = {2017},\n  url = {http://distill.pub/2017/momentum},\n  doi = {10.23915/distill.00006}\n}\nOn the importance of initialization and momentum in deep learning.\n[PDF]\nI. Sutskever, J. Martens, G.E. Dahl, G.E. Hinton.\nICML (3), Vol 28, pp. 1139—1147. 2013.\nSome methods of speeding up the convergence of iteration methods\n[PDF]\nB.T. Polyak.\nUSSR Computational Mathematics and Mathematical Physics, Vol 4(5), pp. 1—17. Elsevier. 1964.\nDOI: 10.1016/0041-5553(64)90137-5\nTheory of gradient methods\nH. Rutishauser.\nRefined iterative methods for computation of the solution and the eigenvalues of self-adjoint boundary value problems, pp. 24—49. Springer. 1959.\nDOI: 10.1007/978-3-0348-7224-9_2\nAnalysis and design of optimization algorithms via integral quadratic constraints\n[PDF]\nL. Lessard, B. Recht, A. Packard.\nSIAM Journal on Optimization, Vol 26(1), pp. 57—95. SIAM. 2016.\nIntroductory lectures on convex optimization: A basic course\nY. Nesterov.\n, Vol 87. Springer Science \\& Business Media. 2013.\nDOI: 10.1007/978-1-4419-8853-9\nNatural gradient works efficiently in learning\n[link]\nS. Amari.\nNeural computation, Vol 10(2), pp. 251—276. MIT Press. 1998.\nDOI: 10.1162/089976698300017746\nDeep Learning, NIPS′2015 Tutorial\n[PDF]\nG. Hinton, Y. Bengio, Y. LeCun.\n2015.\nAdaptive restart for accelerated gradient schemes\n[PDF]\nB. O’Donoghue, E. Candes.\nFoundations of computational mathematics, Vol 15(3), pp. 715—732. Springer. 2015.\nDOI: 10.1007/s10208-013-9150-3\nThe Nth Power of a 2x2 Matrix.\n[PDF]\nK. Williams.\nMathematics Magazine, Vol 65(5), pp. 336. MAA. 1992.\nDOI: 10.2307/2691246\nFrom Averaging to Acceleration, There is Only a Step-size.\n[PDF]\nN. Flammarion, F.R. Bach.\nCOLT, pp. 658—695. 2015.\nOn the momentum term in gradient descent learning algorithms\n[PDF]\nN. Qian.\nNeural networks, Vol 12(1), pp. 145—151. Elsevier. 1999.\nDOI: 10.1016/s0893-6080(98)00116-6\nIntroductory lectures on convex optimization: A basic course\nY. Nesterov.\n, Vol 87. Springer Science \\& Business Media. 2013.\nDOI: 10.1007/978-1-4419-8853-9\nFrom Averaging to Acceleration, There is Only a Step-size.\n[PDF]\nN. Flammarion, F.R. Bach.\nCOLT, pp. 658—695. 2015.\nOn the importance of initialization and momentum in deep learning.\n[PDF]\nI. Sutskever, J. Martens, G.E. Dahl, G.E. Hinton.\nICML (3), Vol 28, pp. 1139—1147. 2013.\nOn the importance of initialization and momentum in deep learning.\n[PDF]\nI. Sutskever, J. Martens, G.E. Dahl, G.E. Hinton.\nICML (3), Vol 28, pp. 1139—1147. 2013.\nUnderstanding deep learning requires rethinking generalization\n[PDF]\nC. Zhang, S. Bengio, M. Hardt, B. Recht, O. Vinyals.\narXiv preprint arXiv:1611.03530. 2016.\nA differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights\n[PDF]\nW. Su, S. Boyd, E. Candes.\nAdvances in Neural Information Processing Systems, pp. 2510—2518. 2014.\nTheory of gradient methods\nH. Rutishauser.\nRefined iterative methods for computation of the solution and the eigenvalues of self-adjoint boundary value problems, pp. 24—49. Springer. 1959.\nDOI: 10.1007/978-3-0348-7224-9_2\nThe Zen of Gradient Descent\n[HTML]\nM. Hardt.  2013.\nA geometric alternative to Nesterov’s accelerated gradient descent\n[PDF]\nS. Bubeck, Y.T. Lee, M. Singh.\narXiv preprint arXiv:1506.08187. 2015.\nAn optimal first order method based on optimal quadratic averaging\n[PDF]\nD. Drusvyatskiy, M. Fazel, S. Roy.\narXiv preprint arXiv:1604.06543. 2016.\nLinear coupling: An ultimate unification of gradient and mirror descent\n[PDF]\nZ. Allen-Zhu, L. Orecchia.\narXiv preprint arXiv:1407.1537. 2014.\nAccelerating the cubic regularization of Newton’s method on convex problems\n[PDF]\nY. Nesterov.\nMathematical Programming, Vol 112(1), pp. 159—181. Springer. 2008.\nDOI: 10.1007/s10107-006-0089-x\nIt is possible, however, to construct very specific counterexamples where momentum does not converge, even on convex functions. See\n[4]\nfor a counterexample.\nIn Tikhonov Regression we add a quadratic penalty to the regression, minimizing\nm\ni\nn\ni\nm\ni\nz\ne\n1\n2\n∥\nZ\nw\n−\nd\n∥\n2\n+\nη\n2\n∥\nw\n∥\n2\n=\n1\n2\nw\nT\n(\nZ\nT\nZ\n+\nη\nI\n)\nw\n−\n(\nZ\nd\n)\nT\nw\n\\text{minimize}\\qquad\\tfrac{1}{2}\\|Zw-d\\|^{2}+\\frac{\\eta}{2}\\|w\\|^{2}=\\tfrac{1}{2}w^{T}(Z^{T}Z+\\eta I)w-(Zd)^{T}w\nminimize\n​\n2\n​\n​\n1\n​\n​\n∥\nZ\nw\n−\nd\n∥\n​\n2\n​\n​\n+\n​\n2\n​\n​\nη\n​\n​\n∥\nw\n∥\n​\n2\n​\n​\n=\n​\n2\n​\n​\n1\n​\n​\nw\n​\nT\n​\n​\n(\nZ\n​\nT\n​\n​\nZ\n+\nη\nI\n)\nw\n−\n(\nZ\nd\n)\n​\nT\n​\n​\nw\nRecall that\nZ\nT\nZ\n=\nQ\nd\ni\na\ng\n(\nΛ\n1\n,\n…\n,\nΛ\nn\n)\nQ\nT\nZ^{T}Z=Q\\ \\text{diag}(\\Lambda_{1},\\ldots,\\Lambda_{n})\\ Q^T\nZ\n​\nT\n​\n​\nZ\n=\nQ\ndiag\n(\nΛ\n​\n1\n​\n​\n,\n…\n,\nΛ\n​\nn\n​\n​\n)\nQ\n​\nT\n​\n​\n. The solution to Tikhonov Regression is therefore\n(\nZ\nT\nZ\n+\nη\nI\n)\n−\n1\n(\nZ\nd\n)\n=\nQ\nd\ni\na\ng\n(\n1\nλ\n1\n+\nη\n,\n⋯\n,\n1\nλ\nn\n+\nη\n)\nQ\nT\n(\nZ\nd\n)\n(Z^{T}Z+\\eta I)^{-1}(Zd)=Q\\ \\text{diag}\\left(\\frac{1}{\\lambda_{1}+\\eta},\\cdots,\\frac{1}{\\lambda_{n}+\\eta}\\right)Q^T(Zd)\n(\nZ\n​\nT\n​\n​\nZ\n+\nη\nI\n)\n​\n−\n1\n​\n​\n(\nZ\nd\n)\n=\nQ\ndiag\n(\n​\nλ\n​\n1\n​\n​\n+\nη\n​\n​\n1\n​\n​\n,\n⋯\n,\n​\nλ\n​\nn\n​\n​\n+\nη\n​\n​\n1\n​\n​\n)\nQ\n​\nT\n​\n​\n(\nZ\nd\n)\nWe can think of regularization as a function which decays the largest eigenvalues, as follows:\nT\ni\nk\nh\no\nn\no\nv\nR\ne\ng\nu\nl\na\nr\ni\nz\ne\nd\nλ\ni\n=\n1\nλ\ni\n+\nη\n=\n1\nλ\ni\n(\n1\n−\n(\n1\n+\nλ\ni\n/\nη\n)\n−\n1\n)\n.\n\\text{Tikhonov Regularized } \\lambda_i = \\frac{1}{\\lambda_{i}+\\eta}=\\frac{1}{\\lambda_{i}}\\left(1-\\left(1+\\lambda_{i}/\\eta\\right)^{-1}\\right).\nTikhonov Regularized\nλ\n​\ni\n​\n​\n=\n​\nλ\n​\ni\n​\n​\n+\nη\n​\n​\n1\n​\n​\n=\n​\nλ\n​\ni\n​\n​\n​\n​\n1\n​\n​\n(\n1\n−\n(\n1\n+\nλ\n​\ni\n​\n​\n/\nη\n)\n​\n−\n1\n​\n​\n)\n.\nGradient descent can be seen as employing a similar decay, but with the decay rate\nG\nr\na\nd\ni\ne\nn\nt\nD\ne\ns\nc\ne\nn\nt\nR\ne\ng\nu\nl\na\nr\ni\nz\ne\nd\nλ\ni\n=\n1\nλ\ni\n(\n1\n−\n(\n1\n−\nα\nλ\ni\n)\nk\n)\n\\text{ Gradient Descent Regularized } \\lambda_i = \\frac{1}{\\lambda_i} \\left( 1-\\left(1-\\alpha\\lambda_{i}\\right)^{k} \\right)\nGradient Descent Regularized\nλ\n​\ni\n​\n​\n=\n​\nλ\n​\ni\n​\n​\n​\n​\n1\n​\n​\n(\n1\n−\n(\n1\n−\nα\nλ\n​\ni\n​\n​\n)\n​\nk\n​\n​\n)\ninstead. Note that this decay is dependent on the step-size.\nThis is true as we can write updates in matrix form as\n(\n1\n0\nα\n1\n)\n(\ny\ni\nk\n+\n1\nx\ni\nk\n+\n1\n)\n=\n(\nβ\nλ\ni\n0\n1\n)\n(\ny\ni\nk\nx\ni\nk\n)\n\\left(\\!\\!\\begin{array}{cc}\n  1 & 0\\\\\n  \\alpha & 1\n  \\end{array}\\!\\!\\right)\\Bigg(\\!\\!\\begin{array}{c}\n  y_{i}^{k+1}\\\\\n  x_{i}^{k+1}\n  \\end{array}\\!\\!\\Bigg)=\\left(\\!\\!\\begin{array}{cc}\n  \\beta & \\lambda_{i}\\\\\n  0 & 1\n  \\end{array}\\!\\!\\right)\\left(\\!\\!\\begin{array}{c}\n  y_{i}^{k}\\\\\n  x_{i}^{k}\n  \\end{array}\\!\\!\\right)\n(\n​\n1\n​\nα\n​\n​\n​\n0\n​\n1\n​\n​\n)\n(\n​\ny\n​\ni\n​\nk\n+\n1\n​\n​\n​\nx\n​\ni\n​\nk\n+\n1\n​\n​\n​\n​\n)\n=\n(\n​\nβ\n​\n0\n​\n​\n​\nλ\n​\ni\n​\n​\n​\n1\n​\n​\n)\n(\n​\ny\n​\ni\n​\nk\n​\n​\n​\nx\n​\ni\n​\nk\n​\n​\n​\n​\n)\nwhich implies, by inverting the matrix on the left,\n(\ny\ni\nk\n+\n1\nx\ni\nk\n+\n1\n)\n=\n(\nβ\nλ\ni\n−\nα\nβ\n1\n−\nα\nλ\ni\n)\n(\ny\ni\nk\nx\ni\nk\n)\n=\nR\nk\n+\n1\n(\nx\ni\n0\ny\ni\n0\n)\n\\Bigg(\\!\\!\\begin{array}{c}\n  y_{i}^{k+1}\\\\\n  x_{i}^{k+1}\n  \\end{array}\\!\\!\\Bigg)=\\left(\\!\\!\\begin{array}{cc}\n  \\beta & \\lambda_{i}\\\\\n  -\\alpha\\beta & 1-\\alpha\\lambda_{i}\n  \\end{array}\\!\\!\\right)\\left(\\!\\!\\begin{array}{c}\n  y_{i}^{k}\\\\\n  x_{i}^{k}\n  \\end{array}\\!\\!\\right)=R^{k+1}\\left(\\!\\!\\begin{array}{c}\n  x_{i}^{0}\\\\\n  y_{i}^{0}\n  \\end{array}\\!\\!\\right)\n(\n​\ny\n​\ni\n​\nk\n+\n1\n​\n​\n​\nx\n​\ni\n​\nk\n+\n1\n​\n​\n​\n​\n)\n=\n(\n​\nβ\n​\n−\nα\nβ\n​\n​\n​\nλ\n​\ni\n​\n​\n​\n1\n−\nα\nλ\n​\ni\n​\n​\n​\n​\n)\n(\n​\ny\n​\ni\n​\nk\n​\n​\n​\nx\n​\ni\n​\nk\n​\n​\n​\n​\n)\n=\nR\n​\nk\n+\n1\n​\n​\n(\n​\nx\n​\ni\n​\n0\n​\n​\n​\ny\n​\ni\n​\n0\n​\n​\n​\n​\n)\nWe can write out the convergence rates explicitly. The eigenvalues are\nσ\n1\n=\n1\n2\n(\n1\n−\nα\nλ\n+\nβ\n+\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n2\n−\n4\nβ\n)\nσ\n2\n=\n1\n2\n(\n1\n−\nα\nλ\n+\nβ\n−\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n2\n−\n4\nβ\n)\n\\begin{aligned}\n\\sigma_{1} & =\\frac{1}{2}\\left(1-\\alpha\\lambda+\\beta+\\sqrt{(-\\alpha\\lambda+\\beta+1)^{2}-4\\beta}\\right)\\\\[0.6em]\n\\sigma_{2} & =\\frac{1}{2}\\left(1-\\alpha\\lambda+\\beta-\\sqrt{(-\\alpha\\lambda+\\beta+1)^{2}-4\\beta}\\right)\n\\end{aligned}\n​\nσ\n​\n1\n​\n​\n​\nσ\n​\n2\n​\n​\n​\n​\n​\n=\n​\n2\n​\n​\n1\n​\n​\n(\n1\n−\nα\nλ\n+\nβ\n+\n√\n​\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n​\n2\n​\n​\n−\n4\nβ\n​\n​\n​\n)\n​\n=\n​\n2\n​\n​\n1\n​\n​\n(\n1\n−\nα\nλ\n+\nβ\n−\n√\n​\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n​\n2\n​\n​\n−\n4\nβ\n​\n​\n​\n)\n​\n​\nWhen the\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n2\n−\n4\nβ\n<\n0\n(-\\alpha\\lambda+\\beta+1)^{2}-4\\beta<0\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n​\n2\n​\n​\n−\n4\nβ\n<\n0\nis less than zero,\nthen the roots are complex and the convergence rate is\n∣\nσ\n1\n∣\n=\n∣\nσ\n2\n∣\n=\n(\n1\n−\nα\nλ\n+\nβ\n)\n2\n+\n∣\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n2\n−\n4\nβ\n∣\n=\n2\nβ\n\\begin{aligned}\n|\\sigma_{1}|=|\\sigma_{2}| & =\\sqrt{(1-\\alpha\\lambda+\\beta)^{2}+|(-\\alpha\\lambda+\\beta+1)^{2}-4\\beta|}=2\\sqrt{\\beta}\n\\end{aligned}\n​\n∣\nσ\n​\n1\n​\n​\n∣\n=\n∣\nσ\n​\n2\n​\n​\n∣\n​\n​\n​\n=\n√\n​\n(\n1\n−\nα\nλ\n+\nβ\n)\n​\n2\n​\n​\n+\n∣\n(\n−\nα\nλ\n+\nβ\n+\n1\n)\n​\n2\n​\n​\n−\n4\nβ\n∣\n​\n​\n​\n=\n2\n√\n​\nβ\n​\n​\n​\n​\n​\nWhich is, surprisingly, independent of the step-size or the eigenvalue\nα\nλ\n\\alpha\\lambda\nα\nλ\n. When the roots are real, the convergence rate is\nmax\n{\n∣\nσ\n1\n∣\n,\n∣\nσ\n2\n∣\n}\n=\n1\n2\nmax\n{\n∣\n1\n−\nα\nλ\ni\n+\nβ\n±\n(\n1\n−\nα\nλ\ni\n+\nβ\n)\n2\n−\n4\nβ\n∣\n}\n\\max\\{|\\sigma_{1}|,|\\sigma_{2}|\\}=\\tfrac{1}{2}\\max\\left\\{ |1-\\alpha\\lambda_{i}+\\beta\\pm\\sqrt{(1-\\alpha\\lambda_{i}+\\beta)^{2}-4\\beta}|\\right\\}\nmax\n{\n∣\nσ\n​\n1\n​\n​\n∣\n,\n∣\nσ\n​\n2\n​\n​\n∣\n}\n=\n​\n2\n​\n​\n1\n​\n​\nmax\n{\n∣\n1\n−\nα\nλ\n​\ni\n​\n​\n+\nβ\n±\n√\n​\n(\n1\n−\nα\nλ\n​\ni\n​\n​\n+\nβ\n)\n​\n2\n​\n​\n−\n4\nβ\n​\n​\n​\n∣\n}\nThis can be derived by reducing the inequalities for all 4 + 1 cases in the explicit form of the convergence rate above.\nWe must optimize over\nmin\nα\n,\nβ\nmax\n{\n∥\n(\nβ\nλ\ni\n−\nα\nβ\n1\n−\nα\nλ\ni\n)\n∥\n,\n…\n,\n∥\n(\nβ\nλ\nn\n−\nα\nβ\n1\n−\nα\nλ\nn\n)\n∥\n}\n.\n\\min_{\\alpha,\\beta}\\max\\left\\{ \\bigg\\| \\! \\left(\\begin{array}{cc}\n  \\beta & \\lambda_{i}\\\\\n  -\\alpha\\beta & 1-\\alpha\\lambda_{i}\n  \\end{array}\\right) \\! \\bigg\\|,\\ldots,\\bigg\\| \\! \\left(\\begin{array}{cc}\n  \\beta & \\lambda_{n}\\\\\n  -\\alpha\\beta & 1-\\alpha\\lambda_{n}\n  \\end{array}\\right)\\! \\bigg\\|\\right\\}.\n​\nα\n,\nβ\n​\nmin\n​\n​\nmax\n{\n​\n∥\n​\n∥\n​\n∥\n​\n∥\n​\n​\n(\n​\nβ\n​\n−\nα\nβ\n​\n​\n​\nλ\n​\ni\n​\n​\n​\n1\n−\nα\nλ\n​\ni\n​\n​\n​\n​\n)\n​\n∥\n​\n∥\n​\n∥\n​\n∥\n​\n​\n,\n…\n,\n​\n∥\n​\n∥\n​\n∥\n​\n∥\n​\n​\n(\n​\nβ\n​\n−\nα\nβ\n​\n​\n​\nλ\n​\nn\n​\n​\n​\n1\n−\nα\nλ\n​\nn\n​\n​\n​\n​\n)\n​\n∥\n​\n∥\n​\n∥\n​\n∥\n​\n​\n}\n.\n(\n∥\n⋅\n∥\n\\|\\cdot \\|\n∥\n⋅\n∥\nhere denotes the magnitude of the maximum eigenvalue), and occurs when the roots of the characteristic polynomial are repeated for the matrices corresponding to the extremal eigenvalues.\nThe above optimization problem is bounded from below by\n0\n0\n0\n, and vector of all\n1\n1\n1\n’s achieve this.\nThis can be written explicitly as\n[\nL\nG\n]\ni\nj\n=\n{\nd\ne\ng\nr\ne\ne\no\nf\nv\ne\nr\nt\ne\nx\ni\ni\n=\nj\n−\n1\ni\n≠\nj\n,\n(\ni\n,\nj\n)\no\nr\n(\nj\n,\ni\n)\n∈\nE\n0\no\nt\nh\ne\nr\nw\ni\ns\ne\n[L_{G}]_{ij}=\\begin{cases} \\text{degree of vertex }i & i=j\\\\ -1 & i\\neq j,(i,j)\\text{ or }(j,i)\\in E\\\\ 0 & \\text{otherwise} \\end{cases}\n[\nL\n​\nG\n​\n​\n]\n​\ni\nj\n​\n​\n=\n​\n⎩\n​\n⎪\n​\n⎨\n​\n⎪\n​\n⎧\n​\n​\n​\ndegree of vertex\ni\n​\n−\n1\n​\n0\n​\n​\n​\ni\n=\nj\n​\ni\n≠\nj\n,\n(\ni\n,\nj\n)\nor\n(\nj\n,\ni\n)\n∈\nE\n​\notherwise\n​\n​\nWe use the infinity norm to measure our error, similar results can be derived for the 1 and 2 norms.\nThe momentum iterations are\nz\nk\n+\n1\n=\nβ\nz\nk\n+\nA\nw\nk\n+\ne\nr\nr\no\nr\n(\nw\nk\n)\nw\nk\n+\n1\n=\nw\nk\n−\nα\nz\nk\n+\n1\n.\n\\begin{aligned}\n  z^{k+1}&=\\beta z^{k}+ A w^{k}  + \\text{error}(w^k) \\\\[0.4em]\n  w^{k+1}&=w^{k}-\\alpha z^{k+1}.\n  \\end{aligned}\n​\nz\n​\nk\n+\n1\n​\n​\n​\nw\n​\nk\n+\n1\n​\n​\n​\n​\n​\n=\nβ\nz\n​\nk\n​\n​\n+\nA\nw\n​\nk\n​\n​\n+\nerror\n(\nw\n​\nk\n​\n​\n)\n​\n=\nw\n​\nk\n​\n​\n−\nα\nz\n​\nk\n+\n1\n​\n​\n.\n​\n​\nwhich, after a change of variables, become\n(\n1\n0\nα\n1\n)\n(\ny\ni\nk\n+\n1\nx\ni\nk\n+\n1\n)\n=\n(\nβ\nλ\ni\n0\n1\n)\n(\ny\ni\nk\nx\ni\nk\n)\n+\n(\nϵ\ni\nk\n0\n)\n\\left(\\!\\!\\begin{array}{cc}\n1 & 0\\\\\n\\alpha & 1\n\\end{array}\\!\\!\\right)\\Bigg(\\!\\!\\begin{array}{c}\ny_{i}^{k+1}\\\\\nx_{i}^{k+1}\n\\end{array}\\!\\!\\Bigg)=\\left(\\!\\!\\begin{array}{cc}\n\\beta & \\lambda_{i}\\\\\n0 & 1\n\\end{array}\\!\\!\\right)\\left(\\!\\!\\begin{array}{c}\ny_{i}^{k}\\\\\nx_{i}^{k}\n\\end{array}\\!\\!\\right)+\\left(\\!\\!\\begin{array}{c}\n\\epsilon_{i}^{k}\\\\\n0\n\\end{array}\\!\\!\\right)\n(\n​\n1\n​\nα\n​\n​\n​\n0\n​\n1\n​\n​\n)\n(\n​\ny\n​\ni\n​\nk\n+\n1\n​\n​\n​\nx\n​\ni\n​\nk\n+\n1\n​\n​\n​\n​\n)\n=\n(\n​\nβ\n​\n0\n​\n​\n​\nλ\n​\ni\n​\n​\n​\n1\n​\n​\n)\n(\n​\ny\n​\ni\n​\nk\n​\n​\n​\nx\n​\ni\n​\nk\n​\n​\n​\n​\n)\n+\n(\n​\nϵ\n​\ni\n​\nk\n​\n​\n​\n0\n​\n​\n)\nInverting the\n2\n×\n2\n2 \\times 2\n2\n×\n2\nmatrix on the left, and applying the formula recursively yields the final solution.\nOn the 1D function\nf\n(\nx\n)\n=\nλ\n2\nx\n2\nf(x)=\\frac{\\lambda}{2}x^{2}\nf\n(\nx\n)\n=\n​\n2\n​\n​\nλ\n​\n​\nx\n​\n2\n​\n​\n, the objective value is\nE\nf\n(\nx\nk\n)\n=\nλ\n2\nE\n[\n(\nx\nk\n)\n2\n]\n=\nλ\n2\nE\n(\ne\n2\nT\nR\nk\n(\ny\n0\nx\n0\n)\n+\nϵ\nk\ne\n2\nT\n∑\ni\n=\n1\nk\nR\nk\n−\ni\n(\n1\n−\nα\n)\n)\n2\n=\nλ\n2\ne\n2\nT\nR\nk\n(\ny\n0\nx\n0\n)\n+\nλ\n2\nE\n(\nϵ\nk\ne\n2\nT\n∑\ni\n=\n1\nk\nR\nk\n−\ni\n(\n1\n−\nα\n)\n)\n2\n=\nλ\n2\ne\n2\nT\nR\nk\n(\ny\n0\nx\n0\n)\n+\nλ\n2\nE\n[\nϵ\nk\n]\n⋅\n∑\ni\n=\n1\nk\n(\ne\n2\nT\nR\nk\n−\ni\n(\n1\n−\nα\n)\n)\n2\n=\nλ\n2\ne\n2\nT\nR\nk\n(\ny\n0\nx\n0\n)\n+\nλ\nE\n[\nϵ\nk\n2\n⋅\n∑\ni\n=\n1\nk\nγ\ni\n2\n,\nγ\ni\n=\ne\n2\nT\nR\nk\n−\ni\n(\n1\n−\nα\n)\n\\begin{aligned}\n\\mathbf{E}f(x^{k})&=\\frac{\\lambda}{2}\\mathbf{E}[(x^{k})^{2}]\\\\&=\\frac{\\lambda}{2}\\mathbf{E}\\left(e_{2}^{T}R^{k}\\left(\\begin{array}{c}\ny^{0}\\\\\nx^{0}\n\\end{array}\\right)+\\epsilon^{k}e_{2}^{T}\\sum_{i=1}^{k}R^{k-i}\\left(\\begin{array}{c}\n1\\\\\n-\\alpha\n\\end{array}\\right)\\right)^{2}\\\\&=\\frac{\\lambda}{2}e_{2}^{T}R^{k}\\left(\\begin{array}{c}\ny^{0}\\\\\nx^{0}\n\\end{array}\\right)+\\frac{\\lambda}{2}\\mathbf{E}\\left(\\epsilon^{k}e_{2}^{T}\\sum_{i=1}^{k}R^{k-i}\\left(\\begin{array}{c}\n1\\\\\n-\\alpha\n\\end{array}\\right)\\right)^{2}\\\\&=\\frac{\\lambda}{2}e_{2}^{T}R^{k}\\left(\\begin{array}{c}\ny^{0}\\\\\nx^{0}\n\\end{array}\\right)+\\frac{\\lambda}{2}\\mathbf{E}[\\epsilon^{k}]\\,\\cdot\\,\\sum_{i=1}^{k}\\left(e_{2}^{T}R^{k-i}\\left(\\begin{array}{c}\n1\\\\\n-\\alpha\n\\end{array}\\right)\\right)^{2}\\\\&=\\frac{\\lambda}{2}e_{2}^{T}R^{k}\\left(\\begin{array}{c}\ny^{0}\\\\\nx^{0}\n\\end{array}\\right)+\\frac{\\lambda\\mathbf{E}[\\epsilon^{k}}{2}\\cdot\\sum_{i=1}^{k}\\gamma_{i}^{2}, \\qquad \\gamma_i = e_{2}^{T}R^{k-i}\\left(\\begin{array}{c}\n1\\\\\n-\\alpha\n\\end{array}\\right)\n\\end{aligned}\n​\nE\nf\n(\nx\n​\nk\n​\n​\n)\n​\n​\n​\n​\n​\n​\n​\n=\n​\n2\n​\n​\nλ\n​\n​\nE\n[\n(\nx\n​\nk\n​\n​\n)\n​\n2\n​\n​\n]\n​\n=\n​\n2\n​\n​\nλ\n​\n​\nE\n(\ne\n​\n2\n​\nT\n​\n​\nR\n​\nk\n​\n​\n(\n​\ny\n​\n0\n​\n​\n​\nx\n​\n0\n​\n​\n​\n​\n)\n+\nϵ\n​\nk\n​\n​\ne\n​\n2\n​\nT\n​\n​\n​\ni\n=\n1\n​\n∑\n​\nk\n​\n​\nR\n​\nk\n−\ni\n​\n​\n(\n​\n1\n​\n−\nα\n​\n​\n)\n)\n​\n2\n​\n​\n​\n=\n​\n2\n​\n​\nλ\n​\n​\ne\n​\n2\n​\nT\n​\n​\nR\n​\nk\n​\n​\n(\n​\ny\n​\n0\n​\n​\n​\nx\n​\n0\n​\n​\n​\n​\n)\n+\n​\n2\n​\n​\nλ\n​\n​\nE\n(\nϵ\n​\nk\n​\n​\ne\n​\n2\n​\nT\n​\n​\n​\ni\n=\n1\n​\n∑\n​\nk\n​\n​\nR\n​\nk\n−\ni\n​\n​\n(\n​\n1\n​\n−\nα\n​\n​\n)\n)\n​\n2\n​\n​\n​\n=\n​\n2\n​\n​\nλ\n​\n​\ne\n​\n2\n​\nT\n​\n​\nR\n​\nk\n​\n​\n(\n​\ny\n​\n0\n​\n​\n​\nx\n​\n0\n​\n​\n​\n​\n)\n+\n​\n2\n​\n​\nλ\n​\n​\nE\n[\nϵ\n​\nk\n​\n​\n]\n⋅\n​\ni\n=\n1\n​\n∑\n​\nk\n​\n​\n(\ne\n​\n2\n​\nT\n​\n​\nR\n​\nk\n−\ni\n​\n​\n(\n​\n1\n​\n−\nα\n​\n​\n)\n)\n​\n2\n​\n​\n​\n=\n​\n2\n​\n​\nλ\n​\n​\ne\n​\n2\n​\nT\n​\n​\nR\n​\nk\n​\n​\n(\n​\ny\n​\n0\n​\n​\n​\nx\n​\n0\n​\n​\n​\n​\n)\n+\n​\n2\n​\n​\nλ\nE\n[\nϵ\n​\nk\n​\n​\n​\n​\n⋅\n​\ni\n=\n1\n​\n∑\n​\nk\n​\n​\nγ\n​\ni\n​\n2\n​\n​\n,\nγ\n​\ni\n​\n​\n=\ne\n​\n2\n​\nT\n​\n​\nR\n​\nk\n−\ni\n​\n​\n(\n​\n1\n​\n−\nα\n​\n​\n)\n​\n​\nThe third inequality uses the fact that\nE\nϵ\nk\n=\n0\n\\mathbf{E} \\epsilon^k = 0\nE\nϵ\n​\nk\n​\n​\n=\n0\nand the fourth uses the fact they are uncorrelated.\nDistill\nis dedicated to clear explanations of machine learning\nAbout\nSubmit\nPrize\nArchive\nRSS\nGitHub\nTwitter\nISSN 2476-0757",
  "metadata": {
    "title": "Why Momentum Really Works",
    "description": null,
    "keywords": null,
    "author": null,
    "og:type": "article",
    "og:title": "Why Momentum Really Works",
    "og:description": "We often think of optimization with momentum as a ball rolling down a hill. This isn't wrong, but there is much more to the story.",
    "og:url": "http://distill.pub/2017/momentum",
    "og:image": "http://distill.pub/2017/momentum/thumbnail.jpg",
    "og:locale": "en_US",
    "og:site_name": "Distill",
    "twitter:card": "summary_large_image",
    "twitter:title": "Why Momentum Really Works",
    "twitter:description": "We often think of optimization with momentum as a ball rolling down a hill. This isn't wrong, but there is much more to the story.",
    "twitter:url": "http://distill.pub/2017/momentum",
    "twitter:image": "http://distill.pub/2017/momentum/thumbnail.jpg",
    "twitter:image:width": "560",
    "twitter:image:height": "295",
    "article:published": "2017-04-04",
    "article:created": "Tue Apr 04 2017 16:00:00 GMT-0400 (Eastern Daylight Time)",
    "article:modified": "Wed Jul 11 2018 04:30:17 GMT-0400 (Eastern Daylight Time)",
    "article:author": "Gabriel Goh"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/d6f0f4773deb/code/body-000.txt",
      "chars": 86,
      "language": "text"
    },
    {
      "file": "attachments/d6f0f4773deb/code/body-001.txt",
      "chars": 205,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://distill.pub/",
      "label": "Distill",
      "type": "a"
    },
    {
      "href": "https://distill.pub/about/",
      "label": "About",
      "type": "a"
    },
    {
      "href": "https://distill.pub/prize/",
      "label": "Prize",
      "type": "a"
    },
    {
      "href": "https://distill.pub/journal/",
      "label": "Submit",
      "type": "a"
    },
    {
      "href": "http://gabgoh.github.io/",
      "label": "Gabriel Goh",
      "type": "a"
    },
    {
      "href": "http://math.ucdavis.edu/",
      "label": "UC Davis",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--momentum/issues/34",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--momentum/issues/29",
      "label": "Reviewer A - Matt Hoffman",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--momentum/issues/51",
      "label": "Discussion with User derifatives",
      "type": "a"
    },
    {
      "href": "http://www.jmlr.org/proceedings/papers/v28/sutskever13.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://www.researchgate.net/profile/Boris_Polyak2/publication/243648538_Some_methods_of_speeding_up_the_convergence_of_iteration_methods/links/5666fa3808ae34c89a01fda1.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.1016/0041-5553(64)90137-5",
      "label": "DOI: 10.1016/0041-5553(64)90137-5",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.1007/978-3-0348-7224-9_2",
      "label": "DOI: 10.1007/978-3-0348-7224-9_2",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1408.3595.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.1007/978-1-4419-8853-9",
      "label": "DOI: 10.1007/978-1-4419-8853-9",
      "type": "a"
    },
    {
      "href": "http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.452.7280&rep=rep1&type=pdf",
      "label": "[link]",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.1162/089976698300017746",
      "label": "DOI: 10.1162/089976698300017746",
      "type": "a"
    },
    {
      "href": "http://www.iro.umontreal.ca/~bengioy/talks/DL-Tutorial-NIPS2015.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1204.3982.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.1007/s10208-013-9150-3",
      "label": "DOI: 10.1007/s10208-013-9150-3",
      "type": "a"
    },
    {
      "href": "http://people.math.carleton.ca/~williams/papers/pdf/175.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.2307/2691246",
      "label": "DOI: 10.2307/2691246",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1504.01577.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://pdfs.semanticscholar.org/735d/4220d5579cc6afe956d9f6ea501a96ae99e2.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.1016/s0893-6080(98)00116-6",
      "label": "DOI: 10.1016/s0893-6080(98)00116-6",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1611.03530.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://arxiv.org/pdf/1503.01243.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://blog.mrtz.org/2013/09/07/the-zen-of-gradient-descent.html",
      "label": "[HTML]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1506.08187.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1604.06543.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1407.1537.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "http://folk.uib.no/ssu029/Pdf_file/Nesterov08.pdf",
      "label": "[PDF]",
      "type": "a"
    },
    {
      "href": "https://doi.org/10.1007/s10107-006-0089-x",
      "label": "DOI: 10.1007/s10107-006-0089-x",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--momentum/compare/95506b079372cee3aa7fbc9bd29ee078aaff12e7...691048b9d00b4b49b830c602b970755781df332c",
      "label": "View all changes",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--momentum/issues/new",
      "label": "create an issue on GitHub",
      "type": "a"
    },
    {
      "href": "https://creativecommons.org/licenses/by/2.0/",
      "label": "CC-BY 2.0",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub/post--momentum",
      "label": "source available on GitHub",
      "type": "a"
    },
    {
      "href": "http://distill.pub/about/",
      "label": "About",
      "type": "a"
    },
    {
      "href": "http://distill.pub/journal/",
      "label": "Submit",
      "type": "a"
    },
    {
      "href": "http://distill.pub/prize/",
      "label": "Prize",
      "type": "a"
    },
    {
      "href": "http://distill.pub/archive/",
      "label": "Archive",
      "type": "a"
    },
    {
      "href": "http://distill.pub/rss.xml",
      "label": "RSS",
      "type": "a"
    },
    {
      "href": "https://github.com/distillpub",
      "label": "GitHub",
      "type": "a"
    },
    {
      "href": "https://twitter.com/distillpub",
      "label": "Twitter",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:54:15Z",
  "source_url": "https://distill.pub/2017/momentum/",
  "papers": [
    {
      "status": "http_404",
      "url": "http://www.jmlr.org/proceedings/papers/v28/sutskever13.pdf",
      "retried_at": "2026-10-20T17:47:34Z"
    },
    {
      "status": "http_403",
      "url": "https://www.researchgate.net/profile/Boris_Polyak2/publication/243648538_Some_methods_of_speeding_up_the_convergence_of_iteration_methods/links/5666fa3808ae34c89a01fda1.pdf",
      "retried_at": "2026-10-20T17:47:34Z"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1408.3595.pdf",
      "bytes": 814784,
      "reused": false,
      "local_path": "papers/repair/92d6213ac3ecbe592195.pdf"
    },
    {
      "status": "invalid_file_type",
      "url": "http://www.iro.umontreal.ca/~bengioy/talks/DL-Tutorial-NIPS2015.pdf",
      "content_type": "text/html; charset=utf-8",
      "retried_at": "2026-10-20T17:47:34Z"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1204.3982.pdf",
      "bytes": 724363,
      "reused": false,
      "local_path": "papers/repair/66c40fc61f2a29f3f5c8.pdf"
    },
    {
      "status": "ok",
      "url": "http://people.math.carleton.ca/~williams/papers/pdf/175.pdf",
      "bytes": 49632,
      "reused": false,
      "local_path": "papers/repair/772c4f4b669ec83ea909.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1504.01577.pdf",
      "bytes": 624288,
      "reused": false,
      "local_path": "papers/repair/e3cd48a10bb7d5c96fae.pdf"
    },
    {
      "status": "http_202",
      "url": "https://pdfs.semanticscholar.org/735d/4220d5579cc6afe956d9f6ea501a96ae99e2.pdf",
      "retried_at": "2026-10-20T17:47:35Z"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1611.03530.pdf",
      "bytes": 403563,
      "reused": false,
      "local_path": "papers/repair/47357558334d5cddb35e.pdf"
    },
    {
      "status": "ok",
      "url": "http://arxiv.org/pdf/1503.01243.pdf",
      "bytes": 1161056,
      "reused": false,
      "local_path": "papers/repair/67260f21364219422fac.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1506.08187.pdf",
      "bytes": 198231,
      "reused": false,
      "local_path": "papers/repair/4ae96e8c784087e02fb1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1604.06543.pdf",
      "bytes": 1441729,
      "reused": false,
      "local_path": "papers/repair/82cb39f81961f9a1aa60.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1407.1537.pdf",
      "bytes": 594243,
      "reused": false,
      "local_path": "papers/repair/fbad3c828e2ce70c64f6.pdf"
    },
    {
      "status": "http_404",
      "url": "http://folk.uib.no/ssu029/Pdf_file/Nesterov08.pdf",
      "retried_at": "2026-10-20T17:48:18Z"
    }
  ],
  "actual_video_files": [],
  "body_chars": 72006,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "http_202；http_403；http_404；invalid_file_type",
  "raw_html": "attachments/d6f0f4773deb/pages/source.html",
  "local_html": "attachments/d6f0f4773deb/pages/article.html",
  "body_base_path": "attachments/d6f0f4773deb/pages/",
  "body_markdown_path": "attachments/d6f0f4773deb/pages/article.md"
}