{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Lilian Weng / Lil'Log",
  "title_listed": "Policy Gradient Algorithms",
  "keywords": "策略梯度,PG,强化学习",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/",
  "publish_date": "2018-04",
  "corpus_stage": "强化学习/对齐",
  "notes": "策略梯度算法大全（REINFORCE、PPO等）",
  "id": "ebc73787cbe5",
  "collected_at": "2026-10-02T14:13:21",
  "title": "Policy Gradient Algorithms | Lil'Log",
  "body_markdown": "[Updated on 2018-06-30: add two new policy gradient methods,\nSAC\nand\nD4PG\n.]\n[Updated on 2018-09-30: add a new policy gradient method,\nTD3\n.]\n[Updated on 2019-02-09: add\nSAC with automatically adjusted temperature\n].\n[Updated on 2019-06-26: Thanks to Chanseok, we have a version of this post in\nKorean\n].\n[Updated on 2019-09-12: add a new policy gradient method\nSVPG\n.]\n[Updated on 2019-12-22: add a new policy gradient method\nIMPALA\n.]\n[Updated on 2020-10-15: add a new policy gradient method\nPPG\n& some new discussion in\nPPO\n.]\n[Updated on 2021-09-19: Thanks to Wenhao & 爱吃猫的鱼, we have this post in\nChinese1\n&\nChinese2\n].\nWhat is Policy Gradient\n#\nPolicy gradient is an approach to solve reinforcement learning problems. If you haven’t looked into the field of reinforcement learning, please first read the section\n“A (Long) Peek into Reinforcement Learning » Key Concepts”\nfor the problem definition and key concepts.\nNotations\n#\nHere is a list of notations to help you read through equations in the post easily.\nSymbol\nMeaning\n$s \\in \\mathcal{S}$\nStates.\n$a \\in \\mathcal{A}$\nActions.\n$r \\in \\mathcal{R}$\nRewards.\n$S_t, A_t, R_t$\nState, action, and reward at time step $t$ of one trajectory. I may occasionally use $s_t, a_t, r_t$ as well.\n$\\gamma$\nDiscount factor; penalty to uncertainty of future rewards; $0<\\gamma \\leq 1$.\n$G_t$\nReturn; or discounted future reward; $G_t = \\sum_{k=0}^{\\infty} \\gamma^k R_{t+k+1}$.\n$P(s’, r \\vert s, a)$\nTransition probability of getting to the next state $s’$ from the current state $s$ with action $a$ and reward $r$.\n$\\pi(a \\vert s)$\nStochastic policy (agent behavior strategy); $\\pi_\\theta(.)$ is a policy parameterized by $\\theta$.\n$\\mu(s)$\nDeterministic policy; we can also label this as $\\pi(s)$, but using a different letter gives better distinction so that we can easily tell when the policy is stochastic or deterministic without further explanation. Either $\\pi$ or $\\mu$ is what a reinforcement learning algorithm aims to learn.\n$V(s)$\nState-value function measures the expected return of state $s$; $V_w(.)$ is a value function parameterized by $w$.\n$V^\\pi(s)$\nThe value of state $s$ when we follow a policy $\\pi$; $V^\\pi (s) = \\mathbb{E}_{a\\sim \\pi} [G_t \\vert S_t = s]$.\n$Q(s, a)$\nAction-value function is similar to $V(s)$, but it assesses the expected return of a pair of state and action $(s, a)$; $Q_w(.)$ is a action value function parameterized by $w$.\n$Q^\\pi(s, a)$\nSimilar to $V^\\pi(.)$, the value of (state, action) pair when we follow a policy $\\pi$; $Q^\\pi(s, a) = \\mathbb{E}_{a\\sim \\pi} [G_t \\vert S_t = s, A_t = a]$.\n$A(s, a)$\nAdvantage function, $A(s, a) = Q(s, a) - V(s)$; it can be considered as another version of Q-value with lower variance by taking the state-value off as the baseline.\nPolicy Gradient\n#\nThe goal of reinforcement learning is to find an optimal behavior strategy for the agent to obtain optimal rewards. The\npolicy gradient\nmethods target at modeling and optimizing the policy directly. The policy is usually modeled with a parameterized function respect to $\\theta$, $\\pi_\\theta(a \\vert s)$. The value of the reward (objective) function depends on this policy and then various algorithms can be applied to optimize $\\theta$ for the best reward.\nThe reward function is defined as:\n$$\nJ(\\theta) \n= \\sum_{s \\in \\mathcal{S}} d^\\pi(s) V^\\pi(s) \n= \\sum_{s \\in \\mathcal{S}} d^\\pi(s) \\sum_{a \\in \\mathcal{A}} \\pi_\\theta(a \\vert s) Q^\\pi(s, a)\n$$\nwhere $d^\\pi(s)$ is the stationary distribution of Markov chain for $\\pi_\\theta$ (on-policy state distribution under $\\pi$). For simplicity, the parameter $\\theta$ would be omitted for the policy $\\pi_\\theta$ when the policy is present in the subscript of other functions; for example, $d^{\\pi}$ and $Q^\\pi$ should be $d^{\\pi_\\theta}$ and $Q^{\\pi_\\theta}$ if written in full.\nImagine that you can travel along the Markov chain’s states forever, and eventually, as the time progresses, the probability of you ending up with one state becomes unchanged — this is the stationary probability for $\\pi_\\theta$. $d^\\pi(s) = \\lim_{t \\to \\infty} P(s_t = s \\vert s_0, \\pi_\\theta)$ is the probability that $s_t=s$ when starting from $s_0$ and following policy $\\pi_\\theta$ for t steps. Actually, the existence of the stationary distribution of Markov chain is one main reason for why PageRank algorithm works. If you want to read more, check\nthis\n.\nIt is natural to expect policy-based methods are more useful in the continuous space. Because there is an infinite number of actions and (or) states to estimate the values for and hence value-based approaches are way too expensive computationally in the continuous space. For example, in\ngeneralized policy iteration\n, the policy improvement step $\\arg\\max_{a \\in \\mathcal{A}} Q^\\pi(s, a)$ requires a full scan of the action space, suffering from the\ncurse of dimensionality\n.\nUsing\ngradient ascent\n, we can move $\\theta$ toward the direction suggested by the gradient $\\nabla_\\theta J(\\theta)$ to find the best $\\theta$ for $\\pi_\\theta$ that produces the highest return.\nPolicy Gradient Theorem\n#\nComputing the gradient $\\nabla_\\theta J(\\theta)$ is tricky because it depends on both the action selection (directly determined by $\\pi_\\theta$) and the stationary distribution of states following the target selection behavior (indirectly determined by $\\pi_\\theta$). Given that the environment is generally unknown, it is difficult to estimate the effect on the state distribution by a policy update.\nLuckily, the\npolicy gradient theorem\ncomes to save the world! Woohoo! It provides a nice reformation of the derivative of the objective function to not involve the derivative of the state distribution $d^\\pi(.)$ and simplify the gradient computation $\\nabla_\\theta J(\\theta)$ a lot.\n$$\n\\begin{aligned}\n\\nabla_\\theta J(\\theta) \n&= \\nabla_\\theta \\sum_{s \\in \\mathcal{S}} d^\\pi(s) \\sum_{a \\in \\mathcal{A}} Q^\\pi(s, a) \\pi_\\theta(a \\vert s) \\\\\n&\\propto \\sum_{s \\in \\mathcal{S}} d^\\pi(s) \\sum_{a \\in \\mathcal{A}} Q^\\pi(s, a) \\nabla_\\theta \\pi_\\theta(a \\vert s) \n\\end{aligned}\n$$\nProof of Policy Gradient Theorem\n#\nThis session is pretty dense, as it is the time for us to go through the proof (\nSutton & Barto, 2017\n; Sec. 13.1) and figure out why the policy gradient theorem is correct.\nWe first start with the derivative of the state value function:\n$$\n\\begin{aligned}\n& \\nabla_\\theta V^\\pi(s) \\\\\n=& \\nabla_\\theta \\Big(\\sum_{a \\in \\mathcal{A}} \\pi_\\theta(a \\vert s)Q^\\pi(s, a) \\Big) & \\\\\n=& \\sum_{a \\in \\mathcal{A}} \\Big( \\nabla_\\theta \\pi_\\theta(a \\vert s)Q^\\pi(s, a) + \\pi_\\theta(a \\vert s) \\color{red}{\\nabla_\\theta Q^\\pi(s, a)} \\Big) & \\scriptstyle{\\text{; Derivative product rule.}} \\\\\n=& \\sum_{a \\in \\mathcal{A}} \\Big( \\nabla_\\theta \\pi_\\theta(a \\vert s)Q^\\pi(s, a) + \\pi_\\theta(a \\vert s) \\color{red}{\\nabla_\\theta \\sum_{s', r} P(s',r \\vert s,a)(r + V^\\pi(s'))} \\Big) & \\scriptstyle{\\text{; Extend } Q^\\pi \\text{ with future state value.}} \\\\\n=& \\sum_{a \\in \\mathcal{A}} \\Big( \\nabla_\\theta \\pi_\\theta(a \\vert s)Q^\\pi(s, a) + \\pi_\\theta(a \\vert s) \\color{red}{\\sum_{s', r} P(s',r \\vert s,a) \\nabla_\\theta V^\\pi(s')} \\Big) & \\scriptstyle{P(s',r \\vert s,a) \\text{ or } r \\text{ is not a func of }\\theta}\\\\\n=& \\sum_{a \\in \\mathcal{A}} \\Big( \\nabla_\\theta \\pi_\\theta(a \\vert s)Q^\\pi(s, a) + \\pi_\\theta(a \\vert s) \\color{red}{\\sum_{s'} P(s' \\vert s,a) \\nabla_\\theta V^\\pi(s')} \\Big) & \\scriptstyle{\\text{; Because }  P(s' \\vert s, a) = \\sum_r P(s', r \\vert s, a)}\n\\end{aligned}\n$$\nNow we have:\n$$\n\\color{red}{\\nabla_\\theta V^\\pi(s)} \n= \\sum_{a \\in \\mathcal{A}} \\Big( \\nabla_\\theta \\pi_\\theta(a \\vert s)Q^\\pi(s, a) + \\pi_\\theta(a \\vert s) \\sum_{s'} P(s' \\vert s,a) \\color{red}{\\nabla_\\theta V^\\pi(s')} \\Big)\n$$\nThis equation has a nice recursive form (see the red parts!) and the future state value function $V^\\pi(s’)$ can be repeated unrolled by following the same equation.\nLet’s consider the following visitation sequence and label the probability of transitioning from state s to state x with policy $\\pi_\\theta$ after k step as $\\rho^\\pi(s \\to x, k)$.\n$$\ns \\xrightarrow[]{a \\sim \\pi_\\theta(.\\vert s)} s' \\xrightarrow[]{a \\sim \\pi_\\theta(.\\vert s')} s'' \\xrightarrow[]{a \\sim \\pi_\\theta(.\\vert s'')} \\dots\n$$\nWhen k = 0: $\\rho^\\pi(s \\to s, k=0) = 1$.\nWhen k = 1, we scan through all possible actions and sum up the transition probabilities to the target state: $\\rho^\\pi(s \\to s’, k=1) = \\sum_a \\pi_\\theta(a \\vert s) P(s’ \\vert s, a)$.\nImagine that the goal is to go from state s to x after k+1 steps while following policy $\\pi_\\theta$. We can first travel from s to a middle point s’ (any state can be a middle point, $s’ \\in \\mathcal{S}$) after k steps and then go to the final state x during the last step. In this way, we are able to update the visitation probability recursively: $\\rho^\\pi(s \\to x, k+1) = \\sum_{s’} \\rho^\\pi(s \\to s’, k) \\rho^\\pi(s’ \\to x, 1)$.\nThen we go back to unroll the recursive representation of $\\nabla_\\theta V^\\pi(s)$! Let $\\phi(s) = \\sum_{a \\in \\mathcal{A}} \\nabla_\\theta \\pi_\\theta(a \\vert s)Q^\\pi(s, a)$ to simplify the maths. If we keep on extending $\\nabla_\\theta V^\\pi(.)$ infinitely, it is easy to find out that we can transition from the starting state s to any state after any number of steps in this unrolling process and by summing up all the visitation probabilities, we get $\\nabla_\\theta V^\\pi(s)$!\n$$\n\\begin{aligned}\n& \\color{red}{\\nabla_\\theta V^\\pi(s)} \\\\\n=& \\phi(s) + \\sum_a \\pi_\\theta(a \\vert s) \\sum_{s'} P(s' \\vert s,a) \\color{red}{\\nabla_\\theta V^\\pi(s')} \\\\\n=& \\phi(s) + \\sum_{s'} \\sum_a \\pi_\\theta(a \\vert s) P(s' \\vert s,a) \\color{red}{\\nabla_\\theta V^\\pi(s')} \\\\\n=& \\phi(s) + \\sum_{s'} \\rho^\\pi(s \\to s', 1) \\color{red}{\\nabla_\\theta V^\\pi(s')} \\\\\n=& \\phi(s) + \\sum_{s'} \\rho^\\pi(s \\to s', 1) \\color{red}{\\sum_{a \\in \\mathcal{A}} \\Big( \\nabla_\\theta \\pi_\\theta(a \\vert s')Q^\\pi(s', a) + \\pi_\\theta(a \\vert s') \\sum_{s'} P(s'' \\vert s',a) \\nabla_\\theta V^\\pi(s'') \\Big)} \\\\\n=& \\phi(s) + \\sum_{s'} \\rho^\\pi(s \\to s', 1) \\color{red}{[ \\phi(s') + \\sum_{s''} \\rho^\\pi(s' \\to s'', 1) \\nabla_\\theta V^\\pi(s'')]} \\\\\n=& \\phi(s) + \\sum_{s'} \\rho^\\pi(s \\to s', 1) \\phi(s') + \\sum_{s''} \\rho^\\pi(s \\to s'', 2)\\color{red}{\\nabla_\\theta V^\\pi(s'')} \\scriptstyle{\\text{ ; Consider }s'\\text{ as the middle point for }s \\to s''}\\\\\n=& \\phi(s) + \\sum_{s'} \\rho^\\pi(s \\to s', 1) \\phi(s') + \\sum_{s''} \\rho^\\pi(s \\to s'', 2)\\phi(s'') + \\sum_{s'''} \\rho^\\pi(s \\to s''', 3)\\color{red}{\\nabla_\\theta V^\\pi(s''')} \\\\\n=& \\dots \\scriptstyle{\\text{; Repeatedly unrolling the part of }\\nabla_\\theta V^\\pi(.)} \\\\\n=& \\sum_{x\\in\\mathcal{S}}\\sum_{k=0}^\\infty \\rho^\\pi(s \\to x, k) \\phi(x)\n\\end{aligned}\n$$\nThe nice rewriting above allows us to exclude the derivative of Q-value function, $\\nabla_\\theta Q^\\pi(s, a)$. By plugging it into the objective function $J(\\theta)$, we are getting the following:\n$$\n\\begin{aligned}\n\\nabla_\\theta J(\\theta)\n&= \\nabla_\\theta V^\\pi(s_0) & \\scriptstyle{\\text{; Starting from a random state } s_0} \\\\\n&= \\sum_{s}\\color{blue}{\\sum_{k=0}^\\infty \\rho^\\pi(s_0 \\to s, k)} \\phi(s) &\\scriptstyle{\\text{; Let }\\color{blue}{\\eta(s) = \\sum_{k=0}^\\infty \\rho^\\pi(s_0 \\to s, k)}} \\\\\n&= \\sum_{s}\\eta(s) \\phi(s) & \\\\\n&= \\Big( {\\sum_s \\eta(s)} \\Big)\\sum_{s}\\frac{\\eta(s)}{\\sum_s \\eta(s)} \\phi(s) & \\scriptstyle{\\text{; Normalize } \\eta(s), s\\in\\mathcal{S} \\text{ to be a probability distribution.}}\\\\\n&\\propto \\sum_s \\frac{\\eta(s)}{\\sum_s \\eta(s)} \\phi(s) & \\scriptstyle{\\sum_s \\eta(s)\\text{  is a constant}} \\\\\n&= \\sum_s d^\\pi(s) \\sum_a \\nabla_\\theta \\pi_\\theta(a \\vert s)Q^\\pi(s, a) & \\scriptstyle{d^\\pi(s) = \\frac{\\eta(s)}{\\sum_s \\eta(s)}\\text{ is stationary distribution.}}\n\\end{aligned}\n$$\nIn the episodic case, the constant of proportionality ($\\sum_s \\eta(s)$) is the average length of an episode; in the continuing case, it is 1 (\nSutton & Barto, 2017\n; Sec. 13.2). The gradient can be further written as:\n$$\n\\begin{aligned}\n\\nabla_\\theta J(\\theta) \n&\\propto \\sum_{s \\in \\mathcal{S}} d^\\pi(s) \\sum_{a \\in \\mathcal{A}} Q^\\pi(s, a) \\nabla_\\theta \\pi_\\theta(a \\vert s)  &\\\\\n&= \\sum_{s \\in \\mathcal{S}} d^\\pi(s) \\sum_{a \\in \\mathcal{A}} \\pi_\\theta(a \\vert s) Q^\\pi(s, a) \\frac{\\nabla_\\theta \\pi_\\theta(a \\vert s)}{\\pi_\\theta(a \\vert s)} &\\\\\n&= \\mathbb{E}_\\pi [Q^\\pi(s, a) \\nabla_\\theta \\ln \\pi_\\theta(a \\vert s)] & \\scriptstyle{\\text{; Because } (\\ln x)' = 1/x}\n\\end{aligned}\n$$\nWhere $\\mathbb{E}_\\pi$ refers to $\\mathbb{E}_{s \\sim d_\\pi, a \\sim \\pi_\\theta}$ when both state and action distributions follow the policy $\\pi_\\theta$ (on policy).\nThe policy gradient theorem lays the theoretical foundation for various policy gradient algorithms. This vanilla policy gradient update has no bias but high variance. Many following algorithms were proposed to reduce the variance while keeping the bias unchanged.\n$$\n\\nabla_\\theta J(\\theta)  = \\mathbb{E}_\\pi [Q^\\pi(s, a) \\nabla_\\theta \\ln \\pi_\\theta(a \\vert s)]\n$$\nHere is a nice summary of a general form of policy gradient methods borrowed from the\nGAE\n(general advantage estimation) paper (\nSchulman et al., 2016\n) and this\npost\nthoroughly discussed several components in GAE , highly recommended.\nA general form of policy gradient methods. (Image source:\nSchulman et al., 2016\n)\nPolicy Gradient Algorithms\n#\nTons of policy gradient algorithms have been proposed during recent years and there is no way for me to exhaust them. I’m introducing some of them that I happened to know and read about.\nREINFORCE\n#\nREINFORCE\n(Monte-Carlo policy gradient) relies on an estimated return by\nMonte-Carlo\nmethods using episode samples to update the policy parameter $\\theta$. REINFORCE works because the expectation of the sample gradient is equal to the actual gradient:\n$$\n\\begin{aligned}\n\\nabla_\\theta J(\\theta)\n&= \\mathbb{E}_\\pi [Q^\\pi(s, a) \\nabla_\\theta \\ln \\pi_\\theta(a \\vert s)] & \\\\\n&= \\mathbb{E}_\\pi [G_t \\nabla_\\theta \\ln \\pi_\\theta(A_t \\vert S_t)] & \\scriptstyle{\\text{; Because } Q^\\pi(S_t, A_t) = \\mathbb{E}_\\pi[G_t \\vert S_t, A_t]}\n\\end{aligned}\n$$\nTherefore we are able to measure $G_t$ from real sample trajectories and use that to update our policy gradient. It relies on a full trajectory and that’s why it is a Monte-Carlo method.\nThe process is pretty straightforward:\nInitialize the policy parameter $\\theta$ at random.\nGenerate one trajectory on policy $\\pi_\\theta$: $S_1, A_1, R_2, S_2, A_2, \\dots, S_T$.\nFor t=1, 2, … , T:\nEstimate the the return $G_t$;\nUpdate policy parameters: $\\theta \\leftarrow \\theta + \\alpha \\gamma^t G_t \\nabla_\\theta \\ln \\pi_\\theta(A_t \\vert S_t)$\nA widely used variation of REINFORCE is to subtract a baseline value from the return $G_t$ to\nreduce the variance of gradient estimation while keeping the bias unchanged\n(Remember we always want to do this when possible). For example, a common baseline is to subtract state-value from action-value, and if applied, we would use advantage $A(s, a) = Q(s, a) - V(s)$ in the gradient ascent update. This\npost\nnicely explained why a baseline works for reducing the variance, in addition to a set of fundamentals of policy gradient.\nActor-Critic\n#\nTwo main components in policy gradient are the policy model and the value function. It makes a lot of sense to learn the value function in addition to the policy, since knowing the value function can assist the policy update, such as by reducing gradient variance in vanilla policy gradients, and that is exactly what the\nActor-Critic\nmethod does.\nActor-critic methods consist of two models, which may optionally share parameters:\nCritic\nupdates the value function parameters w and depending on the algorithm it could be action-value $Q_w(a \\vert s)$ or state-value $V_w(s)$.\nActor\nupdates the policy parameters $\\theta$ for $\\pi_\\theta(a \\vert s)$, in the direction suggested by the critic.\nLet’s see how it works in a simple action-value actor-critic algorithm.\nInitialize $s, \\theta, w$ at random; sample $a \\sim \\pi_\\theta(a \\vert s)$.\nFor $t = 1 \\dots T$:\nSample reward $r_t \\sim R(s, a)$ and next state $s’ \\sim P(s’ \\vert s, a)$;\nThen sample the next action $a’ \\sim \\pi_\\theta(a’ \\vert s’)$;\nUpdate the policy parameters: $\\theta \\leftarrow \\theta + \\alpha_\\theta Q_w(s, a) \\nabla_\\theta \\ln \\pi_\\theta(a \\vert s)$;\nCompute the correction (TD error) for action-value at time t:\n$\\delta_t = r_t + \\gamma Q_w(s’, a’) - Q_w(s, a)$\nand use it to update the parameters of action-value function:\n$w \\leftarrow w + \\alpha_w \\delta_t \\nabla_w Q_w(s, a)$\nUpdate $a \\leftarrow a’$ and $s \\leftarrow s’$.\nTwo learning rates, $\\alpha_\\theta$ and $\\alpha_w$, are predefined for policy and value function parameter updates respectively.\nOff-Policy Policy Gradient\n#\nBoth REINFORCE and the vanilla version of actor-critic method are on-policy: training samples are collected according to the target policy — the very same policy that we try to optimize for. Off policy methods, however, result in several additional advantages:\nThe off-policy approach does not require full trajectories and can reuse any past episodes (\n“experience replay”\n) for much better sample efficiency.\nThe sample collection follows a behavior policy different from the target policy, bringing better\nexploration\n.\nNow let’s see how off-policy policy gradient is computed. The behavior policy for collecting samples is a known policy (predefined just like a hyperparameter), labelled as $\\beta(a \\vert s)$. The objective function sums up the reward over the state distribution defined by this behavior policy:\n$$\nJ(\\theta)\n= \\sum_{s \\in \\mathcal{S}} d^\\beta(s) \\sum_{a \\in \\mathcal{A}} Q^\\pi(s, a) \\pi_\\theta(a \\vert s)\n= \\mathbb{E}_{s \\sim d^\\beta} \\big[ \\sum_{a \\in \\mathcal{A}} Q^\\pi(s, a) \\pi_\\theta(a \\vert s) \\big]\n$$\nwhere $d^\\beta(s)$ is the stationary distribution of the behavior policy $\\beta$; recall that $d^\\beta(s) = \\lim_{t \\to \\infty} P(S_t = s \\vert S_0, \\beta)$; and $Q^\\pi$ is the action-value function estimated with regard to the target policy $\\pi$ (not the behavior policy!).\nGiven that the training observations are sampled by $a \\sim \\beta(a \\vert s)$, we can rewrite the gradient as:\n$$\n\\begin{aligned}\n\\nabla_\\theta J(\\theta)\n&= \\nabla_\\theta \\mathbb{E}_{s \\sim d^\\beta} \\Big[ \\sum_{a \\in \\mathcal{A}} Q^\\pi(s, a) \\pi_\\theta(a \\vert s)  \\Big] & \\\\ \n&= \\mathbb{E}_{s \\sim d^\\beta} \\Big[ \\sum_{a \\in \\mathcal{A}} \\big( Q^\\pi(s, a) \\nabla_\\theta \\pi_\\theta(a \\vert s) + \\color{red}{\\pi_\\theta(a \\vert s) \\nabla_\\theta Q^\\pi(s, a)} \\big) \\Big] & \\scriptstyle{\\text{; Derivative product rule.}}\\\\\n&\\stackrel{(i)}{\\approx} \\mathbb{E}_{s \\sim d^\\beta} \\Big[ \\sum_{a \\in \\mathcal{A}} Q^\\pi(s, a) \\nabla_\\theta \\pi_\\theta(a \\vert s) \\Big] & \\scriptstyle{\\text{; Ignore the red part: } \\color{red}{\\pi_\\theta(a \\vert s) \\nabla_\\theta Q^\\pi(s, a)}}. \\\\\n&= \\mathbb{E}_{s \\sim d^\\beta} \\Big[ \\sum_{a \\in \\mathcal{A}} \\beta(a \\vert s) \\frac{\\pi_\\theta(a \\vert s)}{\\beta(a \\vert s)} Q^\\pi(s, a) \\frac{\\nabla_\\theta \\pi_\\theta(a \\vert s)}{\\pi_\\theta(a \\vert s)} \\Big] & \\\\\n&= \\mathbb{E}_\\beta \\Big[\\frac{\\color{blue}{\\pi_\\theta(a \\vert s)}}{\\color{blue}{\\beta(a \\vert s)}} Q^\\pi(s, a) \\nabla_\\theta \\ln \\pi_\\theta(a \\vert s) \\Big] & \\scriptstyle{\\text{; The blue part is the importance weight.}}\n\\end{aligned}\n$$\nwhere $\\frac{\\pi_\\theta(a \\vert s)}{\\beta(a \\vert s)}$ is the\nimportance weight\n. Because $Q^\\pi$ is a function of the target policy and thus a function of policy parameter $\\theta$,  we should take the derivative of $\\nabla_\\theta Q^\\pi(s, a)$ as well according to the product rule. However, it is super hard to compute $\\nabla_\\theta Q^\\pi(s, a)$ in reality. Fortunately if we use an approximated gradient with the gradient of Q ignored, we still guarantee the policy improvement and eventually achieve the true local minimum. This is justified in the proof\nhere\n(Degris, White & Sutton, 2012).\nIn summary, when applying policy gradient in the off-policy setting, we can simple adjust it with a weighted sum and the weight is the ratio of the target policy to the behavior policy, $\\frac{\\pi_\\theta(a \\vert s)}{\\beta(a \\vert s)}$.\nA3C\n#\n[\npaper\n|\ncode\n]\nAsynchronous Advantage Actor-Critic\n(\nMnih et al., 2016\n), short for\nA3C\n, is a classic policy gradient method with a special focus on parallel training.\nIn A3C, the critics learn the value function while multiple actors are trained in parallel and get synced with global parameters from time to time. Hence, A3C is designed to work well for parallel training.\nLet’s use the state-value function as an example. The loss function for state value is to minimize the mean squared error, $J_v(w) = (G_t - V_w(s))^2$ and gradient descent can be applied to find the optimal w. This state-value function is used as the baseline in the policy gradient update.\nHere is the algorithm outline:\nWe have global parameters, $\\theta$ and $w$; similar thread-specific parameters, $\\theta’$ and $w’$.\nInitialize the time step $t = 1$\nWhile $T \\leq T_\\text{MAX}$:\nReset gradient: $\\mathrm{d}\\theta = 0$ and $\\mathrm{d}w = 0$.\nSynchronize thread-specific parameters with global ones: $\\theta’ = \\theta$ and $w’ = w$.\n$t_\\text{start}$ = t and sample a starting state $s_t$.\nWhile ($s_t$ != TERMINAL) and $t - t_\\text{start} \\leq t_\\text{max}$:\nPick the action $A_t \\sim \\pi_{\\theta’}(A_t \\vert S_t)$ and receive a new reward $R_t$ and a new state $s_{t+1}$.\nUpdate $t = t + 1$ and $T = T + 1$\nInitialize the variable that holds the return estimation\n$$\n R = \\begin{cases} \n 0 & \\text{if } s_t \\text{ is TERMINAL} \\\\\n V_{w'}(s_t) & \\text{otherwise}\n \\end{cases}\n $$\n6. For $i = t-1, \\dots, t\\_\\text{start}$:\n     1. $R \\leftarrow \\gamma R + R\\_i$; here R is a MC measure of $G\\_i$.\n     2. Accumulate gradients w.r.t. $\\theta'$: $d\\theta \\leftarrow d\\theta + \\nabla\\_{\\theta'} \\log \\pi\\_{\\theta'}(a\\_i \\vert s\\_i)(R - V\\_{w'}(s\\_i))$;\nAccumulate gradients w.r.t. w': $dw \\leftarrow dw + 2 (R - V\\_{w'}(s\\_i)) \\nabla\\_{w'} (R - V\\_{w'}(s\\_i))$.\nUpdate asynchronously $\\theta$ using $\\mathrm{d}\\theta$, and $w$ using $\\mathrm{d}w$.\nA3C enables the parallelism in multiple agent training. The gradient accumulation step (6.2) can be considered as a parallelized reformation of minibatch-based stochastic gradient update: the values of $w$ or $\\theta$ get corrected by a little bit in the direction of each training thread independently.\nA2C\n#\n[\npaper\n|\ncode\n]\nA2C\nis a synchronous, deterministic version of A3C; that’s why it is named as “A2C” with the first “A” (“asynchronous”) removed. In A3C each agent talks to the global parameters independently, so it is possible sometimes the thread-specific agents would be playing with policies of different versions and therefore the aggregated update would not be optimal. To resolve the inconsistency, a coordinator in A2C waits for all the parallel actors to finish their work before updating the global parameters and then in the next iteration parallel actors starts from the same policy. The synchronized gradient update keeps the training more cohesive and potentially to make convergence faster.\nA2C has been\nshown\nto be able to utilize GPUs more efficiently and work better with large batch sizes while achieving same or better performance than A3C.\nThe architecture of A3C versus A2C.\nDPG\n#\n[\npaper\n|code]\nIn methods described above, the policy function $\\pi(. \\vert s)$ is always modeled as a probability distribution over actions $\\mathcal{A}$ given the current state and thus it is\nstochastic\n.\nDeterministic policy gradient (DPG)\ninstead models the policy as a deterministic decision: $a = \\mu(s)$. It may look bizarre — how can you calculate the gradient of the action probability when it outputs a single action? Let’s look into it step by step.\nRefresh on a few notations to facilitate the discussion:\n$\\rho_0(s)$: The initial distribution over states\n$\\rho^\\mu(s \\to s’, k)$: Starting from state s, the visitation probability density at state s’ after moving k steps by policy $\\mu$.\n$\\rho^\\mu(s’)$: Discounted state distribution, defined as $\\rho^\\mu(s’) = \\int_\\mathcal{S} \\sum_{k=1}^\\infty \\gamma^{k-1} \\rho_0(s) \\rho^\\mu(s \\to s’, k) ds$.\nThe objective function to optimize for is listed as follows:\n$$\nJ(\\theta) = \\int_\\mathcal{S} \\rho^\\mu(s) Q(s, \\mu_\\theta(s)) ds\n$$\nDeterministic policy gradient theorem\n: Now it is the time to compute the gradient! According to the chain rule, we first take the gradient of Q w.r.t. the action a and then take the gradient of the deterministic policy function $\\mu$ w.r.t. $\\theta$:\n$$\n\\begin{aligned}\n\\nabla_\\theta J(\\theta) \n&= \\int_\\mathcal{S} \\rho^\\mu(s) \\nabla_a Q^\\mu(s, a) \\nabla_\\theta \\mu_\\theta(s) \\rvert_{a=\\mu_\\theta(s)} ds \\\\\n&= \\mathbb{E}_{s \\sim \\rho^\\mu} [\\nabla_a Q^\\mu(s, a) \\nabla_\\theta \\mu_\\theta(s) \\rvert_{a=\\mu_\\theta(s)}]\n\\end{aligned}\n$$\nWe can consider the deterministic policy as a\nspecial case\nof the stochastic one, when the probability distribution contains only one extreme non-zero value over one action. Actually, in the DPG\npaper\n, the authors have shown that if the stochastic policy $\\pi_{\\mu_\\theta, \\sigma}$ is re-parameterized by a deterministic policy $\\mu_\\theta$ and a variation variable $\\sigma$, the stochastic policy is eventually equivalent to the deterministic case when $\\sigma=0$. Compared to the deterministic policy, we expect the stochastic policy to require more samples as it integrates the data over the whole state and action space.\nThe deterministic policy gradient theorem can be plugged into common policy gradient frameworks.\nLet’s consider an example of on-policy actor-critic algorithm to showcase the procedure. In each iteration of on-policy actor-critic, two actions are taken deterministically $a = \\mu_\\theta(s)$ and the\nSARSA\nupdate on policy parameters relies on the new gradient that we just computed above:\n$$\n\\begin{aligned}\n\\delta_t &= R_t + \\gamma Q_w(s_{t+1}, a_{t+1}) - Q_w(s_t, a_t) & \\small{\\text{; TD error in SARSA}}\\\\\nw_{t+1} &= w_t + \\alpha_w \\delta_t \\nabla_w Q_w(s_t, a_t) & \\\\\n\\theta_{t+1} &= \\theta_t + \\alpha_\\theta \\color{red}{\\nabla_a Q_w(s_t, a_t) \\nabla_\\theta \\mu_\\theta(s) \\rvert_{a=\\mu_\\theta(s)}} & \\small{\\text{; Deterministic policy gradient theorem}}\n\\end{aligned}\n$$\nHowever, unless there is sufficient noise in the environment, it is very hard to guarantee enough\nexploration\ndue to the determinacy of the policy. We can either add noise into the policy (ironically this makes it nondeterministic!) or learn it off-policy-ly by following a different stochastic behavior policy to collect samples.\nSay, in the off-policy approach, the training trajectories are generated by a stochastic policy $\\beta(a \\vert s)$ and thus the state distribution follows the corresponding discounted state density $\\rho^\\beta$:\n$$\n\\begin{aligned}\nJ_\\beta(\\theta) &= \\int_\\mathcal{S} \\rho^\\beta Q^\\mu(s, \\mu_\\theta(s)) ds \\\\\n\\nabla_\\theta J_\\beta(\\theta) &= \\mathbb{E}_{s \\sim \\rho^\\beta} [\\nabla_a Q^\\mu(s, a) \\nabla_\\theta \\mu_\\theta(s)  \\rvert_{a=\\mu_\\theta(s)} ]\n\\end{aligned}\n$$\nNote that because the policy is deterministic, we only need $Q^\\mu(s, \\mu_\\theta(s))$ rather than $\\sum_a \\pi(a \\vert s) Q^\\pi(s, a)$ as the estimated reward of a given state s.\nIn the off-policy approach with a stochastic policy, importance sampling is often used to correct the mismatch between behavior and target policies, as what we have described\nabove\n. However, because the deterministic policy gradient removes the integral over actions, we can avoid importance sampling.\nDDPG\n#\n[\npaper\n|\ncode\n]\nDDPG\n(\nLillicrap, et al., 2015\n), short for\nDeep Deterministic Policy Gradient\n, is a model-free off-policy actor-critic algorithm, combining\nDPG\nwith\nDQN\n. Recall that DQN (Deep Q-Network) stabilizes the learning of Q-function by experience replay and the frozen target network. The original DQN works in discrete space, and DDPG extends it to continuous space with the actor-critic framework while learning a deterministic policy.\nIn order to do better exploration, an exploration policy $\\mu’$ is constructed by adding noise $\\mathcal{N}$:\n$$\n\\mu'(s) = \\mu_\\theta(s) + \\mathcal{N}\n$$\nIn addition, DDPG does soft updates (“conservative policy iteration”) on the parameters of both actor and critic, with $\\tau \\ll 1$: $\\theta’ \\leftarrow \\tau \\theta + (1 - \\tau) \\theta’$. In this way, the target network values are constrained to change slowly, different from the design in DQN that the target network stays frozen for some period of time.\nOne detail in the paper that is particularly useful in robotics is on how to normalize the different physical units of low dimensional features. For example, a model is designed to learn a policy with the robot’s positions and velocities as input; these physical statistics are different by nature and even statistics of the same type may vary a lot across multiple robots.\nBatch normalization\nis applied to fix it by normalizing every dimension across samples in one minibatch.\nFig 3. DDPG Algorithm. (Image source:\nLillicrap, et al., 2015\n)\nD4PG\n#\n[\npaper\n|code (Search “github d4pg” and you will see a few.)]\nDistributed Distributional DDPG (D4PG)\napplies a set of improvements on DDPG to make it run in the distributional fashion.\n(1)\nDistributional Critic\n: The critic estimates the expected Q value as a random variable ~ a distribution $Z_w$ parameterized by $w$ and therefore $Q_w(s, a) = \\mathbb{E} Z_w(x, a)$. The loss for learning the distribution parameter is to minimize some measure of the distance between two distributions — distributional TD error: $L(w) = \\mathbb{E}[d(\\mathcal{T}_{\\mu_\\theta}, Z_{w’}(s, a), Z_w(s, a)]$, where $\\mathcal{T}_{\\mu_\\theta}$ is the Bellman operator.\nThe deterministic policy gradient update becomes:\n$$\n\\begin{aligned}\n\\nabla_\\theta J(\\theta) \n&\\approx \\mathbb{E}_{\\rho^\\mu} [\\nabla_a Q_w(s, a) \\nabla_\\theta \\mu_\\theta(s) \\rvert_{a=\\mu_\\theta(s)}] & \\scriptstyle{\\text{; gradient update in DPG}} \\\\\n&= \\mathbb{E}_{\\rho^\\mu} [\\mathbb{E}[\\nabla_a Z_w(s, a)] \\nabla_\\theta \\mu_\\theta(s) \\rvert_{a=\\mu_\\theta(s)}] & \\scriptstyle{\\text{; expectation of the Q-value distribution.}}\n\\end{aligned}\n$$\n(2)\n$N$-step returns\n: When calculating the TD error, D4PG computes $N$-step TD target rather than one-step to incorporate rewards in more future steps. Thus the new TD target is:\n$$\nr(s_0, a_0) + \\mathbb{E}[\\sum_{n=1}^{N-1} r(s_n, a_n) + \\gamma^N Q(s_N, \\mu_\\theta(s_N)) \\vert s_0, a_0 ]\n$$\n(3)\nMultiple Distributed Parallel Actors\n: D4PG utilizes $K$ independent actors, gathering experience in parallel and feeding data into the same replay buffer.\n(4)\nPrioritized Experience Replay (\nPER\n)\n: The last piece of modification is to do sampling from the replay buffer of size $R$ with an non-uniform probability $p_i$. In this way, a sample $i$ has the probability $(Rp_i)^{-1}$ to be selected and thus the importance weight is $(Rp_i)^{-1}$.\nD4PG algorithm (Image source:\nBarth-Maron, et al. 2018\n); Note that in the original paper, the variable letters are chosen slightly differently from what in the post; i.e. I use $\\mu(.)$ for representing a deterministic policy instead of $\\pi(.)$.\nMADDPG\n#\n[\npaper\n|\ncode\n]\nMulti-agent DDPG\n(\nMADDPG\n) (\nLowe et al., 2017\n) extends DDPG to an environment where multiple agents are coordinating to complete tasks with only local information. In the viewpoint of one agent, the environment is non-stationary as policies of other agents are quickly upgraded and remain unknown. MADDPG is an actor-critic model redesigned particularly for handling such a changing environment and interactions between agents.\nThe problem can be formalized in the multi-agent version of MDP, also known as\nMarkov games\n. MADDPG is proposed for partially observable Markov games. Say, there are N agents in total with a set of states $\\mathcal{S}$. Each agent owns a set of possible action, $\\mathcal{A}_1, \\dots, \\mathcal{A}_N$, and a set of observation, $\\mathcal{O}_1, \\dots, \\mathcal{O}_N$. The state transition function involves all states, action and observation spaces  $\\mathcal{T}: \\mathcal{S} \\times \\mathcal{A}_1 \\times \\dots \\mathcal{A}_N \\mapsto \\mathcal{S}$. Each agent’s stochastic policy only involves its own state and action: $\\pi_{\\theta_i}: \\mathcal{O}_i \\times \\mathcal{A}_i \\mapsto [0, 1]$, a probability distribution over actions given its own observation, or a deterministic policy: $\\mu_{\\theta_i}: \\mathcal{O}_i \\mapsto \\mathcal{A}_i$.\nLet $\\vec{o} = {o_1, \\dots, o_N}$, $\\vec{\\mu} = {\\mu_1, \\dots, \\mu_N}$ and the policies are parameterized by $\\vec{\\theta} = {\\theta_1, \\dots, \\theta_N}$.\nThe critic in MADDPG learns a centralized action-value function $Q^\\vec{\\mu}_i(\\vec{o}, a_1, \\dots, a_N)$ for the i-th agent, where $a_1 \\in \\mathcal{A}_1, \\dots, a_N \\in \\mathcal{A}_N$ are actions of all agents. Each $Q^\\vec{\\mu}_i$ is learned separately for $i=1, \\dots, N$ and therefore multiple agents can have arbitrary reward structures, including conflicting rewards in a competitive setting. Meanwhile, multiple actors, one for each agent, are exploring and upgrading the policy parameters $\\theta_i$ on their own.\nActor update\n:\n$$\n\\nabla_{\\theta_i} J(\\theta_i) = \\mathbb{E}_{\\vec{o}, a \\sim \\mathcal{D}} [\\nabla_{a_i} Q^{\\vec{\\mu}}_i (\\vec{o}, a_1, \\dots, a_N) \\nabla_{\\theta_i} \\mu_{\\theta_i}(o_i) \\rvert_{a_i=\\mu_{\\theta_i}(o_i)} ]\n$$\nWhere $\\mathcal{D}$ is the memory buffer for experience replay, containing multiple episode samples $(\\vec{o}, a_1, \\dots, a_N, r_1, \\dots, r_N, \\vec{o}’)$ — given current observation $\\vec{o}$, agents take action $a_1, \\dots, a_N$ and get rewards $r_1, \\dots, r_N$, leading to the new observation $\\vec{o}’$.\nCritic update\n:\n$$\n\\begin{aligned}\n\\mathcal{L}(\\theta_i) &= \\mathbb{E}_{\\vec{o}, a_1, \\dots, a_N, r_1, \\dots, r_N, \\vec{o}'}[ (Q^{\\vec{\\mu}}_i(\\vec{o}, a_1, \\dots, a_N) - y)^2 ] & \\\\\n\\text{where } y &= r_i + \\gamma Q^{\\vec{\\mu}'}_i (\\vec{o}', a'_1, \\dots, a'_N) \\rvert_{a'_j = \\mu'_{\\theta_j}} & \\scriptstyle{\\text{; TD target!}}\n\\end{aligned}\n$$\nwhere $\\vec{\\mu}’$ are the target policies with delayed softly-updated parameters.\nIf the policies $\\vec{\\mu}$ are unknown during the critic update, we can ask each agent to learn and evolve its own approximation of others’ policies. Using the approximated policies, MADDPG still can learn efficiently although the inferred policies might not be accurate.\nTo mitigate the high variance triggered by the interaction between competing or collaborating agents in the environment, MADDPG proposed one more element -\npolicy ensembles\n:\nTrain K policies for one agent;\nPick a random policy for episode rollouts;\nTake an ensemble of these K policies to do gradient update.\nIn summary, MADDPG added three additional ingredients on top of DDPG to make it adapt to the multi-agent environment:\nCentralized critic + decentralized actors;\nActors are able to use estimated policies of other agents for learning;\nPolicy ensembling is good for reducing variance.\nThe architecture design of MADDPG. (Image source:\nLowe et al., 2017\n)\nTRPO\n#\n[\npaper\n|\ncode\n]\nTo improve training stability, we should avoid parameter updates that change the policy too much at one step.\nTrust region policy optimization (TRPO)\n(\nSchulman, et al., 2015\n) carries out this idea by enforcing a\nKL divergence\nconstraint on the size of policy update at each iteration.\nConsider the case when we are doing off-policy RL, the policy $\\beta$ used for collecting trajectories on rollout workers is different from the policy $\\pi$ to optimize for. The objective function in an off-policy model measures the total advantage over the state visitation distribution and actions, while the mismatch between the training data distribution and the true policy state distribution is compensated by importance sampling estimator:\n$$\n\\begin{aligned}\nJ(\\theta)\n&= \\sum_{s \\in \\mathcal{S}} \\rho^{\\pi_{\\theta_\\text{old}}} \\sum_{a \\in \\mathcal{A}} \\big( \\pi_\\theta(a \\vert s) \\hat{A}_{\\theta_\\text{old}}(s, a) \\big) & \\\\\n&= \\sum_{s \\in \\mathcal{S}} \\rho^{\\pi_{\\theta_\\text{old}}} \\sum_{a \\in \\mathcal{A}} \\big( \\beta(a \\vert s) \\frac{\\pi_\\theta(a \\vert s)}{\\beta(a \\vert s)} \\hat{A}_{\\theta_\\text{old}}(s, a) \\big) & \\scriptstyle{\\text{; Importance sampling}} \\\\\n&= \\mathbb{E}_{s \\sim \\rho^{\\pi_{\\theta_\\text{old}}}, a \\sim \\beta} \\big[ \\frac{\\pi_\\theta(a \\vert s)}{\\beta(a \\vert s)} \\hat{A}_{\\theta_\\text{old}}(s, a) \\big] &\n\\end{aligned}\n$$\nwhere $\\theta_\\text{old}$ is the policy parameters before the update and thus known to us; $\\rho^{\\pi_{\\theta_\\text{old}}}$ is defined in the same way as\nabove\n; $\\beta(a \\vert s)$ is the behavior policy for collecting trajectories. Noted that we use an estimated advantage $\\hat{A}(.)$ rather than the true advantage function $A(.)$ because the true rewards are usually unknown.\nWhen training on policy, theoretically the policy for collecting data is same as the policy that we want to optimize. However, when rollout workers and optimizers are running in parallel asynchronously, the behavior policy can get stale. TRPO considers this subtle difference: It labels the behavior policy as $\\pi_{\\theta_\\text{old}}(a \\vert s)$ and thus the objective function becomes:\n$$\nJ(\\theta) = \\mathbb{E}_{s \\sim \\rho^{\\pi_{\\theta_\\text{old}}}, a \\sim \\pi_{\\theta_\\text{old}}} \\big[ \\frac{\\pi_\\theta(a \\vert s)}{\\pi_{\\theta_\\text{old}}(a \\vert s)} \\hat{A}_{\\theta_\\text{old}}(s, a) \\big]\n$$\nTRPO aims to maximize the objective function $J(\\theta)$ subject to,\ntrust region constraint\nwhich enforces the distance between old and new policies measured by\nKL-divergence\nto be small enough, within a parameter δ:\n$$\n\\mathbb{E}_{s \\sim \\rho^{\\pi_{\\theta_\\text{old}}}} [D_\\text{KL}(\\pi_{\\theta_\\text{old}}(.\\vert s) \\| \\pi_\\theta(.\\vert s)] \\leq \\delta\n$$\nIn this way, the old and new policies would not diverge too much when this hard constraint is met. While still, TRPO can guarantee a monotonic improvement over policy iteration (Neat, right?). Please read the proof in the\npaper\nif interested :)\nPPO\n#\n[\npaper\n|\ncode\n]\nGiven that TRPO is relatively complicated and we still want to implement a similar constraint,\nproximal policy optimization (PPO)\nsimplifies it by using a clipped surrogate objective while retaining similar performance.\nFirst, let’s denote the probability ratio between old and new policies as:\n$$\nr(\\theta) = \\frac{\\pi_\\theta(a \\vert s)}{\\pi_{\\theta_\\text{old}}(a \\vert s)}\n$$\nThen, the objective function of TRPO (on policy) becomes:\n$$\nJ^\\text{TRPO} (\\theta) = \\mathbb{E} [ r(\\theta) \\hat{A}_{\\theta_\\text{old}}(s, a) ]\n$$\nWithout a limitation on the distance between $\\theta_\\text{old}$ and $\\theta$, to maximize $J^\\text{TRPO} (\\theta)$ would lead to instability with extremely large parameter updates and big policy ratios. PPO imposes the constraint by forcing $r(\\theta)$ to stay within a small interval around 1, precisely $[1-\\epsilon, 1+\\epsilon]$, where $\\epsilon$ is a hyperparameter.\n$$\nJ^\\text{CLIP} (\\theta) = \\mathbb{E} [ \\min( r(\\theta) \\hat{A}_{\\theta_\\text{old}}(s, a), \\text{clip}(r(\\theta), 1 - \\epsilon, 1 + \\epsilon) \\hat{A}_{\\theta_\\text{old}}(s, a))]\n$$\nThe function $\\text{clip}(r(\\theta), 1 - \\epsilon, 1 + \\epsilon)$ clips the ratio to be no more than $1+\\epsilon$ and no less than $1-\\epsilon$. The objective function of PPO takes the minimum one between the original value and the clipped version and therefore we lose the motivation for increasing the policy update to extremes for better rewards.\nWhen applying PPO on the network architecture with shared parameters for both policy (actor) and value (critic) functions, in addition to the clipped reward, the objective function is augmented with an error term on the value estimation (formula in red) and an entropy term (formula in blue) to encourage sufficient exploration.\n$$\nJ^\\text{CLIP'} (\\theta) = \\mathbb{E} [ J^\\text{CLIP} (\\theta) - \\color{red}{c_1 (V_\\theta(s) - V_\\text{target})^2} + \\color{blue}{c_2 H(s, \\pi_\\theta(.))} ]\n$$\nwhere Both $c_1$ and $c_2$ are two hyperparameter constants.\nPPO has been tested on a set of benchmark tasks and proved to produce awesome results with much greater simplicity.\nIn a later paper by\nHsu et al., 2020\n, two common design choices in PPO are revisited, precisely (1) clipped probability ratio for policy regularization and (2) parameterize policy action space by continuous Gaussian or discrete softmax distribution. They first identified three failure modes in PPO and proposed replacements for these two designs.\nThe failure modes are:\nOn continuous action spaces, standard PPO is unstable when rewards vanish outside bounded support.\nOn discrete action spaces with sparse high rewards, standard PPO often gets stuck at suboptimal actions.\nThe policy is sensitive to initialization when there are locally optimal actions close to initialization.\nDiscretizing the action space or use Beta distribution helps avoid failure mode 1&3 associated with Gaussian policy. Using KL regularization (same motivation as in\nTRPO\n) as an alternative surrogate model helps resolve failure mode 1&2.\nPPG\n#\n[\npaper\n|\ncode\n]\nSharing parameters between policy and value networks have pros and cons. It allows policy and value functions to share the learned features with each other, but it may cause conflicts between competing objectives and demands the same data for training two networks at the same time.\nPhasic policy gradient\n(\nPPG\n;\nCobbe, et al 2020\n) modifies the traditional on-policy\nactor-critic\npolicy gradient algorithm. precisely\nPPO\n, to have separate training phases for policy and value functions. In two alternating phases:\nThe\npolicy phase\n: updates the policy network by optimizing the PPO\nobjective\n$L^\\text{CLIP} (\\theta)$;\nThe\nauxiliary phase\n: optimizes an auxiliary objective alongside a behavioral cloning loss. In the paper, value function error is the sole auxiliary objective, but it can be quite general and includes any other additional auxiliary losses.\n$$\n\\begin{aligned}\nL^\\text{joint} &= L^\\text{aux} + \\beta_\\text{clone} \\cdot \\mathbb{E}_t[\\text{KL}[\\pi_{\\theta_\\text{old}}(\\cdot\\mid s_t), \\pi_\\theta(\\cdot\\mid s_t)]] \\\\\nL^\\text{aux} &= L^\\text{value} = \\mathbb{E}_t \\big[\\frac{1}{2}\\big( V_w(s_t) - \\hat{V}_t^\\text{targ} \\big)^2\\big]\n\\end{aligned}\n$$\nwhere $\\beta_\\text{clone}$ is a hyperparameter for controlling how much we would like to keep the policy not diverge too much from its original behavior while optimizing the auxiliary objectives.\nThe algorithm of PPG. (Image source:\nCobbe, et al 2020\n)\nwhere\n$N_\\pi$ is the number of policy update iterations in the policy phase. Note that the policy phase performs multiple iterations of updates per single auxiliary phase.\n$E_\\pi$ and $E_V$ control the sample reuse (i.e. the number of training epochs performed across data in the reply buffer) for the policy and value functions, respectively. Note that this happens within the policy phase and thus $E_V$ affects the learning of true value function not the auxiliary value function.\n$E_\\text{aux}$ defines the sample reuse in the auxiliary phrase. In PPG, value function optimization can tolerate a much higher level sample reuse; for example, in the experiments of the paper, $E_\\text{aux} = 6$ while $E_\\pi = E_V = 1$.\nPPG leads to a significant improvement on sample efficiency compared to PPO.\nThe mean normalized performance of PPG vs PPO on the\nProcgen\nbenchmark. (Image source:\nCobbe, et al 2020\n)\nACER\n#\n[\npaper\n|\ncode\n]\nACER\n, short for\nactor-critic with experience replay\n(\nWang, et al., 2017\n), is an off-policy actor-critic model with experience replay, greatly increasing the sample efficiency and decreasing the data correlation. A3C builds up the foundation for ACER, but it is on policy; ACER is A3C’s off-policy counterpart. The major obstacle to making A3C off policy is how to control the stability of the off-policy estimator. ACER proposes three designs to overcome it:\nUse Retrace Q-value estimation;\nTruncate the importance weights with bias correction;\nApply efficient TRPO.\nRetrace Q-value Estimation\nRetrace\nis an off-policy return-based Q-value estimation algorithm with a nice guarantee for convergence for any target and behavior policy pair $(\\pi, \\beta)$, plus good data efficiency.\nRecall how TD learning works for prediction:\nCompute TD error: $\\delta_t = R_t + \\gamma \\mathbb{E}_{a \\sim \\pi} Q(S_{t+1}, a) - Q(S_t, A_t)$; the term $r_t + \\gamma \\mathbb{E}_{a \\sim \\pi} Q(s_{t+1}, a) $ is known as “TD target”. The expectation $\\mathbb{E}_{a \\sim \\pi}$ is used because for the future step the best estimation we can make is what the return would be if we follow the current policy $\\pi$.\nUpdate the value by correcting the error to move toward the goal: $Q(S_t, A_t) \\leftarrow Q(S_t, A_t) + \\alpha \\delta_t$. In other words, the incremental update on Q is proportional to the TD error: $\\Delta Q(S_t, A_t) = \\alpha \\delta_t$.\nWhen the rollout is off policy, we need to apply importance sampling on the Q update:\n$$\n\\Delta Q^\\text{imp}(S_t, A_t) \n= \\gamma^t \\prod_{1 \\leq \\tau \\leq t} \\frac{\\pi(A_\\tau \\vert S_\\tau)}{\\beta(A_\\tau \\vert S_\\tau)} \\delta_t\n$$\nThe product of importance weights looks pretty scary when we start imagining how it can cause super high variance and even explode. Retrace Q-value estimation method modifies $\\Delta Q$ to have importance weights truncated by no more than a constant $c$:\n$$\n\\Delta Q^\\text{ret}(S_t, A_t) \n= \\gamma^t \\prod_{1 \\leq \\tau \\leq t} \\min(c, \\frac{\\pi(A_\\tau \\vert S_\\tau)}{\\beta(A_\\tau \\vert S_\\tau)})  \\delta_t\n$$\nACER uses $Q^\\text{ret}$ as the target to train the critic by minimizing the L2 error term: $(Q^\\text{ret}(s, a) - Q(s, a))^2$.\nImportance weights truncation\nTo reduce the high variance of the policy gradient $\\hat{g}$, ACER truncates the importance weights by a constant c, plus a correction term. The label $\\hat{g}_t^\\text{acer}$ is the ACER policy gradient at time t.\n$$\n\\begin{aligned}\n\\hat{g}_t^\\text{acer}\n= & \\omega_t \\big( Q^\\text{ret}(S_t, A_t) - V_{\\theta_v}(S_t) \\big) \\nabla_\\theta \\ln \\pi_\\theta(A_t \\vert S_t) \n  & \\scriptstyle{\\text{; Let }\\omega_t=\\frac{\\pi(A_t \\vert S_t)}{\\beta(A_t \\vert S_t)}} \\\\\n= & \\color{blue}{\\min(c, \\omega_t) \\big( Q^\\text{ret}(S_t, A_t) - V_w(S_t) \\big) \\nabla_\\theta \\ln \\pi_\\theta(A_t \\vert S_t)} \\\\\n  & + \\color{red}{\\mathbb{E}_{a \\sim \\pi} \\big[ \\max(0, \\frac{\\omega_t(a) - c}{\\omega_t(a)}) \\big( Q_w(S_t, a) - V_w(S_t) \\big) \\nabla_\\theta \\ln \\pi_\\theta(a \\vert S_t) \\big]}\n  & \\scriptstyle{\\text{; Let }\\omega_t (a) =\\frac{\\pi(a \\vert S_t)}{\\beta(a \\vert S_t)}}\n\\end{aligned}\n$$\nwhere $Q_w(.)$ and $V_w(.)$ are value functions predicted by the critic with parameter w. The first term (blue) contains the clipped important weight. The clipping helps reduce the variance, in addition to subtracting state value function $V_w(.)$ as a baseline. The second term (red) makes a correction to achieve unbiased estimation.\nEfficient TRPO\nFurthermore, ACER adopts the idea of TRPO but with a small adjustment to make it more computationally efficient: rather than measuring the KL divergence between policies before and after one update, ACER maintains a running average of past policies and forces the updated policy to not deviate far from this average.\nThe ACER\npaper\nis pretty dense with many equations. Hopefully, with the prior knowledge on TD learning, Q-learning, importance sampling and TRPO, you will find the\npaper\nslightly easier to follow :)\nACTKR\n#\n[\npaper\n|\ncode\n]\nACKTR (actor-critic using Kronecker-factored trust region)\n(\nYuhuai Wu, et al., 2017\n) proposed to use Kronecker-factored approximation curvature (\nK-FAC\n) to do the gradient update for both the critic and actor. K-FAC made an improvement on the computation of\nnatural gradient\n, which is quite different from our\nstandard gradient\n.\nHere\nis a nice, intuitive explanation of natural gradient. One sentence summary is probably:\n“we first consider all combinations of parameters that result in a new network a constant KL divergence away from the old network. This constant value can be viewed as the step size or learning rate. Out of all these possible combinations, we choose the one that minimizes our loss function.”\nI listed ACTKR here mainly for the completeness of this post, but I would not dive into details, as it involves a lot of theoretical knowledge on natural gradient and optimization methods. If interested, check these papers/posts, before reading the ACKTR paper:\nAmari.\nNatural Gradient Works Efficiently in Learning\n. 1998\nKakade.\nA Natural Policy Gradient\n. 2002\nA intuitive explanation of natural gradient descent\nWiki: Kronecker product\nMartens & Grosse.\nOptimizing neural networks with kronecker-factored approximate curvature.\n2015.\nHere is a high level summary from the K-FAC\npaper\n:\n“This approximation is built in two stages. In the first, the rows and columns of the Fisher are divided into groups, each of which corresponds to all the weights in a given layer, and this gives rise to a block-partitioning of the matrix. These blocks are then approximated as Kronecker products between much smaller matrices, which we show is equivalent to making certain approximating assumptions regarding the statistics of the network’s gradients.\nIn the second stage, this matrix is further approximated as having an inverse which is either block-diagonal or block-tridiagonal. We justify this approximation through a careful examination of the relationships between inverse covariances, tree-structured graphical models, and linear regression. Notably, this justification doesn’t apply to the Fisher itself, and our experiments confirm that while the inverse Fisher does indeed possess this structure (approximately), the Fisher itself does not.”\nSAC\n#\n[\npaper\n|\ncode\n]\nSoft Actor-Critic (SAC)\n(\nHaarnoja et al. 2018\n) incorporates the entropy measure of the policy into the reward to encourage exploration: we expect to learn a policy that acts as randomly as possible while it is still able to succeed at the task. It is an off-policy actor-critic model following the maximum entropy reinforcement learning framework. A precedent work is\nSoft Q-learning\n.\nThree key components in SAC:\nAn\nactor-critic\narchitecture with separate policy and value function networks;\nAn\noff-policy\nformulation that enables reuse of previously collected data for efficiency;\nEntropy maximization to enable stability and exploration.\nThe policy is trained with the objective to maximize the expected return and the entropy at the same time:\n$$\nJ(\\theta) = \\sum_{t=1}^T \\mathbb{E}_{(s_t, a_t) \\sim \\rho_{\\pi_\\theta}} [r(s_t, a_t) + \\alpha \\mathcal{H}(\\pi_\\theta(.\\vert s_t))]\n$$\nwhere $\\mathcal{H}(.)$ is the entropy measure and $\\alpha$ controls how important the entropy term is, known as\ntemperature\nparameter. The entropy maximization leads to policies that can (1) explore more and (2) capture multiple modes of near-optimal strategies (i.e., if there exist multiple options that seem to be equally good, the policy should assign each with an equal probability to be chosen).\nPrecisely, SAC aims to learn three functions:\nThe policy with parameter $\\theta$, $\\pi_\\theta$.\nSoft Q-value function parameterized by $w$, $Q_w$.\nSoft state value function parameterized by $\\psi$, $V_\\psi$; theoretically we can infer $V$ by knowing $Q$ and $\\pi$, but in practice, it helps stabilize the training.\nSoft Q-value and soft state value are defined as:\n$$\n\\begin{aligned}\nQ(s_t, a_t) &= r(s_t, a_t) + \\gamma \\mathbb{E}_{s_{t+1} \\sim \\rho_{\\pi}(s)} [V(s_{t+1})] & \\text{; according to Bellman equation.}\\\\\n\\text{where }V(s_t) &= \\mathbb{E}_{a_t \\sim \\pi} [Q(s_t, a_t) - \\alpha \\log \\pi(a_t \\vert s_t)] & \\text{; soft state value function.}\n\\end{aligned}\n$$\n$$\n\\text{Thus, } Q(s_t, a_t) = r(s_t, a_t) + \\gamma \\mathbb{E}_{(s_{t+1}, a_{t+1}) \\sim \\rho_{\\pi}} [Q(s_{t+1}, a_{t+1}) - \\alpha \\log \\pi(a_{t+1} \\vert s_{t+1})]\n$$\n$\\rho_\\pi(s)$ and $\\rho_\\pi(s, a)$ denote the state and the state-action marginals of the state distribution induced by the policy $\\pi(a \\vert s)$; see the similar definitions in\nDPG\nsection.\nThe soft state value function is trained to minimize the mean squared error:\n$$\n\\begin{aligned}\nJ_V(\\psi) &= \\mathbb{E}_{s_t \\sim \\mathcal{D}} [\\frac{1}{2} \\big(V_\\psi(s_t) - \\mathbb{E}[Q_w(s_t, a_t) - \\log \\pi_\\theta(a_t \\vert s_t)] \\big)^2] \\\\\n\\text{with gradient: }\\nabla_\\psi J_V(\\psi) &= \\nabla_\\psi V_\\psi(s_t)\\big( V_\\psi(s_t) - Q_w(s_t, a_t) + \\log \\pi_\\theta (a_t \\vert s_t) \\big)\n\\end{aligned}\n$$\nwhere $\\mathcal{D}$ is the replay buffer.\nThe soft Q function is trained to minimize the soft Bellman residual:\n$$\n\\begin{aligned}\nJ_Q(w) &= \\mathbb{E}_{(s_t, a_t) \\sim \\mathcal{D}} [\\frac{1}{2}\\big( Q_w(s_t, a_t) - (r(s_t, a_t) + \\gamma \\mathbb{E}_{s_{t+1} \\sim \\rho_\\pi(s)}[V_{\\bar{\\psi}}(s_{t+1})]) \\big)^2] \\\\\n\\text{with gradient: } \\nabla_w J_Q(w) &= \\nabla_w Q_w(s_t, a_t) \\big( Q_w(s_t, a_t) - r(s_t, a_t) - \\gamma V_{\\bar{\\psi}}(s_{t+1})\\big) \n\\end{aligned}\n$$\nwhere $\\bar{\\psi}$ is the target value function which is the exponential moving average (or only gets updated periodically in a “hard” way), just like how the parameter of the target Q network is treated in\nDQN\nto stabilize the training.\nSAC updates the policy to minimize the\nKL-divergence\n:\n$$\n\\begin{aligned}\n\\pi_\\text{new} \n&= \\arg\\min_{\\pi' \\in \\Pi} D_\\text{KL} \\Big( \\pi'(.\\vert s_t) \\| \\frac{\\exp(Q^{\\pi_\\text{old}}(s_t, .))}{Z^{\\pi_\\text{old}}(s_t)} \\Big) \\\\[6pt]\n&= \\arg\\min_{\\pi' \\in \\Pi} D_\\text{KL} \\big( \\pi'(.\\vert s_t) \\| \\exp(Q^{\\pi_\\text{old}}(s_t, .) - \\log Z^{\\pi_\\text{old}}(s_t)) \\big) \\\\[6pt]\n\\text{objective for update: } J_\\pi(\\theta) &= \\nabla_\\theta D_\\text{KL} \\big( \\pi_\\theta(. \\vert s_t) \\| \\exp(Q_w(s_t, .) - \\log Z_w(s_t)) \\big) \\\\[6pt]\n&= \\mathbb{E}_{a_t\\sim\\pi} \\Big[ - \\log \\big( \\frac{\\exp(Q_w(s_t, a_t) - \\log Z_w(s_t))}{\\pi_\\theta(a_t \\vert s_t)} \\big) \\Big] \\\\[6pt]\n&= \\mathbb{E}_{a_t\\sim\\pi} [ \\log \\pi_\\theta(a_t \\vert s_t) - Q_w(s_t, a_t) + \\log Z_w(s_t) ]\n\\end{aligned}\n$$\nwhere $\\Pi$ is the set of potential policies that we can model our policy as to keep them tractable; for example, $\\Pi$ can be the family of Gaussian mixture distributions, expensive to model but highly expressive and still tractable. $Z^{\\pi_\\text{old}}(s_t)$ is the partition function to normalize the distribution. It is usually intractable but does not contribute to the gradient. How to minimize $J_\\pi(\\theta)$ depends our choice of $\\Pi$.\nThis update guarantees that $Q^{\\pi_\\text{new}}(s_t, a_t) \\geq Q^{\\pi_\\text{old}}(s_t, a_t)$, please check the proof on this lemma in the Appendix B.2 in the original\npaper\n.\nOnce we have defined the objective functions and gradients for soft action-state value, soft state value and the policy network, the soft actor-critic algorithm is straightforward:\nThe soft actor-critic algorithm. (Image source:\noriginal paper\n)\nSAC with Automatically Adjusted Temperature\n#\n[\npaper\n|\ncode\n]\nSAC is brittle with respect to the temperature parameter. Unfortunately it is difficult to adjust temperature, because the entropy can vary unpredictably both across tasks and during training as the policy becomes better. An improvement on SAC formulates a constrained optimization problem: while maximizing the expected return, the policy should satisfy a minimum entropy constraint:\n$$\n\\max_{\\pi_0, \\dots, \\pi_T} \\mathbb{E} \\Big[ \\sum_{t=0}^T r(s_t, a_t)\\Big] \\text{s.t. } \\forall t\\text{, } \\mathcal{H}(\\pi_t) \\geq \\mathcal{H}_0\n$$\nwhere $\\mathcal{H}_0$ is a predefined minimum policy entropy threshold.\nThe expected return $\\mathbb{E} \\Big[ \\sum_{t=0}^T r(s_t, a_t)\\Big]$ can be decomposed into a sum of rewards at all the time steps. Because the policy $\\pi_t$ at time t has no effect on the policy at the earlier time step, $\\pi_{t-1}$, we can maximize the return at different steps backward in time — this is essentially\nDP\n.\n$$\n\\underbrace{\\max_{\\pi_0} \\Big( \\mathbb{E}[r(s_0, a_0)]+ \\underbrace{\\max_{\\pi_1} \\Big(\\mathbb{E}[...] + \\underbrace{\\max_{\\pi_T} \\mathbb{E}[r(s_T, a_T)]}_\\text{1st maximization} \\Big)}_\\text{second but last maximization} \\Big)}_\\text{last maximization}\n$$\nwhere we consider $\\gamma=1$.\nSo we start the optimization from the last timestep $T$:\n$$\n\\text{maximize } \\mathbb{E}_{(s_T, a_T) \\sim \\rho_{\\pi}} [ r(s_T, a_T) ] \\text{ s.t. } \\mathcal{H}(\\pi_T) - \\mathcal{H}_0 \\geq 0\n$$\nFirst, let us define the following functions:\n$$\n\\begin{aligned}\nh(\\pi_T) &= \\mathcal{H}(\\pi_T) - \\mathcal{H}_0 = \\mathbb{E}_{(s_T, a_T) \\sim \\rho_{\\pi}} [-\\log \\pi_T(a_T\\vert s_T)] - \\mathcal{H}_0\\\\\nf(\\pi_T) &= \\begin{cases}\n\\mathbb{E}_{(s_T, a_T) \\sim \\rho_{\\pi}} [ r(s_T, a_T) ], & \\text{if }h(\\pi_T) \\geq 0 \\\\\n-\\infty, & \\text{otherwise}\n\\end{cases}\n\\end{aligned}\n$$\nAnd the optimization becomes:\n$$\n\\text{maximize } f(\\pi_T) \\text{ s.t. } h(\\pi_T) \\geq 0\n$$\nTo solve the maximization optimization with inequality constraint, we can construct a\nLagrangian expression\nwith a Lagrange multiplier (also known as “dual variable”), $\\alpha_T$:\n$$\nL(\\pi_T, \\alpha_T) = f(\\pi_T) + \\alpha_T h(\\pi_T)\n$$\nConsidering the case when we try to\nminimize $L(\\pi_T, \\alpha_T)$ with respect to $\\alpha_T$\n- given a particular value $\\pi_T$,\nIf the constraint is satisfied, $h(\\pi_T) \\geq 0$, at best we can set $\\alpha_T=0$ since we have no control over the value of $f(\\pi_T)$. Thus, $L(\\pi_T, 0) = f(\\pi_T)$.\nIf the constraint is invalidated, $h(\\pi_T) < 0$, we can achieve $L(\\pi_T, \\alpha_T) \\to -\\infty$ by taking  $\\alpha_T \\to \\infty$. Thus, $L(\\pi_T, \\infty) = -\\infty = f(\\pi_T)$.\nIn either case, we can recover the following equation,\n$$\nf(\\pi_T) = \\min_{\\alpha_T \\geq 0} L(\\pi_T, \\alpha_T)\n$$\nAt the same time, we want to maximize $f(\\pi_T)$,\n$$\n\\max_{\\pi_T} f(\\pi_T) = \\min_{\\alpha_T \\geq 0} \\max_{\\pi_T} L(\\pi_T, \\alpha_T)\n$$\nTherefore, to maximize $f(\\pi_T)$, the dual problem is listed as below. Note that to make sure $\\max_{\\pi_T} f(\\pi_T)$ is properly maximized and would not become $-\\infty$, the constraint has to be satisfied.\n$$\n\\begin{aligned}\n\\max_{\\pi_T} \\mathbb{E}[ r(s_T, a_T) ]\n&= \\max_{\\pi_T} f(\\pi_T) \\\\\n&= \\min_{\\alpha_T \\geq 0}  \\max_{\\pi_T} L(\\pi_T, \\alpha_T) \\\\\n&= \\min_{\\alpha_T \\geq 0}  \\max_{\\pi_T} f(\\pi_T) + \\alpha_T h(\\pi_T) \\\\ \n&= \\min_{\\alpha_T \\geq 0}  \\max_{\\pi_T} \\mathbb{E}_{(s_T, a_T) \\sim \\rho_{\\pi}} [ r(s_T, a_T) ] + \\alpha_T ( \\mathbb{E}_{(s_T, a_T) \\sim \\rho_{\\pi}} [-\\log \\pi_T(a_T\\vert s_T)] - \\mathcal{H}_0) \\\\ \n&= \\min_{\\alpha_T \\geq 0}  \\max_{\\pi_T} \\mathbb{E}_{(s_T, a_T) \\sim \\rho_{\\pi}} [ r(s_T, a_T)  - \\alpha_T \\log \\pi_T(a_T\\vert s_T)] - \\alpha_T \\mathcal{H}_0 \\\\\n&= \\min_{\\alpha_T \\geq 0}  \\max_{\\pi_T} \\mathbb{E}_{(s_T, a_T) \\sim \\rho_{\\pi}} [ r(s_T, a_T)  + \\alpha_T \\mathcal{H}(\\pi_T) - \\alpha_T \\mathcal{H}_0 ]\n\\end{aligned}\n$$\nWe could compute the optimal $\\pi_T$ and $\\alpha_T$ iteratively. First given the current $\\alpha_T$, get the best policy $\\pi_T^{*}$ that maximizes $L(\\pi_T^{*}, \\alpha_T)$. Then plug in $\\pi_T^{*}$ and compute $\\alpha_T^{*}$ that minimizes $L(\\pi_T^{*}, \\alpha_T)$. Assuming we have one neural network for policy and one network for temperature parameter, the iterative update process is more aligned with how we update network parameters during training.\n$$\n\\begin{aligned}\n\\pi^{*}_T\n&= \\arg\\max_{\\pi_T} \\mathbb{E}_{(s_T, a_T) \\sim \\rho_{\\pi}} [ r(s_T, a_T)  + \\alpha_T \\mathcal{H}(\\pi_T) - \\alpha_T \\mathcal{H}_0 ] \\\\\n\\color{blue}{\\alpha^{*}_T}\n&\\color{blue}{=} \\color{blue}{\\arg\\min_{\\alpha_T \\geq 0} \\mathbb{E}_{(s_T, a_T) \\sim \\rho_{\\pi^{*}}} [\\alpha_T \\mathcal{H}(\\pi^{*}_T) - \\alpha_T \\mathcal{H}_0 ]}\n\\end{aligned}\n$$\n$$\n\\text{Thus, }\\max_{\\pi_T} \\mathbb{E} [ r(s_T, a_T) ] \n= \\mathbb{E}_{(s_T, a_T) \\sim \\rho_{\\pi^{*}}} [ r(s_T, a_T)  + \\alpha^{*}_T \\mathcal{H}(\\pi^{*}_T) - \\alpha^{*}_T \\mathcal{H}_0 ]\n$$\nNow let’s go back to the soft Q value function:\n$$\n\\begin{aligned}\nQ_{T-1}(s_{T-1}, a_{T-1}) \n&= r(s_{T-1}, a_{T-1}) + \\mathbb{E} [Q(s_T, a_T) - \\alpha_T \\log \\pi(a_T \\vert s_T)] \\\\\n&= r(s_{T-1}, a_{T-1}) + \\mathbb{E} [r(s_T, a_T)] + \\alpha_T \\mathcal{H}(\\pi_T) \\\\\nQ_{T-1}^{*}(s_{T-1}, a_{T-1}) \n&= r(s_{T-1}, a_{T-1}) + \\max_{\\pi_T} \\mathbb{E} [r(s_T, a_T)] +  \\alpha_T \\mathcal{H}(\\pi^{*}_T) & \\text{; plug in the optimal }\\pi_T^{*}\n\\end{aligned}\n$$\nTherefore the expected return is as follows, when we take one step further back to the time step $T-1$:\n$$\n\\begin{aligned}\n&\\max_{\\pi_{T-1}}\\Big(\\mathbb{E}[r(s_{T-1}, a_{T-1})] + \\max_{\\pi_T} \\mathbb{E}[r(s_T, a_T] \\Big) \\\\\n&= \\max_{\\pi_{T-1}} \\Big( Q^{*}_{T-1}(s_{T-1}, a_{T-1}) - \\alpha^{*}_T \\mathcal{H}(\\pi^{*}_T) \\Big) & \\text{; should s.t. } \\mathcal{H}(\\pi_{T-1}) - \\mathcal{H}_0 \\geq 0 \\\\\n&= \\min_{\\alpha_{T-1} \\geq 0}  \\max_{\\pi_{T-1}} \\Big( Q^{*}_{T-1}(s_{T-1}, a_{T-1}) - \\alpha^{*}_T \\mathcal{H}(\\pi^{*}_T) + \\alpha_{T-1} \\big( \\mathcal{H}(\\pi_{T-1}) - \\mathcal{H}_0 \\big) \\Big) & \\text{; dual problem w/ Lagrangian.} \\\\\n&= \\min_{\\alpha_{T-1} \\geq 0}  \\max_{\\pi_{T-1}} \\Big( Q^{*}_{T-1}(s_{T-1}, a_{T-1}) + \\alpha_{T-1} \\mathcal{H}(\\pi_{T-1}) - \\alpha_{T-1}\\mathcal{H}_0 \\Big) - \\alpha^{*}_T \\mathcal{H}(\\pi^{*}_T)\n\\end{aligned}\n$$\nSimilar to the previous step,\n$$\n\\begin{aligned}\n\\pi^{*}_{T-1} &= \\arg\\max_{\\pi_{T-1}} \\mathbb{E}_{(s_{T-1}, a_{T-1}) \\sim \\rho_\\pi} [Q^{*}_{T-1}(s_{T-1}, a_{T-1}) + \\alpha_{T-1} \\mathcal{H}(\\pi_{T-1}) - \\alpha_{T-1} \\mathcal{H}_0 ] \\\\\n\\color{green}{\\alpha^{*}_{T-1}} &\\color{green}{=} \\color{green}{\\arg\\min_{\\alpha_{T-1} \\geq 0} \\mathbb{E}_{(s_{T-1}, a_{T-1}) \\sim \\rho_{\\pi^{*}}} [ \\alpha_{T-1} \\mathcal{H}(\\pi^{*}_{T-1}) - \\alpha_{T-1}\\mathcal{H}_0 ]}\n\\end{aligned}\n$$\nThe equation for updating $\\alpha_{T-1}$ in\ngreen\nhas the same format as the equation for updating $\\alpha_{T-1}$ in\nblue\nabove. By repeating this process, we can learn the optimal temperature parameter in every step by minimizing the same objective function:\n$$\nJ(\\alpha) = \\mathbb{E}_{a_t \\sim \\pi_t} [-\\alpha \\log \\pi_t(a_t \\mid s_t) - \\alpha \\mathcal{H}_0]\n$$\nThe final algorithm is same as SAC except for learning $\\alpha$ explicitly with respect to the objective $J(\\alpha)$ (see Fig. 7):\nThe soft actor-critic algorithm with automatically adjusted temperature. (Image source:\noriginal paper\n)\nTD3\n#\n[\npaper\n|\ncode\n]\nThe Q-learning algorithm is commonly known to suffer from the overestimation of the value function. This overestimation can propagate through the training iterations and negatively affect the policy. This property directly motivated\nDouble Q-learning\nand\nDouble DQN\n: the action selection and Q-value update are decoupled by using two value networks.\nTwin Delayed Deep Deterministic\n(short for\nTD3\n;\nFujimoto et al., 2018\n) applied a couple of tricks on\nDDPG\nto prevent the overestimation of the value function:\n(1)\nClipped Double Q-learning\n: In Double Q-Learning, the action selection and Q-value estimation are made by two networks separately. In the DDPG setting, given two deterministic actors $(\\mu_{\\theta_1}, \\mu_{\\theta_2})$ with two corresponding critics $(Q_{w_1}, Q_{w_2})$, the Double Q-learning Bellman targets look like:\n$$\n\\begin{aligned}\ny_1 &= r + \\gamma Q_{w_2}(s', \\mu_{\\theta_1}(s'))\\\\\ny_2 &= r + \\gamma Q_{w_1}(s', \\mu_{\\theta_2}(s'))\n\\end{aligned}\n$$\nHowever, due to the slow changing policy, these two networks could be too similar to make independent decisions. The\nClipped Double Q-learning\ninstead uses the minimum estimation among two so as to favor underestimation bias which is hard to propagate through training:\n$$\n\\begin{aligned}\ny_1 &= r + \\gamma \\min_{i=1,2}Q_{w_i}(s', \\mu_{\\theta_1}(s'))\\\\\ny_2 &= r + \\gamma \\min_{i=1,2} Q_{w_i}(s', \\mu_{\\theta_2}(s'))\n\\end{aligned}\n$$\n(2)\nDelayed update of Target and Policy Networks\n:  In the\nactor-critic\nmodel, policy and value updates are deeply coupled: Value estimates diverge through overestimation when the policy is poor, and the policy will become poor if the value estimate itself is inaccurate.\nTo reduce the variance, TD3 updates the policy at a lower frequency than the Q-function. The policy network stays the same until the value error is small enough after several updates. The idea is similar to how the periodically-updated target network stay as a stable objective in\nDQN\n.\n(3)\nTarget Policy Smoothing\n: Given a concern with deterministic policies that they can overfit to narrow peaks in the value function, TD3 introduced a smoothing regularization strategy on the value function: adding a small amount of clipped random noises to the selected action and averaging over mini-batches.\n$$\n\\begin{aligned}\ny &= r + \\gamma Q_w (s', \\mu_{\\theta}(s') + \\epsilon) & \\\\\n\\epsilon &\\sim \\text{clip}(\\mathcal{N}(0, \\sigma), -c, +c) & \\scriptstyle{\\text{ ; clipped random noises.}}\n\\end{aligned}\n$$\nThis approach mimics the idea of\nSARSA\nupdate and enforces that similar actions should have similar values.\nHere is the final algorithm:\nTD3 Algorithm. (Image source:\nFujimoto et al., 2018\n)\nSVPG\n#\n[\npaper\n|\ncode\nfor SVPG]\nStein Variational Policy Gradient (\nSVPG\n;\nLiu et al, 2017\n) applies the\nStein\nvariational gradient descent (\nSVGD\n;\nLiu and Wang, 2016\n) algorithm to update the policy parameter $\\theta$.\nIn the setup of maximum entropy policy optimization, $\\theta$ is considered as a random variable $\\theta \\sim q(\\theta)$ and the model is expected to learn this distribution $q(\\theta)$. Assuming we know a prior on how $q$ might look like, $q_0$, and we would like to guide the learning process to not make $\\theta$ too far away from $q_0$ by optimizing the following objective function:\n$$\n\\hat{J}(\\theta) = \\mathbb{E}_{\\theta \\sim q} [J(\\theta)] - \\alpha D_\\text{KL}(q\\|q_0)\n$$\nwhere $\\mathbb{E}_{\\theta \\sim q} [R(\\theta)]$ is the expected reward when $\\theta \\sim q(\\theta)$ and $D_\\text{KL}$ is the KL divergence.\nIf we don’t have any prior information, we might set $q_0$ as a uniform distribution and set $q_0(\\theta)$ to a constant. Then the above objective function becomes\nSAC\n, where the entropy term encourages exploration:\n$$\n\\begin{aligned}\n\\hat{J}(\\theta) \n&= \\mathbb{E}_{\\theta \\sim q} [J(\\theta)] - \\alpha D_\\text{KL}(q\\|q_0) \\\\\n&= \\mathbb{E}_{\\theta \\sim q} [J(\\theta)] - \\alpha \\mathbb{E}_{\\theta \\sim q} [\\log q(\\theta) - \\log q_0(\\theta)] \\\\\n&= \\mathbb{E}_{\\theta \\sim q} [J(\\theta)] + \\alpha H(q(\\theta))\n\\end{aligned}\n$$\nLet’s take the derivative of $\\hat{J}(\\theta) = \\mathbb{E}_{\\theta \\sim q} [J(\\theta)] - \\alpha D_\\text{KL}(q|q_0)$ w.r.t. $q$:\n$$\n\\begin{aligned}\n\\nabla_q \\hat{J}(\\theta) \n&= \\nabla_q \\big( \\mathbb{E}_{\\theta \\sim q} [J(\\theta)] - \\alpha D_\\text{KL}(q\\|q_0) \\big) \\\\\n&= \\nabla_q \\int_\\theta \\big( q(\\theta) J(\\theta) - \\alpha q(\\theta)\\log q(\\theta) + \\alpha q(\\theta) \\log q_0(\\theta) \\big) \\\\\n&= \\int_\\theta \\big( J(\\theta) - \\alpha \\log q(\\theta) -\\alpha + \\alpha \\log q_0(\\theta) \\big) \\\\\n&= 0\n\\end{aligned}\n$$\nThe optimal distribution is:\n$$\n\\log q^{*}(\\theta) = \\frac{1}{\\alpha} J(\\theta) + \\log q_0(\\theta) - 1 \\text{ thus } \\underbrace{ q^{*}(\\theta) }_\\textrm{\"posterior\"} \\propto \\underbrace{\\exp ( J(\\theta) / \\alpha )}_\\textrm{\"likelihood\"} \\underbrace{q_0(\\theta)}_\\textrm{prior}\n$$\nThe temperature $\\alpha$ decides a tradeoff between exploitation and exploration. When $\\alpha \\rightarrow 0$, $\\theta$ is updated only according to the expected return $J(\\theta)$. When $\\alpha \\rightarrow \\infty$, $\\theta$ always follows the prior belief.\nWhen using the SVGD method to estimate the target posterior distribution $q(\\theta)$, it relies on a set of particle $\\{\\theta_i\\}_{i=1}^n$ (independently trained policy agents) and each is updated:\n$$\n\\theta_i \\gets \\theta_i + \\epsilon \\phi^{*}(\\theta_i) \\text{ where } \\phi^{*} = \\max_{\\phi \\in \\mathcal{H}} \\{ - \\nabla_\\epsilon D_\\text{KL} (q'_{[\\theta + \\epsilon \\phi(\\theta)]} \\| q) \\text{ s.t. } \\|\\phi\\|_{\\mathcal{H}} \\leq 1\\} \n$$\nwhere $\\epsilon$ is a learning rate and $\\phi^{*}$ is the unit ball of a\nRKHS\n(reproducing kernel Hilbert space) $\\mathcal{H}$ of $\\theta$-shaped value vectors that maximally decreases the KL divergence between the particles and the target distribution. $q’(.)$ is the distribution of $\\theta + \\epsilon \\phi(\\theta)$.\nComparing different gradient-based update methods:\nMethod\nUpdate space\nPlain gradient\n$\\Delta \\theta$ on the parameter space\nNatural gradient\n$\\Delta \\theta$ on the search distribution space\nSVGD\n$\\Delta \\theta$ on the kernel function space (edited)\nOne\nestimation\nof $\\phi^{*}$ has the following form. A positive definite kernel $k(\\vartheta, \\theta)$, i.e. a Gaussian\nradial basis function\n, measures the similarity between particles.\n$$\n\\begin{aligned}\n\\phi^{*}(\\theta_i) \n&= \\mathbb{E}_{\\vartheta \\sim q'} [\\nabla_\\vartheta \\log q(\\vartheta) k(\\vartheta, \\theta_i) + \\nabla_\\vartheta k(\\vartheta, \\theta_i)]\\\\\n&= \\frac{1}{n} \\sum_{j=1}^n [\\color{red}{\\nabla_{\\theta_j} \\log q(\\theta_j) k(\\theta_j, \\theta_i)} + \\color{green}{\\nabla_{\\theta_j} k(\\theta_j, \\theta_i)}] & \\scriptstyle{\\text{;approximate }q'\\text{ with current particle values}}\n\\end{aligned}\n$$\nThe first term in\nred\nencourages $\\theta_i$ learning towards the high probability regions of $q$ that is shared across similar particles. => to be similar to other particles\nThe second term in\ngreen\npushes particles away from each other and therefore diversifies the policy. => to be dissimilar to other particles\nUsually the temperature $\\alpha$ follows an annealing scheme so that the training process does more exploration at the beginning but more exploitation at a later stage.\nIMPALA\n#\n[\npaper\n|\ncode\n]\nIn order to scale up RL training to achieve a very high throughput,\nIMPALA\n(“Importance Weighted Actor-Learner Architecture”) framework decouples acting from learning on top of basic actor-critic setup and learns from all experience trajectories with\nV-trace\noff-policy correction.\nMultiple actors generate experience in parallel, while the learner optimizes both policy and value function parameters using all the generated experience. Actors update their parameters with the latest policy from the learner periodically. Because acting and learning are decoupled, we can add many more actor machines to generate a lot more trajectories per time unit. As the training policy and the behavior policy are not totally synchronized, there is a\ngap\nbetween them and thus we need off-policy corrections.\nLet the value function $V_\\theta$ parameterized by $\\theta$ and the policy $\\pi_\\phi$ parameterized by $\\phi$. Also we know the trajectories in the replay buffer are collected by a slightly older policy $\\mu$.\nAt the training time $t$, given $(s_t, a_t, s_{t+1}, r_t)$, the value function parameter $\\theta$ is learned through an L2 loss between the current value and a V-trace value target. The $n$-step V-trace target is defined as:\n$$\n\\begin{aligned}\nv_t  &= V_\\theta(s_t) + \\sum_{i=t}^{t+n-1} \\gamma^{i-t} \\big(\\prod_{j=t}^{i-1} c_j\\big) \\color{red}{\\delta_i V} \\\\\n&= V_\\theta(s_t) + \\sum_{i=t}^{t+n-1} \\gamma^{i-t} \\big(\\prod_{j=t}^{i-1} c_j\\big) \\color{red}{\\rho_i (r_i + \\gamma V_\\theta(s_{i+1}) - V_\\theta(s_i))}\n\\end{aligned}\n$$\nwhere the\nred\npart $\\delta_i V$ is a temporal difference for $V$. $\\rho_i = \\min\\big(\\bar{\\rho}, \\frac{\\pi(a_i \\vert s_i)}{\\mu(a_i \\vert s_i)}\\big)$ and $c_j = \\min\\big(\\bar{c}, \\frac{\\pi(a_j \\vert s_j)}{\\mu(a_j \\vert s_j)}\\big)$ are\ntruncated\nimportance sampling (IS)\nweights\n. The product of $c_t, \\dots, c_{i-1}$ measures how much a temporal difference $\\delta_i V$ observed at time $i$ impacts the update of the value function at a previous time $t$. In the on-policy case, we have $\\rho_i=1$ and $c_j=1$ (assuming $\\bar{c} \\geq 1$) and therefore the V-trace target becomes on-policy $n$-step Bellman target.\n$\\bar{\\rho}$ and $\\bar{c}$ are two truncation constants with $\\bar{\\rho} \\geq \\bar{c}$. $\\bar{\\rho}$ impacts the fixed-point of the value function we converge to and $\\bar{c}$ impacts the speed of convergence. When $\\bar{\\rho} =\\infty$ (untruncated), we converge to the value function of the target policy $V^\\pi$; when $\\bar{\\rho}$ is close to 0, we evaluate the value function of the behavior policy $V^\\mu$; when in-between, we evaluate a policy between $\\pi$ and $\\mu$.\nThe value function parameter is therefore updated in the direction of:\n$$\n\\Delta\\theta = (v_t - V_\\theta(s_t))\\nabla_\\theta V_\\theta(s_t)\n$$\nThe policy parameter $\\phi$ is updated through policy gradient,\n$$\n\\begin{aligned}\n\\Delta \\phi \n&= \\rho_t \\nabla_\\phi \\log \\pi_\\phi(a_t \\vert s_t) \\big(r_t + \\gamma v_{t+1} - V_\\theta(s_t)\\big) + \\nabla_\\phi H(\\pi_\\phi)\\\\\n&= \\rho_t \\nabla_\\phi \\log \\pi_\\phi(a_t \\vert s_t) \\big(r_t + \\gamma v_{t+1} - V_\\theta(s_t)\\big) - \\nabla_\\phi \\sum_a \\pi_\\phi(a\\vert s_t)\\log \\pi_\\phi(a\\vert s_t)\n\\end{aligned}\n$$\nwhere $r_t + \\gamma v_{t+1}$ is the estimated Q value, from which a state-dependent baseline $V_\\theta(s_t)$ is subtracted. $H(\\pi_\\phi)$ is an entropy bonus to encourage exploration.\nIn the experiments, IMPALA is used to train one agent over multiple tasks. Two different model architectures are involved, a shallow model (left) and a deep residual model (right).\nQuick Summary\n#\nAfter reading through all the algorithms above, I list a few building blocks or principles that seem to be common among them:\nTry to reduce the variance and keep the bias unchanged to stabilize learning.\nOff-policy gives us better exploration and helps us use data samples more efficiently.\nExperience replay (training data sampled from a replay memory buffer);\nTarget network that is either frozen periodically or updated slower than the actively learned policy network;\nBatch normalization;\nEntropy-regularized reward;\nThe critic and actor can share lower layer parameters of the network and two output heads for policy and value functions.\nIt is possible to learn with deterministic policy rather than stochastic one.\nPut constraint on the divergence between policy updates.\nNew optimization methods (such as K-FAC).\nEntropy maximization of the policy helps encourage exploration.\nTry not to overestimate the value function.\nThink twice whether the policy and value network should share parameters.\nTBA more.\nCited as:\n@article{weng2018PG,\n  title   = \"Policy Gradient Algorithms\",\n  author  = \"Weng, Lilian\",\n  journal = \"lilianweng.github.io\",\n  year    = \"2018\",\n  url     = \"https://lilianweng.github.io/posts/2018-04-08-policy-gradient/\"\n}\nReferences\n#\n[1] jeremykun.com\nMarkov Chain Monte Carlo Without all the Bullshit\n[2] Richard S. Sutton and Andrew G. Barto.\nReinforcement Learning: An Introduction; 2nd Edition\n. 2017.\n[3] John Schulman, et al.\n“High-dimensional continuous control using generalized advantage estimation.”\nICLR 2016.\n[4] Thomas Degris, Martha White, and Richard S. Sutton.\n“Off-policy actor-critic.”\nICML 2012.\n[5] timvieira.github.io\nImportance sampling\n[6] Mnih, Volodymyr, et al.\n“Asynchronous methods for deep reinforcement learning.”\nICML. 2016.\n[7] David Silver, et al.\n“Deterministic policy gradient algorithms.”\nICML. 2014.\n[8] Timothy P. Lillicrap, et al.\n“Continuous control with deep reinforcement learning.”\narXiv preprint arXiv:1509.02971 (2015).\n[9] Ryan Lowe, et al.\n“Multi-agent actor-critic for mixed cooperative-competitive environments.”\nNIPS. 2017.\n[10] John Schulman, et al.\n“Trust region policy optimization.”\nICML. 2015.\n[11] Ziyu Wang, et al.\n“Sample efficient actor-critic with experience replay.”\nICLR 2017.\n[12] Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare.\n“Safe and efficient off-policy reinforcement learning”\nNIPS. 2016.\n[13] Yuhuai Wu, et al.\n“Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation.”\nNIPS. 2017.\n[14] kvfrans.com\nA intuitive explanation of natural gradient descent\n[15] Sham Kakade.\n“A Natural Policy Gradient.”\n. NIPS. 2002.\n[16]\n“Going Deeper Into Reinforcement Learning: Fundamentals of Policy Gradients.”\n- Seita’s Place, Mar 2017.\n[17]\n“Notes on the Generalized Advantage Estimation Paper.”\n- Seita’s Place, Apr, 2017.\n[18] Gabriel Barth-Maron, et al.\n“Distributed Distributional Deterministic Policy Gradients.”\nICLR 2018 poster.\n[19] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.\n“Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.”\narXiv preprint arXiv:1801.01290 (2018).\n[20] Scott Fujimoto, Herke van Hoof, and Dave Meger.\n“Addressing Function Approximation Error in Actor-Critic Methods.”\narXiv preprint arXiv:1802.09477 (2018).\n[21] Tuomas Haarnoja, et al.\n“Soft Actor-Critic Algorithms and Applications.”\narXiv preprint arXiv:1812.05905 (2018).\n[22] David Knowles.\n“Lagrangian Duality for Dummies”\nNov 13, 2010.\n[23] Yang Liu, et al.\n“Stein variational policy gradient.”\narXiv preprint arXiv:1704.02399 (2017).\n[24] Qiang Liu and Dilin Wang.\n“Stein variational gradient descent: A general purpose bayesian inference algorithm.”\nNIPS. 2016.\n[25] Lasse Espeholt, et al.\n“IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures”\narXiv preprint 1802.01561 (2018).\n[26] Karl Cobbe, et al.\n“Phasic Policy Gradient.”\narXiv preprint arXiv:2009.04416 (2020).\n[27] Chloe Ching-Yun Hsu, et al.\n“Revisiting Design Choices in Proximal Policy Optimization.”\narXiv preprint arXiv:2009.10897 (2020).",
  "metadata": {
    "title": "Policy Gradient Algorithms | Lil'Log",
    "description": "[Updated on 2018-06-30: add two new policy gradient methods, SAC and D4PG.]\n\n[Updated on 2018-09-30: add a new policy gradient method, TD3.]\n\n[Updated on 2019-02-09: add SAC with automatically adjusted temperature].\n\n[Updated on 2019-06-26: Thanks to Chanseok, we have a version of this post in Korean].\n\n[Updated on 2019-09-12: add a new policy gradient method SVPG.]\n\n[Updated on 2019-12-22: add a new policy gradient method IMPALA.]\n\n[Updated on 2020-10-15: add a new policy gradient method PPG & some new discussion in PPO.]\n\n[Updated on 2021-09-19: Thanks to Wenhao & 爱吃猫的鱼, we have this post in Chinese1 & Chinese2].",
    "keywords": "reinforcement-learning, long-read, math-heavy",
    "author": "Lilian Weng",
    "og:title": "Policy Gradient Algorithms",
    "og:description": "[Updated on 2018-06-30: add two new policy gradient methods, SAC and D4PG.]\n\n[Updated on 2018-09-30: add a new policy gradient method, TD3.]\n\n[Updated on 2019-02-09: add SAC with automatically adjusted temperature].\n\n[Updated on 2019-06-26: Thanks to Chanseok, we have a version of this post in Korean].\n\n[Updated on 2019-09-12: add a new policy gradient method SVPG.]\n\n[Updated on 2019-12-22: add a new policy gradient method IMPALA.]\n\n[Updated on 2020-10-15: add a new policy gradient method PPG & some new discussion in PPO.]\n\n[Updated on 2021-09-19: Thanks to Wenhao & 爱吃猫的鱼, we have this post in Chinese1 & Chinese2].",
    "og:type": "article",
    "og:url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/",
    "twitter:card": "summary",
    "twitter:title": "Policy Gradient Algorithms",
    "twitter:description": "[Updated on 2018-06-30: add two new policy gradient methods, SAC and D4PG.]\n\n[Updated on 2018-09-30: add a new policy gradient method, TD3.]\n\n[Updated on 2019-02-09: add SAC with automatically adjusted temperature].\n\n[Updated on 2019-06-26: Thanks to Chanseok, we have a version of this post in Korean].\n\n[Updated on 2019-09-12: add a new policy gradient method SVPG.]\n\n[Updated on 2019-12-22: add a new policy gradient method IMPALA.]\n\n[Updated on 2020-10-15: add a new policy gradient method PPG & some new discussion in PPO.]\n\n[Updated on 2021-09-19: Thanks to Wenhao & 爱吃猫的鱼, we have this post in Chinese1 & Chinese2].",
    "article:section": "posts",
    "article:published_time": "2018-04-08T00:00:00+00:00",
    "article:modified_time": "2018-04-08T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/ebc73787cbe5/code/body-000.txt",
      "chars": 226,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/general_form_policy_gradient.png",
      "bytes": 259350,
      "reused": false,
      "local_path": "attachments/ebc73787cbe5/images/body-394ec3f8b6a6ccc3dace.png",
      "src": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/general_form_policy_gradient.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/A3C_vs_A2C.png",
      "bytes": 205392,
      "reused": false,
      "local_path": "attachments/ebc73787cbe5/images/body-9c60abbe659d28457d7a.png",
      "src": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/A3C_vs_A2C.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/DDPG_algo.png",
      "bytes": 275060,
      "reused": false,
      "local_path": "attachments/ebc73787cbe5/images/body-a8004105b4a88c05a952.png",
      "src": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/DDPG_algo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/D4PG_algo.png",
      "bytes": 581593,
      "reused": false,
      "local_path": "attachments/ebc73787cbe5/images/body-7bc12620d13d41e17f0a.png",
      "src": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/D4PG_algo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/MADDPG.png",
      "bytes": 215084,
      "reused": false,
      "local_path": "attachments/ebc73787cbe5/images/body-a1617c1aadd4262ae84a.png",
      "src": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/MADDPG.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/ppo-loss-functions.png",
      "bytes": 181566,
      "reused": false,
      "local_path": "attachments/ebc73787cbe5/images/body-01ca9a7ea1e7d1cd82a2.png",
      "src": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/ppo-loss-functions.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/PPG_algo.png",
      "bytes": 294810,
      "reused": false,
      "local_path": "attachments/ebc73787cbe5/images/body-a362a645159f0dafa82d.png",
      "src": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/PPG_algo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/PPG_exp.png",
      "bytes": 860130,
      "reused": false,
      "local_path": "attachments/ebc73787cbe5/images/body-fafbbe0c69813ea86bf2.png",
      "src": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/PPG_exp.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/SAC_algo.png",
      "bytes": 742699,
      "reused": false,
      "local_path": "attachments/ebc73787cbe5/images/body-d58e26716f1f269dae8a.png",
      "src": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/SAC_algo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/SAC2_algo.png",
      "bytes": 300409,
      "reused": false,
      "local_path": "attachments/ebc73787cbe5/images/body-5e0d3d90650b6cca848e.png",
      "src": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/SAC2_algo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/TD3.png",
      "bytes": 497568,
      "reused": false,
      "local_path": "attachments/ebc73787cbe5/images/body-a29af53d479214903180.png",
      "src": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/TD3.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/SVPG.png",
      "bytes": 332210,
      "reused": false,
      "local_path": "attachments/ebc73787cbe5/images/body-0061164751ee7e193a70.png",
      "src": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/SVPG.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/IMPALA.png",
      "bytes": 253972,
      "reused": false,
      "local_path": "attachments/ebc73787cbe5/images/body-58ac4a854eb578c151aa.png",
      "src": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/IMPALA.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/IMPALA-arch.png",
      "bytes": 567524,
      "reused": false,
      "local_path": "attachments/ebc73787cbe5/images/body-6b398117113938e66d50.png",
      "src": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/IMPALA-arch.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://talkingaboutme.tistory.com/entry/RL-Policy-Gradient-Algorithms",
      "label": "Korean",
      "type": "a"
    },
    {
      "href": "https://tomaxent.com/2019/04/14/%E7%AD%96%E7%95%A5%E6%A2%AF%E5%BA%A6%E6%96%B9%E6%B3%95/",
      "label": "Chinese1",
      "type": "a"
    },
    {
      "href": "https://paperexplained.cn/articles/article/detail/31/",
      "label": "Chinese2",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/",
      "label": "“A (Long) Peek into Reinforcement Learning » Key Concepts”",
      "type": "a"
    },
    {
      "href": "https://jeremykun.com/2015/04/06/markov-chain-monte-carlo-without-all-the-bullshit/",
      "label": "this",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Curse_of_dimensionality",
      "label": "curse of dimensionality",
      "type": "a"
    },
    {
      "href": "http://incompleteideas.net/book/bookdraft2017nov5.pdf",
      "label": "Sutton & Barto, 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1506.02438.pdf",
      "label": "GAE",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1506.02438",
      "label": "Schulman et al., 2016",
      "type": "a"
    },
    {
      "href": "https://danieltakeshi.github.io/2017/04/02/notes-on-the-generalized-advantage-estimation-paper/",
      "label": "post",
      "type": "a"
    },
    {
      "href": "https://danieltakeshi.github.io/2017/03/28/going-deeper-into-reinforcement-learning-fundamentals-of-policy-gradients/",
      "label": "post",
      "type": "a"
    },
    {
      "href": "http://timvieira.github.io/blog/post/2014/12/21/importance-sampling/",
      "label": "importance weight",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1205.4839.pdf",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1602.01783",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://github.com/dennybritz/reinforcement-learning/tree/master/PolicyGradient/a3c",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/baselines/blob/master/baselines/a2c/a2c.py",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://blog.openai.com/baselines-acktr-a2c/",
      "label": "shown",
      "type": "a"
    },
    {
      "href": "https://hal.inria.fr/file/index/docid/938992/filename/dpg-icml2014.pdf",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1509.02971.pdf",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/baselines/tree/master/baselines/ddpg",
      "label": "code",
      "type": "a"
    },
    {
      "href": "http://proceedings.mlr.press/v37/ioffe15.pdf",
      "label": "Batch normalization",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=SyZipzbCb",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1511.05952",
      "label": "PER",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1706.02275.pdf",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/maddpg",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1502.05477.pdf",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/baselines/tree/master/baselines/trpo_mpi",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2017-08-20-gan/",
      "label": "KL divergence",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence",
      "label": "KL-divergence",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1707.06347.pdf",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/baselines/tree/master/baselines/ppo1",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.10897",
      "label": "Hsu et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.04416",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/phasic-policy-gradient",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1912.01588",
      "label": "Procgen",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1611.01224.pdf",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/baselines/tree/master/baselines/acer",
      "label": "code",
      "type": "a"
    },
    {
      "href": "http://papers.nips.cc/paper/6538-safe-and-efficient-off-policy-reinforcement-learning.pdf",
      "label": "Retrace",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1708.05144.pdf",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/baselines/tree/master/baselines/acktr",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1503.05671.pdf",
      "label": "K-FAC",
      "type": "a"
    },
    {
      "href": "http://kvfrans.com/a-intuitive-explanation-of-natural-gradient-descent/",
      "label": "Here",
      "type": "a"
    },
    {
      "href": "http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.452.7280&rep=rep1&type=pdf",
      "label": "Natural Gradient Works Efficiently in Learning",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/2073-a-natural-policy-gradient.pdf",
      "label": "A Natural Policy Gradient",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Kronecker_product",
      "label": "Wiki: Kronecker product",
      "type": "a"
    },
    {
      "href": "http://proceedings.mlr.press/v37/martens15.pdf",
      "label": "Optimizing neural networks with kronecker-factored approximate curvature.",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1801.01290",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://github.com/haarnoja/sac",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1702.08165",
      "label": "Soft Q-learning",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1812.05905",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://github.com/rail-berkeley/softlearning",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://cs.stanford.edu/people/davidknowles/lagrangian_duality.pdf",
      "label": "Lagrangian expression",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1802.09477",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://github.com/sfujim/TD3",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/3964-double-q-learning",
      "label": "Double Q-learning",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1509.06461",
      "label": "Double DQN",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1704.02399",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://github.com/dilinwang820/Stein-Variational-Gradient-Descent",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://www.cs.dartmouth.edu/~qliu/stein.html",
      "label": "Stein",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1608.04471",
      "label": "Liu and Wang, 2016",
      "type": "a"
    },
    {
      "href": "http://mlss.tuebingen.mpg.de/2015/slides/gretton/part_1.pdf",
      "label": "RKHS",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2019-09-05-evolution-strategies/",
      "label": "Natural gradient",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Radial_basis_function",
      "label": "radial basis function",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1802.01561",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://github.com/deepmind/scalable_agent",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1804.08617.pdf",
      "label": "“Distributed Distributional Deterministic Policy Gradients.”",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1801.01290.pdf",
      "label": "“Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.”",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/6338-stein-variational-gradient-descent-a-general-purpose-bayesian-inference-algorithm.pdf",
      "label": "“Stein variational gradient descent: A general purpose bayesian inference algorithm.”",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:34Z",
  "source_url": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/",
  "papers": [
    {
      "status": "ok",
      "url": "http://incompleteideas.net/book/bookdraft2017nov5.pdf",
      "bytes": 12613382,
      "reused": false,
      "local_path": "papers/repair/7ea9e7044b38fbe00b3f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1506.02438.pdf",
      "bytes": 1798117,
      "reused": false,
      "local_path": "papers/repair/be9d8b8466e33236e35e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1506.02438",
      "bytes": 1798117,
      "reused": true,
      "local_path": "papers/repair/0e1163be1a9e279df56e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1205.4839.pdf",
      "bytes": 1610077,
      "reused": false,
      "local_path": "papers/repair/5ab0424e24a7205667b6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1602.01783",
      "bytes": 2302720,
      "reused": true,
      "local_path": "papers/repair/4ebd69e62e1319a151ba.pdf"
    },
    {
      "status": "invalid_file_type",
      "url": "https://hal.inria.fr/file/index/docid/938992/filename/dpg-icml2014.pdf",
      "content_type": "text/html; charset=utf-8",
      "retried_at": "2026-10-20T17:39:22Z"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1509.02971.pdf",
      "bytes": 663643,
      "reused": false,
      "local_path": "papers/repair/cdfdf91c078ecbc5323f.pdf"
    },
    {
      "status": "ok",
      "url": "http://proceedings.mlr.press/v37/ioffe15.pdf",
      "bytes": 318669,
      "reused": false,
      "local_path": "papers/repair/278c974d701b74b5258e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1511.05952",
      "bytes": 1685573,
      "reused": false,
      "local_path": "papers/repair/232c5d788b53c3e83f28.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.02275.pdf",
      "bytes": 1502182,
      "reused": false,
      "local_path": "papers/repair/472ab942fc65a818d588.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1502.05477.pdf",
      "bytes": 1026237,
      "reused": false,
      "local_path": "papers/repair/40909aa64b5fd4aacb8a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1707.06347.pdf",
      "bytes": 2923532,
      "reused": false,
      "local_path": "papers/repair/9e750b40bed5bcd1df74.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.10897",
      "bytes": 2074951,
      "reused": false,
      "local_path": "papers/repair/cb042047afd4ec37247c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.04416",
      "bytes": 2019980,
      "reused": false,
      "local_path": "papers/repair/e6984c4570bddcc10cbe.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1912.01588",
      "bytes": 5158205,
      "reused": false,
      "local_path": "papers/repair/3231a0847e3982baa0a7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1611.01224.pdf",
      "bytes": 1856923,
      "reused": false,
      "local_path": "papers/repair/49a104905756e51b1082.pdf"
    },
    {
      "status": "ok",
      "url": "http://papers.nips.cc/paper/6538-safe-and-efficient-off-policy-reinforcement-learning.pdf",
      "bytes": 309421,
      "reused": false,
      "local_path": "papers/repair/a3c6ca6c0f5fa7ac392b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1708.05144.pdf",
      "bytes": 5151566,
      "reused": false,
      "local_path": "papers/repair/17bd1711b6b1c975cd6c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1503.05671.pdf",
      "bytes": 1801721,
      "reused": false,
      "local_path": "papers/repair/8348cabb06fa9a77e970.pdf"
    },
    {
      "status": "ok",
      "url": "https://papers.nips.cc/paper/2073-a-natural-policy-gradient.pdf",
      "bytes": 1517587,
      "reused": false,
      "local_path": "papers/repair/5e5447a001f6bb409591.pdf"
    },
    {
      "status": "ok",
      "url": "http://proceedings.mlr.press/v37/martens15.pdf",
      "bytes": 866438,
      "reused": false,
      "local_path": "papers/repair/cf98bb852a85a8c5a9be.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1801.01290",
      "bytes": 4388102,
      "reused": false,
      "local_path": "papers/repair/27be080213df12bc727f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1702.08165",
      "bytes": 3634452,
      "reused": false,
      "local_path": "papers/repair/cc481afce087c678bd26.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1812.05905",
      "bytes": 6750542,
      "reused": false,
      "local_path": "papers/repair/eb681f8429e54c661586.pdf"
    },
    {
      "status": "ok",
      "url": "https://cs.stanford.edu/people/davidknowles/lagrangian_duality.pdf",
      "bytes": 157901,
      "reused": false,
      "local_path": "papers/repair/6d312b63b4d7133c0639.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1802.09477",
      "bytes": 2744342,
      "reused": false,
      "local_path": "papers/repair/b1cc41c6920f45d34ed8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1509.06461",
      "bytes": 789062,
      "reused": false,
      "local_path": "papers/repair/1b6dc947b906b3bd9192.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1704.02399",
      "bytes": 1321151,
      "reused": false,
      "local_path": "papers/repair/dd42530ceff4594ef267.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1608.04471",
      "bytes": 595069,
      "reused": false,
      "local_path": "papers/repair/55aec624e4d940526da7.pdf"
    },
    {
      "status": "download_error",
      "url": "http://mlss.tuebingen.mpg.de/2015/slides/gretton/part_1.pdf",
      "reason": "ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))",
      "retried_at": "2026-10-20T22:23:38Z"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1802.01561",
      "bytes": 5601159,
      "reused": false,
      "local_path": "papers/repair/14d26b190854999d5443.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1804.08617.pdf",
      "bytes": 7734735,
      "reused": false,
      "local_path": "papers/repair/2db1af3554211ce5fb37.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1801.01290.pdf",
      "bytes": 4388102,
      "reused": false,
      "local_path": "papers/repair/b36e949651aa96489b98.pdf"
    },
    {
      "status": "ok",
      "url": "https://papers.nips.cc/paper/6338-stein-variational-gradient-descent-a-general-purpose-bayesian-inference-algorithm.pdf",
      "bytes": 477157,
      "reused": false,
      "local_path": "papers/repair/bb0557ad03965422dda1.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 76635,
  "expected_body_images": 14,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "download_error；invalid_file_type",
  "raw_html": "attachments/ebc73787cbe5/pages/source.html",
  "local_html": "attachments/ebc73787cbe5/pages/article.html",
  "body_base_path": "attachments/ebc73787cbe5/pages/",
  "body_markdown_path": "attachments/ebc73787cbe5/pages/article.md"
}