{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Lilian Weng / Lil'Log",
  "title_listed": "A (Long) Peek into Reinforcement Learning",
  "keywords": "强化学习,RL,入门",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/",
  "publish_date": "2018-02",
  "corpus_stage": "强化学习/对齐",
  "notes": "强化学习入门经典综述",
  "id": "9e1d58627b38",
  "collected_at": "2026-10-02T14:13:19",
  "title": "A (Long) Peek into Reinforcement Learning | Lil'Log",
  "body_markdown": "[Updated on 2020-09-03: Updated the algorithm of\nSARSA\nand\nQ-learning\nso that the difference is more pronounced.\n[Updated on 2021-09-19: Thanks to 爱吃猫的鱼, we have this post in\nChinese\n].\nA couple of exciting news in Artificial Intelligence (AI) has just happened in recent years.  AlphaGo defeated the best professional human player in the game of Go. Very soon the extended algorithm AlphaGo Zero beat AlphaGo by 100-0 without supervised learning on human knowledge. Top professional game players lost to the bot developed by OpenAI on DOTA2 1v1 competition. After knowing these, it is pretty hard not to be curious about the magic behind these algorithms — Reinforcement Learning (RL). I’m writing this post to briefly go over the field. We will first introduce several fundamental concepts and then dive into classic approaches to solving RL problems. Hopefully, this post could be a good starting point for newbies, bridging the future study on the cutting-edge research.\nWhat is Reinforcement Learning?\n#\nSay, we have an agent in an unknown environment and this agent can obtain some rewards by interacting with the environment. The agent ought to take actions so as to maximize cumulative rewards. In reality, the scenario could be a bot playing a game to achieve high scores, or a robot trying to complete physical tasks with physical items; and not just limited to these.\nAn agent interacts with the environment, trying to take smart actions to maximize cumulative rewards.\nThe goal of Reinforcement Learning (RL) is to learn a good strategy for the agent from experimental trials and relative simple feedback received. With the optimal strategy, the agent is capable to actively adapt to the environment to maximize future rewards.\nKey Concepts\n#\nNow Let’s formally define a set of key concepts in RL.\nThe agent is acting in an\nenvironment\n. How the environment reacts to certain actions is defined by a\nmodel\nwhich we may or may not know. The agent can stay in one of many\nstates\n($s \\in \\mathcal{S}$) of the environment, and choose to take one of many\nactions\n($a \\in \\mathcal{A}$) to switch from one state to another. Which state the agent will arrive in is decided by transition probabilities between states ($P$). Once an action is taken, the environment delivers a\nreward\n($r \\in \\mathcal{R}$) as feedback.\nThe model defines the reward function and transition probabilities. We may or may not know how the model works and this differentiate two circumstances:\nKnow the model\n: planning with perfect information; do model-based RL. When we fully know the environment, we can find the optimal solution by\nDynamic Programming\n(DP). Do you still remember “longest increasing subsequence” or “traveling salesmen problem” from your Algorithms 101 class? LOL. This is not the focus of this post though.\nDoes not know the model\n: learning with incomplete information; do model-free RL or try to learn the model explicitly as part of the algorithm. Most of the following content serves the scenarios when the model is unknown.\nThe agent’s\npolicy\n$\\pi(s)$ provides the guideline on what is the optimal action to take in a certain state with\nthe goal to maximize the total rewards\n. Each state is associated with a\nvalue\nfunction $V(s)$ predicting the expected amount of future rewards we are able to receive in this state by acting the corresponding policy. In other words, the value function quantifies how good a state is. Both policy and value functions are what we try to learn in reinforcement learning.\nSummary of approaches in RL based on whether we want to model the value, policy, or the environment. (Image source: reproduced from David Silver's RL course\nlecture 1\n.)\nThe interaction between the agent and the environment involves a sequence of actions and observed rewards in time, $t=1, 2, \\dots, T$. During the process, the agent accumulates the knowledge about the environment, learns the optimal policy, and makes decisions on which action to take next so as to efficiently learn the best policy. Let’s label the state, action, and reward at time step t as $S_t$, $A_t$, and $R_t$, respectively. Thus the interaction sequence is fully described by one\nepisode\n(also known as “trial” or “trajectory”) and the sequence ends at the terminal state $S_T$:\n$$\nS_1, A_1, R_2, S_2, A_2, \\dots, S_T\n$$\nTerms you will encounter a lot when diving into different categories of RL algorithms:\nModel-based\n: Rely on the model of the environment; either the model is known or the algorithm learns it explicitly.\nModel-free\n: No dependency on the model during learning.\nOn-policy\n: Use the deterministic outcomes or samples from the target policy to train the algorithm.\nOff-policy\n: Training on a distribution of transitions or episodes produced by a different behavior policy rather than that produced by the target policy.\nModel: Transition and Reward\n#\nThe model is a descriptor of the environment. With the model, we can learn or infer how the environment would interact with and provide feedback to the agent. The model has two major parts, transition probability function $P$ and reward function $R$.\nLet’s say when we are in state s, we decide to take action a to arrive in the next state s’ and obtain reward r. This is known as one\ntransition\nstep, represented by a tuple (s, a, s’, r).\nThe transition function P records the probability of transitioning from state s to s’ after taking action a while obtaining reward r. We use $\\mathbb{P}$ as a symbol of “probability”.\n$$\nP(s', r \\vert s, a)  = \\mathbb{P} [S_{t+1} = s', R_{t+1} = r \\vert S_t = s, A_t = a]\n$$\nThus the state-transition function can be defined as a function of $P(s’, r \\vert s, a)$:\n$$\nP_{ss'}^a = P(s' \\vert s, a)  = \\mathbb{P} [S_{t+1} = s' \\vert S_t = s, A_t = a] = \\sum_{r \\in \\mathcal{R}} P(s', r \\vert s, a)\n$$\nThe reward function R predicts the next reward triggered by one action:\n$$\nR(s, a) = \\mathbb{E} [R_{t+1} \\vert S_t = s, A_t = a] = \\sum_{r\\in\\mathcal{R}} r \\sum_{s' \\in \\mathcal{S}} P(s', r \\vert s, a)\n$$\nPolicy\n#\nPolicy, as the agent’s behavior function $\\pi$, tells us which action to take in state s. It is a mapping from state s to action a and can be either deterministic or stochastic:\nDeterministic: $\\pi(s) = a$.\nStochastic: $\\pi(a \\vert s) = \\mathbb{P}_\\pi [A=a \\vert S=s]$.\nValue Function\n#\nValue function measures the goodness of a state or how rewarding a state or an action is by a prediction of future reward. The future reward, also known as\nreturn\n, is a total sum of discounted rewards going forward. Let’s compute the return $G_t$ starting from time t:\n$$\nG_t = R_{t+1} + \\gamma R_{t+2} + \\dots = \\sum_{k=0}^{\\infty} \\gamma^k R_{t+k+1}\n$$\nThe discounting factor $\\gamma \\in [0, 1]$ penalize the rewards in the future, because:\nThe future rewards may have higher uncertainty; i.e. stock market.\nThe future rewards do not provide immediate benefits; i.e. As human beings, we might prefer to have fun today rather than 5 years later ;).\nDiscounting provides mathematical convenience; i.e., we don’t need to track future steps forever to compute return.\nWe don’t need to worry about the infinite loops in the state transition graph.\nThe\nstate-value\nof a state s is the expected return if we are in this state at time t, $S_t = s$:\n$$\nV_{\\pi}(s) = \\mathbb{E}_{\\pi}[G_t \\vert S_t = s]\n$$\nSimilarly, we define the\naction-value\n(“Q-value”; Q as “Quality” I believe?) of a state-action pair as:\n$$\nQ_{\\pi}(s, a) = \\mathbb{E}_{\\pi}[G_t \\vert S_t = s, A_t = a]\n$$\nAdditionally, since we follow the target policy $\\pi$, we can make use of the probility distribution over possible actions and the Q-values to recover the state-value:\n$$\nV_{\\pi}(s) = \\sum_{a \\in \\mathcal{A}} Q_{\\pi}(s, a) \\pi(a \\vert s)\n$$\nThe difference between action-value and state-value is the action\nadvantage\nfunction (“A-value”):\n$$\nA_{\\pi}(s, a) = Q_{\\pi}(s, a) - V_{\\pi}(s)\n$$\nOptimal Value and Policy\n#\nThe optimal value function produces the maximum return:\n$$\nV_{*}(s) = \\max_{\\pi} V_{\\pi}(s),\nQ_{*}(s, a) = \\max_{\\pi} Q_{\\pi}(s, a)\n$$\nThe optimal policy achieves optimal value functions:\n$$\n\\pi_{*} = \\arg\\max_{\\pi} V_{\\pi}(s),\n\\pi_{*} = \\arg\\max_{\\pi} Q_{\\pi}(s, a)\n$$\nAnd of course, we have $V_{\\pi_{*}}(s)=V_{*}(s)$ and $Q_{\\pi_{*}}(s, a) = Q_{*}(s, a)$.\nMarkov Decision Processes\n#\nIn more formal terms, almost all the RL problems can be framed as\nMarkov Decision Processes\n(MDPs). All states in MDP has “Markov” property, referring to the fact that the future only depends on the current state, not the history:\n$$\n\\mathbb{P}[ S_{t+1} \\vert S_t ] = \\mathbb{P} [S_{t+1} \\vert S_1, \\dots, S_t]\n$$\nOr in other words, the future and the past are\nconditionally independent\ngiven the present, as the current state encapsulates all the statistics we need to decide the future.\nThe agent-environment interaction in a Markov decision process. (Image source: Sec. 3.1 Sutton & Barto (2017).)\nA Markov deicison process consists of five elements $\\mathcal{M} = \\langle \\mathcal{S}, \\mathcal{A}, P, R, \\gamma \\rangle$, where the symbols carry the same meanings as key concepts in the\nprevious\nsection, well aligned with RL problem settings:\n$\\mathcal{S}$ - a set of states;\n$\\mathcal{A}$ - a set of actions;\n$P$ - transition probability function;\n$R$ - reward function;\n$\\gamma$ - discounting factor for future rewards.\nIn an unknown environment, we do not have perfect knowledge about $P$ and $R$.\nA fun example of Markov decision process: a typical work day. (Image source:\nrandomant.net/reinforcement-learning-concepts\n)\nBellman Equations\n#\nBellman equations refer to a set of equations that decompose the value function into the immediate reward plus the discounted future values.\n$$\n\\begin{aligned}\nV(s) &= \\mathbb{E}[G_t \\vert S_t = s] \\\\\n&= \\mathbb{E} [R_{t+1} + \\gamma R_{t+2} + \\gamma^2 R_{t+3} + \\dots \\vert S_t = s] \\\\\n&= \\mathbb{E} [R_{t+1} + \\gamma (R_{t+2} + \\gamma R_{t+3} + \\dots) \\vert S_t = s] \\\\\n&= \\mathbb{E} [R_{t+1} + \\gamma G_{t+1} \\vert S_t = s] \\\\\n&= \\mathbb{E} [R_{t+1} + \\gamma V(S_{t+1}) \\vert S_t = s]\n\\end{aligned}\n$$\nSimilarly for Q-value,\n$$\n\\begin{aligned}\nQ(s, a) \n&= \\mathbb{E} [R_{t+1} + \\gamma V(S_{t+1}) \\mid S_t = s, A_t = a] \\\\\n&= \\mathbb{E} [R_{t+1} + \\gamma \\mathbb{E}_{a\\sim\\pi} Q(S_{t+1}, a) \\mid S_t = s, A_t = a]\n\\end{aligned}\n$$\nBellman Expectation Equations\n#\nThe recursive update process can be further decomposed to be equations built on both state-value and action-value functions. As we go further in future action steps, we extend V and Q alternatively by following the policy $\\pi$.\nIllustration of how Bellman expection equations update state-value and action-value functions.\n$$\n\\begin{aligned}\nV_{\\pi}(s) &= \\sum_{a \\in \\mathcal{A}} \\pi(a \\vert s) Q_{\\pi}(s, a) \\\\\nQ_{\\pi}(s, a) &= R(s, a) + \\gamma \\sum_{s' \\in \\mathcal{S}} P_{ss'}^a V_{\\pi} (s') \\\\\nV_{\\pi}(s) &= \\sum_{a \\in \\mathcal{A}} \\pi(a \\vert s) \\big( R(s, a) + \\gamma \\sum_{s' \\in \\mathcal{S}} P_{ss'}^a V_{\\pi} (s') \\big) \\\\\nQ_{\\pi}(s, a) &= R(s, a) + \\gamma \\sum_{s' \\in \\mathcal{S}} P_{ss'}^a \\sum_{a' \\in \\mathcal{A}} \\pi(a' \\vert s') Q_{\\pi} (s', a')\n\\end{aligned}\n$$\nBellman Optimality Equations\n#\nIf we are only interested in the optimal values, rather than computing the expectation following a policy, we could jump right into the maximum returns during the alternative updates without using a policy. RECAP: the optimal values $V_*$ and $Q_*$ are the best returns we can obtain, defined\nhere\n.\n$$\n\\begin{aligned}\nV_*(s) &= \\max_{a \\in \\mathcal{A}} Q_*(s,a)\\\\\nQ_*(s, a) &= R(s, a) + \\gamma \\sum_{s' \\in \\mathcal{S}} P_{ss'}^a V_*(s') \\\\\nV_*(s) &= \\max_{a \\in \\mathcal{A}} \\big( R(s, a) + \\gamma \\sum_{s' \\in \\mathcal{S}} P_{ss'}^a V_*(s') \\big) \\\\\nQ_*(s, a) &= R(s, a) + \\gamma \\sum_{s' \\in \\mathcal{S}} P_{ss'}^a \\max_{a' \\in \\mathcal{A}} Q_*(s', a')\n\\end{aligned}\n$$\nUnsurprisingly they look very similar to Bellman expectation equations.\nIf we have complete information of the environment, this turns into a planning problem, solvable by DP. Unfortunately, in most scenarios, we do not know $P_{ss’}^a$ or $R(s, a)$, so we cannot solve MDPs by directly applying Bellmen equations, but it lays the theoretical foundation for many RL algorithms.\nCommon Approaches\n#\nNow it is the time to go through the major approaches and classic algorithms for solving RL problems. In future posts, I plan to dive into each approach further.\nDynamic Programming\n#\nWhen the model is fully known, following Bellman equations, we can use\nDynamic Programming\n(DP) to iteratively evaluate value functions and improve policy.\nPolicy Evaluation\n#\nPolicy Evaluation is to compute the state-value $V_\\pi$ for a given policy $\\pi$:\n$$\nV_{t+1}(s) \n= \\mathbb{E}_\\pi [r + \\gamma V_t(s') | S_t = s]\n= \\sum_a \\pi(a \\vert s) \\sum_{s', r} P(s', r \\vert s, a) (r + \\gamma V_t(s'))\n$$\nPolicy Improvement\n#\nBased on the value functions, Policy Improvement generates a better policy $\\pi’ \\geq \\pi$ by acting greedily.\n$$\nQ_\\pi(s, a) \n= \\mathbb{E} [R_{t+1} + \\gamma V_\\pi(S_{t+1}) \\vert S_t=s, A_t=a]\n= \\sum_{s', r} P(s', r \\vert s, a) (r + \\gamma V_\\pi(s'))\n$$\nPolicy Iteration\n#\nThe\nGeneralized Policy Iteration (GPI)\nalgorithm refers to an iterative procedure to improve the policy when combining policy evaluation and improvement.\n$$\n\\pi_0 \\xrightarrow[]{\\text{evaluation}} V_{\\pi_0} \\xrightarrow[]{\\text{improve}}\n\\pi_1 \\xrightarrow[]{\\text{evaluation}} V_{\\pi_1} \\xrightarrow[]{\\text{improve}}\n\\pi_2 \\xrightarrow[]{\\text{evaluation}} \\dots \\xrightarrow[]{\\text{improve}}\n\\pi_* \\xrightarrow[]{\\text{evaluation}} V_*\n$$\nIn GPI, the value function is approximated repeatedly to be closer to the true value of the current policy and in the meantime, the policy is improved repeatedly to approach optimality. This policy iteration process works and always converges to the optimality, but why this is the case?\nSay, we have a policy $\\pi$ and then generate an improved version $\\pi’$ by greedily taking actions, $\\pi’(s) = \\arg\\max_{a \\in \\mathcal{A}} Q_\\pi(s, a)$. The value of this improved $\\pi’$ is guaranteed to be better because:\n$$\n\\begin{aligned}\nQ_\\pi(s, \\pi'(s))\n&= Q_\\pi(s, \\arg\\max_{a \\in \\mathcal{A}} Q_\\pi(s, a)) \\\\\n&= \\max_{a \\in \\mathcal{A}} Q_\\pi(s, a) \\geq Q_\\pi(s, \\pi(s)) = V_\\pi(s)\n\\end{aligned}\n$$\nMonte-Carlo Methods\n#\nFirst, let’s recall that $V(s) = \\mathbb{E}[ G_t \\vert S_t=s]$. Monte-Carlo (MC) methods uses a simple idea: It learns from episodes of raw experience without modeling the environmental dynamics and computes the observed mean return as an approximation of the expected return. To compute the empirical return $G_t$, MC methods need to learn from\ncomplete\nepisodes $S_1, A_1, R_2, \\dots, S_T$ to compute $G_t = \\sum_{k=0}^{T-t-1} \\gamma^k R_{t+k+1}$ and all the episodes must eventually terminate.\nThe empirical mean return for state s is:\n$$\nV(s) = \\frac{\\sum_{t=1}^T \\mathbb{1}[S_t = s] G_t}{\\sum_{t=1}^T \\mathbb{1}[S_t = s]}\n$$\nwhere $\\mathbb{1}[S_t = s]$ is a binary indicator function. We may count the visit of state s every time so that there could exist multiple visits of one state in one episode (“every-visit”), or only count it the first time we encounter a state in one episode (“first-visit”). This way of approximation can be easily extended to action-value functions by counting (s, a) pair.\n$$\nQ(s, a) = \\frac{\\sum_{t=1}^T \\mathbb{1}[S_t = s, A_t = a] G_t}{\\sum_{t=1}^T \\mathbb{1}[S_t = s, A_t = a]}\n$$\nTo learn the optimal policy by MC, we iterate it by following a similar idea to\nGPI\n.\nImprove the policy greedily with respect to the current value function: $\\pi(s) = \\arg\\max_{a \\in \\mathcal{A}} Q(s, a)$.\nGenerate a new episode with the new policy $\\pi$ (i.e. using algorithms like\nε-greedy\nhelps us balance between exploitation and exploration.)\nEstimate Q using the new episode: $q_\\pi(s, a) = \\frac{\\sum_{t=1}^T \\big( \\mathbb{1}[S_t = s, A_t = a] \\sum_{k=0}^{T-t-1} \\gamma^k R_{t+k+1} \\big)}{\\sum_{t=1}^T \\mathbb{1}[S_t = s, A_t = a]}$\nTemporal-Difference Learning\n#\nSimilar to Monte-Carlo methods, Temporal-Difference (TD) Learning is model-free and learns from episodes of experience. However, TD learning can learn from\nincomplete\nepisodes and hence we don’t need to track the episode up to termination. TD learning is so important that Sutton & Barto (2017) in their RL book describes it as “one idea … central and novel to reinforcement learning”.\nBootstrapping\n#\nTD learning methods update targets with regard to existing estimates rather than exclusively relying on actual rewards and complete returns as in MC methods. This approach is known as\nbootstrapping\n.\nValue Estimation\n#\nThe key idea in TD learning is to update the value function $V(S_t)$ towards an estimated return $R_{t+1} + \\gamma V(S_{t+1})$ (known as “\nTD target\n”). To what extent we want to update the value function is controlled by the learning rate hyperparameter α:\n$$\n\\begin{aligned}\nV(S_t) &\\leftarrow (1- \\alpha) V(S_t) + \\alpha G_t \\\\\nV(S_t) &\\leftarrow V(S_t) + \\alpha (G_t - V(S_t)) \\\\\nV(S_t) &\\leftarrow V(S_t) + \\alpha (R_{t+1} + \\gamma V(S_{t+1}) - V(S_t))\n\\end{aligned}\n$$\nSimilarly, for action-value estimation:\n$$\nQ(S_t, A_t) \\leftarrow Q(S_t, A_t) + \\alpha (R_{t+1} + \\gamma Q(S_{t+1}, A_{t+1}) - Q(S_t, A_t))\n$$\nNext, let’s dig into the fun part on how to learn optimal policy in TD learning (aka “TD control”). Be prepared, you are gonna see many famous names of classic algorithms in this section.\nSARSA: On-Policy TD control\n#\n“SARSA” refers to the procedure of updaing Q-value by following a sequence of $\\dots, S_t, A_t, R_{t+1}, S_{t+1}, A_{t+1}, \\dots$. The idea follows the same route of\nGPI\n. Within one episode, it works as follows:\nInitialize $t=0$.\nStart with $S_0$ and choose action $A_0 = \\arg\\max_{a \\in \\mathcal{A}} Q(S_0, a)$, where\n$\\epsilon$-greedy\nis commonly applied.\nAt time $t$, after applying action $A_t$, we observe reward $R_{t+1}$ and get into the next state $S_{t+1}$.\nThen pick the next action in the same way as in step 2: $A_{t+1} = \\arg\\max_{a \\in \\mathcal{A}} Q(S_{t+1}, a)$.\nUpdate the Q-value function: $ Q(S_t, A_t) \\leftarrow Q(S_t, A_t) + \\alpha (R_{t+1} + \\gamma Q(S_{t+1}, A_{t+1}) - Q(S_t, A_t)) $.\nSet $t = t+1$ and repeat from step 3.\nIn each step of SARSA, we need to choose the\nnext\naction according to the\ncurrent\npolicy.\nQ-Learning: Off-policy TD control\n#\nThe development of Q-learning (\nWatkins & Dayan, 1992\n) is a big breakout in the early days of Reinforcement Learning. Within one episode, it works as follows:\nInitialize $t=0$.\nStarts with $S_0$.\nAt time step $t$, we pick the action according to Q values, $A_t = \\arg\\max_{a \\in \\mathcal{A}} Q(S_t, a)$ and $\\epsilon$-greedy is commonly applied.\nAfter applying action $A_t$, we observe reward $R_{t+1}$ and get into the next state $S_{t+1}$.\nUpdate the Q-value function: $Q(S_t, A_t) \\leftarrow Q(S_t, A_t) + \\alpha (R_{t+1} + \\gamma \\max_{a \\in \\mathcal{A}} Q(S_{t+1}, a) - Q(S_t, A_t))$.\n$t = t+1$ and repeat from step 3.\nThe key difference from SARSA is that Q-learning does not follow the current policy to pick the second action $A_{t+1}$. It estimates $Q^*$ out of the best Q values, but which action (denoted as $a^*$) leads to this maximal Q does not matter and in the next step Q-learning may not follow $a^*$.\nThe backup diagrams for Q-learning and SARSA. (Image source: Replotted based on Figure 6.5 in Sutton & Barto (2017))\nDeep Q-Network\n#\nTheoretically, we can memorize $Q_*(.)$ for all state-action pairs in Q-learning, like in a gigantic table. However, it quickly becomes computationally infeasible when the state and action space are large. Thus people use functions (i.e. a machine learning model) to approximate Q values and this is called\nfunction approximation\n. For example, if we use a function with parameter $\\theta$ to calculate Q values, we can label Q value function as $Q(s, a; \\theta)$.\nUnfortunately Q-learning may suffer from instability and divergence when combined with an nonlinear Q-value function approximation and\nbootstrapping\n(See\nProblems #2\n).\nDeep Q-Network (“DQN”; Mnih et al. 2015) aims to greatly improve and stabilize the training procedure of Q-learning by two innovative mechanisms:\nExperience Replay\n: All the episode steps $e_t = (S_t, A_t, R_t, S_{t+1})$ are stored in one replay memory $D_t = \\{ e_1, \\dots, e_t \\}$. $D_t$ has experience tuples over many episodes. During Q-learning updates, samples are drawn at random from the replay memory and thus one sample could be used multiple times. Experience replay improves data efficiency, removes correlations in the observation sequences, and smooths over changes in the data distribution.\nPeriodically Updated Target\n: Q is optimized towards target values that are only periodically updated. The Q network is cloned and kept frozen as the optimization target every C steps (C is a hyperparameter). This modification makes the training more stable as it overcomes the short-term oscillations.\nThe loss function looks like this:\n$$\n\\mathcal{L}(\\theta) = \\mathbb{E}_{(s, a, r, s') \\sim U(D)} \\Big[ \\big( r + \\gamma \\max_{a'} Q(s', a'; \\theta^{-}) - Q(s, a; \\theta) \\big)^2 \\Big]\n$$\nwhere $U(D)$ is a uniform distribution over the replay memory D; $\\theta^{-}$ is the parameters of the frozen target Q-network.\nIn addition, it is also found to be helpful to clip the error term to be between [-1, 1]. (I always get mixed feeling with parameter clipping, as many studies have shown that it works empirically but it makes the math much less pretty. :/)\nAlgorithm for DQN with experience replay and occasionally frozen optimization target. The prepossessed sequence is the output of some processes running on the input images of Atari games. Don't worry too much about it; just consider them as input feature vectors. (Image source: Mnih et al. 2015)\nThere are many extensions of DQN to improve the original design, such as DQN with dueling architecture (Wang et al. 2016) which estimates state-value function V(s) and advantage function A(s, a) with shared network parameters.\nCombining TD and MC Learning\n#\nIn the previous\nsection\non value estimation in TD learning, we only trace one step further down the action chain when calculating the TD target. One can easily extend it to take multiple steps to estimate the return.\nLet’s label the estimated return following n steps as $G_t^{(n)}, n=1, \\dots, \\infty$, then:\n$n$\n$G_t$\nNotes\n$n=1$\n$G_t^{(1)} = R_{t+1} + \\gamma V(S_{t+1})$\nTD learning\n$n=2$\n$G_t^{(2)} = R_{t+1} + \\gamma R_{t+2} + \\gamma^2 V(S_{t+2})$\n…\n$n=n$\n$ G_t^{(n)} = R_{t+1} + \\gamma R_{t+2} + \\dots + \\gamma^{n-1} R_{t+n} + \\gamma^n V(S_{t+n}) $\n…\n$n=\\infty$\n$G_t^{(\\infty)} = R_{t+1} + \\gamma R_{t+2} + \\dots + \\gamma^{T-t-1} R_T + \\gamma^{T-t} V(S_T) $\nMC estimation\nThe generalized n-step TD learning still has the\nsame\nform for updating the value function:\n$$\nV(S_t) \\leftarrow V(S_t) + \\alpha (G_t^{(n)} - V(S_t))\n$$\nWe are free to pick any $n$ in TD learning as we like. Now the question becomes what is the best $n$? Which $G_t^{(n)}$ gives us the best return approximation? A common yet smart solution is to apply a weighted sum of all possible n-step TD targets rather than to pick a single best n. The weights decay by a factor λ with n, $\\lambda^{n-1}$; the intuition is similar to\nwhy\nwe want to discount future rewards when computing the return: the more future we look into the less confident we would be. To make all the weight (n → ∞) sum up to 1, we multiply every weight by (1-λ), because:\n$$\n\\begin{aligned}\n\\text{let } S &= 1 + \\lambda + \\lambda^2 + \\dots \\\\\nS &= 1 + \\lambda(1 + \\lambda + \\lambda^2 + \\dots) \\\\\nS &= 1 + \\lambda S \\\\\nS &= 1 / (1-\\lambda)\n\\end{aligned}\n$$\nThis weighted sum of many n-step returns is called λ-return $G_t^{\\lambda} = (1-\\lambda) \\sum_{n=1}^{\\infty} \\lambda^{n-1} G_t^{(n)}$. TD learning that adopts λ-return for value updating is labeled as\nTD(λ)\n. The original version we introduced\nabove\nis equivalent to\nTD(0)\n.\nComparison of the backup diagrams of Monte-Carlo, Temporal-Difference learning, and Dynamic Programming for state value functions. (Image source: David Silver's RL course\nlecture 4\n: \"Model-Free Prediction\")\nPolicy Gradient\n#\nAll the methods we have introduced above aim to learn the state/action value function and then to select actions accordingly. Policy Gradient methods instead learn the policy directly with a parameterized function respect to $\\theta$, $\\pi(a \\vert s; \\theta)$. Let’s define the reward function (opposite of loss function) as\nthe expected return\nand train the algorithm with the goal to maximize the reward function. My\nnext post\ndescribed why the policy gradient theorem works (proof) and introduced a number of policy gradient algorithms.\nIn discrete space:\n$$\n\\mathcal{J}(\\theta) = V_{\\pi_\\theta}(S_1) = \\mathbb{E}_{\\pi_\\theta}[V_1]\n$$\nwhere $S_1$ is the initial starting state.\nOr in continuous space:\n$$\n\\mathcal{J}(\\theta) = \\sum_{s \\in \\mathcal{S}} d_{\\pi_\\theta}(s) V_{\\pi_\\theta}(s) = \\sum_{s \\in \\mathcal{S}} \\Big( d_{\\pi_\\theta}(s) \\sum_{a \\in \\mathcal{A}} \\pi(a \\vert s, \\theta) Q_\\pi(s, a) \\Big)\n$$\nwhere $d_{\\pi_\\theta}(s)$ is stationary distribution of Markov chain for $\\pi_\\theta$. If you are unfamiliar with the definition of a “stationary distribution,” please check this\nreference\n.\nUsing\ngradient ascent\nwe can find the best θ that produces the highest return. It is natural to expect policy-based methods are more useful in continuous space, because there is an infinite number of actions and/or states to estimate the values for in continuous space and hence value-based approaches are computationally much more expensive.\nPolicy Gradient Theorem\n#\nComputing the gradient\nnumerically\ncan be done by perturbing θ by a small amount ε in the k-th dimension. It works even when $J(\\theta)$ is not differentiable (nice!), but unsurprisingly very slow.\n$$\n\\frac{\\partial \\mathcal{J}(\\theta)}{\\partial \\theta_k} \\approx \\frac{\\mathcal{J}(\\theta + \\epsilon u_k) - \\mathcal{J}(\\theta)}{\\epsilon}\n$$\nOr\nanalytically\n,\n$$\n\\mathcal{J}(\\theta) = \\mathbb{E}_{\\pi_\\theta} [r] = \\sum_{s \\in \\mathcal{S}} d_{\\pi_\\theta}(s) \\sum_{a \\in \\mathcal{A}} \\pi(a \\vert s; \\theta) R(s, a)\n$$\nActually we have nice theoretical support for (replacing $d(.)$ with $d_\\pi(.)$):\n$$\n\\mathcal{J}(\\theta) = \\sum_{s \\in \\mathcal{S}} d_{\\pi_\\theta}(s) \\sum_{a \\in \\mathcal{A}} \\pi(a \\vert s; \\theta) Q_\\pi(s, a) \\propto \\sum_{s \\in \\mathcal{S}} d(s) \\sum_{a \\in \\mathcal{A}} \\pi(a \\vert s; \\theta) Q_\\pi(s, a)\n$$\nCheck Sec 13.1 in Sutton & Barto (2017) for why this is the case.\nThen,\n$$\n\\begin{aligned}\n\\mathcal{J}(\\theta) &= \\sum_{s \\in \\mathcal{S}} d(s) \\sum_{a \\in \\mathcal{A}} \\pi(a \\vert s; \\theta) Q_\\pi(s, a) \\\\\n\\nabla \\mathcal{J}(\\theta) &= \\sum_{s \\in \\mathcal{S}} d(s) \\sum_{a \\in \\mathcal{A}} \\nabla \\pi(a \\vert s; \\theta) Q_\\pi(s, a) \\\\\n&= \\sum_{s \\in \\mathcal{S}} d(s) \\sum_{a \\in \\mathcal{A}} \\pi(a \\vert s; \\theta) \\frac{\\nabla \\pi(a \\vert s; \\theta)}{\\pi(a \\vert s; \\theta)} Q_\\pi(s, a) \\\\\n& = \\sum_{s \\in \\mathcal{S}} d(s) \\sum_{a \\in \\mathcal{A}} \\pi(a \\vert s; \\theta) \\nabla \\ln \\pi(a \\vert s; \\theta) Q_\\pi(s, a) \\\\\n& = \\mathbb{E}_{\\pi_\\theta} [\\nabla \\ln \\pi(a \\vert s; \\theta) Q_\\pi(s, a)]\n\\end{aligned}\n$$\nThis result is named “Policy Gradient Theorem” which lays the theoretical foundation for various policy gradient algorithms:\n$$\n\\nabla \\mathcal{J}(\\theta) = \\mathbb{E}_{\\pi_\\theta} [\\nabla \\ln \\pi(a \\vert s, \\theta) Q_\\pi(s, a)]\n$$\nREINFORCE\n#\nREINFORCE, also known as Monte-Carlo policy gradient, relies on $Q_\\pi(s, a)$, an estimated return by\nMC\nmethods using episode samples, to update the policy parameter $\\theta$.\nA commonly used variation of REINFORCE is to subtract a baseline value from the return $G_t$ to reduce the variance of gradient estimation while keeping the bias unchanged. For example, a common baseline is state-value, and if applied, we would use $A(s, a) = Q(s, a) - V(s)$ in the gradient ascent update.\nInitialize θ at random\nGenerate one episode $S_1, A_1, R_2, S_2, A_2, \\dots, S_T$\nFor t=1, 2, … , T:\nEstimate the the return G_t since the time step t.\n$\\theta \\leftarrow \\theta + \\alpha \\gamma^t G_t \\nabla \\ln \\pi(A_t \\vert S_t, \\theta)$.\nActor-Critic\n#\nIf the value function is learned in addition to the policy, we would get Actor-Critic algorithm.\nCritic\n: updates value function parameters w and depending on the algorithm it could be action-value $Q(a \\vert s; w)$ or state-value $V(s; w)$.\nActor\n: updates policy parameters θ, in the direction suggested by the critic, $\\pi(a \\vert s; \\theta)$.\nLet’s see how it works in an action-value actor-critic algorithm.\nInitialize s, θ, w at random; sample $a \\sim \\pi(a \\vert s; \\theta)$.\nFor t = 1… T:\nSample reward $r_t  \\sim R(s, a)$ and next state $s’ \\sim P(s’ \\vert s, a)$.\nThen sample the next action $a’ \\sim \\pi(s’, a’; \\theta)$.\nUpdate policy parameters: $\\theta \\leftarrow \\theta + \\alpha_\\theta Q(s, a; w) \\nabla_\\theta \\ln \\pi(a \\vert s; \\theta)$.\nCompute the correction for action-value at time t:\n$G_{t:t+1} = r_t + \\gamma Q(s’, a’; w) - Q(s, a; w)$\nand use it to update value function parameters:\n$w \\leftarrow w + \\alpha_w G_{t:t+1} \\nabla_w Q(s, a; w) $.\nUpdate $a \\leftarrow a’$ and $s \\leftarrow s’$.\n$\\alpha_\\theta$ and $\\alpha_w$ are two learning rates for policy and value function parameter updates, respectively.\nA3C\n#\nAsynchronous Advantage Actor-Critic\n(Mnih et al., 2016), short for A3C, is a classic policy gradient method with the special focus on parallel training.\nIn A3C, the critics learn the state-value function, $V(s; w)$, while multiple actors are trained in parallel and get synced with global parameters from time to time. Hence, A3C is good for parallel training by default, i.e. on one machine with multi-core CPU.\nThe loss function for state-value is to minimize the mean squared error, $\\mathcal{J}_v (w) = (G_t - V(s; w))^2$ and we use gradient descent to find the optimal w. This state-value function is used as the baseline in the policy gradient update.\nHere is the algorithm outline:\nWe have global parameters, θ and w; similar thread-specific parameters, θ’ and w'.\nInitialize the time step t = 1\nWhile T <= T_MAX:\nReset gradient: dθ = 0 and dw = 0.\nSynchronize thread-specific parameters with global ones: θ’ = θ and w’ = w.\n$t_\\text{start}$ = t and get $s_t$.\nWhile ($s_t \\neq \\text{TERMINAL}$) and ($t - t_\\text{start} <= t_\\text{max}$):\nPick the action $a_t \\sim \\pi(a_t \\vert s_t; \\theta’)$ and receive a new reward $r_t$ and a new state $s_{t+1}$.\nUpdate t = t + 1 and T = T + 1.\nInitialize the variable that holds the return estimation $$R = \\begin{cases}\n0 & \\text{if } s_t \\text{ is TERMINAL} \\\nV(s_t; w’) & \\text{otherwise}\n\\end{cases}$$.\nFor $i = t-1, \\dots, t_\\text{start}$:\n$R \\leftarrow r_i + \\gamma R$; here R is a MC measure of $G_i$.\nAccumulate gradients w.r.t. θ’: $d\\theta \\leftarrow d\\theta + \\nabla_{\\theta’} \\log \\pi(a_i \\vert s_i; \\theta’)(R - V(s_i; w’))$;\nAccumulate gradients w.r.t. w’: $dw \\leftarrow dw + \\nabla_{w’} (R - V(s_i; w’))^2$.\nUpdate synchronously θ using dθ, and w using dw.\nA3C enables the parallelism in multiple agent training. The gradient accumulation step (6.2) can be considered as a reformation of minibatch-based stochastic gradient update: the values of w or θ get corrected by a little bit in the direction of each training thread independently.\nEvolution Strategies\n#\nEvolution Strategies\n(ES) is a type of model-agnostic optimization approach. It learns the optimal solution by imitating Darwin’s theory of the evolution of species by natural selection. Two prerequisites for applying ES: (1) our solutions can freely interact with the environment and see whether they can solve the problem; (2) we are able to compute a\nfitness\nscore of how good each solution is. We don’t have to know the environment configuration to solve the problem.\nSay, we start with a population of random solutions. All of them are capable of interacting with the environment and only candidates with high fitness scores can survive (\nonly the fittest can survive in a competition for limited resources\n). A new generation is then created by recombining the settings (\ngene mutation\n) of high-fitness survivors. This process is repeated until the new solutions are good enough.\nVery different from the popular MDP-based approaches as what we have introduced above, ES aims to learn the policy parameter $\\theta$ without value approximation. Let’s assume the distribution over the parameter $\\theta$ is an\nisotropic\nmultivariate Gaussian with mean $\\mu$ and fixed covariance $\\sigma^2I$. The gradient of $F(\\theta)$ is calculated:\n$$\n\\begin{aligned}\n& \\nabla_\\theta \\mathbb{E}_{\\theta \\sim N(\\mu, \\sigma^2)} F(\\theta) \\\\\n=& \\nabla_\\theta \\int_\\theta F(\\theta) \\Pr(\\theta) && \\text{Pr(.) is the Gaussian density function.} \\\\\n=& \\int_\\theta F(\\theta) \\Pr(\\theta) \\frac{\\nabla_\\theta \\Pr(\\theta)}{\\Pr(\\theta)} \\\\\n=& \\int_\\theta F(\\theta) \\Pr(\\theta) \\nabla_\\theta \\log \\Pr(\\theta) \\\\\n=& \\mathbb{E}_{\\theta \\sim N(\\mu, \\sigma^2)} [F(\\theta) \\nabla_\\theta \\log \\Pr(\\theta)] && \\text{Similar to how we do policy gradient update.} \\\\\n=& \\mathbb{E}_{\\theta \\sim N(\\mu, \\sigma^2)} \\Big[ F(\\theta) \\nabla_\\theta \\log \\Big( \\frac{1}{\\sqrt{2\\pi\\sigma^2}} e^{-\\frac{(\\theta - \\mu)^2}{2 \\sigma^2 }} \\Big) \\Big] \\\\\n=& \\mathbb{E}_{\\theta \\sim N(\\mu, \\sigma^2)} \\Big[ F(\\theta) \\nabla_\\theta \\Big( -\\log \\sqrt{2\\pi\\sigma^2} - \\frac{(\\theta - \\mu)^2}{2 \\sigma^2} \\Big) \\Big] \\\\\n=& \\mathbb{E}_{\\theta \\sim N(\\mu, \\sigma^2)} \\Big[ F(\\theta) \\frac{\\theta - \\mu}{\\sigma^2} \\Big]\n\\end{aligned}\n$$\nWe can rewrite this formula in terms of a “mean” parameter $\\theta$ (different from the $\\theta$ above; this $\\theta$ is the base gene for further mutation), $\\epsilon \\sim N(0, I)$ and therefore $\\theta + \\epsilon \\sigma \\sim N(\\theta, \\sigma^2)$. $\\epsilon$ controls how much Gaussian noises should be added to create mutation:\n$$\n\\nabla_\\theta \\mathbb{E}_{\\epsilon \\sim N(0, I)} F(\\theta + \\sigma \\epsilon) = \\frac{1}{\\sigma} \\mathbb{E}_{\\epsilon \\sim N(0, I)} [F(\\theta + \\sigma \\epsilon) \\epsilon]\n$$\nA simple parallel evolution-strategies-based RL algorithm. Parallel workers share the random seeds so that they can reconstruct the Gaussian noises with tiny communication bandwidth. (Image source: Salimans et al. 2017.)\nES, as a black-box optimization algorithm, is another approach to RL problems (\nIn my original writing, I used the phrase “a nice alternative”;\nSeita\npointed me to this\ndiscussion\nand thus I updated my wording.\n). It has a couple of good characteristics (Salimans et al., 2017) keeping it fast and easy to train:\nES does not need value function approximation;\nES does not perform gradient back-propagation;\nES is invariant to delayed or long-term rewards;\nES is highly parallelizable with very little data communication.\nKnown Problems\n#\nExploration-Exploitation Dilemma\n#\nThe problem of exploration vs exploitation dilemma has been discussed in my previous\npost\n. When the RL problem faces an unknown environment, this issue is especially a key to finding a good solution: without enough exploration, we cannot learn the environment well enough; without enough exploitation, we cannot complete our reward optimization task.\nDifferent RL algorithms balance between exploration and exploitation in different ways. In\nMC\nmethods,\nQ-learning\nor many on-policy algorithms, the exploration is commonly implemented by\nε-greedy\n; In\nES\n, the exploration is captured by the policy parameter perturbation. Please keep this into consideration when developing a new RL algorithm.\nDeadly Triad Issue\n#\nWe do seek the efficiency and flexibility of TD methods that involve bootstrapping. However, when off-policy, nonlinear function approximation, and bootstrapping are combined in one RL algorithm, the training could be unstable and hard to converge. This issue is known as the\ndeadly triad\n(Sutton & Barto, 2017). Many architectures using deep learning models were proposed to resolve the problem, including DQN to stabilize the training with experience replay and occasionally frozen target network.\nCase Study: AlphaGo Zero\n#\nThe game of\nGo\nhas been an extremely hard problem in the field of Artificial Intelligence for decades until recent years. AlphaGo and AlphaGo Zero are two programs developed by a team at DeepMind. Both involve deep Convolutional Neural Networks (\nCNN\n) and Monte Carlo Tree Search (MCTS) and both have been approved to achieve the level of professional human Go players. Different from AlphaGo that relied on supervised learning from expert human moves, AlphaGo Zero used only reinforcement learning and self-play without human knowledge beyond the basic rules.\nThe board of Go. Two players play black and white stones alternatively on the vacant intersections of a board with 19 x 19 lines. A group of stones must have at least one open point (an intersection, called a \"liberty\") to remain on the board and must have at least two or more enclosed liberties (called \"eyes\") to stay \"alive\". No stone shall repeat a previous position.\nWith all the knowledge of RL above, let’s take a look at how AlphaGo Zero works. The main component is a deep\nCNN\nover the game board configuration (precisely, a\nResNet\nwith batch normalization and ReLU). This network outputs two values:\n$$\n(p, v) = f_\\theta(s)\n$$\n$s$: the game board configuration, 19 x 19 x 17 stacked feature planes; 17 features for each position, 8 past configurations (including current) for the current player + 8 past configurations for the opponent + 1 feature indicating the color (1=black, 0=white). We need to code the color specifically because the network is playing with itself and the colors of current player and opponents are switching between steps.\n$p$: the probability of selecting a move over 19^2 + 1 candidates (19^2 positions on the board, in addition to passing).\n$v$: the winning probability given the current setting.\nDuring self-play, MCTS further improves the action probability distribution $\\pi \\sim p(.)$ and then the action $a_t$ is sampled from this improved policy. The reward $z_t$ is a binary value indicating whether the current player\neventually\nwins the game. Each move generates an episode tuple $(s_t, \\pi_t, z_t)$ and it is saved into the replay memory. The details on MCTS are skipped for the sake of space in this post; please read the original\npaper\nif you are interested.\nAlphaGo Zero is trained by self-play while MCTS improves the output policy further in every step. (Image source: Figure 1a in Silver et al., 2017).\nThe network is trained with the samples in the replay memory to minimize the loss:\n$$\n\\mathcal{L} = (z - v)^2 - \\pi^\\top \\log p + c \\| \\theta \\|^2\n$$\nwhere $c$ is a hyperparameter controlling the intensity of L2 penalty to avoid overfitting.\nAlphaGo Zero simplified AlphaGo by removing supervised learning and merging separated policy and value networks into one. It turns out that AlphaGo Zero achieved largely improved performance with a much shorter training time! I strongly recommend reading these\ntwo\npapers\nside by side and compare the difference, super fun.\nI know this is a long read, but hopefully worth it.\nIf you notice mistakes and errors in this post, don’t hesitate to contact me at [lilian dot wengweng at gmail dot com].\nSee you in the next post! :)\nCited as:\n@article{weng2018bandit,\n  title   = \"A (Long) Peek into Reinforcement Learning\",\n  author  = \"Weng, Lilian\",\n  journal = \"lilianweng.github.io\",\n  year    = \"2018\",\n  url     = \"https://lilianweng.github.io/posts/2018-02-19-rl-overview/\"\n}\nReferences\n#\n[1] Yuxi Li.\nDeep reinforcement learning: An overview.\narXiv preprint arXiv:1701.07274. 2017.\n[2] Richard S. Sutton and Andrew G. Barto.\nReinforcement Learning: An Introduction; 2nd Edition\n. 2017.\n[3] Volodymyr Mnih, et al.\nAsynchronous methods for deep reinforcement learning.\nICML. 2016.\n[4] Tim Salimans, et al.\nEvolution strategies as a scalable alternative to reinforcement learning.\narXiv preprint arXiv:1703.03864 (2017).\n[5] David Silver, et al.\nMastering the game of go without human knowledge\n. Nature 550.7676 (2017): 354.\n[6] David Silver, et al.\nMastering the game of Go with deep neural networks and tree search.\nNature 529.7587 (2016): 484-489.\n[7] Volodymyr Mnih, et al.\nHuman-level control through deep reinforcement learning.\nNature 518.7540 (2015): 529.\n[8] Ziyu Wang, et al.\nDueling network architectures for deep reinforcement learning.\nICML. 2016.\n[9]\nReinforcement Learning lectures\nby David Silver on YouTube.\n[10] OpenAI Blog:\nEvolution Strategies as a Scalable Alternative to Reinforcement Learning\n[11] Frank Sehnke, et al.\nParameter-exploring policy gradients.\nNeural Networks 23.4 (2010): 551-559.\n[12] Csaba Szepesvári.\nAlgorithms for reinforcement learning.\n1st Edition. Synthesis lectures on artificial intelligence and machine learning 4.1 (2010): 1-103.\nIf you notice mistakes and errors in this post, please don’t hesitate to contact me at [lilian dot wengweng at gmail dot com] and I would be super happy to correct them right away!",
  "metadata": {
    "title": "A (Long) Peek into Reinforcement Learning | Lil'Log",
    "description": "[Updated on 2020-09-03: Updated the algorithm of SARSA and Q-learning so that the difference is more pronounced.\n\n[Updated on 2021-09-19: Thanks to 爱吃猫的鱼, we have this post in Chinese].",
    "keywords": "reinforcement-learning, long-read, math-heavy",
    "author": "Lilian Weng",
    "og:title": "A (Long) Peek into Reinforcement Learning",
    "og:description": "[Updated on 2020-09-03: Updated the algorithm of SARSA and Q-learning so that the difference is more pronounced.\n\n[Updated on 2021-09-19: Thanks to 爱吃猫的鱼, we have this post in Chinese].",
    "og:type": "article",
    "og:url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/",
    "twitter:card": "summary",
    "twitter:title": "A (Long) Peek into Reinforcement Learning",
    "twitter:description": "[Updated on 2020-09-03: Updated the algorithm of SARSA and Q-learning so that the difference is more pronounced.\n\n[Updated on 2021-09-19: Thanks to 爱吃猫的鱼, we have this post in Chinese].",
    "article:section": "posts",
    "article:published_time": "2018-02-19T00:00:00+00:00",
    "article:modified_time": "2018-02-19T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/9e1d58627b38/code/body-000.txt",
      "chars": 241,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/RL_illustration.png",
      "bytes": 58269,
      "reused": false,
      "local_path": "attachments/9e1d58627b38/images/body-418c8992a5cc9d9deeb1.png",
      "src": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/RL_illustration.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/RL_algorithm_categorization.png",
      "bytes": 107025,
      "reused": false,
      "local_path": "attachments/9e1d58627b38/images/body-d53eaab123d7a3b8d561.png",
      "src": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/RL_algorithm_categorization.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/agent_environment_MDP.png",
      "bytes": 33772,
      "reused": false,
      "local_path": "attachments/9e1d58627b38/images/body-1a3ee27cdec9dd4e95e7.png",
      "src": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/agent_environment_MDP.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/mdp_example.jpg",
      "bytes": 55693,
      "reused": false,
      "local_path": "attachments/9e1d58627b38/images/body-8b930e0942e1d249d7cd.jpg",
      "src": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/mdp_example.jpg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/bellman_equation.png",
      "bytes": 27143,
      "reused": false,
      "local_path": "attachments/9e1d58627b38/images/body-508e643f8751bb71fcf3.png",
      "src": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/bellman_equation.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/MC_control.png",
      "bytes": 32728,
      "reused": false,
      "local_path": "attachments/9e1d58627b38/images/body-aa4ac628fe85693e0ba3.png",
      "src": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/MC_control.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/sarsa_vs_q_learning.png",
      "bytes": 21618,
      "reused": false,
      "local_path": "attachments/9e1d58627b38/images/body-8917c12943284b1507b6.png",
      "src": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/sarsa_vs_q_learning.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/DQN_algorithm.png",
      "bytes": 133435,
      "reused": false,
      "local_path": "attachments/9e1d58627b38/images/body-c1fcc0568374ce65ea91.png",
      "src": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/DQN_algorithm.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/TD_lambda.png",
      "bytes": 35850,
      "reused": false,
      "local_path": "attachments/9e1d58627b38/images/body-be7712dd68a63d561005.png",
      "src": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/TD_lambda.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/TD_MC_DP_backups.png",
      "bytes": 128382,
      "reused": false,
      "local_path": "attachments/9e1d58627b38/images/body-d374974862f12f0f0ada.png",
      "src": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/TD_MC_DP_backups.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/EA_RL_parallel.png",
      "bytes": 91431,
      "reused": false,
      "local_path": "attachments/9e1d58627b38/images/body-0d8b962a59d7ad31b2ef.png",
      "src": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/EA_RL_parallel.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/go_config.png",
      "bytes": 136647,
      "reused": false,
      "local_path": "attachments/9e1d58627b38/images/body-3ceb5a7713444f264e70.png",
      "src": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/go_config.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/alphago-zero-selfplay.png",
      "bytes": 61555,
      "reused": false,
      "local_path": "attachments/9e1d58627b38/images/body-070437b06f4b4d8a0287.png",
      "src": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/alphago-zero-selfplay.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=2pWv7GOvuf0",
      "src": "https://youtu.be/2pWv7GOvuf0",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/9e1d58627b38/videos/repair-2pWv7GOvuf0.mp4",
      "bytes": 397520008,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "5292.421000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/playlist?list=PL7-jPKtc4r78-wCZcQn5IqyuWhBZ8fOxT",
      "src": "https://www.youtube.com/playlist?list=PL7-jPKtc4r78-wCZcQn5IqyuWhBZ8fOxT",
      "status": "playlist_discovery_failed",
      "kind": "playlist",
      "reason": "播放列表返回空或不完整结果，保留原记录并等待重试",
      "error": "WARNING: [youtube:tab] YouTube said: The playlist does not exist.\nERROR: [youtube:tab] PL7-jPKtc4r78-wCZcQn5IqyuWhBZ8fOxT: YouTube said: The playlist does not exist.\n",
      "retried_at": "2026-10-20T22:23:33Z"
    }
  ],
  "links": [
    {
      "href": "https://paperexplained.cn/articles/article/detail/33/",
      "label": "Chinese",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Dynamic_programming",
      "label": "Dynamic Programming",
      "type": "a"
    },
    {
      "href": "https://youtu.be/2pWv7GOvuf0",
      "label": "lecture 1",
      "type": "a"
    },
    {
      "href": "https://randomant.net/reinforcement-learning-concepts/",
      "label": "randomant.net/reinforcement-learning-concepts",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-01-23-multi-armed-bandit/",
      "label": "ε-greedy",
      "type": "a"
    },
    {
      "href": "https://link.springer.com/content/pdf/10.1007/BF00992698.pdf",
      "label": "Watkins & Dayan, 1992",
      "type": "a"
    },
    {
      "href": "http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/MC-TD.pdf",
      "label": "lecture 4",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/",
      "label": "next post",
      "type": "a"
    },
    {
      "href": "https://jeremykun.com/2015/04/06/markov-chain-monte-carlo-without-all-the-bullshit/",
      "label": "reference",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Evolution_strategy",
      "label": "Evolution Strategies",
      "type": "a"
    },
    {
      "href": "https://math.stackexchange.com/questions/1991961/gaussian-distribution-is-isotropic",
      "label": "isotropic",
      "type": "a"
    },
    {
      "href": "https://danieltakeshi.github.io/",
      "label": "Seita",
      "type": "a"
    },
    {
      "href": "https://www.reddit.com/r/MachineLearning/comments/6gke6a/d_requesting_openai_to_justify_the_grandiose/dir9wde/",
      "label": "discussion",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Go_(game)",
      "label": "Go",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2017-12-15-object-recognition-part-2/",
      "label": "CNN",
      "type": "a"
    },
    {
      "href": "https://www.dropbox.com/s/yva172qos2u15hf/2017-silver.pdf?dl=0",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://pdfs.semanticscholar.org/1740/eb993cc8ca81f1e46ddaadce1f917e8000b5.pdf",
      "label": "two",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1701.07274.pdf",
      "label": "Deep reinforcement learning: An overview.",
      "type": "a"
    },
    {
      "href": "http://incompleteideas.net/book/bookdraft2017nov5.pdf",
      "label": "Reinforcement Learning: An Introduction; 2nd Edition",
      "type": "a"
    },
    {
      "href": "http://proceedings.mlr.press/v48/mniha16.pdf",
      "label": "Asynchronous methods for deep reinforcement learning.",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1703.03864.pdf",
      "label": "Evolution strategies as a scalable alternative to reinforcement learning.",
      "type": "a"
    },
    {
      "href": "https://www.cs.swarthmore.edu/~meeden/cs63/s15/nature15b.pdf",
      "label": "Human-level control through deep reinforcement learning.",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1511.06581.pdf",
      "label": "Dueling network architectures for deep reinforcement learning.",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/playlist?list=PL7-jPKtc4r78-wCZcQn5IqyuWhBZ8fOxT",
      "label": "Reinforcement Learning lectures",
      "type": "a"
    },
    {
      "href": "https://blog.openai.com/evolution-strategies/",
      "label": "Evolution Strategies as a Scalable Alternative to Reinforcement Learning",
      "type": "a"
    },
    {
      "href": "https://mediatum.ub.tum.de/doc/1287490/file.pdf",
      "label": "Parameter-exploring policy gradients.",
      "type": "a"
    },
    {
      "href": "https://sites.ualberta.ca/~szepesva/papers/RLAlgsInMDPs.pdf",
      "label": "Algorithms for reinforcement learning.",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:33Z",
  "source_url": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/",
  "papers": [
    {
      "status": "invalid_file_type",
      "url": "https://link.springer.com/content/pdf/10.1007/BF00992698.pdf",
      "content_type": "text/html; charset=utf-8",
      "retried_at": "2026-10-20T17:36:14Z"
    },
    {
      "status": "http_404",
      "url": "http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/MC-TD.pdf",
      "retried_at": "2026-10-20T17:36:31Z"
    },
    {
      "status": "invalid_file_type",
      "url": "https://www.dropbox.com/s/yva172qos2u15hf/2017-silver.pdf?dl=0",
      "content_type": "text/html; charset=utf-8",
      "retried_at": "2026-10-20T17:36:15Z"
    },
    {
      "status": "http_202",
      "url": "https://pdfs.semanticscholar.org/1740/eb993cc8ca81f1e46ddaadce1f917e8000b5.pdf",
      "retried_at": "2026-10-20T17:36:15Z"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1701.07274.pdf",
      "bytes": 1380080,
      "reused": false,
      "local_path": "papers/repair/b6ffc91d59527939bdfc.pdf"
    },
    {
      "status": "ok",
      "url": "http://incompleteideas.net/book/bookdraft2017nov5.pdf",
      "bytes": 12613382,
      "reused": true,
      "local_path": "papers/repair/7ea9e7044b38fbe00b3f.pdf"
    },
    {
      "status": "ok",
      "url": "http://proceedings.mlr.press/v48/mniha16.pdf",
      "bytes": 2455923,
      "reused": false,
      "local_path": "papers/repair/de6e461fa7ff09fc7981.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1703.03864.pdf",
      "bytes": 277340,
      "reused": false,
      "local_path": "papers/repair/d7a6ba3d9a896a29908d.pdf"
    },
    {
      "status": "ok",
      "url": "https://www.cs.swarthmore.edu/~meeden/cs63/s15/nature15b.pdf",
      "bytes": 4600582,
      "reused": false,
      "local_path": "papers/repair/a86f4e3a12fbcea87e0c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1511.06581.pdf",
      "bytes": 688509,
      "reused": false,
      "local_path": "papers/repair/f1c991f0c8eda83cb980.pdf"
    },
    {
      "status": "invalid_file_type",
      "url": "https://mediatum.ub.tum.de/doc/1287490/file.pdf",
      "content_type": "text/html; charset=utf-8",
      "retried_at": "2026-10-20T17:36:15Z"
    },
    {
      "status": "ok",
      "url": "https://sites.ualberta.ca/~szepesva/papers/RLAlgsInMDPs.pdf",
      "bytes": 1623436,
      "reused": false,
      "local_path": "papers/repair/fbf59542cdbde527181b.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 40855,
  "expected_body_images": 13,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "http_202；http_404；invalid_file_type；playlist_discovery_failed",
  "raw_html": "attachments/9e1d58627b38/pages/source.html",
  "local_html": "attachments/9e1d58627b38/pages/article.html",
  "body_base_path": "attachments/9e1d58627b38/pages/",
  "body_markdown_path": "attachments/9e1d58627b38/pages/article.md"
}