{
  "source_file": "0930",
  "source_platform": "研究机构",
  "author": "PyTorch Tutorials",
  "title_listed": "Reinforcement Learning (DQN)",
  "keywords": "DQN,强化学习,CartPole",
  "domain": "机器学习",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html",
  "publish_date": "持续更新",
  "corpus_stage": "机器学习",
  "notes": "DQN 强化学习",
  "id": "10d19930e334",
  "collected_at": "2026-10-02T16:54:33",
  "title": "Reinforcement Learning (DQN) Tutorial — PyTorch Tutorials 2.14.0+cu130 documentation",
  "body_markdown": "Note\nGo to the end\nto download the full example code.\nReinforcement Learning (DQN) Tutorial\n#\nCreated On: Mar 24, 2017 | Last Updated: Jun 16, 2025 | Last Verified: Nov 05, 2024\nAuthor\n:\nAdam Paszke\nMark Towers\nThis tutorial shows how to use PyTorch to train a Deep Q Learning (DQN) agent\non the CartPole-v1 task from\nGymnasium\n.\nYou might find it helpful to read the original\nDeep Q Learning (DQN)\npaper\nTask\nThe agent has to decide between two actions - moving the cart left or\nright - so that the pole attached to it stays upright. You can find more\ninformation about the environment and other more challenging environments at\nGymnasium’s website\n.\nCartPole\n#\nAs the agent observes the current state of the environment and chooses\nan action, the environment\ntransitions\nto a new state, and also\nreturns a reward that indicates the consequences of the action. In this\ntask, rewards are +1 for every incremental timestep and the environment\nterminates if the pole falls over too far or the cart moves more than 2.4\nunits away from center. This means better performing scenarios will run\nfor longer duration, accumulating larger return.\nThe CartPole task is designed so that the inputs to the agent are 4 real\nvalues representing the environment state (position, velocity, etc.).\nWe take these 4 inputs without any scaling and pass them through a\nsmall fully-connected network with 2 outputs, one for each action.\nThe network is trained to predict the expected value for each action,\ngiven the input state. The action with the highest expected value is\nthen chosen.\nPackages\nFirst, let’s import needed packages. Firstly, we need\ngymnasium\nfor the environment,\ninstalled by using\npip\n. This is a fork of the original OpenAI\nGym project and maintained by the same team since Gym v0.19.\nIf you are running this in Google Colab, run:\n%%bash\npip3\ninstall\ngymnasium\n[\nclassic_control\n]\nWe’ll also use the following from PyTorch:\nneural networks (\ntorch.nn\n)\noptimization (\ntorch.optim\n)\nautomatic differentiation (\ntorch.autograd\n)\nimport\ngymnasium\nas\ngym\nimport\nmath\nimport\nrandom\nimport\nmatplotlib\nimport\nmatplotlib.pyplot\nas\nplt\nfrom\ncollections\nimport\nnamedtuple\n,\ndeque\nfrom\nitertools\nimport\ncount\nimport\ntorch\nimport\ntorch.nn\nas\nnn\nimport\ntorch.optim\nas\noptim\nimport\ntorch.nn.functional\nas\nF\nenv\n=\ngym\n.\nmake\n(\n\"CartPole-v1\"\n)\n# set up matplotlib\nis_ipython\n=\n'inline'\nin\nmatplotlib\n.\nget_backend\n()\nif\nis_ipython\n:\nfrom\nIPython\nimport\ndisplay\nplt\n.\nion\n()\n# if GPU is to be used\ndevice\n=\ntorch\n.\ndevice\n(\n\"cuda\"\nif\ntorch\n.\ncuda\n.\nis_available\n()\nelse\n\"mps\"\nif\ntorch\n.\nbackends\n.\nmps\n.\nis_available\n()\nelse\n\"cpu\"\n)\n# To ensure reproducibility during training, you can fix the random seeds\n# by uncommenting the lines below. This makes the results consistent across\n# runs, which is helpful for debugging or comparing different approaches.\n#\n# That said, allowing randomness can be beneficial in practice, as it lets\n# the model explore different training trajectories.\n# seed = 42\n# random.seed(seed)\n# torch.manual_seed(seed)\n# env.reset(seed=seed)\n# env.action_space.seed(seed)\n# env.observation_space.seed(seed)\n# if torch.cuda.is_available():\n#     torch.cuda.manual_seed(seed)\nGym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.\nPlease upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.\nUsers of this version of Gym should be able to simply replace 'import gym' with 'import gymnasium as gym' in the vast majority of cases.\nSee the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.\nReplay Memory\n#\nWe’ll be using experience replay memory for training our DQN. It stores\nthe transitions that the agent observes, allowing us to reuse this data\nlater. By sampling from it randomly, the transitions that build up a\nbatch are decorrelated. It has been shown that this greatly stabilizes\nand improves the DQN training procedure.\nFor this, we’re going to need two classes:\nTransition\n- a named tuple representing a single transition in\nour environment. It essentially maps (state, action) pairs\nto their (next_state, reward) result, with the state being the\nscreen difference image as described later on.\nReplayMemory\n- a cyclic buffer of bounded size that holds the\ntransitions observed recently. It also implements a\n.sample()\nmethod for selecting a random batch of transitions for training.\nTransition\n=\nnamedtuple\n(\n'Transition'\n,\n(\n'state'\n,\n'action'\n,\n'next_state'\n,\n'reward'\n))\nclass\nReplayMemory\n(\nobject\n):\ndef\n__init__\n(\nself\n,\ncapacity\n):\nself\n.\nmemory\n=\ndeque\n([],\nmaxlen\n=\ncapacity\n)\ndef\npush\n(\nself\n,\n*\nargs\n):\n\"\"\"Save a transition\"\"\"\nself\n.\nmemory\n.\nappend\n(\nTransition\n(\n*\nargs\n))\ndef\nsample\n(\nself\n,\nbatch_size\n):\nreturn\nrandom\n.\nsample\n(\nself\n.\nmemory\n,\nbatch_size\n)\ndef\n__len__\n(\nself\n):\nreturn\nlen\n(\nself\n.\nmemory\n)\nNow, let’s define our model. But first, let’s quickly recap what a DQN is.\nDQN algorithm\n#\nOur environment is deterministic, so all equations presented here are\nalso formulated deterministically for the sake of simplicity. In the\nreinforcement learning literature, they would also contain expectations\nover stochastic transitions in the environment.\nOur aim will be to train a policy that tries to maximize the discounted,\ncumulative reward\n\\(R_{t_0} = \\sum_{t=t_0}^{\\infty} \\gamma^{t - t_0} r_t\\)\n, where\n\\(R_{t_0}\\)\nis also known as the\nreturn\n. The discount,\n\\(\\gamma\\)\n, should be a constant between\n\\(0\\)\nand\n\\(1\\)\nthat ensures the sum converges. A lower\n\\(\\gamma\\)\nmakes\nrewards from the uncertain far future less important for our agent\nthan the ones in the near future that it can be fairly confident\nabout. It also encourages agents to collect reward closer in time\nthan equivalent rewards that are temporally far away in the future.\nThe main idea behind Q-learning is that if we had a function\n\\(Q^*: State \\times Action \\rightarrow \\mathbb{R}\\)\n, that could tell\nus what our return would be, if we were to take an action in a given\nstate, then we could easily construct a policy that maximizes our\nrewards:\n\\[\\pi^*(s) = \\arg\\!\\max_a \\ Q^*(s, a)\n\n\\]\nHowever, we don’t know everything about the world, so we don’t have\naccess to\n\\(Q^*\\)\n. But, since neural networks are universal function\napproximators, we can simply create one and train it to resemble\n\\(Q^*\\)\n.\nFor our training update rule, we’ll use a fact that every\n\\(Q\\)\nfunction for some policy obeys the Bellman equation:\n\\[Q^{\\pi}(s, a) = r + \\gamma Q^{\\pi}(s', \\pi(s'))\n\n\\]\nThe difference between the two sides of the equality is known as the\ntemporal difference error,\n\\(\\delta\\)\n:\n\\[\\delta = Q(s, a) - (r + \\gamma \\max_a' Q(s', a))\n\n\\]\nTo minimize this error, we will use the\nHuber\nloss\n. The Huber loss acts\nlike the mean squared error when the error is small, but like the mean\nabsolute error when the error is large - this makes it more robust to\noutliers when the estimates of\n\\(Q\\)\nare very noisy. We calculate\nthis over a batch of transitions,\n\\(B\\)\n, sampled from the replay\nmemory:\n\\[\\mathcal{L} = \\frac{1}{|B|}\\sum_{(s, a, s', r) \\ \\in \\ B} \\mathcal{L}(\\delta)\\]\n\\[\\text{where} \\quad \\mathcal{L}(\\delta) = \\begin{cases}\n  \\frac{1}{2}{\\delta^2}  & \\text{for } |\\delta| \\le 1, \\\\\n  |\\delta| - \\frac{1}{2} & \\text{otherwise.}\n\\end{cases}\\]\nQ-network\n#\nOur model will be a feed forward  neural network that takes in the\ndifference between the current and previous screen patches. It has two\noutputs, representing\n\\(Q(s, \\mathrm{left})\\)\nand\n\\(Q(s, \\mathrm{right})\\)\n(where\n\\(s\\)\nis the input to the\nnetwork). In effect, the network is trying to predict the\nexpected return\nof\ntaking each action given the current input.\nclass\nDQN\n(\nnn\n.\nModule\n):\ndef\n__init__\n(\nself\n,\nn_observations\n,\nn_actions\n):\nsuper\n(\nDQN\n,\nself\n)\n.\n__init__\n()\nself\n.\nlayer1\n=\nnn\n.\nLinear\n(\nn_observations\n,\n128\n)\nself\n.\nlayer2\n=\nnn\n.\nLinear\n(\n128\n,\n128\n)\nself\n.\nlayer3\n=\nnn\n.\nLinear\n(\n128\n,\nn_actions\n)\n# Called with either one element to determine next action, or a batch\n# during optimization. Returns tensor([[left0exp,right0exp]...]).\ndef\nforward\n(\nself\n,\nx\n):\nx\n=\nF\n.\nrelu\n(\nself\n.\nlayer1\n(\nx\n))\nx\n=\nF\n.\nrelu\n(\nself\n.\nlayer2\n(\nx\n))\nreturn\nself\n.\nlayer3\n(\nx\n)\nTraining\n#\nHyperparameters and utilities\n#\nThis cell instantiates our model and its optimizer, and defines some\nutilities:\nselect_action\n- will select an action according to an epsilon\ngreedy policy. Simply put, we’ll sometimes use our model for choosing\nthe action, and sometimes we’ll just sample one uniformly. The\nprobability of choosing a random action will start at\nEPS_START\nand will decay exponentially towards\nEPS_END\n.\nEPS_DECAY\ncontrols the rate of the decay.\nplot_durations\n- a helper for plotting the duration of episodes,\nalong with an average over the last 100 episodes (the measure used in\nthe official evaluations). The plot will be underneath the cell\ncontaining the main training loop, and will update after every\nepisode.\n# BATCH_SIZE is the number of transitions sampled from the replay buffer\n# GAMMA is the discount factor as mentioned in the previous section\n# EPS_START is the starting value of epsilon\n# EPS_END is the final value of epsilon\n# EPS_DECAY controls the rate of exponential decay of epsilon, higher means a slower decay\n# TAU is the update rate of the target network\n# LR is the learning rate of the ``AdamW`` optimizer\nBATCH_SIZE\n=\n128\nGAMMA\n=\n0.99\nEPS_START\n=\n0.9\nEPS_END\n=\n0.01\nEPS_DECAY\n=\n2500\nTAU\n=\n0.005\nLR\n=\n3e-4\n# Get number of actions from gym action space\nn_actions\n=\nenv\n.\naction_space\n.\nn\n# Get the number of state observations\nstate\n,\ninfo\n=\nenv\n.\nreset\n()\nn_observations\n=\nlen\n(\nstate\n)\npolicy_net\n=\nDQN\n(\nn_observations\n,\nn_actions\n)\n.\nto\n(\ndevice\n)\ntarget_net\n=\nDQN\n(\nn_observations\n,\nn_actions\n)\n.\nto\n(\ndevice\n)\ntarget_net\n.\nload_state_dict\n(\npolicy_net\n.\nstate_dict\n())\noptimizer\n=\noptim\n.\nAdamW\n(\npolicy_net\n.\nparameters\n(),\nlr\n=\nLR\n,\namsgrad\n=\nTrue\n)\nmemory\n=\nReplayMemory\n(\n10000\n)\nsteps_done\n=\n0\ndef\nselect_action\n(\nstate\n):\nglobal\nsteps_done\nsample\n=\nrandom\n.\nrandom\n()\neps_threshold\n=\nEPS_END\n+\n(\nEPS_START\n-\nEPS_END\n)\n*\n\\\nmath\n.\nexp\n(\n-\n1.\n*\nsteps_done\n/\nEPS_DECAY\n)\nsteps_done\n+=\n1\nif\nsample\n>\neps_threshold\n:\nwith\ntorch\n.\nno_grad\n():\n# t.max(1) will return the largest column value of each row.\n# second column on max result is index of where max element was\n# found, so we pick action with the larger expected reward.\nreturn\npolicy_net\n(\nstate\n)\n.\nmax\n(\n1\n)\n.\nindices\n.\nview\n(\n1\n,\n1\n)\nelse\n:\nreturn\ntorch\n.\ntensor\n([[\nenv\n.\naction_space\n.\nsample\n()]],\ndevice\n=\ndevice\n,\ndtype\n=\ntorch\n.\nlong\n)\nepisode_durations\n=\n[]\ndef\nplot_durations\n(\nshow_result\n=\nFalse\n):\nplt\n.\nfigure\n(\n1\n)\ndurations_t\n=\ntorch\n.\ntensor\n(\nepisode_durations\n,\ndtype\n=\ntorch\n.\nfloat\n)\nif\nshow_result\n:\nplt\n.\ntitle\n(\n'Result'\n)\nelse\n:\nplt\n.\nclf\n()\nplt\n.\ntitle\n(\n'Training...'\n)\nplt\n.\nxlabel\n(\n'Episode'\n)\nplt\n.\nylabel\n(\n'Duration'\n)\nplt\n.\nplot\n(\ndurations_t\n.\nnumpy\n())\n# Take 100 episode averages and plot them too\nif\nlen\n(\ndurations_t\n)\n>=\n100\n:\nmeans\n=\ndurations_t\n.\nunfold\n(\n0\n,\n100\n,\n1\n)\n.\nmean\n(\n1\n)\n.\nview\n(\n-\n1\n)\nmeans\n=\ntorch\n.\ncat\n((\ntorch\n.\nzeros\n(\n99\n),\nmeans\n))\nplt\n.\nplot\n(\nmeans\n.\nnumpy\n())\nplt\n.\npause\n(\n0.001\n)\n# pause a bit so that plots are updated\nif\nis_ipython\n:\nif\nnot\nshow_result\n:\ndisplay\n.\ndisplay\n(\nplt\n.\ngcf\n())\ndisplay\n.\nclear_output\n(\nwait\n=\nTrue\n)\nelse\n:\ndisplay\n.\ndisplay\n(\nplt\n.\ngcf\n())\nTraining loop\n#\nFinally, the code for training our model.\nHere, you can find an\noptimize_model\nfunction that performs a\nsingle step of the optimization. It first samples a batch, concatenates\nall the tensors into a single one, computes\n\\(Q(s_t, a_t)\\)\nand\n\\(V(s_{t+1}) = \\max_a Q(s_{t+1}, a)\\)\n, and combines them into our\nloss. By definition we set\n\\(V(s) = 0\\)\nif\n\\(s\\)\nis a terminal\nstate. We also use a target network to compute\n\\(V(s_{t+1})\\)\nfor\nadded stability. The target network is updated at every step with a\nsoft update\ncontrolled by\nthe hyperparameter\nTAU\n, which was previously defined.\ndef\noptimize_model\n():\nif\nlen\n(\nmemory\n)\n<\nBATCH_SIZE\n:\nreturn\ntransitions\n=\nmemory\n.\nsample\n(\nBATCH_SIZE\n)\n# Transpose the batch (see https://stackoverflow.com/a/19343/3343043 for\n# detailed explanation). This converts batch-array of Transitions\n# to Transition of batch-arrays.\nbatch\n=\nTransition\n(\n*\nzip\n(\n*\ntransitions\n))\n# Compute a mask of non-final states and concatenate the batch elements\n# (a final state would've been the one after which simulation ended)\nnon_final_mask\n=\ntorch\n.\ntensor\n(\ntuple\n(\nmap\n(\nlambda\ns\n:\ns\nis\nnot\nNone\n,\nbatch\n.\nnext_state\n)),\ndevice\n=\ndevice\n,\ndtype\n=\ntorch\n.\nbool\n)\nnon_final_next_states\n=\ntorch\n.\ncat\n([\ns\nfor\ns\nin\nbatch\n.\nnext_state\nif\ns\nis\nnot\nNone\n])\nstate_batch\n=\ntorch\n.\ncat\n(\nbatch\n.\nstate\n)\naction_batch\n=\ntorch\n.\ncat\n(\nbatch\n.\naction\n)\nreward_batch\n=\ntorch\n.\ncat\n(\nbatch\n.\nreward\n)\n# Compute Q(s_t, a) - the model computes Q(s_t), then we select the\n# columns of actions taken. These are the actions which would've been taken\n# for each batch state according to policy_net\nstate_action_values\n=\npolicy_net\n(\nstate_batch\n)\n.\ngather\n(\n1\n,\naction_batch\n)\n# Compute V(s_{t+1}) for all next states.\n# Expected values of actions for non_final_next_states are computed based\n# on the \"older\" target_net; selecting their best reward with max(1).values\n# This is merged based on the mask, such that we'll have either the expected\n# state value or 0 in case the state was final.\nnext_state_values\n=\ntorch\n.\nzeros\n(\nBATCH_SIZE\n,\ndevice\n=\ndevice\n)\nwith\ntorch\n.\nno_grad\n():\nnext_state_values\n[\nnon_final_mask\n]\n=\ntarget_net\n(\nnon_final_next_states\n)\n.\nmax\n(\n1\n)\n.\nvalues\n# Compute the expected Q values\nexpected_state_action_values\n=\n(\nnext_state_values\n*\nGAMMA\n)\n+\nreward_batch\n# Compute Huber loss\ncriterion\n=\nnn\n.\nSmoothL1Loss\n()\nloss\n=\ncriterion\n(\nstate_action_values\n,\nexpected_state_action_values\n.\nunsqueeze\n(\n1\n))\n# Optimize the model\noptimizer\n.\nzero_grad\n()\nloss\n.\nbackward\n()\n# In-place gradient clipping\ntorch\n.\nnn\n.\nutils\n.\nclip_grad_value_\n(\npolicy_net\n.\nparameters\n(),\n100\n)\noptimizer\n.\nstep\n()\nBelow, you can find the main training loop. At the beginning we reset\nthe environment and obtain the initial\nstate\nTensor. Then, we sample\nan action, execute it, observe the next state and the reward (always\n1), and optimize our model once. When the episode ends (our model\nfails), we restart the loop.\nBelow,\nnum_episodes\nis set to 600 if a GPU is available, otherwise 50\nepisodes are scheduled so training does not take too long. However, 50\nepisodes is insufficient for to observe good performance on CartPole.\nYou should see the model constantly achieve 500 steps within 600 training\nepisodes. Training RL agents can be a noisy process, so restarting training\ncan produce better results if convergence is not observed.\nif\ntorch\n.\ncuda\n.\nis_available\n()\nor\ntorch\n.\nbackends\n.\nmps\n.\nis_available\n():\nnum_episodes\n=\n600\nelse\n:\nnum_episodes\n=\n50\nfor\ni_episode\nin\nrange\n(\nnum_episodes\n):\n# Initialize the environment and get its state\nstate\n,\ninfo\n=\nenv\n.\nreset\n()\nstate\n=\ntorch\n.\ntensor\n(\nstate\n,\ndtype\n=\ntorch\n.\nfloat32\n,\ndevice\n=\ndevice\n)\n.\nunsqueeze\n(\n0\n)\nfor\nt\nin\ncount\n():\naction\n=\nselect_action\n(\nstate\n)\nobservation\n,\nreward\n,\nterminated\n,\ntruncated\n,\n_\n=\nenv\n.\nstep\n(\naction\n.\nitem\n())\nreward\n=\ntorch\n.\ntensor\n([\nreward\n],\ndevice\n=\ndevice\n)\ndone\n=\nterminated\nor\ntruncated\nif\nterminated\n:\nnext_state\n=\nNone\nelse\n:\nnext_state\n=\ntorch\n.\ntensor\n(\nobservation\n,\ndtype\n=\ntorch\n.\nfloat32\n,\ndevice\n=\ndevice\n)\n.\nunsqueeze\n(\n0\n)\n# Store the transition in memory\nmemory\n.\npush\n(\nstate\n,\naction\n,\nnext_state\n,\nreward\n)\n# Move to the next state\nstate\n=\nnext_state\n# Perform one step of the optimization (on the policy network)\noptimize_model\n()\n# Soft update of the target network's weights\n# θ′ ← τ θ + (1 −τ )θ′\ntarget_net_state_dict\n=\ntarget_net\n.\nstate_dict\n()\npolicy_net_state_dict\n=\npolicy_net\n.\nstate_dict\n()\nfor\nkey\nin\npolicy_net_state_dict\n:\ntarget_net_state_dict\n[\nkey\n]\n=\npolicy_net_state_dict\n[\nkey\n]\n*\nTAU\n+\ntarget_net_state_dict\n[\nkey\n]\n*\n(\n1\n-\nTAU\n)\ntarget_net\n.\nload_state_dict\n(\ntarget_net_state_dict\n)\nif\ndone\n:\nepisode_durations\n.\nappend\n(\nt\n+\n1\n)\nplot_durations\n()\nbreak\nprint\n(\n'Complete'\n)\nplot_durations\n(\nshow_result\n=\nTrue\n)\nplt\n.\nioff\n()\nplt\n.\nshow\n()\n/usr/local/lib/python3.10/dist-packages/gymnasium/utils/passive_env_checker.py:249: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)\n  if not isinstance(terminated, (bool, np.bool8)):\nComplete\nHere is the diagram that illustrates the overall resulting data flow.\nActions are chosen either randomly or based on a policy, getting the next\nstep sample from the gym environment. We record the results in the\nreplay memory and also run optimization step on every iteration.\nOptimization picks a random batch from the replay memory to do training of the\nnew policy. The “older” target_net is also used in optimization to compute the\nexpected Q values. A soft update of its weights are performed at every step.\nTotal running time of the script:\n(9 minutes 40.579 seconds)\nDownload\nJupyter\nnotebook:\nreinforcement_q_learning.ipynb\nDownload\nPython\nsource\ncode:\nreinforcement_q_learning.py\nDownload\nzipped:\nreinforcement_q_learning.zip",
  "metadata": {
    "title": "Reinforcement Learning (DQN) Tutorial — PyTorch Tutorials 2.14.0+cu130 documentation",
    "description": null,
    "keywords": null,
    "author": null,
    "og:image": "https://docs.pytorch.org/docs/stable/_static/img/pytorch_seo.png",
    "article:modified_time": "2022-12-08T19:56:14+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/10d19930e334/code/body-000.txt",
      "chars": 47,
      "language": "text"
    },
    {
      "file": "attachments/10d19930e334/code/body-001.txt",
      "chars": 1153,
      "language": "text"
    },
    {
      "file": "attachments/10d19930e334/code/body-002.txt",
      "chars": 500,
      "language": "text"
    },
    {
      "file": "attachments/10d19930e334/code/body-003.txt",
      "chars": 472,
      "language": "text"
    },
    {
      "file": "attachments/10d19930e334/code/body-004.txt",
      "chars": 524,
      "language": "text"
    },
    {
      "file": "attachments/10d19930e334/code/body-005.txt",
      "chars": 2443,
      "language": "text"
    },
    {
      "file": "attachments/10d19930e334/code/body-006.txt",
      "chars": 2164,
      "language": "text"
    },
    {
      "file": "attachments/10d19930e334/code/body-007.txt",
      "chars": 1566,
      "language": "text"
    },
    {
      "file": "attachments/10d19930e334/code/body-008.txt",
      "chars": 238,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://docs.pytorch.org/tutorials/_images/cartpole.gif",
      "bytes": 272424,
      "reused": false,
      "local_path": "attachments/10d19930e334/images/body-8da0cc19cd8e93fb2c78.gif",
      "src": "https://docs.pytorch.org/tutorials/_images/cartpole.gif",
      "alt": "CartPole",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://docs.pytorch.org/tutorials/_images/sphx_glr_reinforcement_q_learning_001.png",
      "bytes": 30019,
      "reused": false,
      "local_path": "attachments/10d19930e334/images/body-f69194d5aaeb994cebf0.png",
      "src": "https://docs.pytorch.org/tutorials/_images/sphx_glr_reinforcement_q_learning_001.png",
      "alt": "Result",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://docs.pytorch.org/tutorials/_images/reinforcement_learning_diagram.jpg",
      "bytes": 34793,
      "reused": false,
      "local_path": "attachments/10d19930e334/images/body-03f36edec5f0050b487d.jpg",
      "src": "https://docs.pytorch.org/tutorials/_images/reinforcement_learning_diagram.jpg",
      "alt": "../_images/reinforcement_learning_diagram.jpg",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/apaszke",
      "label": "Adam Paszke",
      "type": "a"
    },
    {
      "href": "https://github.com/pseudo-rnd-thoughts",
      "label": "Mark Towers",
      "type": "a"
    },
    {
      "href": "https://gymnasium.farama.org/",
      "label": "Gymnasium",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1312.5602",
      "label": "Deep Q Learning (DQN)",
      "type": "a"
    },
    {
      "href": "https://gymnasium.farama.org/environments/classic_control/cart_pole/",
      "label": "Gymnasium’s website",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/tensor_attributes.html",
      "label": "device",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.cuda.is_available.html",
      "label": "torch . cuda . is_available",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/backends.html",
      "label": "torch . backends . mps . is_available",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Huber_loss",
      "label": "Huber\nloss",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html",
      "label": "nn . Module",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.Linear.html",
      "label": "nn . Linear",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.relu.html",
      "label": "F . relu",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/tensors.html",
      "label": "state",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.optim.AdamW.html",
      "label": "optimizer",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.no_grad.html",
      "label": "torch . no_grad",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.tensor.html",
      "label": "torch . tensor",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.cat.html",
      "label": "torch . cat",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.zeros.html",
      "label": "torch . zeros",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1509.02971.pdf",
      "label": "soft update",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.SmoothL1Loss.html",
      "label": "nn . SmoothL1Loss",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_value_.html",
      "label": "torch . nn . utils . clip_grad_value_",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/_downloads/9da0471a9eeb2351a488cd4b44fc6bbf/reinforcement_q_learning.ipynb",
      "label": "Download Jupyter notebook: reinforcement_q_learning.ipynb",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/_downloads/6ea0e49c7d0da2713588ef1a3b64eb35/reinforcement_q_learning.py",
      "label": "Download Python source code: reinforcement_q_learning.py",
      "type": "a"
    },
    {
      "href": "https://docs.pytorch.org/tutorials/_downloads/4688eaf2bedafbdc28e62bfc388d2b6d/reinforcement_q_learning.zip",
      "label": "Download zipped: reinforcement_q_learning.zip",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:54:03Z",
  "source_url": "https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1312.5602",
      "bytes": 483443,
      "reused": false,
      "local_path": "papers/repair/9d9e45e8528378b20a1d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1509.02971.pdf",
      "bytes": 663643,
      "reused": true,
      "local_path": "papers/repair/cdfdf91c078ecbc5323f.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 17188,
  "expected_body_images": 3,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/10d19930e334/pages/source.html",
  "local_html": "attachments/10d19930e334/pages/article.html",
  "body_base_path": "attachments/10d19930e334/pages/",
  "body_markdown_path": "attachments/10d19930e334/pages/article.md"
}