{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Mitansh Gor",
  "title_listed": "RL 3: Bellman Equations and Markov Decision Processes (1950s–1960s)",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/mitanshgor/rl-3-bellman-equations-and-markov-decision-processes-1950s-1960s-2pp0",
  "publish_date": "2026-09-18",
  "corpus_stage": "",
  "notes": "",
  "id": "77684a950aa4",
  "collected_at": "2026-10-02T13:52:27",
  "title": "RL 3: Bellman Equations and Markov Decision Processes (1950s–1960s) - DEV Community",
  "body_markdown": "Where we left off\nTwo quick reminders.\nThorndike (1898)\nwatched cats escape a puzzle box and gave us the\nLaw of Effect\n: do something, get a good result, do it again. Get a bad result, stop. Learning is bookkeeping over consequences.\nBlog 2\nturned that idea into hardware. Shannon's mouse ran a maze and remembered the route. Minsky's SNARC strengthened the wires that led to good outcomes.\nBoth worked. Both were missing the same thing.\nThorndike's cat knew\n\"pulling that loop felt good.\"\nIt did\nnot\nknow it was three moves from freedom. Shannon's mouse could replay a path, but it couldn't tell you why one path was better than another.\nNobody had a word for this yet:\nValue\n— a number attached to a situation that says\nhow good does the future look from right here?\nThat one missing number is what this whole post is about.\nSide note: the biologists were chasing the same number\nWorth thirty seconds, because it makes the maths feel less alien.\nRescorla and Wagner (1972)\nshowed that animals don't learn from repetition. They learn from\nsurprise\n. If a bell already predicts food perfectly, adding a light next to the bell teaches the animal almost nothing about the light. There's no surprise left to learn from.\nTheir rule, in plain English:\nnew belief = old belief + (small step) × (what actually happened − what I expected)\nThat last bracket — reality minus expectation — is called\nprediction error\n. Hold onto it. It comes back in Blog 4 with a much bigger job.\n(Yes, 1972 is out of order here. I'm putting it early because it explains \"value\" better than a grid does.)\nEnter Richard Bellman\nIn 1949, a mathematician named\nRichard Bellman\nstarted working at\nRAND\n, a US think tank. His job was a specific shape of problem: you make a choice, the world changes, then you have to choose again. Missile guidance. Supply planning. Anything where today's decision quietly reshapes tomorrow's options.\nHe cracked it. Then he needed a name for it, So he picked two words that sounded harmless and industrial:\nDynamic Programming\n.\n(The story has probably been tidied up over the years — the dates don't quite line up — but the name stuck, and it's now one of the most important terms in computer science.)\nHere's the problem he was actually solving.\nThe problem, in the form we'll use all post\nForget missiles. Here's the setup we'll use for everything that follows.\nYou're on a grid. You start in one corner. The goal is somewhere else. Every step costs you a little energy. Some cells are bad news. You want to reach the goal having collected as much reward as possible.\nEasy to describe. Genuinely nasty to solve\nsystematically\n.\nThe obvious approach is to list every possible path and pick the best. That works right up until it catastrophically doesn't. Paths multiply exponentially with grid size. 10×10 is already unpleasant. 100×100 is hopeless. Bellman later gave this failure mode a name —\nthe curse of dimensionality\n— and we'll come back to it at the end, because it's the villain of the whole post.\nSo he needed a fundamentally different kind of reasoning.\nDynamic Programming: solve it backward, and only solve it once\nBellman's core insight is called the\nPrinciple of Optimality\n. Here it is in his words:\n\"An optimal policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision.\"\nThat's a mouthful. In human:\nIf the best route from A to C goes through B, then the B-to-C part of it must itself be the best route from B to C.\nBecause if it weren't — if there were a better way from B to C — you'd just swap it in and get a better A-to-C route. Contradiction. Done.\nSounds trivial. It isn't, because of what it licenses you to do:\nYou only ever have to solve each sub-problem once.\nOnce you know the best way from B onward,\nevery\nroute that passes through B can reuse that answer for free. No recomputation. That's Dynamic Programming: chop a giant sequential problem into overlapping sub-problems, solve them from the end backward, and cache the answers.\nThe\nbackward\npart is the counterintuitive bit and it's worth sitting with. Bellman didn't ask \"what's my best first move?\" He asked\n\"what's my best\nlast\nmove?\"\nThen second-to-last. Then third-to-last, propagating backward until every square on the board has an answer.\nThis is\nbackward induction\n, and it's not a historical curiosity — it is still, structurally, how modern RL agents plan.\nHold onto this:\nthe future is easier to reason about than the past, because the future has an ending you can anchor to.\nFive nouns you need before the equation shows up\nSo: five nouns first. Equation after. All five together are called a\nMarkov Decision Process (MDP)\n— the formal container the maths lives in.\n1. State —\ns\nWhere you are. On the grid, \"row 3, column 4.\" In chess, the board position. In a delivery robot, position\nplus\nbattery\nplus\nwhether it's holding a package. The full set of states is written\nS\n.\n2. Action —\na\nWhat you can do from here. On the grid: up, down, left, right. The full set is\nA\n.\n3. Transition —\nP(s' | s, a)\nRead it as:\n\"the probability of landing in state\ns'\n, given that I was in\ns\nand did\na\n.\"\nThe apostrophe just means \"next.\"\n4. Reward —\nR(s, a)\nA number the environment hands you for doing\na\nin\ns\n. This is where you, the designer, encode what \"good\" means. Reach the goal: +10. Take a step: −1. Fall in the pit: −100.\nGet this wrong and your agent will cheerfully optimise the wrong thing. Reward design is 90% of applied RL pain, and I'm not going to pretend otherwise.\n5. Policy —\nπ\n(pi)\nYour strategy. A rule that maps\nstate → action\n. \"When in this square, go right.\" That's it. A policy isn't a plan or a path — it's a lookup table of what to do\nanywhere\nyou might find yourself.\nThe whole field, in one sentence:\nan MDP describes the world; a policy describes your behaviour in it; and the rest of RL is the search for the best policy.\nThe one assumption holding all of this up\nThere's a condition the whole framework quietly depends on, and it's called the\nMarkov Property\n:\nThe future depends only on the present state — not on the history of how you got there.\nFirst reaction to this is usually:\nthat's obviously false, history matters all the time.\nFair. But that's not quite what it says.\nIt says\nhistory must already be baked into the state.\nIf your robot's battery level affects the next decision, then battery level belongs\nin the state definition\n. Once it's in there, you don't need to replay the last 50 timesteps — you just look at where you are now.\nTwo quick examples to make it concrete:\nChess is Markov.\nThe board position tells you everything. It doesn't matter whether you reached it through a brilliant sacrifice or a series of blunders — the position is the position.\nPoker is not Markov if your state is just \"my cards.\"\nWhether your opponent has been bluffing all night genuinely changes what you should do. Fix it by expanding the state to include betting history. Now it's Markov again — and much bigger. (Notice the trade you just made. That's foreshadowing.)\nThe neat way to say it:\nthe\nstate\nis allowed to have memory. The\npolicy\nisn't.\nThis assumption is exactly what makes RL computable. Without it, an agent would have to carry and process its entire past at every step. With it, a policy is just a lookup: current state in, action out.\nOne more assumption, and it's a big one.\nFor this entire post, the agent is assumed to already\nknow\nP\nand\nR\n. It knows the physics of the world and the reward structure before it takes a single step. That's called\nmodel-based\nRL, and it's what makes the maths clean here.\nIt is also completely unrealistic, and Blog 4 takes a sledgehammer to it.\nReturn and γ: why \"how good\" needs a definition\nOne more thing before the equation. If value means \"how promising is the future from here,\" we need to say what\nthe future\nmeans numerically.\nAdd up all the reward you'll collect from now until the end. That total is called the\nreturn\n.\nExcept there's a problem: if the task never ends, that sum runs to infinity, and \"infinity\" is a terrible thing to compare against another infinity. So Bellman introduced a\ndiscount factor,\nγ\n(gamma)\n, a number between 0 and 1. Reward one step away is multiplied by γ. Two steps away, γ². Three steps, γ³. And so on.\nG\nt\n=\nR\nt\n+\nγ\nR\nt\n+\n1\n+\nγ\n2\nR\nt\n+\n2\n+\nγ\n3\nR\nt\n+\n3\n+\n…\nG_t = R_t + \\gamma R_{t+1} + \\gamma^2 R_{t+2} + \\gamma^3 R_{t+3} + \\dots\nG\nt\n​\n=\nR\nt\n​\n+\nγ\nR\nt\n+\n1\n​\n+\nγ\n2\nR\nt\n+\n2\n​\n+\nγ\n3\nR\nt\n+\n3\n​\n+\n…\nTwo things fall out of this, and both matter:\nMathematically:\nthe sum now converges. It's a geometric series. No infinities.\nBehaviourally:\nγ is a\npersonality dial.\nγ near\n0\n— a myopic agent. Only right now exists. It will grab the nearest +1 and ignore a +100 two steps away.\nγ near\n1\n— a patient agent. Happy to eat a −1 today for a +100 later.\nγ = 0.9 — a reasonable default that says \"about ten steps of foresight.\"\nIt's compound interest, run backward. A reward ten steps away at γ = 0.9 is worth about 35% of its face value today.\nValue: the number Thorndike's cat never had\nNow we can define the thing this whole post is chasing.\nV(s)\n— the value of state\ns\n— is the expected return you'll collect if you start in\ns\nand behave well from there on.\nThat's the number the cat didn't have. The cat knew\n\"this felt good.\"\nA value function knows\n\"this square is worth 7.1, that one's worth 5.4, so go left.\"\nAnd that's the whole trick:\nonce every square has a number, choosing an action stops being a search problem and becomes a comparison.\nLook at the neighbouring numbers. Walk toward the big one. Done.\nThe Bellman Equation\nIn his 1957 book\nDynamic Programming\n, Bellman wrote down the recursion that ties all of this together:\nV\n(\ns\n)\n=\nmax\n⁡\na\n[\nR\n(\ns\n,\na\n)\n+\nγ\n∑\ns\n′\nP\n(\ns\n′\n∣\ns\n,\na\n)\nV\n(\ns\n′\n)\n]\nV(s) = \\max_a \\left[ R(s, a) + \\gamma \\sum_{s'} P(s' \\mid s, a)\\, V(s') \\right]\nV\n(\ns\n)\n=\na\nmax\n​\n[\nR\n(\ns\n,\na\n)\n+\nγ\ns\n′\n∑\n​\nP\n(\ns\n′\n∣\ns\n,\na\n)\nV\n(\ns\n′\n)\n]\nIn one sentence:\nThe value of where you are = the best immediate reward you can grab + the discounted value of wherever you end up next.\nEvery symbol is one of the five nouns you already know:\nPiece\nReading\nV(s)\nhow good it is to be here\nmax_a\nyou, choosing.\nTry every action, keep the best\nR(s, a)\nwhat you get right now\nγ\nhow much you care about later\n∑ P(s':s,a) · V(s')\nthe world, responding.Average over every place you might land, weighted by likelihood\nTwo things I want to flag, because they're where people quietly lose the thread.\nFirst:\nmax\nand\n∑\nare doing opposite jobs.\nThe\nmax\nis the agent\nchoosing\n. The\n∑\nis the world\nrolling dice\n. You control your action. You do not control the outcome. If the transitions were deterministic, that\n∑\nwould collapse to a single term and disappear — it only exists because the world is uncertain.\nSecond:\nV(s)\nappears on both sides.\nThat's not sloppy notation, it's the entire idea. The equation is\nself-referential\n— the value of a state is defined in terms of the values of\nother\nstates.\nWhich raises the obvious question: if V is defined in terms of V, how do you ever compute it?\nYou guess, then you improve. Start with V = 0 everywhere (a wrong but harmless guess), apply the equation, get better numbers, apply again. The values stop changing when they're all consistent with each other. Mathematicians call that a\nfixed point\n— the place where applying the rule doesn't change anything anymore.\nThat \"keeps changing until it doesn't\" process is the entire computational game. And it's easier to believe once you've watched it happen, so let's watch it happen.\nLet's actually put numbers on the board\nHere's the smallest example that shows the real behaviour. A corridor, four squares, then the goal:\n[ S1 ][ S2 ][ S3 ][ S4 ][ GOAL ]\nEnter fullscreen mode\nExit fullscreen mode\nRules:\nActions: move left or move right. Deterministic — no ice.\nEvery move costs\n−1\n.\nStepping into GOAL pays\n+10\n.\nγ =\n0.9\n.\nGOAL is terminal, so\nV(GOAL) = 0\n.\nWork backward, exactly as Bellman said to.\nS4\nis one step from GOAL:\nV(S4) = −1 + 10 + 0.9 × 0 =\n9.00\nS3\nis one step from S4:\nV(S3) = −1 + 0.9 × 9.00 =\n7.10\nS2:\nV(S2) = −1 + 0.9 × 7.10 =\n5.39\nS1:\nV(S1) = −1 + 0.9 × 5.39 =\n3.85\n[ 3.85 ][ 5.39 ][ 7.10 ][ 9.00 ][ GOAL ]\nEnter fullscreen mode\nExit fullscreen mode\nLook at what happened.\nThe goal's reward leaked backward through the corridor and left a slope behind it.\nThe agent doesn't need a map, a plan, or a search. It just walks uphill. The value function turned a planning problem into a gradient you can follow like a scent trail.\nAnimated_heatmap_of_the_x_gr.mp4 - Google Drive\ndrive.google.com\nNow watch it converge from nothing\nThat was the answer. Here's the\nprocess\n— starting from V = 0 everywhere and sweeping the equation across all four states repeatedly:\nSweep\nV(S1)\nV(S2)\nV(S3)\nV(S4)\n0\n0.00\n0.00\n0.00\n0.00\n1\n−1.00\n−1.00\n−1.00\n9.00\n2\n−1.90\n−1.90\n7.10\n9.00\n3\n−2.71\n5.39\n7.10\n9.00\n4\n3.85\n5.39\n7.10\n9.00\n5\n3.85\n5.39\n7.10\n9.00\nThree things worth noticing here, and honestly this table is the most important object in the post:\nInformation travels exactly one square per sweep.\nSweep 1, only S4 knows the goal exists. Sweep 2, S3 finds out. It's a wavefront moving backward.\nThe early numbers are hilariously wrong.\nAt sweep 3, S1 thinks it's worth −2.71 — it can only see step costs and no payoff yet. It's a pessimist with incomplete information. It gets over it.\nYou don't need a stopping rule, you need a stopping\ncondition\n.\nWhen a full sweep changes nothing, you're done. That's the fixed point, made of actual numbers.\nHoward's Policy Iteration: from knowing values to having a strategy\nBellman gave us the value of every square. But there's a subtly separate question: how do you systematically build the\nstrategy\n— the policy — that goes with it?\nRonald Howard\n, in his 1960 monograph\nDynamic Programming and Markov Processes\n, noticed the problem is circular:\nTo compute\nV(s)\n, you need to know which policy\nπ\nthe agent is following.\nTo find the best\nπ\n, you need to know\nV(s)\n.\nClassic chicken and egg. Howard's answer:\nstop trying to do both at once. Alternate.\nStep 1 — Policy Evaluation: \"how good is my current strategy?\"\nFreeze the policy. Don't try to improve it. Just ask:\nif I followed this exact strategy forever, what would each state be worth?\nV\nπ\n(\ns\n)\n=\nR\n(\ns\n,\nπ\n(\ns\n)\n)\n+\nγ\n∑\ns\n′\nP\n(\ns\n′\n∣\ns\n,\nπ\n(\ns\n)\n)\nV\nπ\n(\ns\n′\n)\nV^\\pi(s) = R(s, \\pi(s)) + \\gamma \\sum_{s'} P(s' \\mid s, \\pi(s))\\, V^\\pi(s')\nV\nπ\n(\ns\n)\n=\nR\n(\ns\n,\nπ\n(\ns\n))\n+\nγ\ns\n′\n∑\n​\nP\n(\ns\n′\n∣\ns\n,\nπ\n(\ns\n))\nV\nπ\n(\ns\n′\n)\nNotice the\nmax\nis gone.\nThere's no choosing here.\nπ(s)\ntells you what to do; you're just measuring the consequences. This isn't optimisation, it's\nscorekeeping\n.\nStep 2 — Policy Improvement: \"could I do better?\"\nNow take those scores and ask, at every state:\ngiven what I now know things are worth, is there an action better than the one my policy currently prescribes?\nIf yes, switch to it.\nπ\n′\n(\ns\n)\n=\narg\n⁡\nmax\n⁡\na\n[\nR\n(\ns\n,\na\n)\n+\nγ\n∑\ns\n′\nP\n(\ns\n′\n∣\ns\n,\na\n)\nV\nπ\n(\ns\n′\n)\n]\n\\pi'(s) = \\arg\\max_a \\left[ R(s, a) + \\gamma \\sum_{s'} P(s' \\mid s, a)\\, V^\\pi(s') \\right]\nπ\n′\n(\ns\n)\n=\nar\ng\na\nmax\n​\n[\nR\n(\ns\n,\na\n)\n+\nγ\ns\n′\n∑\n​\nP\n(\ns\n′\n∣\ns\n,\na\n)\nV\nπ\n(\ns\n′\n)\n]\narg max\nvs\nmax\n, since this trips everyone up:\nmax\nreturns\nthe best value\n.\narg max\nreturns\nthe action that achieved it\n. Evaluation wants the number; improvement wants the move.\nAnd repeat\nEvaluate. Improve. Evaluate. Improve. Stop when an improvement step changes nothing.\nHoward proved two things about this loop that are genuinely reassuring:\nThe greedy step never makes the policy worse.\nEvery improvement is an improvement or a tie. It can't wander off toward a worse policy — the local, greedy move is globally safe here. (That's a strong guarantee, and one that quietly evaporates once we start using neural networks in later posts.)\nOn a finite MDP, it terminates in a finite number of iterations.\nThere are only finitely many policies, you never repeat one, so you must stop. Usually in a handful of rounds.\nAnd here's the part that should give you a small jolt:\nthat two-role split — one part measuring, one part deciding — is the direct ancestor of the\nActor–Critic\narchitectures we'll get to in Blog 7. The Critic does Policy Evaluation. The Actor does Policy Improvement.\nThat structure wasn't invented for deep learning. It was sitting in a 1960 operations research monograph the whole time.\nValue Iteration: the impatient version\nPolicy Iteration is a bit fussy. It insists on\nfully\nevaluating a policy — running evaluation to convergence — before it'll consider improving anything.\nValue Iteration\nsays: why wait? Fold the improvement directly into the update.\nV\n(\ns\n)\n←\nmax\n⁡\na\n[\nR\n(\ns\n,\na\n)\n+\nγ\n∑\ns\n′\nP\n(\ns\n′\n∣\ns\n,\na\n)\nV\n(\ns\n′\n)\n]\nV(s) \\leftarrow \\max_a \\left[ R(s, a) + \\gamma \\sum_{s'} P(s' \\mid s, a)\\, V(s') \\right]\nV\n(\ns\n)\n←\na\nmax\n​\n[\nR\n(\ns\n,\na\n)\n+\nγ\ns\n′\n∑\n​\nP\n(\ns\n′\n∣\ns\n,\na\n)\nV\n(\ns\n′\n)\n]\nThe\nmax\nis back, and it's inside the loop.\nEvery single update implicitly re-decides the best action. No separate improvement phase exists.\nValue Iteration never stores a policy at all. It just hammers on the value function until it settles, and\nthen\nyou read the policy off: stand in each state, look at the neighbours, take the action pointing at the biggest number.\n(That sweep table you looked at earlier? That was Value Iteration. You've already seen it run.)\nPolicy Iteration vs Value Iteration, side by side\nSame equation, same destination, different temperament.\nPolicy Iteration is the coach who watches the entire game.\nFilms it. Writes a 40-page report. Rewrites the playbook from scratch before the next match. Few iterations, each one expensive and each one a big leap.\nValue Iteration is the coach who stops play after every snap.\n\"Stop — standing three feet left would've been worth two more points.\"\nCheap per update, but it needs a lot of them. Constantly nudging everyone's understanding of the field.\nPolicy Iteration\nValue Iteration\nExplicit policy?\nYes, maintained separately\nNo, implicit in the values\nmax\nin the update?\nOnly in the improvement step\nEvery update\nCost per iteration\nHigh (full evaluation)\nLow (one sweep)\nIterations needed\nFew\nMany\nTerminates when\nPolicy stops changing\nValues stop changing\nBest when\nState space is small and exact\nState space is large or approximate\nWhich should you use? Small, well-defined state space where you want exactness — Policy Iteration. Large or approximate — Value Iteration.\nAnd worth flagging for later:\nalmost every modern RL algorithm is spiritually a descendant of Value Iteration.\nThey update values continuously and never wait for a full evaluation to finish. Q-learning is Value Iteration with the model ripped out. DQN is Q-learning with a neural network glued on.\nThe curse they named but couldn't escape\nNow the bad news, and Bellman himself is the one who delivered it.\nBoth algorithms require sweeping over\nevery state in\nS\n. Every single one. Every iteration.\n4-square corridor: 4 states. Trivial.\n10×10 grid: 100 states. Fine.\nChess: roughly 10⁴⁴ positions.\nGo: more legal positions than there are atoms in the observable universe.\nBellman named this the\ncurse of dimensionality\n, in the same 1957 book that gave us the equation. Add one variable to your state description and the state space doesn't grow — it\nmultiplies\n. A robot tracking position, velocity, battery, and grip status doesn't have four problems. It has (positions × velocities × battery levels × grip states) problems.\nAnd remember the poker example from earlier? Fixing the Markov violation by stuffing betting history into the state made the state space explode. That's the curse arriving in person.\nMaking a problem Markov and making it small are usually opposing goals.\nThis is why RL went quiet for decades after Bellman. It wasn't that the theory was shaky.\nThe theory was perfect and the computer was too small.\nDynamic Programming solves the\nstructure\nof sequential decision-making completely — it just needs a table with one row per state, and sometimes that table can't exist.\nThe escape route — stop storing\nV(s)\nin a table and start\napproximating\nit with a function — wouldn't properly arrive until DeepMind's DQN in 2013. But the problem was named here, in 1957. Nearly everything between then and now is an attempt to get around it.\nWhy 1957 mathematics is running inside your 2026 chatbot\nIt sounds like a stretch that equations written for missile guidance underpin modern reasoning models. It isn't.\nWhen a model like\no1\nor\nDeepSeek-R1\nworks through a hard problem, it isn't just emitting the most likely next token. It's exploring reasoning paths,\nscoring\nhow promising each one looks, and abandoning the ones heading toward a dead end.\nRead that again with this post's vocabulary loaded:\nA partial chain of reasoning is a\nstate\n.\nThe next thing to write is an\naction\n.\n\"Is this line of thinking going anywhere?\" is a\nvalue estimate\n.\nBacktracking out of a bad path is\nbackward induction\n— the same move Bellman made when he asked about the\nlast\nstep instead of the first.\nThe state space changed from grid squares to token sequences. The tables became neural networks. The core question —\nhow good is this situation, really?\n— is unchanged since 1957.\nSix things to keep\nIf every equation falls out of your head by Thursday, keep these:\nThe world is states and actions\n— that's an MDP.\nRewards define what \"good\" means\n— and you're the one writing them, so be careful.\nValue is a score for how promising the future looks from here\n— the number Thorndike's cat never had.\nThe Bellman Equation makes future reward flow backward\nthrough states, leaving a slope you can climb.\nA policy is just a rule for picking actions\n— a lookup table, not a plan.\nIteration fixes the circularity.\nGuess, measure, improve, repeat, stop when nothing changes.\nIf Bellman clicks, the rest of RL is refinement.\nWhat's coming next\nEverything in this post rested on one assumption I flagged and then leaned on hard:\nthe agent already knows\nP\nand\nR\n.\nIt knows the physics. It knows the payoffs. It's not learning about the world — it's\ncalculating\nagainst a world it already has a complete map of.\nReal agents don't get that map.\nIn\nBlog 4 — Early Heuristics and the Birth of Temporal Difference Learning (1959–1968)\n, that assumption gets torn up. We'll meet a program that had to work out what positions were worth by\nplaying\n, not by computing — and we'll finally see what happens when you take Rescorla and Wagner's \"learn from surprise\" idea and let it fire on every single step instead of once per trial.\nRemember the prediction error term from the top of this post? It's about to get a job.\nThe equation was ready. The machine was next.\nNext: Blog 4 — Early Heuristics and the Birth of Temporal Difference Learning (1959–1968).\nPreviously: Blog 1 — Biological Foundations and the Law of Effect (1898–1949) · Blog 2 — The First Machines.",
  "metadata": {
    "title": "RL 3: Bellman Equations and Markov Decision Processes (1950s–1960s) - DEV Community",
    "description": "Where we left off   Two quick reminders.  Thorndike (1898) watched cats escape a puzzle box... Tagged with reinforcementlearning, machinelearning, ai, history.",
    "keywords": "reinforcementlearning, machinelearning, ai, history, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/mitanshgor/rl-3-bellman-equations-and-markov-decision-processes-1950s-1960s-2pp0",
    "og:title": "RL 3: Bellman Equations and Markov Decision Processes (1950s–1960s)",
    "og:description": "Where we left off   Two quick reminders.  Thorndike (1898) watched cats escape a puzzle box...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnpcsa3zzyks2p22eexgu.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "RL 3: Bellman Equations and Markov Decision Processes (1950s–1960s)",
    "twitter:description": "Where we left off   Two quick reminders.  Thorndike (1898) watched cats escape a puzzle box...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnpcsa3zzyks2p22eexgu.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/77684a950aa4/code/body-000.txt",
      "chars": 33,
      "language": "text"
    },
    {
      "file": "attachments/77684a950aa4/code/body-001.txt",
      "chars": 41,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F98aodvd1bi2qni84k7q9.png",
      "bytes": 28254,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/001.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F98aodvd1bi2qni84k7q9.png",
      "alt": "Prediction error diagram",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F28nj9nvict7epd44gyk0.png",
      "bytes": 12586,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/002.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F28nj9nvict7epd44gyk0.png",
      "alt": "Richard Bellman",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1cx1d3ep6pxzlra06vvm.png",
      "bytes": 79030,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/003.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1cx1d3ep6pxzlra06vvm.png",
      "alt": "grid",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F9f9yzz0hsk9op7rj7817.png",
      "bytes": 29712,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/004.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F9f9yzz0hsk9op7rj7817.png",
      "alt": "stare-reardloop",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fvoj8dzboblvy02ckek0h.png",
      "bytes": 135034,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/005.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fvoj8dzboblvy02ckek0h.png",
      "alt": "backward induction",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F460d9r80ynde2ixbgf04.png",
      "bytes": 101812,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/006.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F460d9r80ynde2ixbgf04.png",
      "alt": "MDP diagram",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fbz9309bj0gu2w598yq1s.png",
      "bytes": 43736,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/007.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fbz9309bj0gu2w598yq1s.png",
      "alt": "Reinforcement Learning loop",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0j3w2pfw16tjoscydeb1.png",
      "bytes": 71546,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/008.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0j3w2pfw16tjoscydeb1.png",
      "alt": "stateIsEnough",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fivqu2yfy9rbo9lxbnlf8.png",
      "bytes": 215058,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/009.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fivqu2yfy9rbo9lxbnlf8.png",
      "alt": "Same grid solved three times",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fssl.gstatic.com%2Fdocs%2Fdoclist%2Fimages%2Fdrive_favicon_2026_32dp.png",
      "bytes": 513,
      "reused": false,
      "local_path": "attachments/77684a950aa4/images/body-6195183832963f067bb8.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fssl.gstatic.com%2Fdocs%2Fdoclist%2Fimages%2Fdrive_favicon_2026_32dp.png",
      "alt": "favicon",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe8grw8sbri68uc9p8788.png",
      "bytes": 59143,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/010.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe8grw8sbri68uc9p8788.png",
      "alt": "bar chart per sweep",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fs3xphu5bv5k6sxejgsjx.png",
      "bytes": 94174,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/019.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fs3xphu5bv5k6sxejgsjx.png",
      "alt": "PE vs PI",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fh0vlv9oo30xbnr0swsa5.png",
      "bytes": 136106,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/012.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fh0vlv9oo30xbnr0swsa5.png",
      "alt": "Policy Eval and Improvement",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fj0n70gh4uwe9sqkz4sog.png",
      "bytes": 63450,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/013.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fj0n70gh4uwe9sqkz4sog.png",
      "alt": "Eval Improv General idea",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fy1sj0dabx5pxpe0c7fu0.png",
      "bytes": 48552,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/014.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fy1sj0dabx5pxpe0c7fu0.png",
      "alt": "actor-critic",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkisnepo5w1ik802tpuqo.png",
      "bytes": 82232,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/015.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkisnepo5w1ik802tpuqo.png",
      "alt": "Value iteration code loop",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fol3kxq76qxut4anbugrn.png",
      "bytes": 18354,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/016.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fol3kxq76qxut4anbugrn.png",
      "alt": "Value iteration diagram",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ft67wzcfoyg1eg0yusm8x.png",
      "bytes": 128294,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/017.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ft67wzcfoyg1eg0yusm8x.png",
      "alt": "PI vs VI comparison",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd58xk4wtaecldkdl8ny6.png",
      "bytes": 87258,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/018.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd58xk4wtaecldkdl8ny6.png",
      "alt": "Convergence plot",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fldvjuox3yrc4hxi0ql3p.png",
      "bytes": 76822,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/020.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fldvjuox3yrc4hxi0ql3p.png",
      "alt": "Exponential blowup chart",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fjyhn54bk6dkq914dey11.png",
      "bytes": 172904,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/021.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fjyhn54bk6dkq914dey11.png",
      "alt": "timeline",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmu8z9j1hlh0zh5csusmj.png",
      "bytes": 89344,
      "reused": true,
      "local_path": "attachments/77684a950aa4/images/022.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmu8z9j1hlh0zh5csusmj.png",
      "alt": "Glossary",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F98aodvd1bi2qni84k7q9.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F28nj9nvict7epd44gyk0.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1cx1d3ep6pxzlra06vvm.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F9f9yzz0hsk9op7rj7817.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fvoj8dzboblvy02ckek0h.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F460d9r80ynde2ixbgf04.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fbz9309bj0gu2w598yq1s.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0j3w2pfw16tjoscydeb1.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fivqu2yfy9rbo9lxbnlf8.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://drive.google.com/file/d/1zEZ-xmnFXWbNvPaQeYPOpUamLs6qBp62/view?usp=sharing",
      "label": "Animated_heatmap_of_the_x_gr.mp4 - Google Drive",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe8grw8sbri68uc9p8788.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fs3xphu5bv5k6sxejgsjx.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fh0vlv9oo30xbnr0swsa5.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fj0n70gh4uwe9sqkz4sog.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fy1sj0dabx5pxpe0c7fu0.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkisnepo5w1ik802tpuqo.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fol3kxq76qxut4anbugrn.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ft67wzcfoyg1eg0yusm8x.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd58xk4wtaecldkdl8ny6.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fldvjuox3yrc4hxi0ql3p.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fjyhn54bk6dkq914dey11.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmu8z9j1hlh0zh5csusmj.png",
      "label": "",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:46Z",
  "source_url": "https://dev.to/mitanshgor/rl-3-bellman-equations-and-markov-decision-processes-1950s-1960s-2pp0",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 22913,
  "expected_body_images": 22,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/77684a950aa4/pages/source.html",
  "local_html": "attachments/77684a950aa4/pages/article.html",
  "body_base_path": "attachments/77684a950aa4/pages/",
  "body_markdown_path": "attachments/77684a950aa4/pages/article.md"
}