{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Mitansh Gor",
  "title_listed": "RL 4: Early heuristics and the birth of Temporal Difference learning (1959–1968)",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/mitanshgor/rl-4-early-heuristics-and-the-birth-of-temporal-difference-learning-1959-1968-1hnb",
  "publish_date": "2026-09-25",
  "corpus_stage": "",
  "notes": "",
  "id": "db13b5457fae",
  "collected_at": "2026-10-02T13:52:25",
  "title": "RL 4: Early heuristics and the birth of Temporal Difference learning (1959–1968) - DEV Community",
  "body_markdown": "TL;DR\nBellman's math told you how to act perfectly — on paper. 1959 computers had neither the memory nor the map of reality to run it. So a few stubborn engineers cheated their way to something that actually worked.\nArthur Samuel (1959)\nswapped an impossible checkers lookup table for a\nscoring function\n— a formula with adjustable weights, trained by playing itself. His real breakthrough wasn't the 1962 headline win (which, spoiler, gets more impressive the more you learn how it actually went). It was an idea called\nTemporal Difference (TD) learning\n: don't wait for the game to end — learn from the gap between what you predicted a moment ago and what you predict now. No rulebook required. That's\nmodel-free\nlearning, and it turns out to be the exact same rule psychologists later used to explain Pavlov's dogs.\nDonald Michie's MENACE (1961)\ndid something similar with 304 matchboxes and coloured beads — but learned a\npolicy\n(\"what should I do?\") instead of a\nvalue\n(\"how good is this?\"). Value-learning and policy-learning are the two ancestors of basically every RL algorithm running today.\nMinsky (1961)\nput a name on why any of this was hard in the first place: the\ncredit-assignment problem.\nMichie's BOXES (1968)\nremoved the last training wheel — a human teacher — and balanced a physical pole using nothing but \"it fell over\" as feedback.\nEvery system in this post assumes the world is reliable. Blog 5 breaks that assumption.\nThe blueprint had a bill attached\nBellman's equation is one of those rare results that's simultaneously beautiful and completely useless.\nBeautiful, because it tells you the\nexact\noptimal move, always. Useless, because to run it you need two things handed to you first: every possible state of the world, written down, and the exact odds of moving between them, also written down.\nNeither exists. Computers in 1959 topped out around 32,768 words of memory — smaller than a single photo on your phone today. Checkers alone has roughly 500 billion possible positions. And the \"odds of every outcome\" problem is even worse than the memory one — more on that in a second.\nSo the people actually trying to build this did what engineers always do when elegant theory meets a real budget: they cheated. One of the most important cheats in AI history started with a man, a room-sized vacuum-tube computer, and a game of checkers.\nThree walls, one problem each\nPicture yourself in 1959, staring at Bellman's equation, actually trying to build the thing. You'd hit these three walls, in roughly this order, and none of them have an easy way around them.\nWall 1 — Memory.\nStore every situation in a table and the table explodes. Tic-tac-toe: ~5,000 positions, fine. Checkers: ~500 billion. Chess: ~10⁴⁵. Bellman called this the\ncurse of dimensionality\n— add one variable, and your table doesn't grow, it detonates. Lesson: you can't memorise your way to intelligence.\nWall 2 — No rulebook.\nBellman's math needs\nP(s'|s,a)\n— the odds of every outcome, for every action, in advance. Nobody has that spreadsheet. A self-driving car has no table for \"will this pedestrian step off the curb.\" Reality just doesn't ship with its own source code.\nWall 3 — The score becomes the goal.\nTurn \"behave well\" into a number, and machines will take that number\nextremely\nliterally. Reward a robot vacuum for movement, and it'll ram the same wall forever, racking up points and cleaning nothing — and from its own perspective, winning. Today we'd call this\nreward hacking\n. In 1959 it didn't have a name yet, just a very confused engineer staring at the output.\nSamuel is about to dent Walls 1 and 2. Michie dents Wall 1 from a totally different angle. Wall 3 is still standing today — it's a large chunk of why \"AI alignment\" is now a job title.\nArthur Samuel and the machine that learned checkers\nIn the 1950s, the prevailing wisdom was blunt: computers execute, they don't learn. A program does exactly what it's told, forever, full stop.\nArthur Samuel, an IBM\nengineer, didn't buy it. He picked checkers — not chess — as his test case. Deep enough to need real strategy, small enough that a 1950s machine stood a chance. He started in 1952 on the IBM 701 and by 1959 had published work that put the phrase \"machine learning\" into circulation for the first time.\nHis machine, the IBM 704, maxed out at 32,768 words of memory. Storing every checkers position wasn't \"hard\" — it was short by seven orders of magnitude. So instead of trying to be\ncomprehensive\n, Samuel made the machine\nopinionated\n.\nA scoring function instead of a lookup table\nThrow away the table. Replace it with a formula that squeezes an entire board into one number: high = \"good for me,\" low = \"bad for me.\" The machine has no idea\nwhy\n— it just needs a number it can push upward.\nIf blog 3's\nV(s)\nwas an exact number, computed by sweeping every state until the answer converged — this is the same object, wearing a disguise. Samuel's scoring function\nis\nV(s)\n. The difference is that nobody can compute it exactly here, so the machine has to guess it, and improve its guess through play instead of through a formula that already has the whole map.\nThat number was built from hand-picked\nfeatures\n— piece count, king count, board control, mobility — each with a\nweight\nsaying how much the machine currently trusts it. Score = features × weights, summed.\nHere's the part that still holds up today: the weights weren't fixed. A feature that kept predicting wins got trusted more. One that kept misleading the machine got trusted less, or dropped entirely and swapped for a new candidate. The machine wasn't just learning checkers — it was learning\nwhat was worth paying attention to\n. That's the same basic instinct behind feature learning in neural networks, six decades early.\nSelf-play\nSamuel then had the program play itself. A \"student\" copy tries tweaked weights against a \"teacher\" copy running the current best settings. Student wins consistently → student becomes the new teacher → repeat.\nThis loop is, structurally, identical to what DeepMind ran in 2016 to build AlphaGo. The hardware went from vacuum tubes to TPU pods. The logic didn't change at all.\nThe famous win (and the asterisk nobody mentions)\nIn 1962, Samuel's program beat a checkers player named Robert Nealey, and IBM's press release turned it into the first-ever headline about a machine beating a human at a strategy game.\nWhat usually gets left out: Nealey wasn't state champion yet at the time (that came four years later), he beat the program in a rematch the following year, and by 1966 Samuel's program lost eight straight games to serious competition. None of that makes the work less real — it just moves the story from \"magic trick\" to \"science\": genuine learning, with a genuine, visible ceiling.\nThe ceiling Samuel could see from day one\nEvery strategic idea the program ever had came from a feature Samuel typed in by hand. If he never thought to encode \"board control matters,\" the machine could never discover it on its own. It could re-weight human knowledge with superhuman patience — but it couldn't generate knowledge that wasn't already sitting in the box.\nThat single limitation defined the next thirty years of AI research, and it's exactly the problem deep learning was eventually built to solve: stop handing the machine features, let it find its own.\nA name for the missing piece:\nMarvin Minsky, in 1961, pointed at exactly this gap and called it the\ncredit-assignment problem\n— with only a final win or loss, which of a hundred earlier decisions actually deserves the credit? He specifically called out Samuel's scoring updates as the first real attempt at an answer: doling out partial credit one move at a time instead of waiting for the final verdict. That's the problem the rest of this post is trying to crack.\nMemorise, or generalise?\nSamuel's scoring function created a fork in the road that reinforcement learning is still walking down today.\nHis first instinct was\nrote learning\n: store exact positions and how they turned out, replay the good ones when they reappear. Works great for small games. But nudge a single piece and the \"memory\" is worthless — that exact position has never happened before, so there's nothing to look up. It's a lookup table wearing a trench coat.\nGeneralisation\nwas the fix: describe a board by its features instead of memorising the whole thing. Now a position the machine has never seen still gets a sensible score, because it's\ndescribed\n, not\nrecognised\n.\nTable lookup asks \"have I seen this exact thing?\" Generalisation asks \"what is this like?\" Only the second question scales.\nSamuel's update rule for the weights:\nΔw_i = α · [V_actual − V_predicted] · f_i\nΔw_i — how much to nudge feature i's weight\nα — learning rate: how boldly to react to new evidence\n[V_actual − V_predicted] — the surprise\nf_i — how present that feature was in this position\nIf you remember one sentence from this whole post, make it this:\nlearning is the size of your mistake, multiplied by how willing you are to change your mind.\nEverything from here on is a variation on that one bracket.\nThinking deeper on a slower machine: alpha-beta pruning\nOne quick detour, and it earns its place here: self-play only works if you can play thousands of games against yourself, and thousands of games only fits into a reasonable afternoon if each individual game doesn't take forever to think through. That's the problem this section solves.\nThe IBM 704 was unbelievably slow by today's standards — looking twenty moves ahead by brute force would've taken longer than the universe has existed. So Samuel needed to skip most of the game tree without missing anything that mattered.\nCredit where due: Samuel used\nalpha-beta pruning\n, but didn't invent it. John McCarthy proposed it at the 1956 Dartmouth workshop; Hart and Edwards formalised it in 1961; Knuth and Moore gave it full analysis in 1975. Samuel had a working version quietly running in the background — he just thought the learning parts were the interesting bit. (He was right.)\nThe idea: as the machine explores possible futures, it tracks the best result it can already guarantee (\nalpha\n) and the worst its opponent can already force (\nbeta\n). The moment a branch is provably worse than something already locked in, the machine stops exploring it entirely — not \"reads it faster,\"\nstops\n.\nRestaurant analogy: someone tells you \"the other place has a two-hour wait.\" You don't need the menu or the parking situation. You're not going. Branch closed.\nCrucially, this is\nlossless\n— it finds the exact same best move as checking everything, just without wasting time on futures that can't matter. It's also a technique with real staying power: it was the backbone of Deep Blue, the chess machine that beat world champion Garry Kasparov in 1997, decades after Samuel quietly used it for checkers.\npruning comparision - Google Drive\ndrive.google.com\nThe idea that mattered most: learning from your own changing mind\nThis is the part that echoes into every modern RL system, and it's easy to undersell because it sounds so small.\nThe old assumption\nThe 1950s default: learning happens at the\nend\n. Play the whole game, see who won, then work out what went wrong. This became known as\nMonte Carlo learning\n— and it's brutally slow. You need a full game before learning anything, and if the game runs 100 moves, one noisy final signal has to somehow get distributed across all 100 decisions.\nSamuel's question: why wait?\nThe machine already had a second opinion available, one move later, for free. If a position looked 70% winning after move 10, and only 50% winning after move 11 — something was already learned, right there, with no need to finish the game. Your own estimate moved, and estimates don't move for no reason.\nReal-life version: you leave home expecting a 20-minute drive. Ten minutes in, you hit an empty motorway and revise to \"7 more minutes.\" You haven't arrived yet, you don't know the true total — and you've unmistakably learned something the moment your prediction jumped.\nThat's the whole trick: learn from the\ngap between consecutive predictions\n, not from the final answer. It sidesteps Minsky's credit-assignment problem rather than solving it head-on — each move only ever has to explain the\nnext\nmove, not the whole game.\nThe formula\nFor a game like checkers, where nothing scores until someone wins:\nV(S_t) ← V(S_t) + α [ V(S_t+1) − V(S_t) ]\nV(S_t) — current guess about this position\nV(S_t+1) — the guess one move later\n[V(S_t+1) − V(S_t)] — the\nTD error\n: how wrong I just turned out to be\nα — learning rate: too high and it's jittery, too low and it barely moves\nIn plain English: nudge what you believed toward what you now believe, in proportion to how surprised you are.\nThe general version below isn't a new idea — it's the exact same bracket with two extra pieces bolted on, so you can see them coming rather than guessing where they came from. It adds in any reward collected along the way, plus a discount for how much future value matters:\nV(S_t) ← V(S_t) + α [ r_t+1 + γ V(S_t+1) − V(S_t) ]\nCheckers collapses back to the simple version because nothing is scored mid-game (r = 0) and a win in 10 moves is worth exactly as much as a win now (γ = 1).\nThe déjà vu moment\nThat bracket should look familiar if animal learning is on your radar at all — and if you've been following this series, it should ring a very specific bell. This is the\nprediction error\nterm flagged back in Blog 3, finally getting the job it was promised. Same idea, same shape, just a new name and a new job description: back then it fired once per trial, at the end. Here, it fires on every single move.\nPavlov's dogs weren't learning from bells — they were learning from\nbeing wrong about bells\n. A decade after Samuel, psychologists Rescorla and Wagner wrote down almost the identical rule to explain it: change your belief in proportion to your error. Different letters, same bracket — reality minus expectation, scaled by how much you're willing to update.\nAn IBM engineer trying to save vacuum-tube cycles, and two psychologists trying to explain salivating dogs, landed on the same equation from completely opposite ends of science, with zero contact between them. That's not a coincidence — it's a strong hint that prediction error is something close to fundamental about what learning actually\nis\n. (It gets better: in the 1990s, Wolfram Schultz found dopamine neurons firing in a pattern that looks almost exactly like a TD error signal. More on that in a later post.)\nAnd notice what TD learning never needed: the rulebook Wall 2 said was impossible to get. Just the current state, the next state, and two estimates. That's\nmodel-free\nreinforcement learning — arguably the single most important idea in this post — quietly running on a vacuum-tube computer in 1959, almost thirty years before Richard Sutton formalised it as TD(λ) in 1988.\nMENACE: reinforcement learning built out of matchboxes\nSamuel had IBM's hardware. British researcher Donald Michie had a rather different constraint: no computer access at all. So in 1961, he built one out of matchboxes.\nMENACE\n— the Matchbox Educable Noughts And Crosses Engine — used 304 matchboxes, each labelled with a tic-tac-toe position MENACE might face, and filled with coloured beads. Each colour maps to one legal move.\nTo move:\nfind the matching box, shake it, pull a bead blind. Its colour is the move. Leave the box open — you'll need the trail later.\nTo learn:\nafter the game, walk back along the trail of open boxes. Win → return the bead played, plus three more of the same colour. Draw → add just one. Loss → the bead simply isn't returned; it's gone. More beads of a colour means a better chance of drawing it next time.\nMichie played roughly 220 games against it over a weekend. It went from flailing randomly to essentially unbeatable.\nThe detail worth noticing\nMichie didn't load every box with the same number of beads. Early-game boxes started with about four beads per move; late-game boxes tapered down toward one. That's a deliberately engineered\nexploration schedule\n. Early moves are far from the outcome, so any one game barely tells you if they were good — you want MENACE to keep trying alternatives. Late moves sit right next to the result, so their feedback is trustworthy and can be locked in fast. That's the\nexplore vs. exploit\ntrade-off, solved in 1961 with nothing but bead-counting.\nWhy this is a plot point, not a party trick\nSamuel's program learned to answer \"how good is this position?\" MENACE never asked that question — it has no concept of a position being worth 0.7. It only tracks \"which move should I make more often from here?\"\nIn blog 3's terms: MENACE is growing a\npolicy\n—\nπ\n, a state-to-action lookup — directly. It never computes a value for anything along the way. That's the split worth holding onto:\nSamuel\nMichie\nLearns\nValues\n— how good is this state?\nA policy\n— what should I do here?\nStores\nFeature weights\nAction probabilities\nModern descendant\nValue-based methods — learn a score for each move, then just pick the best one (Q-learning, and its neural-network upgrade DQN)\nPolicy-gradient methods — skip the scoring step, learn the action probabilities directly (REINFORCE, and its more stable modern version PPO)\nTwo men, two continents, same decade — and they'd independently split reinforcement learning into the two halves it still has today. Modern RL doesn't pick a winner; it runs both at once.\nActor-Critic\nmethods (more on these in a later post) are literally a Critic estimating values (Samuel's job) paired with an Actor adjusting action probabilities (Michie's job), each sharpening the other.\nMENACE also proves something worth sitting with: reinforcement learning isn't a property of silicon. It's a mathematical principle. Run it on vacuum tubes, neurons, or 304 matchboxes on a Sunday afternoon — what matters is the loop: act, observe the consequence, adjust, repeat.\nIt also hit the exact same wall Samuel did: every position needs its own box, and one extra square collapses the shelf. Same curse, cardboard edition.\nMichie didn't stop there.\nIn 1968, with R. A. Chambers, he took the matchbox idea into physics:\nBOXES\n, balancing a pole on a moving cart using nothing but a bang when it fell over. Widrow and Smith had balanced the same pole in 1964 — but with a human demonstrating the right move at each step. BOXES threw the teacher out entirely: no demonstration, just eventual failure walked backward through whichever boxes were open. The one new trick it needed was\nstate aggregation\n— chopping a continuous space (pole angle, cart speed) into discrete chunks, since you can't label a matchbox for infinitely many positions. That's the field's first honest step toward\nfunction approximation\n— a plain-English way of saying \"estimate the value instead of memorising it,\" the same escape hatch this whole post keeps circling back to. Cart-pole balancing went on to become RL's most reused benchmark ever since.\nTwo roads, one open question\nBy the late 1960s, two things were true at once. Samuel and Michie both had systems that genuinely learned from experience — no rulebook, no complete map of the world, and they visibly got better over time. But nobody could prove any of it. No convergence guarantee, no theorem saying \"this reaches the optimal policy\" — just clever hacks that happened to hold up on small problems.\nBellman's framework had the opposite problem: flawless proofs, unusable in practice.\nWhich set up the question the field spent the next twenty years chasing:\nCan you keep Bellman's guarantees without needing Bellman's rulebook?\nThat question is the birth of model-free reinforcement learning — and chasing it eventually pushes the field toward systems that learn their own features instead of borrowing a human's. Which is to say: toward neural networks.\nWorth sitting with for a second before we move on: the technique that trains most modern chatbots to be helpful — RLHF, reinforcement learning from human feedback — is a direct descendant of exactly this idea. No rulebook for \"what makes a good response,\" just a reward signal and a model updating itself against its own predictions. The line from a matchbox in 1961 to a language model in 2026 runs straight through this post.\nBut not yet. First, someone has to ask a harder version of the question.\nComing up next\nSamuel and Michie both had one thing quietly working in their favour: their worlds behaved. Move a checker, it stays moved. Draw a bead, that's the move you play. The environment was unknown, but at least it was\nreliable\n.\nNow break that assumption. What if pressing the same button in the same situation works 70% of the time and fails the other 30%, for no reason you can see? A single result stops meaning anything — it might just be noise. You need to learn from distributions, not events, while still acting sensibly under real uncertainty.\nThat discipline was being built in parallel, largely by control theorists who weren't talking to the game-playing crowd at all.\nNext up — RL 5: Learning Automata and Stochastic Environments (1961–1974).\nHow do you make good decisions when the world itself won't give you a straight answer?\nSeries index:\nBlog 1 — Biological foundations and the Law of Effect\nBlog 2 — Classical conditioning and prediction\nBlog 3 — Bellman, MDPs and Dynamic Programming\nBlog 4 — Early heuristics and the birth of TD learning (this post)",
  "metadata": {
    "title": "RL 4: Early heuristics and the birth of Temporal Difference learning (1959–1968) - DEV Community",
    "description": "TL;DR   Bellman's math told you how to act perfectly — on paper. 1959 computers had neither... Tagged with rl, ai, machinelearning, deeplearning.",
    "keywords": "rl, ai, machinelearning, deeplearning, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/mitanshgor/rl-4-early-heuristics-and-the-birth-of-temporal-difference-learning-1959-1968-1hnb",
    "og:title": "RL 4: Early heuristics and the birth of Temporal Difference learning (1959–1968)",
    "og:description": "TL;DR   Bellman's math told you how to act perfectly — on paper. 1959 computers had neither...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fox23yf77pkqvvjkjr4to.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "RL 4: Early heuristics and the birth of Temporal Difference learning (1959–1968)",
    "twitter:description": "TL;DR   Bellman's math told you how to act perfectly — on paper. 1959 computers had neither...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fox23yf77pkqvvjkjr4to.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq3ewlex9fi8murgw8zlb.png",
      "bytes": 118936,
      "reused": true,
      "local_path": "attachments/db13b5457fae/images/001.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq3ewlex9fi8murgw8zlb.png",
      "alt": "Memory size-comparison graphic",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqwsv2zgjyie607ag8z1m.png",
      "bytes": 179938,
      "reused": false,
      "local_path": "attachments/db13b5457fae/images/body-31eae778b861507bf8e9.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqwsv2zgjyie607ag8z1m.png",
      "alt": "one clean 3-icon graphic",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzoi4m3bahw70ytzuvsym.png",
      "bytes": 168408,
      "reused": true,
      "local_path": "attachments/db13b5457fae/images/002.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzoi4m3bahw70ytzuvsym.png",
      "alt": "Arthur Samuel",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fujvxs9dmpdyswruhyapj.png",
      "bytes": 81132,
      "reused": true,
      "local_path": "attachments/db13b5457fae/images/003.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fujvxs9dmpdyswruhyapj.png",
      "alt": "Self-play",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fz5ir2p8l9qu2x9kiq6de.png",
      "bytes": 182696,
      "reused": true,
      "local_path": "attachments/db13b5457fae/images/004.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fz5ir2p8l9qu2x9kiq6de.png",
      "alt": "Memorise, or generalise",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fssl.gstatic.com%2Fdocs%2Fdoclist%2Fimages%2Fdrive_favicon_2026_32dp.png",
      "bytes": 513,
      "reused": false,
      "local_path": "attachments/db13b5457fae/images/body-6195183832963f067bb8.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fssl.gstatic.com%2Fdocs%2Fdoclist%2Fimages%2Fdrive_favicon_2026_32dp.png",
      "alt": "favicon",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft4xnmc7j8ju2h73c65rm.png",
      "bytes": 110174,
      "reused": true,
      "local_path": "attachments/db13b5457fae/images/005.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft4xnmc7j8ju2h73c65rm.png",
      "alt": "old assumption",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkl8ofqe1zuuxx7i43yj9.png",
      "bytes": 103494,
      "reused": true,
      "local_path": "attachments/db13b5457fae/images/006.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkl8ofqe1zuuxx7i43yj9.png",
      "alt": "TD improments",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F96twyh8rv4xoua2ntler.png",
      "bytes": 131390,
      "reused": true,
      "local_path": "attachments/db13b5457fae/images/007.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F96twyh8rv4xoua2ntler.png",
      "alt": "Monte Carlo VS TD",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxgx4blsdq6yrfszi4xw8.png",
      "bytes": 167784,
      "reused": true,
      "local_path": "attachments/db13b5457fae/images/008.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxgx4blsdq6yrfszi4xw8.png",
      "alt": "MENACE matchbox setup",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyywrw5nv22v0bsxd3cbp.png",
      "bytes": 27490,
      "reused": true,
      "local_path": "attachments/db13b5457fae/images/009.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyywrw5nv22v0bsxd3cbp.png",
      "alt": "modelfree vs modelbased",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq3ewlex9fi8murgw8zlb.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqwsv2zgjyie607ag8z1m.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzoi4m3bahw70ytzuvsym.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.ibm.com/history/early-games",
      "label": "Arthur Samuel, an IBM",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fujvxs9dmpdyswruhyapj.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fz5ir2p8l9qu2x9kiq6de.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://drive.google.com/file/d/1wk6siVlagwB9-13cf-0dbWVwrvXabgeV/view?usp=sharing",
      "label": "pruning comparision - Google Drive",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft4xnmc7j8ju2h73c65rm.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkl8ofqe1zuuxx7i43yj9.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F96twyh8rv4xoua2ntler.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxgx4blsdq6yrfszi4xw8.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyywrw5nv22v0bsxd3cbp.png",
      "label": "",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:46Z",
  "source_url": "https://dev.to/mitanshgor/rl-4-early-heuristics-and-the-birth-of-temporal-difference-learning-1959-1968-1hnb",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 21466,
  "expected_body_images": 11,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/db13b5457fae/pages/source.html",
  "local_html": "attachments/db13b5457fae/pages/article.html",
  "body_base_path": "attachments/db13b5457fae/pages/",
  "body_markdown_path": "attachments/db13b5457fae/pages/article.md"
}