{ "source_file": "0930", "source_platform": "Hashnode", "author": "Muhammad Fahad Bashir", "title_listed": "Q-Learning in Practice: Understanding the Update Equation with One Simple Example", "keywords": "", "domain": "机器学习", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://ml-diaries-by-fahd.hashnode.dev/q-learning-in-practice-understanding-the-update-equation-with-one-simple-example", "publish_date": "2026-07-22", "corpus_stage": "", "notes": "", "id": "b44bd577d7bd", "collected_at": "2026-10-02T14:10:52", "title": "Q-Learning in Practice: Understanding the Update Equation with One Simple Example", "body_markdown": "Q-Learning in Practice: Understanding the Update Equation with One Simple Example\nUpdated\nJuly 22, 2026\n•\n8\nmin read\n•\nView as Markdown\nM\nMuhammad Fahad Bashir\nMachine Learning, Artificial Intelligence\nPart of series\nReinforcement Learning\nIn the\nprevious article\n, we built the intuition behind\nQ-Learning\n. We learned that an agent improves its decisions through trial and error, stores its experience in a\nQ-table\n, and gradually discovers which actions lead to better outcomes.\nHowever, one important question was intentionally left unanswered:\nHow does the agent actually update the values stored in its Q-table?\nImagine you're lost in a desert with no map and no prior knowledge of the area. Your goal is to find water. Every decision you make teaches you something new. If moving east repeatedly leads to dead ends, you'll become less confident about choosing that direction again. If moving north eventually brings you closer to water, you'll start trusting that choice more.\nA reinforcement learning agent learns in exactly the same way. After every action, it receives feedback from the environment and uses that experience to improve its future decisions.\nBut how does it perform this update mathematically?\nIn this article, we'll answer that question by exploring the\nQ-Learning update equation\n. We'll build a tiny example, understand what every part of the equation means, and manually update a Q-value step by step. By the end, you'll understand how an agent transforms experience into knowledge\nBuilding a Tiny Grid World\nBefore looking at the mathematics, let's create a simple environment where our agent can learn. We'll use a\n3 × 3 grid world\n. The agent starts in the top-left corner and must reach the goal in the bottom-right corner while avoiding an obstacle.\n+-------+-------+-------+\n| START | | |\n+-------+-------+-------+\n| | X | |\n+-------+-------+-------+\n| | | GOAL |\n+-------+-------+-------+\nThe agent can perform four actions from any valid position, including Up, Down, Left, and Right. Every action results in a reward from the environment.\nSituation\nReward\nReach the goal\n+10\nHit the obstacle\n-10\nNormal move\n-1\nNotice that even a normal move receives a small negative reward. This encourages the agent to find the\nshortest path\ninstead of wandering around the environment forever.\nAt the beginning, the agent has no idea where the goal is or which path is best. Every direction looks equally good because it hasn't gathered any experience yet. So where does it store everything it learns? That's the role of the\nQ-table\n.\nThe Q-Table: The Agent's Memory\nThink of the\nQ-table\nas the agent's memory. It stores how valuable each action appears to be in every state based on the experiences the agent has collected so far. Since our grid world contains\n9 states\nand the agent has\n4 possible actions\n, the Q-table contains one value for every\nstate-action pair\n.\nWhen training begins, the agent knows absolutely nothing about the environment. It hasn't explored any paths or received any rewards, so every value in the Q-table is initialized to\n0\n.\nState\nUp\nDown\nLeft\nRight\nS₀\n0\n0\n0\n0\nS₁\n0\n0\n0\n0\nS₂\n0\n0\n0\n0\n...\n...\n...\n...\n...\nS₈\n0\n0\n0\n0\nFor example, if the agent is in\nState S₀\n, all four actions currently have the same value. This doesn't mean every action is equally good—it simply means the agent hasn't learned enough to distinguish between them.\nAs the agent explores the environment, these values begin to change. Actions that consistently lead to better outcomes receive higher Q-values, while actions that result in poor outcomes receive lower ones. Over time, the Q-table transforms from a collection of zeros into the agent's learned knowledge about the environment.\nThis naturally leads to the most important question:\nHow does a Q-value change after every experience?\nThe answer lies in the\nQ-Learning update equation\n.\nThe Q-Learning Update Equation\nNow we arrive at the heart of the Q-Learning algorithm. Every time the agent takes an action, it learns something new. Instead of throwing away its previous knowledge, it combines what it already knows with what it just experienced to make a better decision next time. This learning process is represented by the following equation:\nAt first glance, this equation might look intimidating, especially if you're new to reinforcement learning. The equation is simply answering one question:\n\"\nAfter taking an action, how should the agent update its current knowledge\n?\"\nEvery time the agent interacts with the environment, it:\nReceives feedback\n.\nLooks at the new state it reached.\nUpdates its previous estimate based on this new experience.\nThat's exactly what this equation does.\nInstead of trying to understand the entire formula at once, let's break it into smaller pieces.\nUnderstanding the Equation\nRather than focusing on mathematical notation, let's understand what each part represents. Suppose our agent is currently in State S₀ and decides to move Right.\nCurrent Knowledge — Q(s,a)\nThis is the value the agent currently has for taking action a in state s. Think of it as the agent's current opinion. For example:\nQ(S₀,Right)=2\nThis means the agent currently believes that moving Right from State S₀ has a value of 2.\nImmediate Reward — r\nAfter taking the action, the environment immediately gives feedback. This feedback is called the reward.\nFor example:\nReach the goal → +10\nHit the obstacle → −10\nNormal move → −1\nThe reward tells the agent whether the action it just took was good or bad.\nBest Future Opportunity — maxQ(s ′ ,a ′ )\nAfter moving, the agent reaches a new state. Now it asks a simple question:\n\"From here, what is the best action I can take next?\" .\nInstead of considering every possible future, Q-Learning simply looks at the highest Q-value in the new state. This allows the agent to think beyond the immediate reward and consider future opportunities.\nLearning Rate — α\nThe learning rate determines how much the agent should trust its new experience. A higher value means the agent learns quickly. A lower value means it updates its knowledge more gradually. Think of it as:\n\"\nHow much should this new experience change what I already know?\n\"\nDiscount Factor — γ\nThe discount factor determines how much importance the agent gives to future rewards. A higher value encourages the agent to think long-term, while a lower value makes it focus more on immediate rewards. Think of it as:\n\"How much do I care about rewards that I'll receive later?\"\nThat's it.\nThose five ideas are all you need to understand the Q-Learning equation. Once you know what each component represents, the formula becomes much less intimidating. Now let's see how these pieces work together by updating a Q-value step by step.\nA Worked Example\nLet's return to our grid world. Suppose the agent is in State S₀ and decides to move right. Before taking the action, the agent has the following information:\nCurrent Q-value = 2\nReward (r) = −1\nLearning Rate (α) = 0.5\nDiscount Factor (γ) = 0.9\nBest future Q-value = 8\nNow we'll substitute these values into the Q-Learning update equation, and after substituting, it becomes\nQ(S₀​,Right)=4.1 ​\nWhat does this result mean?\nBefore taking the action, the agent believed that moving Right was worth 2. After receiving a small negative reward but discovering that the next state offers a promising future, it updates its estimate to 4.1. The agent hasn't memorized the correct path. Instead, it has slightly improved its understanding based on one new experience.\nBy repeating this process over hundreds or thousands of interactions, the Q-table gradually becomes more accurate, allowing the agent to make increasingly better decisions.\nFrom Theory to Practice\nNow that we understand how the Q-Learning update equation works, it's time to put it into practice.\nThe equation itself is only one step of the learning process. During training, the agent repeatedly interacts with the environment, receives rewards, updates its Q-values using the same equation, and gradually improves its decision-making.\nRather than adding a lengthy implementation to this article, I've prepared a\nseparate Google Colab notebook\nwhere we'll build everything from scratch.\nIn the notebook, we'll:\nBuild our own\n3 × 3 Grid World\n.\nCreate and initialize the\nQ-table\n.\nImplement the\nQ-Learning algorithm\nstep by step.\nTrain the agent over multiple episodes.\nVisualize the learned Q-table and the final path chosen by the agent.\nBy separating the implementation from the theory, you can first understand\nwhy\nQ-Learning works and then focus on\nhow\nto implement it without feeling overwhelmed.\n📓\nGoogle Colab Notebook:\nBuilding our First Q-Learning Agent from Scratch\nConclusion\nIn this article, we explored how a Q-Learning agent updates its knowledge after every interaction with the environment. By understanding the Q-table, the update equation, and a simple worked example, we've seen how an agent gradually learns better decisions through experience.\nLet's follow us in our series on\nReinforcement Learning.\n#\nreinforcement-learning\n#\nmachine-learning\n#\nq-learning\n#\ndeep-learning\n10\nviews", "metadata": { "title": "Q-Learning in Practice: Understanding the Update Equation with One Simple Example", "description": "In the previous article, we built the intuition behind Q-Learning. We learned that an agent improves its decisions through trial and error, stores its experience in a Q-table, and gradually discovers", "keywords": null, "author": null, "og:title": "Q-Learning in Practice: Understanding the Update Equation with One Simple Example", "og:description": "In the previous article, we built the intuition behind Q-Learning. We learned that an agent improves its decisions through trial and error, stores its experience in a Q-table, and gradually discovers", "og:url": "https://ml-diaries-by-fahd.hashnode.dev/q-learning-in-practice-understanding-the-update-equation-with-one-simple-example", "og:site_name": "ML Diaries by Fahd", "og:image": "https://cdn.hashnode.com/uploads/covers/66a62c26199a848f896b087f/7e9f46cd-98e0-4040-8ab7-3b2466b40a42.png", "og:image:width": "1200", "og:image:height": "630", "og:type": "article", "twitter:card": "summary_large_image", "twitter:title": "Q-Learning in Practice: Understanding the Update Equation with One Simple Example", "twitter:description": "In the previous article, we built the intuition behind Q-Learning. We learned that an agent improves its decisions through trial and error, stores its experience in a Q-table, and gradually discovers", "twitter:image": "https://cdn.hashnode.com/uploads/covers/66a62c26199a848f896b087f/7e9f46cd-98e0-4040-8ab7-3b2466b40a42.png", "article:published_time": "2026-07-22T08:56:20.733Z", "article:modified_time": "2026-07-22T08:56:20.733Z", "article:author": "https://ml-diaries-by-fahd.hashnode.dev/@mfahadbashir" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/b44bd577d7bd/code/body-000.txt", "chars": 182, "language": "text" } ], "images": [ { "status": "ok", "url": "https://cdn.hashnode.com/uploads/covers/66a62c26199a848f896b087f/7e9f46cd-98e0-4040-8ab7-3b2466b40a42.png", "bytes": 688682, "reused": true, "local_path": "attachments/b44bd577d7bd/images/002.png", "src": "https://cdn.hashnode.com/uploads/covers/66a62c26199a848f896b087f/7e9f46cd-98e0-4040-8ab7-3b2466b40a42.png", "alt": "Q-Learning in Practice: Understanding the Update Equation with One Simple Example", "downloaded": true }, { "status": "ok", "url": "https://cdn.hashnode.com/uploads/covers/66a62c26199a848f896b087f/af3751dd-6ec9-4f37-a136-6a160151e911.png", "bytes": 960245, "reused": true, "local_path": "attachments/b44bd577d7bd/images/004.png", "src": "https://cdn.hashnode.com/uploads/covers/66a62c26199a848f896b087f/af3751dd-6ec9-4f37-a136-6a160151e911.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn.hashnode.com/uploads/covers/66a62c26199a848f896b087f/dc39665e-8381-44c1-96e7-b1fa4356cf72.png", "bytes": 931333, "reused": true, "local_path": "attachments/b44bd577d7bd/images/005.png", "src": "https://cdn.hashnode.com/uploads/covers/66a62c26199a848f896b087f/dc39665e-8381-44c1-96e7-b1fa4356cf72.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn.hashnode.com/uploads/covers/66a62c26199a848f896b087f/34797511-1344-429e-b0ad-594e7da6c612.png", "bytes": 3621, "reused": true, "local_path": "attachments/b44bd577d7bd/images/006.png", "src": "https://cdn.hashnode.com/uploads/covers/66a62c26199a848f896b087f/34797511-1344-429e-b0ad-594e7da6c612.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn.hashnode.com/uploads/covers/66a62c26199a848f896b087f/2ca2f02a-fe1c-4234-b433-d3d6b80a97b2.png", "bytes": 1095819, "reused": true, "local_path": "attachments/b44bd577d7bd/images/007.png", "src": "https://cdn.hashnode.com/uploads/covers/66a62c26199a848f896b087f/2ca2f02a-fe1c-4234-b433-d3d6b80a97b2.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://cdn.hashnode.com/uploads/covers/66a62c26199a848f896b087f/e3f4f780-0a3a-440b-8480-93e9f8703436.png", "bytes": 2391, "reused": false, "local_path": "attachments/b44bd577d7bd/images/body-f0975d3fdfe58df78999.png", "src": "https://cdn.hashnode.com/uploads/covers/66a62c26199a848f896b087f/e3f4f780-0a3a-440b-8480-93e9f8703436.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://ml-diaries-by-fahd.hashnode.dev/q-learning-in-practice-understanding-the-update-equation-with-one-simple-example.md", "label": "View as Markdown", "type": "a" }, { "href": "https://hashnode.com/@mfahadbashir", "label": "M", "type": "a" }, { "href": "https://www.linkedin.com/in/muhammad-fahad-bashir-09568216a/", "label": "", "type": "a" }, { "href": "https://ml-diaries-by-fahd.hashnode.dev/series/reinforcementlearning", "label": "Part of series Reinforcement Learning", "type": "a" }, { "href": "https://ml-diaries-by-fahd.hashnode.dev/q-learning-how-an-agent-learns-from-rewards-intuition-first-no-math", "label": "previous article", "type": "a" }, { "href": "https://colab.research.google.com/drive/1okILDeA535-neYXiOTsIP7F-izT2HZF1?usp=sharing", "label": "Google Colab Notebook: Building our First Q-Learning Agent from Scratch", "type": "a" }, { "href": "https://ml-diaries-by-fahd.hashnode.dev/tag/reinforcement-learning", "label": "# reinforcement-learning", "type": "a" }, { "href": "https://ml-diaries-by-fahd.hashnode.dev/tag/machine-learning", "label": "# machine-learning", "type": "a" }, { "href": "https://ml-diaries-by-fahd.hashnode.dev/tag/q-learning", "label": "# q-learning", "type": "a" }, { "href": "https://ml-diaries-by-fahd.hashnode.dev/tag/deep-learning", "label": "# deep-learning", "type": "a" } ], "retry_at": "2026-10-03T08:06:52", "repair_version": 2, "repaired_at": "2026-10-14T10:50:40Z", "source_url": "https://ml-diaries-by-fahd.hashnode.dev/q-learning-in-practice-understanding-the-update-equation-with-one-simple-example", "papers": [], "actual_video_files": [], "body_chars": 9145, "expected_body_images": 6, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/b44bd577d7bd/pages/source.html", "local_html": "attachments/b44bd577d7bd/pages/article.html", "body_base_path": "attachments/b44bd577d7bd/pages/", "body_markdown_path": "attachments/b44bd577d7bd/pages/article.md" }