In the previous article, we built the intuition behind Q-Learning. We learned that an agent improves its decisions through trial and error, stores its experience in a Q-table, and gradually discovers which actions lead to better outcomes.
However, one important question was intentionally left unanswered:
How does the agent actually update the values stored in its Q-table?
Imagine you're lost in a desert with no map and no prior knowledge of the area. Your goal is to find water. Every decision you make teaches you something new. If moving east repeatedly leads to dead ends, you'll become less confident about choosing that direction again. If moving north eventually brings you closer to water, you'll start trusting that choice more.
A reinforcement learning agent learns in exactly the same way. After every action, it receives feedback from the environment and uses that experience to improve its future decisions.
But how does it perform this update mathematically?
In this article, we'll answer that question by exploring the Q-Learning update equation. We'll build a tiny example, understand what every part of the equation means, and manually update a Q-value step by step. By the end, you'll understand how an agent transforms experience into knowledge
Before looking at the mathematics, let's create a simple environment where our agent can learn. We'll use a 3 × 3 grid world. The agent starts in the top-left corner and must reach the goal in the bottom-right corner while avoiding an obstacle.
+-------+-------+-------+
| START | | |
+-------+-------+-------+
| | X | |
+-------+-------+-------+
| | | GOAL |
+-------+-------+-------+
The agent can perform four actions from any valid position, including Up, Down, Left, and Right. Every action results in a reward from the environment.
| Situation | Reward |
|---|---|
| Reach the goal | +10 |
| Hit the obstacle | -10 |
| Normal move | -1 |
Notice that even a normal move receives a small negative reward. This encourages the agent to find the shortest path instead of wandering around the environment forever.
At the beginning, the agent has no idea where the goal is or which path is best. Every direction looks equally good because it hasn't gathered any experience yet. So where does it store everything it learns? That's the role of the Q-table.
Think of the Q-table as the agent's memory. It stores how valuable each action appears to be in every state based on the experiences the agent has collected so far. Since our grid world contains 9 states and the agent has 4 possible actions, the Q-table contains one value for every state-action pair.
When training begins, the agent knows absolutely nothing about the environment. It hasn't explored any paths or received any rewards, so every value in the Q-table is initialized to 0.
| State | Up | Down | Left | Right |
|---|---|---|---|---|
| S₀ | 0 | 0 | 0 | 0 |
| S₁ | 0 | 0 | 0 | 0 |
| S₂ | 0 | 0 | 0 | 0 |
| ... | ... | ... | ... | ... |
| S₈ | 0 | 0 | 0 | 0 |
For example, if the agent is in State S₀, all four actions currently have the same value. This doesn't mean every action is equally good—it simply means the agent hasn't learned enough to distinguish between them.
As the agent explores the environment, these values begin to change. Actions that consistently lead to better outcomes receive higher Q-values, while actions that result in poor outcomes receive lower ones. Over time, the Q-table transforms from a collection of zeros into the agent's learned knowledge about the environment.
This naturally leads to the most important question:
How does a Q-value change after every experience?
The answer lies in the Q-Learning update equation.
Now we arrive at the heart of the Q-Learning algorithm. Every time the agent takes an action, it learns something new. Instead of throwing away its previous knowledge, it combines what it already knows with what it just experienced to make a better decision next time. This learning process is represented by the following equation:
At first glance, this equation might look intimidating, especially if you're new to reinforcement learning. The equation is simply answering one question:
"After taking an action, how should the agent update its current knowledge?"
Every time the agent interacts with the environment, it:
Receives feedback.
Looks at the new state it reached.
Updates its previous estimate based on this new experience.
That's exactly what this equation does.
Instead of trying to understand the entire formula at once, let's break it into smaller pieces.
Rather than focusing on mathematical notation, let's understand what each part represents. Suppose our agent is currently in State S₀ and decides to move Right.
This is the value the agent currently has for taking action a in state s. Think of it as the agent's current opinion. For example:
Q(S₀,Right)=2
This means the agent currently believes that moving Right from State S₀ has a value of 2.
After taking the action, the environment immediately gives feedback. This feedback is called the reward.
For example:
Reach the goal → +10
Hit the obstacle → −10
Normal move → −1
The reward tells the agent whether the action it just took was good or bad.
After moving, the agent reaches a new state. Now it asks a simple question:
"From here, what is the best action I can take next?" . Instead of considering every possible future, Q-Learning simply looks at the highest Q-value in the new state. This allows the agent to think beyond the immediate reward and consider future opportunities.
The learning rate determines how much the agent should trust its new experience. A higher value means the agent learns quickly. A lower value means it updates its knowledge more gradually. Think of it as:
"How much should this new experience change what I already know?"
The discount factor determines how much importance the agent gives to future rewards. A higher value encourages the agent to think long-term, while a lower value makes it focus more on immediate rewards. Think of it as:
"How much do I care about rewards that I'll receive later?"
That's it.
Those five ideas are all you need to understand the Q-Learning equation. Once you know what each component represents, the formula becomes much less intimidating. Now let's see how these pieces work together by updating a Q-value step by step.
Let's return to our grid world. Suppose the agent is in State S₀ and decides to move right. Before taking the action, the agent has the following information:
Current Q-value = 2
Reward (r) = −1
Learning Rate (α) = 0.5
Discount Factor (γ) = 0.9
Best future Q-value = 8
Now we'll substitute these values into the Q-Learning update equation, and after substituting, it becomes
Q(S₀,Right)=4.1
What does this result mean?
Before taking the action, the agent believed that moving Right was worth 2. After receiving a small negative reward but discovering that the next state offers a promising future, it updates its estimate to 4.1. The agent hasn't memorized the correct path. Instead, it has slightly improved its understanding based on one new experience.
By repeating this process over hundreds or thousands of interactions, the Q-table gradually becomes more accurate, allowing the agent to make increasingly better decisions.
Now that we understand how the Q-Learning update equation works, it's time to put it into practice.
The equation itself is only one step of the learning process. During training, the agent repeatedly interacts with the environment, receives rewards, updates its Q-values using the same equation, and gradually improves its decision-making.
Rather than adding a lengthy implementation to this article, I've prepared a separate Google Colab notebook where we'll build everything from scratch.
In the notebook, we'll:
Build our own 3 × 3 Grid World.
Create and initialize the Q-table.
Implement the Q-Learning algorithm step by step.
Train the agent over multiple episodes.
Visualize the learned Q-table and the final path chosen by the agent.
By separating the implementation from the theory, you can first understand why Q-Learning works and then focus on how to implement it without feeling overwhelmed.
📓 Google Colab Notebook: Building our First Q-Learning Agent from Scratch
In this article, we explored how a Q-Learning agent updates its knowledge after every interaction with the environment. By understanding the Q-table, the update equation, and a simple worked example, we've seen how an agent gradually learns better decisions through experience.
Let's follow us in our series on Reinforcement Learning.