Q-Learning: How an Agent Learns From Rewards — Intuition First, No Math
Updated
July 6, 2026
•
10
min read
•
View as Markdown
M
Muhammad Fahad Bashir
Machine Learning, Artificial Intelligence
Part of series
Reinforcement Learning
In the previous article of this series,
Reinforcement Learning: The Big Picture
, we explored how reinforcement learning differs from supervised and unsupervised learning. We discussed the interaction between an agent and its environment and introduced important concepts such as states, actions, rewards, policies, and episodes.
However, one important question was left unanswered:
How does an agent actually learn which action is good?
Suppose an agent is placed in a completely unfamiliar environment. It does not know the correct path, where the obstacles are, or which actions will move it closer to its goal. At first, it may hit a wall, move in circles, or repeatedly choose poor paths. Yet, after enough interaction with the environment, its decisions can gradually improve. How does that happen? This is where
Q-Learning
comes into the picture.
From Reinforcement Learning to Q-Learning
Just as supervised and unsupervised learning contain different algorithms for different types of problems, reinforcement learning also provides several approaches for learning through interaction.
Q-Learning is one of the most fundamental and beginner-friendly reinforcement learning algorithms.
The central idea is simple: an agent interacts with an environment, tries different actions, observes the consequences, and gradually learns which actions are more valuable in different situations.
Q-Learning is a
model-free reinforcement learning algorithm
. In simple terms, this means that the agent does not need a complete map or predefined model of the environment before it begins learning. Instead, it improves through experience.
Think of a person exploring an unfamiliar area without a map. Initially, every road is unknown. Some roads may lead to dead ends, while others may move the person closer to the destination. By repeatedly exploring and remembering previous outcomes, the person gradually becomes better at choosing a route.
Q-Learning follows a similar intuition.
The Main Problem: How Does the Agent Know Which Action Is Good?
Consider a simple treasure-hunt environment. An agent begins at a starting position and wants to reach a treasure. It can move left, right, up, or down, but it has no prior knowledge of the environment. At the beginning, all actions are uncertain. Moving right may lead toward the treasure, moving left may lead to a wall, and moving upward may take the agent along a longer route. The agent cannot know this in advance because it has not yet gained any experience.
Therefore, the real problem is not simply reaching the goal once. The deeper problem is:
How can the agent learn from previous actions and make better decisions the next time it faces a similar situation?
Q-Learning addresses this problem by learning the value, or quality, of actions.
What Does the “Q” in Q-Learning Mean?
The letter
Q
refers to
Quality
. It represents how useful or valuable it is to take a particular action in a particular situation. The key question behind Q-Learning is:
How good is this action in this situation?
Suppose the agent reaches a junction where four actions are possible. After gaining some experience, it may have learned values similar to these:
Possible Action
Learned Q-Value
Move Left
2
Move Right
8
Move Up
4
Move Down
-3
Based on its current experience, moving right appears to be the most valuable action because it has the highest Q-value.
However, these values are not manually assigned by a programmer. The agent learns and improves them through repeated interaction with the environment. This distinction is important because the intelligence of the agent emerges from experience rather than from explicitly programmed instructions.
The Q-Table: The Agent’s Learning Diary
A beginner-friendly way to understand Q-Learning is to imagine that the agent keeps a diary of its experiences. Whenever it takes an action in a particular situation, it learns something about the usefulness of that decision.
In Q-Learning, this learning diary is commonly represented as a
Q-table
.
State
Left
Right
Up
Down
State A
2
8
1
0
State B
5
3
7
1
State C
1
0
4
9
The rows represent different
states
, while the columns represent possible
actions
. Each cell contains a Q-value that reflects how valuable the corresponding action appears to be in that state.
For example, if the agent is currently in State B, moving upward has a Q-value of 7. Based on what the agent has learned so far, moving up appears more promising than moving left, right, or down.
This is how the agent develops a form of experience-based memory. It does not simply remember a single route from start to finish. Instead, it learns which actions appear valuable in different situations.
How Does the Agent Improve Over Time?
At the beginning of learning, the agent has little or no useful knowledge. Its Q-values may initially be equal because it has not yet discovered which actions are better. The learning process then follows a repeated cycle:
Try an action → Observe the outcome → Receive feedback → Update knowledge → Repeat
Suppose the agent moves right and the outcome is useful. Its confidence in that decision may increase. If it repeatedly moves into a wall, that action becomes less attractive. As the agent collects more experience, the values in its Q-table gradually become more informative.
The important point is that the agent does not become intelligent after one successful attempt. Learning is gradual. A single experience contributes information, while repeated experiences help the agent develop better estimates of which actions are worth taking.
You can think of this like navigating a new city. During your first week, you may take longer routes, miss turns, or enter congested roads. After repeated travel, you begin to recognize shortcuts and avoid poor routes. Your decisions improve because previous experience changes how you evaluate future choices.
Exploration vs. Exploitation: Should the Agent Always Choose the Best-Known Action?
At this point, a natural question appears. If the agent knows which action currently has the highest Q-value, why not always choose it?
Suppose the agent has learned:
Action
Q-Value
Right
8
Left
3
Choosing right seems obvious. However, what if the left path has not been explored enough? Perhaps it eventually leads to a much larger reward.
This creates one of the most important challenges in reinforcement learning: the balance between
exploration
and
exploitation
.
Exploitation
means using existing knowledge. The agent chooses the action that currently appears best because previous experience suggests that it is valuable.
Exploration
means trying uncertain or less familiar actions. The result may be worse, but it may also reveal a better strategy that the agent would never discover by repeatedly choosing the same action.
A simple real-life analogy is choosing a restaurant. You can return to your favorite restaurant because you already know it is good, or you can try a new restaurant that might be even better. The first choice uses existing knowledge; the second gathers new experience.
A successful learning agent needs both.
The Biggest Immediate Reward Is Not Always the Best Decision
Another important idea is that Q-Learning is not simply about collecting the largest reward available at the current moment.
Consider two possible choices:
Action A gives an immediate reward of 5, but the journey ends there.
Action B gives no immediate reward, but eventually leads to a reward of 100.
If the agent only cared about immediate feedback, it might repeatedly choose Action A. However, from a long-term perspective, Action B is clearly more valuable.
Chess provides a useful analogy. A move may give no immediate reward at all, yet several moves later it may contribute to checkmate. The value of the earlier move comes from the future position it helped create.
This is a central intuition in Q-Learning:
A good action is not always the one that gives the biggest reward immediately; it may be the one that leads to better future outcomes.
We will examine how Q-Learning represents this idea mathematically in the next article.
What Does Q-Learning Actually Learn?
A common misunderstanding is that Q-Learning simply memorizes the correct path. That is not quite accurate.
Q-Learning learns the expected quality of taking different actions in different states. Instead of memorizing a fixed sequence such as “right, right, up, left,” it develops knowledge such as:
In State A, moving right appears valuable. In State B, moving upward appears better. In State C, moving left appears poor.
As these local decisions improve, a good overall strategy can emerge.
This distinction matters because the agent is learning the value of
state-action pairs
, not merely memorizing one successful journey.
Why Q-Learning Matters
Q-Learning is an important starting point because it introduces several ideas that appear repeatedly across reinforcement learning: learning through interaction, evaluating actions, balancing exploration and exploitation, and considering future outcomes rather than only immediate rewards.
It also provides useful intuition for understanding more advanced methods. For example, Deep Q-Network. This progression is one reason Q-Learning remains such a useful concept for beginners. It provides a clear bridge between basic reinforcement learning intuition and more advanced deep reinforcement learning methods.
A Small but Important Limitation
If Q-tables are so useful, why do we not use them for every reinforcement learning problem?
The answer is scale.
A small maze may contain only a manageable number of states and actions, making a table practical. However, a complex video game, robotic system, or real-world environment may contain an enormous number of possible situations. In some environments, states are continuous rather than neatly countable.
As the number of possibilities grows, maintaining a separate table entry for every state-action pair can become impractical.
This limitation motivates more advanced approaches, including methods that estimate Q-values rather than storing every value in a literal table. For now, however, the Q-table remains one of the clearest ways to understand how an agent can learn from experience.
Final Thoughts
Q-Learning becomes much easier to understand when we focus on its central question: "
How good is this action in this situation?"
The agent begins with limited knowledge, interacts with its environment, observes outcomes, and gradually improves its estimates of which actions are valuable. The Q-table acts as a learning diary, exploration helps discover new possibilities, exploitation uses existing knowledge, and long-term thinking prevents the agent from chasing only immediate rewards.
At this stage, one major question remains unanswered: "
How exactly does the agent update a Q-value after taking an action? "
That is where the mathematics begins.
In the next article, we will explore the Q-Learning update process, understand the role of immediate and future rewards, and build a small working example to see the algorithm in action.
Let’s follow us
ML Diaries by Fahd
26
views