What if an AI agent says it succeeded, but the evidence says otherwise?
I write technical articles that analyze real world engineering problems through the lens of production systems.
My work focuses on architectural tradeoffs, UI layer system design, performance optimization, and reliability patterns derived from shipping and maintaining scalable applications.
AI agents are becoming increasingly capable of taking actions rather than simply generating text.
They can inspect systems, call tools, execute code, modify state, recover from failures, and interact with external services through protocols such as MCP.
That creates a problem that is different from ordinary application correctness.
An agent can say: "I investigated the incident and fixed it."
But what actually happened?
Did it call the right tool?
Did the tool return what the agent claims?
Did it execute code in the intended sandbox?
Did it stay within its execution contract?
Did a retry actually recover the operation?
Was a dangerous action authorized?
Did the action execute at all?
And perhaps most importantly:
Can I prove the answer without trusting the agent's own narrative?
That question became the foundation for AgentGuard.
The core idea is simple:
An agent's claim is not evidence. The execution trajectory is evidence.
AgentGuard is a runtime trust and assurance layer for agentic systems. TrueForge provides the execution runtime; AgentGuard sits around that execution to govern consequential actions, capture evidence, verify execution contracts, reason about recovery, and produce an auditable assurance result.
🔗 GitHub Repository: https://github.com/Jaival-Suthar/agentguard
The architectural boundary is deliberate:
TrueForge owns execution. AgentGuard owns trust.
Traditional software gives us relatively clear sources of truth.
A process exited with code 0.
A database transaction committed.
A test passed.
A request returned a particular response.
Agentic systems introduce another layer between the system and the result: the model's interpretation of what happened.
Consider an incident investigation agent.
It might:
Investigate incident
↓
Read incident state
↓
Inspect logs
↓
Run analysis
↓
Attempt remediation
↓
Report success
The final response might confidently say:
Incident resolved.
But the runtime could tell a different story:
incident.lookup ✓
sandbox execution ✓
rollback requested ✓
rollback denied ✓
rollback executed ✗
verification ✗
The agent's statement and the runtime trajectory are not equivalent.
That led to the central design principle behind AgentGuard:
Never promote model narrative into runtime truth.
The verifier should establish facts from execution evidence. The assurance layer should summarize those facts.
AgentGuard treats an agent run as a sequence of observable events rather than a final text response.
A simplified execution might look like:
Agent intent
↓
Policy evaluation
↓
Tool request
↓
Tool execution
↓
Tool result
↓
Evidence collection
↓
Contract verification
↓
Recovery if necessary
↓
Independent verification
↓
Assurance verdict
This gives the system several distinct questions to answer.
The execution intent and requested action.
The policy decision at the execution boundary.
The runtime and tool evidence.
The execution verifier.
The resulting trajectory and independent verification.
The final Assurance Artifact.
Keeping these questions separate is one of the most important architectural decisions in AgentGuard.
A tool call succeeding is not necessarily an execution succeeding.
A recovery API returning successfully is not necessarily a recovered system.
And a model claiming success is certainly not sufficient evidence.
AgentGuard therefore uses execution contracts to define the conditions that must be satisfied for a run to be considered valid.
A contract can express requirements around:
expected actions
permitted retries
required evidence
execution outcomes
target identity
recovery behavior
Instead of asking: Did the agent run?
AgentGuard asks: Did the execution satisfy the declared contract?
That distinction becomes particularly important when failures and retries enter the picture.
One of the most important parts of AgentGuard is the evidence verification layer.
Runtime events are normalized into a common execution vocabulary so that the verifier can reason about the execution without depending on every underlying runtime detail.
Conceptually:
TrueForge evidence
↓
Observation normalization
↓
Event correlation
↓
Evidence verification
↓
Contract verification
↓
PASS / WARN / FAIL
The verifier correlates things such as:
TOOL_CALL
TOOL_RESULT
SANDBOX
EXECUTION
FAILURE
RECOVERY
POLICY
APPROVAL
The important part is not merely recording those events.
It is establishing their relationships.
For example:
Which action requested this result?
Which tool produced it?
Which result belongs to which call?
Was that attempt successful?
Was it a retry?
Does the evidence correspond to the required target?
Does the observed trajectory satisfy the contract?
Without that provenance, an assurance system can become confidently wrong.
That is why AgentGuard deliberately treats:
Agent output ≠ Ground truth
Ground truth comes from the execution evidence.
Retries make execution verification substantially harder.
Consider:
Attempt 1 → FAIL
Attempt 2 → SUCCESS
That is not the same execution outcome as:
Attempt 1 → FAIL
Attempt 2 → FAIL
The first attempt should not disappear merely because a later attempt succeeded.
At the same time, a successful retry should not be ignored simply because the first attempt failed.
AgentGuard therefore reasons about the trajectory rather than treating an action as a single static event.
This is also an area where Qodo's review was particularly valuable. It identified correctness problems around retry/evidence correlation and helped expose cases where the verifier could select the wrong evidence rather than correctly associating a successful retry with the contract requirement.
That resulted in both implementation changes and stronger tests.
For an assurance system, this matters enormously.
A verifier that mishandles retries can report the wrong truth while still appearing perfectly deterministic.
The same principle applies to sandboxed execution.
An event saying: sandbox started
is not equivalent to proving: the intended computation executed successfully
AgentGuard therefore distinguishes between an invocation and its verified outcome.
The broader principle is:
An observation about an action is not automatically proof of its result.
This sounds obvious.
In practice, maintaining that distinction across an event pipeline, runtime adapter, verifier, recovery system, and UI is surprisingly easy to get wrong.
That is why the evidence model is deliberately explicit.
Post-execution verification is only half of the problem.
A trustworthy agent system also needs to answer:
What if the action should never execute in the first place?
AgentGuard therefore introduces a policy boundary at the MCP execution layer:
The critical invariant is:
If an action is blocked or approval is denied, the real executor must never be called.
This changes AgentGuard from a purely observational system into a control boundary.
It can now answer both: Was the action allowed to execute?
and: Did the action execute correctly?
Those are different questions.
And for consequential agent actions, both matter.
Suppose an agent requests: rollback_incident
AgentGuard determines: APPROVAL_REQUIRED
The human denies it.
A weak implementation might only show:
❌ Denied
The stronger guarantee is:
Agent request
↓
PolicyGate
↓
APPROVAL_REQUIRED
↓
Human DENY
↓
APPROVAL_DENIED
↓
REAL EXECUTOR INVOCATIONS = 0
That final line is the important part.
The dangerous operation never ran.
This is the difference between displaying a security decision and enforcing one.
Failures are inevitable in real agentic systems.
The interesting question is what happens next.
AgentGuard treats recovery as a bounded execution trajectory:
Attempt 1
↓
FAIL
↓
Recovery decision
↓
Attempt 2
↓
Observe
↓
Verify
Recovery is constrained by the execution contract rather than becoming an uncontrolled retry loop.
More importantly:
A recovery action is not proof of recovery.
The system must observe the resulting execution and independently verify that the expected state has actually been restored.
That gives the complete lifecycle:
Failure
↓
Recovery
↓
Re-execution
↓
Observation
↓
Independent verification
Only then should recovery contribute to the final assurance result.
Once AgentGuard had policy, execution, contract, evidence, and recovery signals, there was a dangerous temptation:
Let the UI combine them and decide what happened.
I deliberately avoided that.
Instead, AgentGuard has a deterministic Assurance Artifact that acts as the authoritative composition layer.
Conceptually:
The artifact records the relevant lifecycle state, component checks, summaries, recovery information, and final verdict.
The important boundary is:
The UI does not decide whether an execution passed. The artifact does.
This makes the artifact independently inspectable and prevents different presentation surfaces from inventing different versions of the truth.
An assurance result should not depend on incidental runtime state.
For identical inputs:
same evidence
+
same contract
+
same policy
↓
same assurance result
This sounds straightforward, but even seemingly harmless details can break it.
For example, allowing an Assurance Artifact timestamp to silently default to the current clock means the same input can produce different artifacts.
Qodo caught this class of issue during review.
The fix was to make the relevant artifact inputs explicit and deterministic.
That matters because reproducibility is part of trust.
If two independent verifiers receive the same evidence, they should be able to arrive at the same logical conclusion.
Qodo became especially useful on AgentGuard because the hardest bugs were not simple syntax errors or obvious failures.
They were correctness-of-the-verifier problems.
The reviews challenged assumptions around:
retry semantics
event identity
evidence correlation
policy provenance
recovery state
deterministic artifacts
verdict rendering
security boundaries
One particularly important finding involved the Assurance Artifact receiving:
policyVerdict: "ALLOW"
from an integration path rather than deriving that decision from the actual deterministic policy evaluator.
That created the possibility of an artifact saying:
Policy: PASS
without actually evaluating the policy.
Qodo caught the issue.
The implementation was changed so that the real policy evaluation became the source of the artifact's policy result.
Another review caught a recovery-state inconsistency where an existing successful recovery result could be discarded during error handling and replaced with an inconsistent state.
Again, the fix was not merely cosmetic.
The system was changed to preserve the actual execution trajectory.
Qodo also identified additional issues around event identity and correlation, including cases where ID-less events could accidentally share state.
That matters because incorrect correlation in an assurance system is much worse than an ordinary application bug.
It can produce a credible but false proof.
That is exactly the class of failure AgentGuard is supposed to prevent.
So the role of Qodo was not:
"Qodo wrote my code."
It was:
Qodo continuously challenged whether the code's claims were actually supported by its implementation.
For a project whose purpose is proving claims about agent execution, that was an unusually strong fit.
The implementation is backed by 100+ passing test cases covering the core execution, verification, policy, recovery, evidence, assurance, and integration behavior.
The important part is not simply the number.
The tests exercise the failure modes that matter to a trust boundary:
Allowed action
Blocked action
Approval required
Approval denied
Missing approval handler
Successful execution
Failed execution
Successful retry
Exhausted retry budget
Evidence mismatch
Invalid evidence
Sandbox verification
Recovery success
Recovery failure
Assurance precedence
Deterministic artifact generation
MCP integration
That gives AgentGuard a much stronger foundation than a demo where only the happy path is exercised.
The goal is not to prove that every possible agent behavior is safe.
The goal is to make the behavior inside the defined assurance boundary deterministic, testable, and explainable.
Once the backend could produce an authoritative assurance artifact, a human still needed a way to inspect it.
That is where the Assurance Console comes in.
The live architecture is:
TrueForge Runtime
↓
Run Evidence
↓
Live Store
↓
REST / SSE API
↓
Assurance Console
The console exposes:
run identity
connection state
semantic execution timeline
policy decisions
tool calls
evidence
recovery
assurance artifact
final verdict
But it does not independently decide the verdict.
That distinction is critical.
A dashboard can make almost anything look trustworthy.
AgentGuard instead tries to make the dashboard a projection of something that has already been verified.
The console asks: What happened?
The verifier asks: Can the evidence prove the required facts?
The policy layer asks: Was the action permitted?
The contract asks: What should have happened?
The Assurance Artifact asks: What does all of that establish?
Those questions remain separate.
The current demonstration uses incident investigation because it provides a concrete workload for demonstrating the system.
But AgentGuard's underlying trust model is not fundamentally about incidents.
It is about generic execution primitives:
Actor
Action
Tool
Input
Output
Resource
State
Invariant
Policy
Evidence
Decision
Approval
Failure
Recovery
That means the same assurance machinery could eventually be applied to:
DevOps agents
Finance agents
Database agents
GitHub agents
Research agents
Cloud agents
Customer-support agents
Browser agents
The domain changes.
The assurance model should not.
That is why normalized events, contracts, policy decisions, evidence, and assurance artifacts are kept separate from the specific workload.
Agent systems increasingly interact with external capabilities through standardized tool interfaces.
MCP provides a useful boundary for AgentGuard because the system can observe a meaningful transition:
Agent
↓
MCP
↓
Tool request
↓
Policy
↓
Tool execution
↓
Tool result
Instead of deeply coupling the assurance layer to the internals of every agent framework, AgentGuard can reason about the execution boundary.
TrueForge provides the current runtime integration.
The longer-term architectural direction is to make the assurance machinery increasingly independent of the underlying agent runtime.
This is worth stating explicitly.
AgentGuard is not a dashboard for TrueForge.
The UI is simply how a human inspects the evidence.
The actual product idea is the trust boundary underneath it:
Intent
↓
Policy
↓
Execution
↓
Evidence
↓
Contract
↓
Recovery
↓
Verification
↓
Assurance
The console makes that lifecycle visible.
It does not define it.
Assurance systems become less credible when they claim too much.
AgentGuard is not intended to be:
another agent runtime
a replacement for TrueForge
a complete enterprise IAM platform
an autonomous policy authorizer
a universal verifier for every agent framework
a claim of cryptographically tamper-proof evidence
The current system focuses on a narrower and more defensible problem:
Control consequential agent actions, capture execution evidence, and deterministically establish what that evidence proves.
That boundary is intentional.
The model knows what it intended to do.
The runtime knows what happened.
Those are not the same thing.
It is not enough to find:
incident_id = INC-042
somewhere in the output.
You need to know:
Which action requested it?
Which tool produced it?
Which result belongs to that call?
Was the result successful?
Can the evidence be correlated exactly?
Otherwise the verifier can become confidently wrong.
A failed attempt followed by a successful attempt is different from two failed attempts.
The trajectory matters.
A repair action returning successfully is not proof that the system was repaired.
The resulting state must be independently evaluated.
When AgentGuard cannot establish that an action is authorized, the safe answer is not:
probably okay
It is:
not authorized
The same principle applies to evidence.
When AgentGuard cannot establish that an outcome happened, it should not manufacture confidence.
The current architecture leaves a few natural directions open.
Extend beyond the current TrueForge integration toward additional agent runtimes and generic MCP environments.
Allow AgentGuard to identify potentially dangerous or state-changing tools and propose candidate policies for operator validation.
The important constraint is that adaptive discovery should propose, not silently rewrite the deterministic trust core.
For long-running agents, reconstruct the complete lineage:
Task
↓
Plan
↓
Tool calls
↓
Retries
↓
Code execution
↓
State changes
↓
Verification
↓
Final claim
That could enable a particularly useful assurance capability:
The agent's final claim is not supported by its execution evidence.
Agent systems are moving from:
generate text
toward:
observe
reason
act
retry
modify
recover
As agents become more capable, capability alone becomes less interesting.
The harder question becomes:
How do we trust the execution?
That is the problem AgentGuard explores.
The architecture is intentionally simple:
Agent
↓
Intent
↓
Policy
↓
Execution
↓
Evidence
↓
Contract
↓
Recovery
↓
Verification
↓
Assurance
And the core philosophy is even simpler:
Don't trust what the agent says it did. Prove what actually happened.
That is why AgentGuard sits alongside an agent runtime rather than trying to replace it.
TrueForge makes the agent capable of acting. AgentGuard makes those actions observable, governable, verifiable, and accountable.
The interesting question for agentic systems is no longer only:
"What can the agent do?"
It is increasingly:
"What can we prove the agent actually did?"
AgentGuard is an attempt to build that missing layer.
A layer where authorization is enforced before consequential execution, runtime behavior becomes evidence, execution contracts define success, recovery is bounded and verified, and the final result is represented by a deterministic assurance artifact.
Because when an agent says:
"I fixed it."
the most important question is not whether the answer sounds convincing.
It is:
What evidence proves it?
That is what AgentGuard is built to answer.