Building an RL Traffic Agent for Indian Roads, Part 7: Unlocked, and Immediately Falling Apart Differently

The critic had curvature. The actor finally had somewhere to go. Which meant, for the first time, the car actually moved. It also meant a new set of problems, because "moving" and "driving well" turned out to be two very different bars to clear.

The Car That Glitched

First symptom: the vehicle started accelerating, then immediately started glitching. Steering violently hard-right, then hard-left, on consecutive steps. Throttle swinging between full power and hard braking, with nothing smooth in between.

The cause traced back to how SAC's actor actually works. It's stochastic by design, outputting a mean and standard deviation rather than a single fixed number, and during training that's exactly right, you sample from N(μ, σ) specifically to keep exploring. The mismatch was that I'd left evaluation running the same way, deterministic=False, which meant every single step during evaluation was still drawing an independent random sample from that distribution. Two back-to-back steps could land on wildly different points of the same distribution purely by chance, and the car experienced that as violent, second-to-second whiplash. The fix was narrow and specific: set deterministic=True at evaluation time, so inference runs strictly along the policy mean, a_t = μ(s_t), and the randomness that's genuinely useful during training stops leaking into how the car actually drives once training is over.

Smooth Isn't the Same as Comfortable

Even with deterministic evaluation fixing the second-to-second chatter, the driving was still jerky over longer stretches. The reward function I'd built was shaped entirely around progress and safety, and it had nothing at all to say about comfort. Which meant a trajectory that swerved violently at high speed scored exactly the same as a trajectory that stayed smooth the whole way through, as long as both ended up making equivalent progress.

So I added a quadratic action-smoothness penalty directly into the environment loop:

R_comfort = −0.1 × (a_steer,t − a_steer,t−1)² − 0.1 × (a_throt,t − a_throt,t−1)²

Caching the previous step's action and comparing it against the current one, then folding that penalty into the reward before it ever reaches the replay buffer. Simple, but it meant the agent was now being scored not just on where it ended up, but on how abruptly it got there.

Two Bugs at the Wrapper Boundary

A couple of structural issues showed up right where my custom wrapper met the underlying environment.

First, an AttributeError: 'FrameStackWrapper' object has no attribute 'action_space'. The wrapper I'd built was acting as a complete barrier, hiding everything about the raw environment underneath it, including attributes other parts of the training code needed direct access to. Fix was straightforward once I saw it: query the inner, unwrapped environment directly, raw_env.action_space.sample(), instead of expecting the wrapper to transparently forward every attribute on the object it was wrapping.

Second, a missing positional argument, 'truncated', crashing the replay buffer. Older Gym APIs merged termination and timeout into one shared done flag. Modern Gymnasium APIs split that into two separate signals on purpose, terminated for a genuine catastrophic end, a crash, where the expected future return really is zero, and truncated for hitting a step-count timer while otherwise driving fine, where the expected future return very much isn't zero. I'd already reasoned through why that distinction mattered back in Part 2, but this was where it actually broke the code until I decoupled the two signals properly and stored them independently.

The Last Problem: Steering Stuck at -0.0102

With the structural issues cleared out, the car was actually accelerating well, hitting speeds around 4.35 m/s. But its steering had locked onto almost exactly -0.0102, consistently, across different scenarios, and the car was slowly drifting off the left edge of the track because of it.

The root cause was the asymmetric setup itself, the same design that had fixed the earlier "cheat sheet" problem back in Part 1 was now creating a new one. The critic, the teacher, was processing the latent vector plus the sixteen-dimensional num_others array, which meant it had direct access to exact lateral error and heading deviation. With that information sitting right there, it could easily compute a properly curved value surface and meaningful steering gradients. The actor, the student, was only ever seeing the detached visual latent, nothing else. And early in training, at only around 15,000 steps in, that visual CNN was still close to uninitialized static noise, producing features with effectively zero meaningful variance relative to actual lane boundaries. With no useful signal coming from its own input, the actor's forward pass ended up dominated entirely by the bias vectors sitting in its output layers, vectors that happened to have initialized somewhere near -0.0102. And because the critic's steering gradient was still comparatively weak relative to everything else going on, the comfort penalty from earlier in this part locked the behavior in place further: the safest way to avoid a change penalty was to just leave the steering exactly where the bias vector had put it, and focus entirely on accelerating instead.

Two ways to fix this presented themselves, and they pointed in opposite directions.

Option one: give the actor privileged telemetry too. Just concatenate num_others onto the actor's input directly, matching what the critic already sees. This worked essentially instantly, steering sensitivity came back and the car started centering itself in the lane right away. But it broke the entire premise of the project. The whole point was an end-to-end visual policy, a car that drives off what it can actually perceive, not one that queries privileged simulator coordinates it would never have access to outside a simulation. Taking this option would have meant winning the metric and losing the actual research question.

Option two: stay pure end-to-end, and fix the underlying imbalance instead. This is what I went with. Two things made it work. First, scaling down the navigation input relative to the visual latent before concatenation:

x_actor,t = [z_visual,t ‖ (n_nav,t × 0.1)]

Scaling the navigation coordinates down by a factor of 0.1 kept them from mathematically overwhelming the still-forming visual representation, giving the CNN room to actually start contributing instead of getting drowned out by a numerically larger but less informative signal. Second, simply extending training out to 100,000 steps. That gave the critic enough temporal volume to keep backpropagating TD errors through the visual encoder long enough for the CNN to actually learn to map raw road geometry into structured spatial features, rather than staying stuck as noise. Once that happened, the actor finally broke free of its bias-vector default, its steering signs aligned with the geometry of the road, and it started gating its own navigation behavior, only actually steering when the lane ahead genuinely opened up for it.

The Whole Arc, Side by Side

Pulling every phase of this into one table, since seeing it laid out end to end says something the individual debugging stories don't quite capture on their own:

Every row of that table is a different flavor of the same underlying lesson, really: a critic that can't feel the action signal will always find some cheaper baseline to hide behind, and every fix in this whole arc, dueling streams, horizon compression, leaky gradients, comfort penalties, nav scaling, was really just different ways of forcing the critic to stop being able to take the easy way out.