Everything in Part 5 worked. The pipeline ran end to end, LIDAR in, occupancy grid out, latent vector through the actor, continuous steering and throttle out the other side. Then I actually let it train for a while, and the car stopped moving. Not crashed, not stuck, not erroring out. It just sat there.
First instinct with something like this is to assume it's broken. It wasn't. The agent wasn't stuck or malfunctioning, it had actually calculated, correctly, that moving was worse than standing still. Zero throttle and zero braking minimized its losses better than anything else it had tried.
That's a strange thing to sit with. The car wasn't failing to learn. It had learned something, just not the thing I wanted.
Looking at it through the Q-function, the thing SAC is actually trying to maximize is expected return. So I engineered a reward topology where driving forward gives a continuous stream of small positive rewards, efficiency and progress, while crashing or going out of bounds triggers a massive negative penalty.
Run the numbers on that. If the agent drives forward for ten steps and then crashes: (10 steps × 0.1 driving reward) − 50 crash penalty = −49 total return. If the agent instead slams the brakes and sits completely still for those same ten steps: (10 steps × 0 reward) + 0 = 0 total return.
Zero beats negative forty-nine. Every time. The agent wasn't broken, it was doing exactly what the math told it to do.
My first read on this was that the policy had collapsed, in the SAC sense. In continuous action spaces, the policy isn't a lookup table, it's an explicit distribution, predicting the parameters of a Gaussian at every state rather than a single fixed action.
Here's how I thought the collapse might have happened. At step 0, the network initializes with random weights, which means a wide standard deviation. Wide standard deviation means highly random actions, which inevitably means the car crashes into something almost immediately. The critic sees that catastrophe and sends back massive negative gradients. And the actor's mathematical objective, in response, is to eliminate the randomness that caused it, since less randomness means fewer catastrophic actions. If the architecture clamps the log standard deviation at some minimum, say -20, the exponential of -20 is a vanishingly small number, and the Gaussian effectively collapses into a single sharp spike, a Dirac delta function in all but name. At that point entropy regularization has basically nothing left to push against. The critic has overpowered the actor, and the actor has decided that absorbing the entropy penalty is cheaper than ever risking the crash penalty again.
It's a clean theory. I went looking for evidence of it in the training logs.
Step 990 → out of road, reward drifting between 0.000 and 0.051. Step 1640 → out of road. Step 3180 → out of road, reward sitting at 0.001 or 0.000. Step 5140 → 0.050, but 0.00 for most of the run. Step 6148 → out of road. Step 8604 → out of road.
Same pattern, over and over. So I tried the obvious fixes. Lowered the crash penalty from 50 down to 15. Raised alpha, the entropy coefficient, up to 0.3, to force more exploration back into the policy. Nothing changed. The agent still wasn't moving. Which meant the actual problem wasn't where I thought it was, and it was time to go looking in the training code directly instead of just adjusting hyperparameters and hoping.
I wrote a small diagnostic to actually check whether the policy had collapsed, rather than just assuming it had. The logic: if the standard deviation values are genuinely tiny, somewhere around 1e-6 or lower, that confirms real collapse, the actor paralyzed by fear of the environment and shutting off all stochastic exploration entirely. If instead the standard deviation is still meaningfully large, above roughly 0.1, then collapse isn't the story, and something else architectural is going on, a dying CNN latent vector, or a critic outputting flat gradients.
Even after reducing alpha down to 0.05, the actor's standard deviation sat at roughly 0.87. Nowhere near a collapse. Which meant my entire first theory, however clean it looked on paper, was wrong.
Before abandoning the collapse theory completely, I wanted to rule out one more thing: whether alpha itself was being tuned correctly. SAC has a mechanism for this, automatic temperature adjustment, which treats entropy as a constrained optimization problem rather than a fixed hyperparameter. The framing is roughly: maximize expected return subject to policy entropy staying above some minimum target, H(π_θ(s)) ≥ H_target. Using dual formulation, that constraint introduces a learnable Lagrange multiplier:
L(α) = E[−α(log π_θ(a|s) + H_target)]
Worked through both directions of that. If the policy is too random, entropy exceeding target, say H_target = −2.0 and the term inside the parenthesis comes out to log π_θ(a|s) + H_target = −2.5 + 2.0 = −0.5, negative. Plug that back in: L(α) = −α × (−0.5) = +0.5α, which drives alpha down toward zero, pulling exploration back in. If the policy is collapsing instead, entropy falling below target, the sign flips, L(α) = −α × (+1.8) = −1.8α, and alpha climbs upward, pushing exploration back up. Target entropy itself is typically chosen as H_target = −dim(A) = −2.0 for a two-dimensional continuous action space, steering and throttle.
None of it mattered here. Even with alpha auto-tuned down to 0.06, the standard deviation stayed stubbornly around 0.87. The temperature mechanism was doing exactly what it was supposed to do. It just wasn't the thing keeping the car frozen.
If the standard deviation wasn't collapsing, then the actor wasn't afraid of exploring. Which meant the problem had to be sitting somewhere else in the loop entirely, in what the critic was actually telling it.
Here's the calculation that cracked it open. Take two nearly identical actions: a1 = [steer: 0, throttle: 0.5], a2 = [steer: 0, throttle: 0.52]. One step later, the vehicle's speed under a1 might land at 10.00 km/h, and under a2 at 10.02 km/h. That's a velocity difference of Δv = 0.00005. Practically nothing.
Which means, for almost the entire space of reasonable driving actions, the reward signal barely changes at all between one action and its near neighbor. The critic only ever sees a sharp, high-variance signal at two specific, discontinuous events: releasing the throttle entirely and triggering the stationary penalty, or crashing outright and eating an immediate penalty. Everywhere in between, for any action sequence that keeps the car safely on the road, the critic's training targets end up nearly identical to each other, and the value landscape it learns ends up exceptionally smooth and correspondingly flat.
That flatness isn't a bug. It's the mathematically correct shape of the actual task. There's a genuinely high-dimensional manifold of "acceptable" driving behaviors, gentle variations in steering and throttle that all lead to roughly the same long-term outcome, and a flat value landscape across that manifold is exactly what an accurate critic should produce. It has no measurable impact on long-term expected return whether you steer 0.01 degrees left or 0.01 degrees right, so an accurate critic correctly reports that they're worth the same.
I tested this directly rather than just trusting the theory. Sampling the critics' Q-value estimates across a batch of similar safe-driving states:
Critic 1 → range [4.0858, 4.0968], max-min delta ΔQ = 0.0110, standard deviation ≈ 0.0030. Critic 2 → range [4.0838, 4.0951], max-min delta ΔQ = 0.0113.
Confirmed the uninformative-critic hypothesis directly. Geometrically, the Q-surface really was close to a perfectly flat table. In that state, it genuinely doesn't matter what the actor does, crashing, braking, steering, speeding, they're all landing at close to mathematically identical value as far as the critic can tell.
This is where the actual freeze comes from, and it's a kind of paradox once you see it laid out. When the critic's landscape is steep, sharp feedback, it violently pulls the actor's mean toward the peak of that landscape and rapidly forces the standard deviation down, since there's a clear best action worth committing to. But when the landscape is flat, which is exactly the situation here, the critic is effectively telling the actor "left or right, faster or slower, doesn't matter, they're all equally good." There's no gradient sharp enough to pull the actor's mean anywhere in particular, and there's nothing pulling it toward "drive forward" specifically either. The safe, do-nothing basin sits right there in that same flat region, easy to fall into and with nothing steep enough nearby to push the agent back out of it.
To get past that deadlock, I needed a way to actually see what the critic was telling the actor at the level of a single gradient, not just infer it from standard deviation values and reward logs.
I froze a single representative starting observation, the car sitting at the track's start line, and queried the critic's gradient with respect to action directly, ∇_aQ(s,a), evaluated at the zero-action baseline. The result:
Evaluated State Q-Value : 4.0913
Analytical Gradient (dQ/da) : Steering: 0.00087807 | Throttle/Brake: 0.00461295
Both gradients were microscopically tiny, on the order of 10⁻⁴ and 10⁻³. And that mattered a lot more than it might sound like, because of exactly how SAC updates its actor. The actor's weights get updated by propagating gradients through the critic, using the reparameterization trick:
∇_θL_actor ≈ E[α∇_θ log π_θ(a|s) − ∇_aQ_φ(s,a) · ∇_θf_θ(ε;s)]
With ∇_aQ_φ(s,a) sitting essentially flat, that entire exploitation term went dead. Backpropagation through the critic was, functionally, severed. The actor's weights weren't receiving zero direction because of some bug, they were receiving zero direction because there was genuinely nothing there to receive. Whatever movement the actor made was coming entirely from the entropy regularization term, not from anything the critic actually knew about driving.
Two things were stacking on top of each other here. First, a dimensionality mismatch: the input state space was huge, a 2,048-dimensional visual latent plus a 16-dimensional neighbor-tracking vector, 2,064 dimensions total, feeding into a control output that was just two numbers, steering and throttle. Second, and more subtly, a signal-to-noise problem. Under a discount factor of γ = 0.99, the expected cumulative discounted return integrates progress rewards over roughly a hundred-step horizon: V(s) ≈ r_progress / (1 − γ) = 0.04 / 0.01 ≈ 4.0. That balloons the baseline expected return up to around 4.09. Meanwhile, the actual step-to-step difference in advantage between, say, a lane-centering steer and a mild swerve, was on the order of 0.01. A signal-to-noise ratio of roughly 400 to 1.
Given that ratio, the Adam optimizer did exactly what any gradient-based optimizer would do: found the path of least resistance. It achieved a 99.9 percent reduction in loss simply by learning to reconstruct that dominant 4.09 baseline, and never bothered engaging with the comparatively tiny action-dependent signal buried underneath it. The critic hadn't failed to learn, it had learned something real, Q(s,a) ≈ V(s), a state-value function wearing a state-action function's name.
The obvious structural fix was to stop asking one network to do both jobs. Split the critic into a state-value stream and an action-advantage stream, Q(s,a) = V_ψ(s) + A_χ(s,a). Since V_ψ(s) only ever sees the state, it's mathematically forced to absorb that 4.09 baseline on its own, freeing the advantage stream to focus purely on the local, action-dependent variation that actually mattered.
I ran it expecting a breakthrough. Instead I got a landscape flatter than before I'd started:
Critic 1 Q-value Range : [4.1446, 4.1446]
Critic 1 Max-Min Delta : 0.0000
Analytical Gradient (dQ/da) : Steering: 0.00000000 | Throttle/Brake: 0.00000000
Gradients at exact, literal zero. The reason came down to something discrete dueling architectures get for free and continuous ones don't: an identifiability constraint. Discrete dueling DQN normalizes its advantage stream by subtracting the mean advantage across the finite set of possible actions, A(s,a) − (1/|A|)Σ A(s,a'). With a continuous, infinite action space, a ∈ [-1,1]², there's no finite set to sum over, no way to apply that same normalization. Without it, the optimizer found a new shortcut instead. Early in training, while the policy was still noisy, any nonzero output from the advantage stream only added variance and increased prediction error. Minimizing MSE meant driving the advantage stream's weights all the way down to zero and leaving them there. The critic hadn't just stayed flat, it had actively learned to switch off the one piece of itself that was supposed to fix the problem.
Two changes together got the loop moving.
Horizon compression. I scaled the discount factor down from γ = 0.99 to γ = 0.90. That directly compresses the baseline: V(s) ≈ 0.04 / (1 − 0.90) = 0.40. Dropping the state baseline from roughly 4.14 down to 0.40 meant the action-dependent reward differences suddenly represented a meaningful fraction of the total loss instead of getting lost underneath it, which gave the optimizer an actual reason to engage with the action-processing layers at all.
Leaky ReLU. Separately, I found a dead ReLU cascade sitting inside the advantage stream. While the oversized baseline had been dominating early updates, the pre-activations feeding the action-processing layers had been pushed permanently negative, and a standard ReLU, max(0, x), clips the gradient to exactly zero for any negative input. Once a unit died that way, it could never receive another update, no matter what changed afterward. Swapping in Leaky ReLU (F.leaky_relu, negative slope 0.01) fixed that directly: for x < 0, the derivative becomes 0.01 instead of 0, small, but nonzero, which is the entire difference between a permanently closed gradient path and one that stays open.
Ran a fresh 15,000-step diagnostic with both interventions in place:
Evaluated State Q-Value : 1.6346
Analytical Gradient (dQ/da) : Steering: -0.00108051 | Throttle/Brake: 0.00395266
Critic Standard Deviation : 0.0026 (curvature unlocked)
Steering gradient negative, throttle gradient positive, and neither one sitting at zero anymore. That negative steering gradient is the critic actually saying something now: from this starting frame, steering further right reduces expected future return, so steer straighter or slightly left. The positive throttle gradient says the opposite kind of thing: increasing throttle directly increases cumulative discounted return. Small numbers, but real direction behind them for the first time.
Worth noting why gradients this small could move anything at all. SAC trains with Adam, and Adam normalizes updates by the running root-mean-square of the gradient, which makes it scale-invariant to consistent sign. It's not the raw magnitude of a gradient that drives the policy somewhere, it's whether that gradient reliably points the same direction, update after update. Once the critic's landscape had any curvature at all, however faint, that was enough for Adam to actually steer the policy with it.