Part 7 ended on a table that looked clean. Nav scaling fixed the steering lock-in, extending training to 100,000 steps gave the CNN room to actually learn road geometry, and every metric in that closing table moved in the right direction at once. It read like an ending.
It wasn't one. This part is what happened the moment I tried to scale that exact same fix past the point where it had only ever been tested.
The obvious next question after Part 5 was whether the fix actually held, or whether it was a 100k-step-scale artifact that would fall apart the second I gave it more room to run. Same architecture, dueling critic, nav_info concatenated into the actor's input, auto-tuned alpha through the Lagrangian entropy trick, gamma still compressed to 0.90, just scaled up to 300,000 steps.
It didn't hold cleanly. To be fair, the actual Part 5 fix stuck, steering wasn't frozen at -0.0102 anymore. But a new pattern showed up in its place, and it was oddly specific. Steering would climb smoothly from near-zero, hold there for a stretch, plateau, then decay back down, a shape closer to a slow sine wave than anything resembling "read the road, correct toward center."
I checked this across six different scenario indices, different maps, different layouts entirely, and the same ramp-then-decay shape kept showing up in close variants across all of them. That consistency was the first real clue that whatever was driving this wasn't something specific to a given road. It was structural.
Which meant the framing from Part 4 needed correcting. Nav scaling and more training steps had solved bias-vector lock-in. That was real, and the table proved it. But bias-vector lock-in and actually reading the road are two different bars, and I'd only cleared the first one.
My first read on the ramp was that the critic could tell actions apart now, probe_critic_at_turn.py was showing smooth, sensibly shaped Q-surfaces instead of the flat tabletop from Part 3, but maybe it still hadn't seen enough correctly-labeled turning behavior to actually generalize "nav_command says left, so steer left" purely from self-play. Early self-play is mostly incompetent exploration. If a critic trained almost entirely on bad turns is the thing shaping the actor, that's a version of the classic spurious-correlation trap, the model latches onto whatever correlate happens to be lying around, elapsed time, distance since reset, instead of the actual causal signal it's supposed to be using.
I considered scripting a rule-based driver by hand for straight sections, turns, and junctions separately, and decided against it almost immediately. MetaDrive already ships a working expert autopilot, env.agent.expert_takeover = True, IDM-style, competent at lanes, turns, and traffic. Using it meant zero new controller logic to write or debug. Just let it drive, and record.
Getting that flag to do anything useful took longer than it should have, and both problems were found by actually reading output, not by guessing.
First attempt, expert_takeover = True on its own did nothing. info["action"] kept handing back my own placeholder [0.0, 0.0], unchanged, step after step. Turned out manual_control has to be set to True in the env config for the takeover flag to actually engage at all. Without it, the flag is silently inert, no error, no warning, it just quietly doesn't do anything.
Second bug, once the car was genuinely under ManualControlPolicy, the real action being executed wasn't sitting under the "action" key like I'd assumed. It was under "raw_action", a tuple of np.float32 values. Only found that by dumping the entire info dict and reading every key in it instead of guessing which one was likely to hold the answer, which is a lesson I seem to relearn every few phases of this project.
collect_demonstrations.py: reset with manual_control=True, num_scenarios=200 for variety, force expert_takeover=True after every reset, then step through full episodes until terminated or truncated fires. Pull raw_action at each step and store the full transition, (obs_grid, obs_num_others, obs_nav_info, action, reward, next_obs_grid, next_obs_num_others, next_obs_nav_info, terminated, truncated), shaped exactly the way the real replay buffer expects it. All of it serialized out to demonstrations.pkl.
Fifty episodes came out to 37,503 transitions total, verified after the fact rather than assumed.
train-v4.py is identical to v3 except for one addition, a load_demonstrations() call right after the replay buffer gets instantiated, run once before the main loop starts, iterating the pickled transitions and pushing each one in through the exact same .add() call the live loop uses later. One deliberate simplification here: demo transitions aren't protected from being overwritten once the circular buffer cycles past capacity during training. I thought about building a reserved region for them and decided against it for this first pass, getting an honest read on whether seeding helped at all mattered more right now than optimizing how it was seeded.
Ran it first at 100k steps as a validation pass, then extended the same setup out to 500k to see if it held at a longer horizon.
Demo-seeding didn't fix anything. Worse, by one metric it made things measurably worse. Episodes in the demo-seeded run were ending after 25 to 49 steps of out_of_road, compared to 85-plus steps in the equivalent v3 run with zero demo data. And the ramp-then-decay steering shape was still right there, now showing up at a strikingly consistent point, a sharp correction landing around step 20 to 21, run after run, regardless of which map it was on.
That consistency was the actually interesting part. It smelled exactly like the Husky-versus-wolf problem, a model technically solving the training objective by keying off a correlate that happens to be available, instead of the causal signal it's actually supposed to be reading.
Rather than keep training and hoping the next run looked different, I ran two direct, falsifiable checks on the demonstration data.
Label balance. Tallied steering sign across all 37,503 transitions. 51.4% left, 48.6% right, mean steering sitting at -0.008. Essentially balanced. Ruled out demo-label imbalance as the cause outright.
Temporal clustering. If "meaningful turn" moments, anything with |steering| > 0.1, clustered tightly around a specific step-count-since-reset across episodes, that would be direct evidence of a temporal confound baked into the demo data itself, most demo episodes just happening to hit their first real turn around a similar point. Measured mean step-since-reset at those moments: 257.99, standard deviation 142.03. That's a wide spread, not a cluster. Ruled that out too.
Both hypotheses about the demonstration data were cleanly falsified by actual measurement. Good process. Wrong location.
Went back and compared a v3 run, no demo data at all, 300k steps, directly against the demo-seeded 500k run, steering value by steering value. Both showed the same shape, a climb from near-zero up to a plateau around 0.13 to 0.15, a brief hold, then a smooth decay. The absolute step where it happened differed, v3 plateaued around step 55 to 56, the demo-seeded run around step 20, but the shape itself, climb, hold, decay, was structurally identical in both.
That's the decisive result. Demonstration seeding is neither the cause of the ramp nor the fix for it. It shows up with zero demo data and it shows up with demo data. Whatever's producing it lives somewhere in the base training setup, not in what's feeding the replay buffer.
Which sent me back to something I'd added two parts ago and hadn't really interrogated since. The comfort penalty from Part 7:
R_comfort = −0.1 × (a_steer,t − a_steer,t−1)² − 0.1 × (a_throt,t − a_throt,t−1)²
That term penalizes how fast the action changes, full stop, independent of whether the change was actually the correct thing to do given the road. An agent training under that penalty has a direct incentive to move smoothly and gradually between action values regardless of target. A slow climb, a plateau, a slow decay is close to exactly the shape you'd predict from a policy that's learned "large jumps are expensive" without a strong enough countervailing signal about where it should actually be steering toward. If the comfort penalty's gradient pressure is currently outweighing the still-forming road-reading signal coming from the critic, "smooth arcs regardless of geometry" is a locally attractive, easy-to-learn strategy to fall into. It's basically the same baseline-dominance problem from Phase I and II, just relocated from the value-magnitude domain into the temporal one.
train-v5.py is identical to v4, dueling critic, nav_info, demo-seeding, auto-alpha, gamma at 0.90, with exactly one change: comfort_penalty forced to 0.0, not just shrunk, fully zeroed out, specifically to keep this a clean single-variable test. Scaled back down to 100k steps, validation scale, so the answer comes back fast enough to actually inform the next move instead of costing another ten-hour wait for one data point.
The falsifiable prediction: if the comfort penalty is actually driving the ramp shape, removing it should produce steering that looks more like sharp, direct corrections tied to observable road geometry, probably noisier and more jerky moment to moment, since nothing's discouraging abrupt changes anymore, but responsive in a way the smooth arcs never were. If the ramp shape survives with the penalty at zero, that rules the comfort penalty out too, and the search moves somewhere else next, most likely the actor-critic learning rate balance, or the possibility that the CNN's spatial features still aren't discriminative enough between "road curves left" and "road curves right" at the resolution this whole pipeline is running at.
Run was still going as of writing this. Don't have the answer yet.
| Phase | What changed | What it fixed | What it didn't fix |
|---|---|---|---|
| IV (end-to-end visual) | Nav scaling ×0.1, extended to 100k steps | Bias-vector steering lock-in (-0.0102 constant) | Road-responsive steering under longer horizons |
| V-a (demo seeding) | 37.5k expert transitions via MetaDrive's built-in autopilot, seeded into replay buffer | Nothing measurable, episodes got shorter, not longer | Ramp-then-decay steering pattern, present with or without demos |
| V-b (in progress) | Comfort penalty forced to zero, single-variable test | (pending) | (pending) |
The pattern across this whole arc is starting to feel less like a series of separate bugs and more like one repeating shape. Every fix has been finding whatever shortcut is currently cheapest for the optimizer to fall into, a flat critic, a frozen bias vector, a temporal confound that turned out not to exist, and now possibly a smoothness penalty that's winning against a signal that hasn't caught up yet, and removing that one specific shortcut before checking what's underneath it. No conclusion yet on this one. That's genuinely where it stands.