{ "source_file": "0930", "source_platform": "Hashnode", "author": "Jeevika S", "title_listed": "Building an RL Traffic Agent for Indian Roads, Part 8: The Ramp That Wouldn't Go Away", "keywords": "", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://local-minima.hashnode.dev/building-an-rl-traffic-agent-for-indian-roads-part-8-the-ramp-that-wouldn-t-go-away", "publish_date": "2026-08-08", "corpus_stage": "", "notes": "", "id": "0a8e2a2fdb48", "collected_at": "2026-10-02T14:12:13", "title": "RL Traffic Agent, Part 8: The Ramp That Wouldn't Go Away", "body_markdown": "Building an RL Traffic Agent for Indian Roads, Part 8: The Ramp That Wouldn't Go Away\nUpdated\nAugust 8, 2026\n•\n10\nmin read\n•\nView as Markdown\nJ\nJeevika S\nDocumenting research and projects across deep learning and reinforcement learning\nPart of series\nSelf-driving cars\nPart 7 ended on a table that looked clean. Nav scaling fixed the steering lock-in, extending training to 100,000 steps gave the CNN room to actually learn road geometry, and every metric in that closing table moved in the right direction at once. It read like an ending.\nIt wasn't one. This part is what happened the moment I tried to scale that exact same fix past the point where it had only ever been tested.\nThe 300k Run: Same Symptom, Different Clothes\nThe obvious next question after Part 5 was whether the fix actually held, or whether it was a 100k-step-scale artifact that would fall apart the second I gave it more room to run. Same architecture, dueling critic, nav_info concatenated into the actor's input, auto-tuned alpha through the Lagrangian entropy trick, gamma still compressed to 0.90, just scaled up to 300,000 steps.\nIt didn't hold cleanly. To be fair, the actual Part 5 fix stuck, steering wasn't frozen at -0.0102 anymore. But a new pattern showed up in its place, and it was oddly specific. Steering would climb smoothly from near-zero, hold there for a stretch, plateau, then decay back down, a shape closer to a slow sine wave than anything resembling \"read the road, correct toward center.\"\nI checked this across six different scenario indices, different maps, different layouts entirely, and the same ramp-then-decay shape kept showing up in close variants across all of them. That consistency was the first real clue that whatever was driving this wasn't something specific to a given road. It was structural.\nWhich meant the framing from Part 4 needed correcting. Nav scaling and more training steps had solved bias-vector lock-in. That was real, and the table proved it. But bias-vector lock-in and actually reading the road are two different bars, and I'd only cleared the first one.\nHypothesis: Borrowing MetaDrive's Own Expert\nMy first read on the ramp was that the critic could tell actions apart now,\nprobe_critic_at_turn.py\nwas showing smooth, sensibly shaped Q-surfaces instead of the flat tabletop from Part 3, but maybe it still hadn't seen enough correctly-labeled turning behavior to actually generalize \"nav_command says left, so steer left\" purely from self-play. Early self-play is mostly incompetent exploration. If a critic trained almost entirely on bad turns is the thing shaping the actor, that's a version of the classic spurious-correlation trap, the model latches onto whatever correlate happens to be lying around, elapsed time, distance since reset, instead of the actual causal signal it's supposed to be using.\nI considered scripting a rule-based driver by hand for straight sections, turns, and junctions separately, and decided against it almost immediately. MetaDrive already ships a working expert autopilot,\nenv.agent.expert_takeover = True\n, IDM-style, competent at lanes, turns, and traffic. Using it meant zero new controller logic to write or debug. Just let it drive, and record.\nTwo Bugs Before the Expert Would Actually Drive\nGetting that flag to do anything useful took longer than it should have, and both problems were found by actually reading output, not by guessing.\nFirst attempt,\nexpert_takeover = True\non its own did nothing.\ninfo[\"action\"]\nkept handing back my own placeholder\n[0.0, 0.0]\n, unchanged, step after step. Turned out\nmanual_control\nhas to be set to\nTrue\nin the env config for the takeover flag to actually engage at all. Without it, the flag is silently inert, no error, no warning, it just quietly doesn't do anything.\nSecond bug, once the car was genuinely under\nManualControlPolicy\n, the real action being executed wasn't sitting under the\n\"action\"\nkey like I'd assumed. It was under\n\"raw_action\"\n, a tuple of\nnp.float32\nvalues. Only found that by dumping the entire\ninfo\ndict and reading every key in it instead of guessing which one was likely to hold the answer, which is a lesson I seem to relearn every few phases of this project.\nThe Collection Pipeline\ncollect_demonstrations.py\n: reset with\nmanual_control=True\n,\nnum_scenarios=200\nfor variety, force\nexpert_takeover=True\nafter every reset, then step through full episodes until\nterminated\nor\ntruncated\nfires. Pull\nraw_action\nat each step and store the full transition,\n(obs_grid, obs_num_others, obs_nav_info, action, reward, next_obs_grid, next_obs_num_others, next_obs_nav_info, terminated, truncated)\n, shaped exactly the way the real replay buffer expects it. All of it serialized out to\ndemonstrations.pkl\n.\nFifty episodes came out to 37,503 transitions total, verified after the fact rather than assumed.\ntrain-v4.py\nis identical to v3 except for one addition, a\nload_demonstrations()\ncall right after the replay buffer gets instantiated, run once before the main loop starts, iterating the pickled transitions and pushing each one in through the exact same\n.add()\ncall the live loop uses later. One deliberate simplification here: demo transitions aren't protected from being overwritten once the circular buffer cycles past capacity during training. I thought about building a reserved region for them and decided against it for this first pass, getting an honest read on whether seeding helped at all mattered more right now than optimizing how it was seeded.\nRan it first at 100k steps as a validation pass, then extended the same setup out to 500k to see if it held at a longer horizon.\nThe Result: Shorter Episodes, Same Ramp\nDemo-seeding didn't fix anything. Worse, by one metric it made things measurably worse. Episodes in the demo-seeded run were ending after 25 to 49 steps of\nout_of_road\n, compared to 85-plus steps in the equivalent v3 run with zero demo data. And the ramp-then-decay steering shape was still right there, now showing up at a strikingly consistent point, a sharp correction landing around step 20 to 21, run after run, regardless of which map it was on.\nThat consistency was the actually interesting part. It smelled exactly like the Husky-versus-wolf problem, a model technically solving the training objective by keying off a correlate that happens to be available, instead of the causal signal it's actually supposed to be reading.\nTwo Checks to Rule Out the Demo Data Itself\nRather than keep training and hoping the next run looked different, I ran two direct, falsifiable checks on the demonstration data.\nLabel balance.\nTallied steering sign across all 37,503 transitions. 51.4% left, 48.6% right, mean steering sitting at -0.008. Essentially balanced. Ruled out demo-label imbalance as the cause outright.\nTemporal clustering.\nIf \"meaningful turn\" moments, anything with\n|steering| > 0.1\n, clustered tightly around a specific step-count-since-reset across episodes, that would be direct evidence of a temporal confound baked into the demo data itself, most demo episodes just happening to hit their first real turn around a similar point. Measured mean step-since-reset at those moments: 257.99, standard deviation 142.03. That's a wide spread, not a cluster. Ruled that out too.\nBoth hypotheses about the demonstration data were cleanly falsified by actual measurement. Good process. Wrong location.\nThe Actual Tell\nWent back and compared a v3 run, no demo data at all, 300k steps, directly against the demo-seeded 500k run, steering value by steering value. Both showed the same shape, a climb from near-zero up to a plateau around 0.13 to 0.15, a brief hold, then a smooth decay. The absolute step where it happened differed, v3 plateaued around step 55 to 56, the demo-seeded run around step 20, but the shape itself, climb, hold, decay, was structurally identical in both.\nThat's the decisive result. Demonstration seeding is neither the cause of the ramp nor the fix for it. It shows up with zero demo data and it shows up with demo data. Whatever's producing it lives somewhere in the base training setup, not in what's feeding the replay buffer.\nCurrent Hypothesis: The Comfort Penalty Might Be the Real Author\nWhich sent me back to something I'd added two parts ago and hadn't really interrogated since. The comfort penalty from Part 7:\nR_comfort = −0.1 × (a_steer,t − a_steer,t−1)² − 0.1 × (a_throt,t − a_throt,t−1)²\nThat term penalizes how fast the action changes, full stop, independent of whether the change was actually the correct thing to do given the road. An agent training under that penalty has a direct incentive to move smoothly and gradually between action values regardless of target. A slow climb, a plateau, a slow decay is close to exactly the shape you'd predict from a policy that's learned \"large jumps are expensive\" without a strong enough countervailing signal about where it should actually be steering toward. If the comfort penalty's gradient pressure is currently outweighing the still-forming road-reading signal coming from the critic, \"smooth arcs regardless of geometry\" is a locally attractive, easy-to-learn strategy to fall into. It's basically the same baseline-dominance problem from Phase I and II, just relocated from the value-magnitude domain into the temporal one.\nThe Test in Progress\ntrain-v5.py\nis identical to v4, dueling critic, nav_info, demo-seeding, auto-alpha, gamma at 0.90, with exactly one change:\ncomfort_penalty\nforced to\n0.0\n, not just shrunk, fully zeroed out, specifically to keep this a clean single-variable test. Scaled back down to 100k steps, validation scale, so the answer comes back fast enough to actually inform the next move instead of costing another ten-hour wait for one data point.\nThe falsifiable prediction: if the comfort penalty is actually driving the ramp shape, removing it should produce steering that looks more like sharp, direct corrections tied to observable road geometry, probably noisier and more jerky moment to moment, since nothing's discouraging abrupt changes anymore, but responsive in a way the smooth arcs never were. If the ramp shape survives with the penalty at zero, that rules the comfort penalty out too, and the search moves somewhere else next, most likely the actor-critic learning rate balance, or the possibility that the CNN's spatial features still aren't discriminative enough between \"road curves left\" and \"road curves right\" at the resolution this whole pipeline is running at.\nRun was still going as of writing this. Don't have the answer yet.\nTable, Continued From Part 7\nPhase\nWhat changed\nWhat it fixed\nWhat it didn't fix\nIV (end-to-end visual)\nNav scaling ×0.1, extended to 100k steps\nBias-vector steering lock-in (-0.0102 constant)\nRoad-responsive steering under longer horizons\nV-a (demo seeding)\n37.5k expert transitions via MetaDrive's built-in autopilot, seeded into replay buffer\nNothing measurable, episodes got shorter, not longer\nRamp-then-decay steering pattern, present with or without demos\nV-b (in progress)\nComfort penalty forced to zero, single-variable test\n(pending)\n(pending)\nThe pattern across this whole arc is starting to feel less like a series of separate bugs and more like one repeating shape. Every fix has been finding whatever shortcut is currently cheapest for the optimizer to fall into, a flat critic, a frozen bias vector, a temporal confound that turned out not to exist, and now possibly a smoothness penalty that's winning against a signal that hasn't caught up yet, and removing that one specific shortcut before checking what's underneath it. No conclusion yet on this one. That's genuinely where it stands.\n#\nreinforcement-learning\n#\ndeep-learning\n#\nself-driving-cars\n#\nai\n4\nviews", "metadata": { "title": "RL Traffic Agent, Part 8: The Ramp That Wouldn't Go Away", "description": "The steering lock-in was fixed. Then a new pattern showed up, a slow climb, hold, and decay, consistent across every map.", "keywords": null, "author": null, "og:title": "RL Traffic Agent, Part 8: The Ramp That Wouldn't Go Away", "og:description": "The steering lock-in was fixed. Then a new pattern showed up, a slow climb, hold, and decay, consistent across every map.", "og:url": "https://local-minima.hashnode.dev/building-an-rl-traffic-agent-for-indian-roads-part-8-the-ramp-that-wouldn-t-go-away", "og:site_name": "Local Minima", "og:image": "https://local-minima.hashnode.dev/api/og/post?og=eyJhdXRob3IiOiJKZWV2aWthJTIwUyIsImlzRGVmYXVsdE1vZGVEYXJrIjpmYWxzZSwicmVhZFRpbWUiOjEwLCJjb21tZW50cyI6MCwicmVhY3Rpb25zIjowLCJkb21haW4iOiJsb2NhbC1taW5pbWEuaGFzaG5vZGUuZGV2IiwidGl0bGUiOiJCdWlsZGluZyUyMGFuJTIwUkwlMjBUcmFmZmljJTIwQWdlbnQlMjBmb3IlMjBJbmRpYW4lMjBSb2FkcyUyQyUyMFBhcnQlMjA4JTNBJTIwVGhlJTIwUmFtcCUyMFRoYXQlMjBXb3VsZG4ndCUyMEdvJTIwQXdheSIsInBob3RvIjoiaHR0cHM6Ly9saDMuZ29vZ2xldXNlcmNvbnRlbnQuY29tL2EvQUNnOG9jS05YQkFHSHZTZlltVVJYNXc0M3pzU2V0NHVqcTI5djIwelVOMl9qOEZYWk03anl3PXM5Ni1jIiwibG9nbyI6Imh0dHBzOi8vY2RuLmhhc2hub2RlLmNvbS9yZXMvaGFzaG5vZGUvaW1hZ2UvdXBsb2FkL3YxNTkzNjgwMjgyODk2L2tOQzdFOElSNC5wbmciLCJmYXZpY29uIjoiaHR0cHM6Ly9jZG4uaGFzaG5vZGUuY29tL3Jlcy9oYXNobm9kZS9pbWFnZS91cGxvYWQvdjE1OTM2ODAyODI4OTYva05DN0U4SVI0LnBuZyIsInB1Ymxpc2hlZEF0IjoiMjAyNi0wOC0wOFQwNDo0NzoxOS43ODVaIiwidXBkYXRlZEF0IjoiMjAyNi0wOC0wOFQwNDo0NzoxOS43ODVaIn0=", "og:image:width": "1200", "og:image:height": "630", "og:type": "article", "twitter:card": "summary_large_image", "twitter:title": "RL Traffic Agent, Part 8: The Ramp That Wouldn't Go Away", "twitter:description": "The steering lock-in was fixed. Then a new pattern showed up, a slow climb, hold, and decay, consistent across every map.", "twitter:image": "https://local-minima.hashnode.dev/api/og/post?og=eyJhdXRob3IiOiJKZWV2aWthJTIwUyIsImlzRGVmYXVsdE1vZGVEYXJrIjpmYWxzZSwicmVhZFRpbWUiOjEwLCJjb21tZW50cyI6MCwicmVhY3Rpb25zIjowLCJkb21haW4iOiJsb2NhbC1taW5pbWEuaGFzaG5vZGUuZGV2IiwidGl0bGUiOiJCdWlsZGluZyUyMGFuJTIwUkwlMjBUcmFmZmljJTIwQWdlbnQlMjBmb3IlMjBJbmRpYW4lMjBSb2FkcyUyQyUyMFBhcnQlMjA4JTNBJTIwVGhlJTIwUmFtcCUyMFRoYXQlMjBXb3VsZG4ndCUyMEdvJTIwQXdheSIsInBob3RvIjoiaHR0cHM6Ly9saDMuZ29vZ2xldXNlcmNvbnRlbnQuY29tL2EvQUNnOG9jS05YQkFHSHZTZlltVVJYNXc0M3pzU2V0NHVqcTI5djIwelVOMl9qOEZYWk03anl3PXM5Ni1jIiwibG9nbyI6Imh0dHBzOi8vY2RuLmhhc2hub2RlLmNvbS9yZXMvaGFzaG5vZGUvaW1hZ2UvdXBsb2FkL3YxNTkzNjgwMjgyODk2L2tOQzdFOElSNC5wbmciLCJmYXZpY29uIjoiaHR0cHM6Ly9jZG4uaGFzaG5vZGUuY29tL3Jlcy9oYXNobm9kZS9pbWFnZS91cGxvYWQvdjE1OTM2ODAyODI4OTYva05DN0U4SVI0LnBuZyIsInB1Ymxpc2hlZEF0IjoiMjAyNi0wOC0wOFQwNDo0NzoxOS43ODVaIiwidXBkYXRlZEF0IjoiMjAyNi0wOC0wOFQwNDo0NzoxOS43ODVaIn0=", "article:published_time": "2026-08-08T04:47:19.785Z", "article:modified_time": "2026-08-08T04:47:19.785Z", "article:author": "https://local-minima.hashnode.dev/@jeevikasrinath" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://local-minima.hashnode.dev/building-an-rl-traffic-agent-for-indian-roads-part-8-the-ramp-that-wouldn-t-go-away.md", "label": "View as Markdown", "type": "a" }, { "href": "https://hashnode.com/@jeevikasrinath", "label": "J", "type": "a" }, { "href": "https://www.linkedin.com/in/jeevika-s-82ab66325/", "label": "", "type": "a" }, { "href": "https://local-minima.hashnode.dev/series/self-driving-rl", "label": "Part of series Self-driving cars", "type": "a" }, { "href": "https://local-minima.hashnode.dev/tag/reinforcement-learning", "label": "# reinforcement-learning", "type": "a" }, { "href": "https://local-minima.hashnode.dev/tag/deep-learning", "label": "# deep-learning", "type": "a" }, { "href": "https://local-minima.hashnode.dev/tag/self-driving-cars", "label": "# self-driving-cars", "type": "a" }, { "href": "https://local-minima.hashnode.dev/tag/ai", "label": "# ai", "type": "a" } ], "retry_at": "2026-10-03T08:10:02", "repair_version": 2, "repaired_at": "2026-10-14T10:51:01Z", "source_url": "https://local-minima.hashnode.dev/building-an-rl-traffic-agent-for-indian-roads-part-8-the-ramp-that-wouldn-t-go-away", "papers": [], "actual_video_files": [], "body_chars": 11711, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/0a8e2a2fdb48/pages/source.html", "local_html": "attachments/0a8e2a2fdb48/pages/article.html", "body_base_path": "attachments/0a8e2a2fdb48/pages/", "body_markdown_path": "attachments/0a8e2a2fdb48/pages/article.md" }