After going through the theory, I didn't want to just jump into code. I wanted a base to build on first, a shape for the whole thing before I touched a single line of Python.
So the first question I had to answer, before making any assumptions, was simple: what does MetaDrive actually hand you by default? I knew it was LIDAR-based, raw distance readings fanning out around the car. But raw readings aren't a decision. Something has to turn them into one.
That left me with two options. Let the network figure the geometry out on its own, straight from the raw scan. Or process everything down into a clean set of, say, sixteen state values first.
Option two sounds smarter on paper, right? Structured, tidy, easier to reason about. Except Indian roads don't give you lanes to structure around. There's no clean line to snap a state vector to. So option one it was, the messier, dynamic one, because it's the only one that doesn't assume a road that doesn't exist.
Bear with me here, there's going to be a lot of buts and what-ifs.
Going with the raw route meant the network had to figure out the geometry itself. And that's where the curse of dimensionality showed up, along with what I've started calling exponential data hunger. With sixteen clean states, the state space stays tight and manageable. Feed it raw LIDAR instead, and that space explodes into the thousands. Which means you lean harder on a deep neural network to make sense of it. But a network that size doesn't learn a workable policy in a thousand steps. Not even close.
So DQN and PPO were out before I'd really started.
Then there was representation learning. Before the model can learn anything about driving, it first has to learn what an edge is, what an object is, basically relearn the world from scratch. And that means every time the car hits new terrain, it's back to square one, re-learning structure it should already know. Which defeats the entire point of a generalizable driving agent.
On top of that: a brutal credit assignment problem. If the car hits a wall, it has no idea which of the last however-many actions actually caused it. Was it the steering three steps back? The braking before that? No signal tells you.
So, how do you fix all of that at once?
State abstraction. Instead of raw points, work off a 2D bird's-eye view, an occupancy grid. Compress the chaos into something spatial and structured before the network ever sees it.
A specialized architecture: CNN plus self-attention. A plain multi-layer perceptron just won't cut it here, there's too much spatial structure for a flat network to hold onto. But run the data through convolutional layers alongside self-attention and you get something workable. The attention layer flags what actually matters locally, a vehicle closing in fast, say. The CNN then compresses the broader spatial picture into a dense, lower-dimensional vector. From there, the agent, whether it's PPO or DQN, operates on that latent vector instead of wrestling with the raw point cloud directly.
That's when something I'd read earlier came back to me, and led straight into the next question.
What about actor-critic? (Spoiler: this is the one that actually works.)
It handles high-dimensional state spaces well, which tracks, since that's exactly what a chaotic LIDAR point cloud is. It solves the continuous control problem too, the steering wheel being the obvious example. And it tames training instability through bootstrapping, which lets you actually trace which action led to which reward. That's my wall-collision problem, solved. Add baseline subtraction on top and you're not fighting numerical instability either.
Is that the whole story? No. Unfortunately, no.
Because now you have to pick which actor-critic algorithm to actually use:
Deep Deterministic Policy Gradient (DDPG)
Twin Delayed DDPG (TD3)
Soft Actor-Critic (SAC)
I'll get into each of those properly in a bit. First, another fork in the road.
Discrete or continuous? Worth flagging: SUMO had me working in discrete action space. MetaDrive's data isn't built that way. Steering angle, throttle, brake pressure, all of it is continuous by default.
And you can't just bolt a value-based policy like DQN onto continuous actions. Think about what DQN actually needs to do: pick the action with the highest expected reward, which means computing the argmax of a Q-function. Fine in a discrete space where you can just check every option. In a continuous one, there is no fixed set of options to check, the argmax itself becomes an optimization problem you'd need to solve at every single step. That's not a policy, that's a bottleneck.
Actor-critic sidesteps that entirely, which is exactly why it kept coming back around.
The discrete path isn't dead on arrival either, to be fair. You don't have to go continuous. Dueling DQN improves on the base architecture by splitting evaluation in two: one stream estimates how good it is to just be in a given state, the other calculates the specific advantage of a particular action from there.
So before deciding, I ran the actual trade-off. Discrete: if your intervals are too coarse, the controls get jerky and the car crashes at higher speeds. Make the intervals finer and the number of possible action combinations explodes, which slows Q-learning down to a crawl. Continuous: heavier training instability, and hyperparameter tuning that gets complicated fast.
Then a dumb but honestly useful question hit me: why wasn't SUMO jerky in the first place? Because SUMO was never a high-fidelity vehicle dynamics simulator to begin with, it's a traffic flow simulator. It didn't need to be smooth at the wheel level. Translate that same discrete logic into something meant to resemble a real car, though, and it falls apart. Take -1 for full left and +1 for full right and the steering only ever sits at the extremes. No in-between. What you actually need is finer resolution, real control, not a switch.
Can discrete threading actually get a car through real traffic? No.
Which brings it back to what MetaDrive expects by default anyway: a 2D continuous vector, a float between -1 and 1 for steering. It runs on a physics engine built around continuous tire friction and momentum. Force discrete steps into that and you get artificial jerkiness that fights the physics rather than working with it, and at speed, that's the kind of thing that destabilizes the whole vehicle.
Conclusion: continuous it is.
Before I get into why I landed on SAC specifically, there's one more problem worth pulling apart, because I actually mislabeled it at first. I called it an "RL algorithm compute problem." It isn't. It's a perception network compute problem.
Here's the distinction: whichever actor-critic algorithm I end up choosing, TD3 or SAC, doesn't change the fact that I still need a feature extractor my CPU can actually run in a reasonable time frame. No dedicated GPU here. So the fix isn't in the RL algorithm at all, it's upstream. Break the raw data down into something abstract, a 2D bird's-eye grid instead of a full point cloud, and lean on optimization tricks like model pruning to keep it lean.
Step one, then, isn't picking SAC or TD3. It's building a genuinely lightweight network for the perception layer, something small enough to actually run on what I've got.
Three actor-critic candidates, and I went through them one at a time.
DDPG. How it works: it takes deep Q-learning and drags it into continuous control, merging the strengths of deep learning with policy gradients to output a specific continuous action directly. On the plus side, it's sample efficient and gives clean gradient signals. On the downside, it's known for overestimation bias, the Q-value estimates run high, and that instability means you basically need a dual critic setup just to keep it stable.
TD3, Twin Delayed DDPG. Built directly on top of DDPG to fix that overestimation problem. The core trick is the "twin" structure: two independent critic networks evaluate the actor's action, and you take the minimum of the two instead of trusting either one alone. It also delays the actor's policy updates so they lag behind the critic, which helps stability. Pros: stable, converges well, fast at inference. The catch is that it's still deterministic at heart, and deterministic policies tend to struggle in chaotic, highly dynamic environments. Which, given what I'm building for, is basically the whole job description.
SAC, Soft Actor-Critic. This one solves a different failure mode entirely, an agent converging prematurely into a narrow policy and never really exploring past it. It optimizes for both maximum expected reward and maximum policy entropy at the same time, entropy regularization, so the agent stays a little uncertain on purpose instead of locking in early. The cost is higher computational overhead.
So, TD3 or SAC? SAC made more intuitive sense for the actual driving problem, but I didn't want to pick it just because it felt right. I wanted to know where that overhead was actually coming from before committing.
Turns out both algorithms share the same underlying structure, dual networks either way. The overhead isn't in the architecture or the input. It's in the output. TD3 outputs a single, exact continuous action. SAC outputs a full continuous probability distribution instead, and sampling from that costs more. But that extra cost is negligible in absolute terms. It only becomes a real bottleneck if you're running something like PPO on a dedicated GPU at scale, not for an actor-critic setup running on CPU. So: SAC it is.
Which meant the real bottleneck was never the RL algorithm to begin with. I'd been calling it an "RL algorithm compute problem," but the honest name for it is a perception network compute problem. Whichever algorithm I picked, TD3 or SAC, I still needed a feature extractor my CPU could run in a reasonable time frame. So step one wasn't the RL side at all. It was designing a genuinely lightweight CNN for perception, and figuring out exactly how lightweight it could get before it stopped seeing anything useful.
A few concrete moves here, each one trading raw capacity for speed:
Drop self-attention. Just abandon the attention layers entirely for the base network. They're expensive, and expensive is the one thing a CPU-only pipeline can't afford by default.
Keep the convolutional depth shallow. Two to three layers, max. The goal isn't a rich hierarchical feature extractor, it's just enough to pull out the macroscopic geometry: road boundaries, obstacle shapes, nothing fussier than that.
Downsample aggressively. Large strides or max-pooling right after the initial convolutions, shrinking the spatial dimensions fast so the CPU isn't pushing floating-point operations through a huge feature map at every step.
Go coarse on input resolution. A 256×256 grid is overkill. Bring it down to 64×64, or even a 32×32 binary grid, and you've cut the workload dramatically without losing the information that actually matters for driving decisions.
Which gave me a rough final shape: MetaDrive's raw LIDAR gets manually projected into a low-resolution bird's-eye occupancy grid, that grid gets fed into a shallow, downsampled CNN, and the CNN compresses the spatial geometry into a dense one-dimensional latent vector. That latent vector gets handed to SAC's actor and critic heads, which evaluate the state and output continuous steering and throttle commands.
But I'd dropped attention entirely to get there, and that came back to bite me almost immediately.
With only two or three conv layers, the effective receptive field, meaning the total area of the original input that actually influences a single node deep in the network, stays small. On a 64×64 grid, that works out to something like a 10×10 local patch. The network can see what's directly around a point. It can't see the whole board at once. Which means it literally cannot calculate that a vehicle three lanes over is closing in fast, because that information never reaches the same part of the network at the same time. I needed something in between full self-attention and no attention at all.
The middle option: place attention strategically instead of everywhere. Let the shallow CNN do its job compressing the 64×64 grid down to something like an 8×8 feature map first, and only then apply self-attention, on that already-compressed representation. At that resolution, the attention layer only has to reason about relationships between 64 elements instead of thousands, which is cheap enough to actually run in real time, and it's exactly what lets the model "thread through traffic" instead of only reacting to whatever's immediately adjacent.
I still wanted to check the assumption that attention is expensive before committing to skipping it everywhere. Computing self-attention on a raw 64×64 grid directly would absolutely stall a CPU control loop. Computing it on an already-compressed 8×8 feature map is a different story entirely.
Final call: a middle-ground architecture. Build a shallow CNN to rapidly downsample the grid, then attach a single self-attention head to the compressed feature map so the agent can reason across distant gaps in the scene, while staying inside a budget that actually fits a real-time CPU loop.
Theory's one thing. The first real step was turning MetaDrive's raw output into a usable BEV grid, and before writing any of that, I wanted to actually see what MetaDrive's LIDAR output looks like structurally, not just conceptually.
I ran a quick script to read the raw observation and just looked at it. Three sensors were active: lidar, side_detector, and lane_line_detector. What came back was a 256-value array, every value between 0 and 1. So the float values were some combination of all three sensors mixed together, and my first real task was just isolating the LIDAR portion from that combined vector.
Digging into MetaDrive's observation code led me to two relevant classes: StateObservation and LidarObservation.
StateObservation.observe() builds a state vector out of the ego state (nine-plus values: speed, steering, heading difference, and so on) concatenated with navigation info (ten values covering checkpoint distance and direction).
LidarObservation.observe() builds on top of that. Its final vector is the state (ego state plus navigation) concatenated with other-vehicle info, meaning nearby vehicle data plus the raw LIDAR points themselves.
The documentation claims 275 total values. I was getting 259. A sixteen-value gap. My first guess was that the missing sixteen was the "other vehicle info" block, so I went digging through the config to check the defaults. Turned out num_lasers = 240 and num_others = 0, meaning zero nearby-vehicle slots were being included at all. 240 plus the 19 remaining values, nine for ego state and ten for navigation, lines up exactly to 259.
So the 259-length vector breaks down as:
[0:9] → ego vehicle state (speed, steering, and so on)
[9:19] → navigation info (checkpoint distance and angle)
[19:259] → 240 raw LIDAR distance readings, sweeping clockwise starting from the vehicle's heading
Now the real question: what do I do with num_others sitting at zero?
Option one, turn it on and use the structured vehicle info directly. Tempting, but it defeats the entire point of the project. If I hand the network relative speed and object identification for free, that's not perception, that's a cheat sheet. A real sensor gives you detailed contextual information the hard way. Using num_others as a shortcut means the model never actually learns to work with an unstructured environment, which is the whole reason this project exists in the first place.
Option two, pure LIDAR only. The problem there is you genuinely can't distinguish a car from a wall using raw distance readings alone. Same signal, different object.
So if I do end up using num_others, there are two ways to structure it. A purist end-to-end approach, strictly LIDAR, full stop, which pushes training time and compute cost up hard. Or an asymmetric actor-critic setup, learning with privileged information, where I use the num_others ground truth without actually violating the constraint I set for myself. The actor network receives only the raw stacked LIDAR occupancy grid and never sees num_others at all. The critic network receives the raw LIDAR plus the perfectly clean num_others array. The actor stays honest to a real sensor's limitations. The critic gets to be the all-seeing teacher grading its decisions. That approach works cleanly.
Before committing, I had to check something: does SAC's standard formulation even support different inputs for the actor versus the critic? Looking at the underlying Markov decision process structure, the actor-critic architecture already maintains two separate functional representations by design. The actor is an explicit policy, π_θ(a|s), mapping a state to a probability distribution over the action space. The critic is a value function, Q_φ(s,a), mapping a state-action pair to a real number representing expected cumulative reward. Nothing in that structure requires the actor and critic to be looking at the same input. So asymmetry isn't a hack bolted onto SAC, it fits the existing shape of the algorithm. It just meant restructuring my pipeline to actually take advantage of it.
A few concrete changes fell out of that:
Redesigning the replay buffer. A standard SAC buffer stores transitions as (state, action, reward, next_state, done). To support asymmetry, that has to expand and decouple the observable state from the privileged one: (obs_actor, obs_critic, action, reward, next_obs_actor, next_obs_critic, done), where obs_actor is the raw LIDAR occupancy grid and obs_critic is that same LIDAR grid plus the clean 16-value num_others array.
Splitting the network input dimensions. The actor network's input size is just the LIDAR latent dimension. The critic networks take a wider input: LIDAR latent dimension plus num_others dimension plus action dimension.
The critic's forward pass, step by step. Think of it almost like a blind student and a sighted teacher. The target actor is the blind student, asked to rely only on next_obs_actor (raw LIDAR) and make a decision from that alone. The target critic is the teacher who can see everything, it takes next_obs_critic (LIDAR plus the clean vehicle data) along with the actor's proposed action and estimates the future value of that decision. Since the critic has the full picture, it computes an accurate target Q-value, the correct grade for that action. The loss update then compares that target against the critic's own earlier estimate and nudges the critic closer to the accurate target.
The actor's forward pass. Because the critic has access to num_others, its Q-value estimates come out sharp, stable, and accurate, which means the gradient signal it feeds back to the actor is precise about exactly why a given steering angle was good or bad. The actor never sees the privileged data directly. It just gets better at acting on what it does have, guided by a critic that understands the scene more fully than it ever will.
The core formulas, for the record:
Target Q-value:target_Q = reward + γ × (min(target_critic1(next_obs_critic, a′), target_critic2(next_obs_critic, a′)) − α × log_prob)
Both target critics evaluate the proposed next action, a′, and we take the minimum of the two to correct for the overestimation problem TD3 first identified. Entropy, the mathematical measure of spread in a probability distribution, gets folded in through α × log_prob, controlling how much the update prioritizes exploration. That combined target is what trains the two primary critic networks, compared against their current Q-value estimate to check how accurate they are.
Actor loss:loss_actor = α × log_prob − min(critic1(obs_critic, a), critic2(obs_critic, a))
The actor is maximizing the negative of that minimum critic estimate, in effect getting graded conservatively on safety, since the critic always takes the more cautious of its two estimates. The +α × log_prob term maximizes entropy on top of that, actively pushing the actor to keep exploring rather than settling too early.
A few smaller details worth nailing down while I was at it: a′ has to be sampled from the target actor network, and that network only ever sees next_obs_actor, never the privileged data. The current action a, meanwhile, comes from the live actor network through the reparameterization trick, a = μ + σ × ε. And obs_critic should never require gradients, it has to stay a frozen input tensor, since it's ground-truth context, not something the network should be learning to reshape.
Small detail, but it mattered. MetaDrive returns a normalized reading of 1.0 whenever there's nothing within the ray's 50-meter range, the ray just never bounces back. Read naively, the model could end up interpreting that as the car being boxed in on every side, trapped in a cylinder of walls that don't actually exist, since "maximum distance" and "wall right here" can end up looking suspiciously similar once everything's normalized between 0 and 1.
The fix: filter out any ray returning 0.99 or higher and treat it as empty space instead, dropping it to 0.0 in the occupancy matrix so open space actually reads as open. I also considered just binarizing the whole grid to 1s and 0s at that point, occupied or not, and confirmed it doesn't break anything downstream either.
That's Phase 1: the reasoning that got me from "what does MetaDrive even give me by default" to a working occupancy grid pipeline and an asymmetric SAC setup that respects the constraints I set for the project. Phase 2 picks up from the actual training run, and everything that broke along the way.