After going through the self-driving landscape, Tesla, Waymo, BYD, and the rest, covered in the first two parts of this series, I spent a couple of weeks studying the architecture and approach companies like Tesla, BYD, Waymo, and Swaayatt took to get to the level they're at. This project was inspired by Swaayatt, who managed to build a self-driving car that hit Level 5 autonomy.
It made me think, how? On Indian roads of all places. Personally, living and driving on Bangalore roads is a near-impossible task even as a human, the unpredictability, the lack of road rules, the aggressive traffic, and so much more.
Looking further into Swaayatt, I understood that they use the concept of passive-aggressive driving. But before I could get into that, I had to learn reinforcement learning first, thanks to a pile of Stanford and MIT lectures on deep RL. There were three main components. What can the agent do? That's the action space. What can the agent see, or what state can it be in? That's the state space. And how do we tell if the decision the agent made was good? That's the reward system.
In theory I had it all down, but there was only one way to know if it actually worked, and that's where the 2D sim came into the picture. Starting off simple, I tested my agent on a straight road, and of course even that had its challenges. So let me start from the beginning.
Given a straight road, starting off with just two lanes, one-directional traffic, I asked myself three main questions.
What do I need the agent to do? Taking reference from how I drive and what I actually do behind the wheel, steering, accelerating, braking were the basic actions. I needed my agent to be able to move left or right, go straight, speed up, or slow down.
What does the agent need to observe or look at? What do I look at when I'm driving? I look at the road, the car in front if there is one, and if there's anything behind, obstacles. I act based on that.
And what do I reward it for? Not crashing, speeding up when it's safe to, overtaking, switching lanes, and so on.
Now that I had a brief idea about the components, what do I actually do with them?
That's where Markov Decision Processes (MDPs) came in. What is an MDP? It's the mathematical framework used to define the environment in RL. It's a 5-tuple system: (S, A, P, R, gamma).
S is all possible states or configurations the agent can be in. A is the set of all valid moves or decisions the agent can make. P is the transition probability, also known as the rules, for example, if the agent is in state x and takes action y, what's the probability it ends up in state z, that's the Markov property. R is the reward function, the numerical feedback the agent gets after an action.
State + action → next state + reward.
The goal was to maximize reward, so the agent has to think about long-term outcomes, not just the next step. That's where gamma comes in, the discount factor, ranging between 0 and 1, controlling how much the agent cares about distant future rewards versus immediate ones. 0 is short-sighted, 1 is long-sighted.
Once I had that working, I moved on to four lanes.
With four lanes, what the agent sees, the state space, changes. Not just the lanes to the right or left of the ego car, but the relative speed of the cars around it too. Rather than just checking distance, I needed the agent to know if the car ahead was speeding up or slowing down.
That's not all either, I needed the agent to observe itself: what speed the ego car was going, what lane it was in. There wasn't much else to explore here, the car just had to travel down a straight road, switching lanes when needed.
Moving on to junctions, there are a few possibilities when a car approaches one: take a U-turn, turn right or left, or go straight.
The car needed to understand it has to slow down and not crash while performing any of these, and not just that, the steering changes too, right?
That was my first thought, that the steering angle varies, and unlike switching lanes, the agent would need to execute a whole U-turn or a full right turn.
Except that's not actually true, and it took me a while to see why.
I sat with it and realised I was mixing up two different things. Turning at a junction and switching lanes are both just "steering left or right" at the end of the day, same wheel, same two hands. What's actually different is when the car does it and why. A human doesn't have a separate turning mechanism for junctions, you just use the same steering wheel you already have, you just know it's time to use it because you can see the junction coming up.
So the fix wasn't a bigger action space. It was giving the agent the same information a human already has: knowing a junction exists, and knowing which way it needs to go. Two new things to sense, not new things to do.
I added two values to what the agent sees, how far away the next junction is, and what turn it's supposed to make there, straight, left, right, or a U-turn. That last part matters because on my sim there's no real destination the agent is picking, I just assign it a route at the start of every episode, kind of like Google Maps already telling you to turn left in 200m instead of the car figuring out on its own where it wants to go. The car's whole job is to execute that safely, not decide it.
Steering stayed exactly the same, left, right, or maintain. Speed too. The agent just had to learn, from those two new numbers plus everything it already knew about the cars around it, when to get into the right lane, when to slow down, and how to time the turn without getting hit.
I genuinely wasn't sure this would work without a bigger action space. Tested it, and it did. I watched the car actually wait for a gap in oncoming traffic and then cut across once it was clear, one of those moments where you realise the model is doing something you didn't explicitly tell it to do, it just learned that waiting is sometimes the better move.
Along the way I caught two mistakes I made myself, worth mentioning because they're the kind of thing that doesn't throw an error, they just quietly produce the wrong answer.
First, I had the relative speed of a car behind me calculated the exact same way as a car in front of me. Sounds fine until you actually think about it. If a car ahead of you is going slower than you, you're closing in on it, that's the dangerous case. But if a car behind you is going slower than you, that's completely safe, they're falling back. Same formula, opposite meaning depending on direction. I had to flip the subtraction for anything behind the car so that "the gap is closing" always means the same thing everywhere, front, back, either side.
Second, and this one was dumber and took longer to catch. I set up a junction route thinking I'd built a right turn, car goes north into the junction then exits east. Except east isn't right, it's right on a map, not right from inside the car. If you're facing north and you turn toward east, that's a left turn from where you're sitting. I'd been reading the map like a bird instead of like a driver. Fixed the route, and once I thought about it properly it was obvious, but for a solid chunk of debugging I was convinced the code was broken when actually my own assumption was wrong.
By the end of the SUMO phase, I had a car that could handle a straight road full of genuinely chaotic traffic, some vehicles polite, some aggressive tailgaters, some crawling along, and a junction where it had to negotiate crossing traffic to turn. I ran it a bunch of times with randomised traffic each run so it wasn't just memorising one lucky scenario, and it held up.
But SUMO isn't a real driving simulator, it's a traffic flow simulator. No real physics, no cameras, no lidar, the car just knows exactly where everything is because the simulation tells it directly. Which is fine for figuring out the decision-making side of things, but it's not testing anything close to how a real self-driving car actually perceives the world. That's where the next part of this comes in.