{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Dwarkesh Podcast",
  "title_listed": "Leopold Aschenbrenner on AI and Compute",
  "keywords": "Leopold,计算,AI",
  "domain": "大模型算法",
  "content_type": "访谈",
  "is_paid_flag": "否",
  "url": "https://www.dwarkeshpatel.com/p/leopold-aschenbrenner",
  "publish_date": "2024-01",
  "corpus_stage": "预训练/SFT",
  "notes": "Leopold谈AI与算力",
  "id": "cfae7be5aa63",
  "collected_at": "2026-10-02T15:12:03",
  "title": "Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history",
  "body_markdown": "Chatted with my friend\nLeopold Aschenbrenner\nabout the trillion dollar cluster, unhobblings + scaling = 2027 AGI, CCP espionage at AI labs, leaving OpenAI and starting an AGI investment firm, dangers of outsourcing clusters to the Middle East, & The Project.\nRead the new essay series from Leopold this episode is based on\nhere\n.\nWatch on\nYouTube\n. Listen on\nApple Podcasts\n,\nSpotify\n, or any other podcast platform. Read the full transcript\nhere\n.\nFollow\nme on Twitter\nfor updates on future episodes. Follow\nLeopold on Twitter\n.\nTimestamps\n(00:00:00) –\nThe trillion-dollar cluster and unhobbling\n(00:20:31) –\nAI 2028: The return of history\n(00:40:26) –\nEspionage & American AI superiority\n(01:08:20) –\nGeopolitical implications of AI\n(01:31:23) –\nState-led vs. private-led AI\n(02:12:23) –\nBecoming Valedictorian of Columbia at 19\n(02:30:35) –\nWhat happened at OpenAI\n(02:45:11) –\nAccelerating AI research progress\n(03:25:58) –\nAlignment\n(03:41:26) –\nOn Germany, and understanding foreign perspectives\n(03:57:04) –\nDwarkesh’s immigration story and path to the podcast\n(04:07:58) –\nLaunching an AGI hedge fund\n(04:19:14) –\nLessons from WWII\n(04:29:08) –\nCoda: Frederick the Great\nTranscript\nEdited by\nTeddy Kim\n.\n(00:00:00) – The trillion-dollar cluster and unhobbling\nDwarkesh Patel\n00:00:00\nToday I’m chatting with my friend\nLeopold Aschenbrenner\n. He grew up in Germany and graduated as valedictorian of Columbia when he was 19. After that, he had a very interesting gap year which we’ll talk about. Then, he was on the\nOpenAI superalignment\nteam, may it rest in peace.\nNow, with some anchor investments — from\nPatrick\nand\nJohn Collison\n,\nDaniel Gross\n, and\nNat Friedman\n— he is launching an investment firm.\nLeopold, you’re off to a slow start but life is long. I wouldn’t worry about it too much. You’ll make up for it in due time. Thanks for coming on the podcast.\nLeopold Aschenbrenner\n00:00:38\nThank you. I first discovered your podcast when your best episode had a couple of hundred views. It’s been amazing to follow your trajectory. It’s a delight to be on.\nDwarkesh Patel\n00:00:48\nIn the\nSholto and Trenton episode\n, I mentioned that a lot of the things I’ve learned about AI I’ve learned from talking with them. The third, and probably most significant, part of this triumvirate has been you. We’ll get all the stuff on the record now.\nHere’s the first thing I want to get on the record. Tell me about the trillion-dollar cluster.\nI should mention this for the context of the podcast. Today you’re releasing a series called Situational Awareness. We’re going to get into it. First question about that is, tell me about the trillion-dollar cluster.\nLeopold Aschenbrenner\n00:01:20\nUnlike most things that have recently come out of Silicon Valley, AI is an industrial process. The next model doesn’t just require some code. It’s building a giant new cluster. It’s building giant new power plants. Pretty soon, it’s going to involve building giant new fabs.\nSince\nChatGPT\n, this extraordinary techno-capital acceleration has been set into motion. Exactly a year ago today, Nvidia had their\nfirst blockbuster earnings call\n. It went up 25% after hours and everyone was like, \"oh my God, AI is a thing.\" Within a year, Nvidia data center revenue has gone from a few billion a quarter to $25 billion a quarter and continues to go up. Big Tech capex is skyrocketing.\nIt’s funny. There’s this crazy scramble going on, but in some sense it’s just the continuation of straight lines on a graph. There’s this long-run trend of almost a decade of training compute for the largest AI systems growing by about half an order of magnitude, 0.5 OOMs a year.\nJust play that forward.\nGPT-4\nwas reported to have finished\npre-training\nin 2022. On\nSemiAnalysis\n, it was rumored to have a cluster size of about 25,000\nA100s\n. That’s roughly a $500 million cluster. Very roughly, it’s 10 megawatts.\nJust play that forward half a year. By 2024, that’s a cluster that’s 100 MW and 100,000\nH100\nequivalents with costs in the billions.\nPlay it forward two more years. By 2026, that’s a gigawatt, the size of a large nuclear reactor. That’s like the power of the Hoover Dam. That costs tens of billions of dollars and requires a million H100 equivalents.\nBy 2028, that’s a cluster that’s ten GW. That’s more power than most US states. That’s 10 million H100 equivalents, costing hundreds of billions of dollars.\nBy 2030, you get the trillion-dollar cluster using 100 gigawatts, over 20% of US electricity production. That’s 100 million H100 equivalents.\nThat’s just the training cluster. There are more\ninference\nGPUs as well. Once there are products, most of them will be inference GPUs. US power production has\nbarely grown for decades\n. Now we’re really in for a ride.\nDwarkesh Patel\n00:03:53\nWhen I had\nZuck on the podcast\n, he was claiming not a plateau per se, but that AI progress would be bottlenecked by this constraint on energy. Specifically, he was like, \"oh, gigawatt data centers, are we going to build another\nThree Gorges Dam\nor something?\"\nAccording to public reports\n, there are companies planning things on the scale of a 1 GW data center. With a 10 GW data center, who’s going to be able to build that? A 100 GW center is like a state project. Are you going to pump that into one physical data center? How is it going to be possible?\nWhat is Zuck missing?\nLeopold Aschenbrenner\n00:04:29\nSix months ago, 10 GW was the talk of the town. Now, people have moved on. 10 GW is happening. There’s\nThe Information\nreport\non OpenAI and Microsoft planning a $100 billion cluster.\nDwarkesh Patel\n00:04:43\nIs that 1 GW? Or is that 10 GW?\nLeopold Aschenbrenner\n00:04:45\nI don’t know but if you try to map out how expensive the 10 GW cluster would be, that’s a couple of hundred billion. It’s sort of on that scale and they’re planning it. It’s not just my crazy take.\nAMD forecasted a $400 billion AI accelerator market by 2027\n. AI accelerators are only part of the expenditures.\nWe’re very much on track for a $1 trillion of total AI investment by 2027. The $1 trillion cluster will take a bit more acceleration. We saw how much ChatGPT unleashed. Every generation, the models are going to be crazy and shift the\nOverton window\n.\nThen the revenue comes in. These are forward-looking investments. The question is, do they pay off? Let’s estimate the GPT-4 cluster at around $500 million. There’s a common mistake people make, saying it was $100 million for GPT-4. That’s just the rental price. If you’re building the biggest cluster, you have to build and pay for the whole cluster. You can’t just rent it for three months.\nDwarkesh Patel\n00:05:50\nCan’t you?\nLeopold Aschenbrenner\n00:05:50\nOnce you’re trying to get into the hundreds of billions, you have to get to like $100 billion a year in revenue. This is where it gets really interesting for the big tech companies because their revenues are on the order of hundreds of billions.\n$10 billion is fine. It’ll pay off the 2024 size training cluster. It’ll really be gangbusters with Big Tech when it costs $100 billion a year. The question is how feasible is $100 billion a year from AI revenue? It’s a lot more than right now. If you believe in the trajectory of AI systems as I do, it’s not that crazy.\nThere are like 300 million Microsoft Office subscribers. They have\nCopilot\nnow. I don’t know what they’re selling it for. Suppose you sold some AI add-on for $100/month to a third of Microsoft Office subscribers. That’d be $100 billion right there. $100/month is a lot.\nDwarkesh Patel\n00:06:43\nThat’s a lot for a third of Office subscribers.\nLeopold Aschenbrenner\n00:06:46\nFor the average knowledge worker, it’s a few hours of productivity a month. You have to be expecting pretty lame AI progress to not hit a few hours of productivity a month.\nDwarkesh Patel\n00:06:55\nSure, let’s assume all this. What happens in the next few years? What can the AI trained on the 1 GW data center do? What about the one on the 10 GW data center? Just map out the next few years of AI progress for me.\nLeopold Aschenbrenner\n00:07:11\nThe 10 GW range is my best guess for when you get true\nAGI\n. Compute is actually overrated. We’ll talk about that.\nBy 2025-2026, we’re going to get models that are basically smarter than most college graduates. A lot of the economic usefulness depends on unhobbling. The models are smart but limited. There are chatbots and then there are things like being able to use a computer and doing agentic long-horizon tasks.\nBy 2027-2028, it’ll get as smart as the smartest experts. The unhobbling trajectory points to it becoming much more like an agent than a chatbot. It’ll almost be like a drop-in remote worker.\nThis is the question around the economic returns. Intermediate AI systems could be really useful, but it takes a lot of\nschlep\nto integrate them. There’s a lot you could do with GPT-4 or GPT-4.5 in a business use case, but you really have to change your workflows to make them useful. It’s a very\nTyler Cowen\n-esque take. It just takes a long time to diffuse. We’re in SF and so we miss that.\nBut in some sense, the way these systems want to be integrated is where you get this kind of sonic boom. Intermediate systems could have done it, but it would have taken schlep. Before you do the schlep to integrate them, you’ll get much more powerful systems that are unhobbled.\nThey’re agents, drop-in remote workers. You’re interacting with them like coworkers. You can do Zoom calls and Slack with them. You can ask them to do a project and they go off and write a first draft, get feedback, run tests on their code, and come back. Then you can tell them more things. That’ll be much easier to integrate.\nYou might need a bit of overkill to make the transition easy and harvest the gains.\nDwarkesh Patel\n00:09:16\nWhat do you mean by overkill? Overkill on model capabilities?\nLeopold Aschenbrenner\n00:09:19\nYeah, the intermediate models could do it but it would take a lot of schlep. The drop-in remote worker AGI can automate cognitive tasks. The intermediate models would have made the software engineer more productive. But will the software engineer adopt it?\nWith the 2027 model, you just don’t need the software engineer. You can interact with it like a software engineer, and it’ll do the work of a software engineer.\nDwarkesh Patel\n00:09:43\nThe\nlast episode\nI did was with\nJohn Schulman\n.\nI was asking about this. We have these models that have come out in the last year and none seem to have significantly surpassed GPT-4, certainly not in an agentic way where they interact with you as a coworker. They’ll brag about a few extra points on\nMMLU\n. Even with\nGPT-4o\n, it’s cool they can talk like Scarlett Johansson (I guess not anymore) but it’s not like a coworker.\nIt makes sense why they’d be good at answering questions. They have data on how to complete Wikipedia text. Where is the equivalent training data to understand a Zoom call? Referring back to your point about a Slack conversation, how can it use context to figure out the cohesive project you’re working on? Where is that training data coming from?\nLeopold Aschenbrenner\n00:10:49\nA key question for AI progress in the next few years is how hard it is to unlock the\ntest time compute overhang\n. Right now, GPT-4 can do a few hundred\ntokens\nwith\nchain-of-thought\n. That’s already a huge improvement. Before, answering a math question was just\nshotgun\n. If you tried to answer a math question by saying the first thing that comes to mind, you wouldn’t be very good.\nGPT-4 thinks for a few hundred tokens. If I think at 100 tokens a minute, that’s like what GPT-4 does. It’s equivalent to me thinking for three minutes. Suppose GPT-4 could think for millions of tokens. That’s +4 OOMs on test time compute on one problem. It can’t do it now. It gets stuck. It writes some code. It can do a little bit of iterative debugging, but eventually gets stuck and can’t correct its errors.\nThere’s a big overhang. In other areas of ML, there’s a\ngreat paper\non\nAlphaGo\n, where you can trade off train time and test time compute. If you can use 4 OOMs more test time compute, that’s almost like a 3.5x OOM bigger model.\nAgain, if it’s 100 tokens a minute, a few million tokens is a few months of working time. There’s a lot more you can do in a few months of working time than just getting an answer right now. The question is how hard is it to unlock that?\nIn the short timelines AI world, it’s not that hard. The reason it might not be that hard is that there are only a few extra tokens to learn. You need to learn things like error correction tokens where you’re like “ah, I made a mistake, let me think about that again.” You need to learn planning tokens where it’s like “I’m going to start by making a plan. Here’s my plan of attack. I’m going to write a draft and now I’m going to critique my draft and think about it.” These aren’t things that models can do now, but the question is how hard it is.\nThere are two paths to agents. When Sholto was on your podcast, he talked about scaling leading to more\nnines of reliability\n. That’s one path. The other path is the unhobbling path. It needs to learn this\nSystem 2\nprocess. If it can learn that, it can use millions of tokens and think coherently.\nHere’s an analogy. When you drive, you’re on autopilot most of the time. Sometimes you hit a weird construction zone or intersection. Sometimes my girlfriend is in the passenger seat and I’m like “ah, be quiet for a moment, I need to figure out what’s going on.”\nYou go from autopilot to System 2 and you’re thinking about how to do it. Scaling improves that System 1 autopilot. The brute force way to get to agents is improving that system. If you can get System 2 working, you can quickly jump to something more agentified and test time compute overhang is unlocked.\nDwarkesh Patel\n00:13:57\nWhat’s the reason to think this is an easy win? Is there some\nloss function\nthat easily enables System 2 thinking? There aren’t many animals with System 2 thinking. It took a long time for evolution to give us System 2 thinking.\nPre-training has trillions of tokens of Internet text, I get that. You match that and get all of these free training capabilities. What’s the reason to think this is an easy unhobbling?\nLeopold Aschenbrenner\n00:14:29\nFirst of all, pre-training is magical. It gave us a huge advantage for models of\ngeneral intelligence\nbecause you can predict the next token. But there’s a common misconception. Predicting the next token lets the model learn incredibly rich representations. Representation learning properties are the magic of\ndeep learning\n. Rather than just learning statistical artifacts, the models learn models of the world. That’s why they can generalize, because it learned the right representations.\nWhen you train a model, you have this raw bundle of capabilities that’s useful. The unhobbling from\nGPT-2\nto GPT-4 took this raw mass and\nRLHF\n’d it into a good chatbot. That was a huge win.\nIn the\noriginal InstructGPT paper\n, comparing RLHF vs. non-RLHF models it’s like a 100x model size win on human preference rating. It started to be able to do simple chain-of-thought and so on. But you still have this advantage of all these raw capabilities, and there’s still a huge amount you’re not doing with them.\nThis pre-training advantage is also the difference to\nrobotics\n. People used to say it was a hardware problem. The hardware is getting solved, but you don’t have this huge advantage of bootstrapping with pre-training. You don’t have all this\nunsupervised learning\nyou can do. You have to start right away with\nRL\nself-play\n.\nThe question is why RL and unhobbling might work. Bootstrapping is an advantage. Your Twitter bio is being pre-trained. You’re not being pre-trained anymore. You were pre-trained in grade school and high school. At some point, you transition to being able to learn by yourself. You weren’t able to do it in elementary school. High school is probably where it started and by college, if you’re smart, you can teach yourself. Models are just starting to enter that regime.\nIt’s a little bit more scaling and then you figure out what goes on top. It won’t be trivial. A lot of deep learning seems obvious in retrospect. There’s some obvious cluster of ideas. There are some ideas that seem a little dumb but work. There are a lot of details you have to get right. We’re not going to get this next month. It’ll take a while to figure out.\nDwarkesh Patel\n00:17:04\nA while for you is like half a year.\nLeopold Aschenbrenner\n00:17:07\nI don’t know, between six months and three years. But it's possible. It’s also very related to the issue of the\ndata wall\n. Here’s one intuition on learning by yourself. Pre-training is kind of like the teacher lecturing to you and the words are flying by. You’re just getting a little bit from it.\nThat's not what you do when you learn by yourself. When you learn by yourself, say you're reading a dense math textbook, you're not just skimming through it once. Some\nwordcels\njust skim through and reread and reread the math textbook and they memorize.\nWhat you do is you read a page, think about it, have some internal monologue going on, and have a conversation with a study buddy. You try a practice problem and fail a bunch of times. At some point it clicks, and you're like, \"this made sense.\" Then you read a few more pages.\nWe've kind of bootstrapped our way to just starting to be able to do that now with models. The question is, can you use all this sort of self-play,\nsynthetic data\n, RL to make that thing work. Right now, there's\nin-context learning\n, which is super\nsample efficient\n. In the\nGemini paper\n, it just learns a language in-context. Pre-training, on the other hand, is not at all sample efficient.\nWhat humans do is a kind of in-context learning. You read a book, think about it, until eventually it clicks. Then you somehow distill that back into the\nweights\n. In some sense, that's what RL is trying to do. RL is super finicky, but when it works it's kind of magical.\nIt's the best possible data for the model. It’s when you try a practice problem, fail, and at some point figure it out in a way that makes sense to you. That's the best possible data for you because it's the way you would have solved the problem, rather than just reading how somebody else solved the problem, which doesn't initially click.\nDwarkesh Patel\n00:19:18\nBy the way, if that take sounds familiar it's because it was part of the question I asked John Schulman. It goes to illustrate the thing I said in the intro. A bunch of the things I've learned about AI comes from these dinners we do before the interviews with me, you, Sholto, and a couple of others. We’re like, “what should I ask John Schulman, what I should ask\nDario\n.”\nSuppose this is the way things go and we get these unhobblings—\nLeopold Aschenbrenner\n00:19:42\nAnd the scaling. You have this baseline of this enormous force of scaling. GPT-2 was amazing. It could string together plausible sentences, but it could barely do anything. It was kind of like a preschooler. GPT-4, on the other hand, could write code and do hard math, like a smart high schooler. This big jump in capability is explored in the essay series. I count the orders of magnitude of compute and scale-up of algorithmic progress.\nScaling alone by 2027-2028 is going to do another preschool to high school jump on top of GPT-4. At a per token level, the models will be incredibly smart. They'll gain more reliability, and with the addition of unhobblings, they'll look less like chatbots and more like agents or drop-in remote workers. That's when things really get going.\n(00:20:31) – AI 2028: The return of history\nDwarkesh Patel\n00:20:31\nI want to ask more questions about this but let's zoom out. Suppose you're right about this.\nThis is because of the 2027 cluster which is at 10 GW?\nLeopold Aschenbrenner\n00:20:45\n2028 is 10 GW. Maybe it'll be pulled forward.\nDwarkesh Patel\n00:20:50\nSomething like a 5.5 level by 2027, whatever that's called. What does the world look like at that point? You have these remote workers who can replace people. What is the reaction to that in terms of the economy, politics, and geopolitics?\nLeopold Aschenbrenner\n00:21:06\n2023 was a really interesting year to experience as somebody who was really following the AI stuff.\nDwarkesh Patel\n00:21:16\nWhat were you doing in 2023?\nLeopold Aschenbrenner\n00:21:18\nOpenAI. When you were at OpenAI in 2023, it was a weird thing. You almost didn't want to talk about AI or AGI. It was kind of a dirty word. Then in 2023, people saw ChatGPT for the first time, they saw GPT-4, and it just exploded.\nIt triggered huge capital expenditures from all these firms and an explosion in revenue from Nvidia and so on. Things have been quiet since then, but the next thing has been in the oven. I expect every generation these\ng-forces\nto intensify. People will see the models. They won’t have counted the OOMs so they're going to be surprised. It'll be kind of crazy.\nRevenue is going to accelerate. Suppose you do hit $10 billion by the end of this year. Suppose it just continues on the trajectory of revenue doubling every six months. It's not actually that far from $100 billion, maybe by 2026. At some point, what happened to Nvidia is going to happen to Big Tech. It's going to explode. A lot more people are going to feel it.\n2023 was the moment for me where AGI went from being this theoretical, abstract thing. I see it, I feel it, and I see the path. I see where it's going. I can see the cluster it's trained on, the rough combination of algorithms, the people, how it's happening. Most of the world is not there yet. Most of the people who feel it are right here. A lot more of the world is going to start feeling it. That's going to start being intense.\nDwarkesh Patel\n00:22:59\nRight now, who feels it? You can go on Twitter and there are these\nGPT wrapper\ncompanies, like, \"whoa, GPT-4 is going to change our business.\"\nLeopold Aschenbrenner\n00:23:06\nI'm so bearish on the wrapper companies because they're betting on stagnation. They're betting that you have these intermediate models and it takes so much schlep to integrate them. I'm really bearish because we're just going to sonic boom you. We're going to get the unhobblings. We're going to get the drop-in remote worker. Your stuff is not going to matter.\nDwarkesh Patel\n00:23:23\nSo that's done. SF, this crowd, is paying attention now. Who is going to be paying attention in 2026 and 2027? Presumably, these are years in which hundreds of billions of capex is being spent on AI.\nLeopold Aschenbrenner\n00:23:40\nThe national security state is going to start paying a lot of attention. I hope we get to talk about that.\nDwarkesh Patel\n00:23:48\nLet’s talk about it now. What happens? What is the immediate political reaction? Looking internationally, I don't know if\nXi Jinping\nsees the GPT-4 news and goes, \"oh, my God, look at the MMLU score on that. What are we doing about this, comrade?\"\nSo what happens when he sees a remote worker replacement and it has $100 billion in revenue? There’s a lot of businesses that have $100 billion in revenue, and people aren't staying up all night talking about it.\nLeopold Aschenbrenner\n00:24:15\nThe question is, when does the\nCCP\nand when does the American national security establishment realize that\nsuperintelligence\nis going to be absolutely decisive for national power? This is where the\nintelligence explosion\nstuff comes in, which we should talk about later.\nYou have AGI. You have this drop-in remote worker that can replace you or me, at least for remote jobs. Fairly quickly, you turn the crank one or two more times and you get a thing that's smarter than humans.\nEven more than just turning the crank a few more times, one of the first jobs to be automated is going to be that of an AI researcher or engineer. If you can automate AI research, things can start going very fast.\nRight now, there's already at this trend of 0.5 OOMs a year of algorithmic progress. At some point, you're going to have GPU fleets in the tens of millions for inference or more. You’re going to be able to run 100 million human equivalents of these automated AI researchers.\nIf you can do that, you can maybe do a decade's worth of\nML\nresearch progress in a year. You get some sort of 10x speed up. You can make the jump to AI that is vastly smarter than humans within a year, a couple of years.\nThat broadens from there. You have this initial acceleration of AI research. You apply R&D to a bunch of other fields of technology. At this point, you have a billion super intelligent researchers, engineers, technicians, everything. They’re superbly competent at all things.\nThey're going to figure out robotics. We talked about that being a software problem. Well, you have a billion super smart — smarter than the smartest human researchers — AI researchers in your cluster. At some point during the intelligence explosion, they're going to be able to figure out robotics. Again, that’ll expand.\nIf you play this picture forward, it is fairly unlike any other technology. A couple years of lead could be utterly decisive in say, military competition. If you look at the\nfirst Gulf War\n, Western coalition forces had a 100:1 kill ratio. They had\nbetter sensors on their tanks\n. They had better precision missiles, GPS, and stealth. They had maybe 20-30 years of technological lead. They just completely crushed them.\nSuperintelligence applied to broad fields of R&D — and the industrial explosion that comes from it, robots making a lot of material — could compress a century’s worth of technological progress into less than a decade. That means that a couple years could mean a Gulf War 1-style advantage in military affairs. That’s including a decisive advantage that even preempts nukes.\nHow do you find nuclear stealth submarines? Right now, you have sensors and software to detect where they are. You can do that. You can find them. You have millions or billions of mosquito-sized drones, and they take out the nuclear submarines. They take out the mobile launchers. They take out the other nukes.\nIt’s potentially enormously destabilizing and enormously important for national power. At some point people are going to realize that. Not yet, but they will. When they do, it won’t just be the AI researchers in charge.\nThe CCP is going to have an all-out effort to infiltrate American AI labs. It’ll involve billions of dollars, thousands of people, and the full force of the\nMinistry of State Security\n. The CCP is going to try to outbuild us.\nThey added as much power in the last decade as an entire US electric grid\n. So the 100 GW cluster, at least the 100 GW part of it, is going to be a lot easier for them to get. By this point, it's going to be an extremely intense international competition.\nDwarkesh Patel\n00:28:26\nOne thing I'm uncertain about in this picture is if it’s like what you say, where it's more of an explosion. You’ve developed an AGI. You make it into an AI researcher. For a while, you're only using this ability to make hundreds of millions of other AI researchers. The thing that comes out of this really frenetic process is a superintelligence. Then that goes out in the world and is developing robotics and helping you take over other countries and whatever.\nLeopold Aschenbrenner\n00:29:04\nIt's a little bit more gradual. It's an explosion that starts narrowly. It can do cognitive jobs. The highest ROI use for cognitive jobs is to make the AI better and solve robotics. As you solve robotics, now you can do R&D in biology and other technology.\nInitially, you start with the factory workers. They're wearing the\nglasses\nand AirPods, and the AI is instructing them because you can make any worker into a skilled technician. Then you have the robots come in. So this process expands.\nDwarkesh Patel\n00:29:30\nMeta's Ray-Bans\nare a complement to\nLlama\n.\nLeopold Aschenbrenner\n00:29:35\nWith the fabs in the US,\ntheir constraint is skilled workers\n. Even if you don't have robots, you have the cognitive superintelligence and can kind of make them all into skilled workers immediately. That's a very brief period. Robots will come soon.\nDwarkesh Patel\n00:29:44\nSuppose this is actually how the tech progresses in the United States, maybe because these companies are already generating hundreds of billions of dollars of AI revenue\nLeopold Aschenbrenner\n00:29:54\nAt this point, companies are borrowing hundreds of billions or more in the corporate debt markets.\nDwarkesh Patel\n00:29:58\nWhy is a CCP bureaucrat, some 60-year-old guy, looking at this and going, \"oh, Copilot has gotten better now\" and now—\nLeopold Aschenbrenner\n00:30:07\nThis is much more than Copilot has gotten better now.\nDwarkesh Patel\n00:30:12\nIt’d require shifting the production of an entire country, dislocating energy that is otherwise being used for consumer goods or something, and feeding all that into the data centers. Part of this whole story is that you realize superintelligence is coming soon. You realize it and maybe I realize it. I'm not sure how much I realize it.\nWill the national security apparatus in the United States and the CCP realize it?\nLeopold Aschenbrenner\n00:30:41\nThis is a really key question. We have a few more years of mid-game. We have a few more 2023s. That just starts updating more and more people. The trend lines will become clear.\nYou will see some amount of the COVID dynamic. COVID in February of 2020 honestly feels a lot like today. It feels like this utterly crazy thing is coming. You see the exponential and yet most of the world just doesn't realize it. The\nmayor of New York\nis like,\n\"go out to the shows,\"\nand\n\"this is just Asian racism.\"\nAt some point, people saw it and then crazy, radical reactions came.\nDwarkesh Patel\n00:31:32\nBy the way, what were you doing during COVID? Was it your freshman or sophomore year?\nLeopold Aschenbrenner\n00:31:39\nJunior.\nDwarkesh Patel\n00:31:40\nStill, you were like a 17-year-old junior or something right? Did you short the market or something? Did you sell at the right time?\nLeopold Aschenbrenner\n00:31:51\nYeah.\nDwarkesh Patel\n00:31:54\nSo there will be a March 2020 moment.\nYou can make the analogy you make in the series that this will cause a reaction like, “we have to do the\nManhattan Project\nagain for America here.” I wonder what the politics of this will be like. The difference here is that it’s not just like, “we need the bomb to beat the Nazis.”\nWe'll be building this thing that makes all our energy prices go up a bunch and it's automating a lot of our jobs. The climate change stuff people are going to be like, \"oh, my God, it's making climate change worse and it's helping Big Tech.\"\nPolitically, this doesn't seem like a dynamic where the national security apparatus or the president is like, \"we have to step on the gas here and make sure America wins.\"\nLeopold Aschenbrenner\n00:32:42\nAgain, a lot of this really depends on how much people are feeling it and how much people are seeing it. Our generation is so used to peace, American hegemony and nothing matters. The historical norm is very much one of extremely intense and extraordinary things happening in the world with intense international competition.\nThere's a 20-year very unique period. In World War II, something like\n50% of GDP went to war production\n. The US borrowed over 60% of GDP. With Germany and Japan I think it was over 100%. In World War I, the UK, France, and Germany all borrowed over 100% of GDP.\nMuch more was on the line. People talk about World War I being so destructive with 20 million Soviet soldiers dying and 20% of Poland. That happened all the time. During the\nSeven Years' War\nsomething like 20-30% of Prussia died. In the\nThirty Years' War\n, up to 50% of a large swath of Germany died.\nWill people see that the stakes here are really high and that\nhistory\nis actually back? The American national security state thinks very seriously about stuff like this. They think very seriously about competition with China. China very much thinks of itself on this historical mission of the rejuvenation of the Chinese nation. They think a lot about national power. They think a lot about the world order.\nThere's a real question on timing. Do they start taking this seriously when the intelligence explosion is already happening quite late. Do they start taking this seriously two years earlier? That matters a lot for how things play out.\nAt some point they will and they will realize that this will be utterly decisive for not just some proxy war but for major questions. Can liberal democracy continue to thrive? Can the CCP continue existing? That will activate forces that we haven't seen in a long time.\nDwarkesh Patel\n00:35:06\nThe great power conflict definitely seems compelling. All kinds of different things seem much more likely when you think from a historical perspective. You zoom out beyond the liberal democracy that we’ve had the pleasure to live in America for say the last 80 years. That includes things like dictatorships, war, famine, etc.\nI was reading\nThe Gulag Archipelago\nand one of the chapters begins with\nSolzhenitsyn\nsaying how if you had told a Russian citizen under the tsars that because of all these new technologies — we wouldn’t see some Great Russian revival with Russia becoming a great power and the citizens made wealthy — you would see tens of millions of Soviet citizens tortured by millions of beasts in the worst possible ways. If you’d told them that that would be the result of the 20th century, they wouldn’t have believed you. They’d have called you a slanderer.\nLeopold Aschenbrenner\n00:35:59\nThe possibilities for dictatorship with superintelligence are even crazier as well. Imagine you have a perfectly loyal military and security force. No more rebellions. No more popular uprisings. You have perfect lie detection. You have surveillance of everybody. You can perfectly figure out who's the dissenter and weed them out. No\nGorbachev\nwho had some doubts about the system would have ever risen to power. No military coup would have ever happened.\nThere's a real way in which part of why things have worked out is that ideas can evolve. There's some sense in which time heals a lot of wounds and solves a lot of debates. Throughout time, a lot of people had really strong convictions, but a lot of those have been overturned over time because there's been continued\npluralism\nand evolution.\nImagine applying a CCP-like approach to truth where truth is what the party says. When you supercharge that with superintelligence, that could just be locked in and enshrined for a long time. The possibilities are pretty terrifying.\nTo your point about history and living in America for the past eight years, this is one of the things I took away from growing up in Germany. A lot of this stuff feels more visceral. My mother grew up in the former\nEast\n, my father in the former\nWest\n. They met shortly after the\nWall fell\n. The end of the Cold War was this extremely pivotal moment for me because it's the reason I exist.\nI grew up in Berlin with the former Wall. My great-grandmother, who is still alive, is very important in my life. She was born in 1934 and grew up during the Nazi era. In World War II, she saw the\nfirebombing of Dresden\nfrom this country cottage where they were as kids. Then she spent most of her life in the East German communist dictatorship.\nShe'd tell me about how Soviet tanks came when there was the\npopular uprising in 1954\n. Her husband was telling her to get home really quickly and get off the streets. She had a son who tried to ride a motorcycle across the\nIron Curtain\nand then was put in a\nStasi\nprison for a while. Finally, when she's almost 60, it was the first time she lived in a free country, and a wealthy country.\nWhen I was a kid, the thing she always really didn't want me to do was get involved in politics. Joining a political party had very bad connotations for her. She raised me when I was young. So it doesn't feel that long ago. It feels very close.\nDwarkesh Patel\n00:38:43\nThere’s one thing I wonder about when we're talking today about the CCP. The people in China who will be doing their version of this project will be AI researchers who are somewhat Westernized. They’ll either have gotten educated in the West or have colleagues in the West.\nAre they going to sign up for the CCP project that's going to hand over control to Xi Jinping? What's your sense of that? Fundamentally, they're just people, right? Can't you convince them about the dangers of superintelligence?\nLeopold Aschenbrenner\n00:39:20\nWill they be in charge though? In some sense, this is also the case in the US. This is like the rapidly depreciating influence of the lab employees. Right now, the AI lab employees have so much power. You saw this\nNovember event\n. It’s so much power.\nBoth are going to get automated and they're going to lose all their power.\nIt'll just be a few people in charge with their armies of automated AIs. It’s also the politicians and the generals and the national security state. There are some of these classic scenes from the\nOppenheimer\nmovie. The scientists built it and then the bomb was shipped away and it was out of their hands.\nIt's good for lab employees to be aware of this. You have a lot of power now, but maybe not for that long. Use it wisely. I do think they would benefit from some more organs of representative democracy.\nDwarkesh Patel\n00:40:13\nWhat do you mean by that?\nLeopold Aschenbrenner\n00:40:14\nIn the OpenAI board events, employee power is exercised in a very direct democracy way. How some of that went about really highlighted the benefits of representative democracy and having some deliberative organs.\n(00:40:26) – Espionage & American AI superiority\nDwarkesh Patel\n00:40:26\nInteresting. Let's go back to the $100 billion revenue question. The companies are trying to build clusters that are this big. Where are they building it? Say it's the amount of energy that would be required for a small or medium-sized US state. Does Colorado then get no power because it's happening in the United States? Is it happening somewhere else?\nLeopold Aschenbrenner\n00:40:47\nThis is the thing that I always find funny, when you talk about Colorado getting no power. The easy way to get the power would be to displace less economically useful stuff. Buy up the aluminum smelting plant that has a gigawatt. We're going to replace it with the data center because that's important. That's not actually happening because a lot of these power contracts are really locked in long-term. Also, people don't like things like this.\nIn practice what it requires, at least right now, is building new power. That might change. That's when things get really interesting, when it's like, “no, we're just dedicating all of the power to the AGI.”\nSo right now it's building new power. 10 GW is quite doable. It's like a few percent of\nUS natural gas production\n. When you have the 10 GW training cluster, you have a lot more inference. 100 gigawatts is where it starts getting pretty wild. That's over 20% of\nUS electricity production\n. It's pretty doable, especially if you're willing to go for natural gas.\nIt is incredibly important that these clusters are in the United States.\nDwarkesh Patel\n00:41:50\nWhy does it matter that it's in the US?\nLeopold Aschenbrenner\n00:41:52\nThere are some people who are trying to build clusters elsewhere. There's a lot of\nfree-flowing Middle Eastern money\nthat's trying to build clusters elsewhere. This comes back to the national security question we talked about. Would you do the Manhattan Project in the UAE?\nYou can put the clusters in the US and you can put them in allied democracies. Once you put them in authoritarian dictatorships, you create this irreversible security risk. Once the cluster is there, it's much easier for them to exfiltrate the weights. They can literally steal the AGI, the superintelligence. It’s like they got a direct copy of the atomic bomb. It makes it much easier for them. They have weird ties to China. They can ship that to China. That's a huge risk.\nAnother thing is they can just seize the compute. The issue here is people right now are thinking of this as ChatGPT, Big Tech product clusters. The clusters being planned now, three to five years out, may well be the AGI, superintelligence clusters. When things get hot, they might just seize the compute.\nSuppose we put 25% of the compute capacity in these Middle Eastern dictatorships. Say they seize that. Now it's a ratio of compute of 3:1. We still have more, but even with only 25% of compute there it starts getting pretty hairy. 3:1 is not that great of a ratio. You can do a lot with that amount of compute.\nSay they don't actually do this. Even if they don't actually seize the compute, even if they actually don't steal the weights, there's just a lot of implicit leverage you get. They get seats at the AGI table. I don't know why we're giving authoritarian dictatorships the seat at the AGI table.\nDwarkesh Patel\n00:43:36\nThere's going to be a lot of compute in the Middle East if these deals go through.\nFirst of all, who is it? Is it just every single Big Tech company trying to figure it out over there?\nLeopold Aschenbrenner\n00:43:44\nIt’s not everybody, some.\nDwarkesh Patel\n00:43:45\nThere are\nreports\n, I think Microsoft. We'll get into it.\nSo say the UAE gets a bunch of compute because we're building the clusters there. Let's say they have 25% of the compute. Why does a compute ratio matter? If it's about them being able to kick off the intelligence explosion, isn't it just some threshold where you have 100 million AI researchers or you don't?\nLeopold Aschenbrenner\n00:44:12\nYou can do a lot with 33 million extremely smart scientists. That might be enough to build the crazy bio weapons. Then you're in a situation where they stole the weights and they seized the compute.\nNow they can make these crazy new\nWMDs\nthat will be possible with superintelligence. Now you've just proliferated the stuff that’ll be really powerful. Also, 3x on compute isn't actually that much.\nThe riskiest situation is if we're in some sort of really neck and neck, feverish international struggle. Say we're really close with the CCP and we're months apart. The situation we want to be in — and could be in if we play our cards right — is a little bit more like the US building the atomic bomb versus the German project years behind. If we have that, we just have so much more wiggle room to get safety right.\nWe're going to be building these crazy new WMDs that completely undermine nuclear deterrence. That's so much easier to deal with if you don't have somebody right on your tails and you have to go at maximum speed. You have no wiggle room. You're worried that at any time they can overtake you.\nThey can also just try to outbuild you. They might literally win. China might literally win if they can steal the weights, because they can outbuild you. They may have less caution, both good and bad caution in terms of whatever unreasonable regulations we have.\nIf you're in this really tight race, this sort of feverish struggle, that's when there's the greatest\nperil of self-destruction\n.\nDwarkesh Patel\n00:45:58\nPresumably the companies that are trying to build clusters in the Middle East realize this. Is it just that it’s impossible to do this in America? If you want American companies to do this at all, do you have to do it in the Middle East or not at all? Then you just have China build a Three Gorges Dam cluster.\nLeopold Aschenbrenner\n00:46:12\nThere’s a few reasons. People aren’t thinking about this as the AGI superintelligence cluster. They’re just like, “ah, cool clusters for my ChatGPT.”\nDwarkesh Patel\n00:46:19\nIf you’re doing ones for inference, presumably you could spread them out across the country or something. The ones they’re building, they’re going to do one training run in a single thing they’re building.\nLeopold Aschenbrenner\n00:46:33\nIt’s just hard to distinguish between inference and training compute. People can claim it’s inference compute, but they might realize that actually this is going to be useful for training compute too.\nDwarkesh Patel\n00:46:45\nBecause of synthetic data and things like that?\nLeopold Aschenbrenner\n00:46:46\nRL looks a lot like inference, for example. Or you just end up connecting them in time. It's a lot like raw materials. It's like placing your uranium refinement facilities there.\nSo there are a few reasons. One, they don't think about this as the AGI cluster. Another is just that there’s easy money coming from the Middle East.\nAnother one is that some people think that you can't do it in the US. We actually face a real system competition here. Some people think that only autocracies that can do this with top-down mobilization of industrial capacity and the power to get stuff done fast.\nAgain, this is the sort of thing we haven't faced in a while. But during the Cold War, there was this intense system competition. East vs. West Germany was this. It was West Germany as liberal democratic capitalism vs. state-planned communism.\nNow it's obvious that the free world would win. But even as late as 1961,\nPaul Samuelson\nwas\npredicting that the Soviet Union would outgrow\nthe United States because they were able to mobilize industry better.\nSo there are some people who shitpost about loving America, but then in private they're betting against America. They're betting against the liberal order. Basically, it's just a bad bet. This stuff is really possible in the US.\nTo make it possible in the US, to some degree we have to get our act together. There are basically two paths to doing it in the US. One is you just have to be willing to do natural gas. There's ample natural gas. You put your cluster in West Texas. You put it in southwest Pennsylvania by the\nMarcellus Shale\n. The 10 GW cluster is super easy. The 100 GW cluster is also pretty doable. I think\nnatural gas production in the United States has almost doubled in a decade\n. You do that one more time over the next seven years, you could power multiple trillion-dollar data centers.\nThe issue there is that a lot of people made these climate commitments, not just the government. It's actually the private companies themselves,\nMicrosoft\n,\nAmazon\n, etc., that have made these climate commitments. So they won't do natural gas. I admire the climate commitments, but at some point the national interest and national security is more important.\nThe other path is doing green energy megaprojects. You do solar and batteries and\nSMR\ns and geothermal. If we want to do that, there needs to be a broad deregulatory push. You can't have permitting take a decade. You have to reform\nFERC\n. You have to have blanket\nNEPA\nexemptions for this stuff.\nThere are inane state-level regulations. You can build the solar panels and batteries next to your data center, but it'll still take years because you actually have to hook it up to the state electrical grid. You have to use governmental powers to create rights of way to have multiple clusters and connect them and have the cables.\nIdeally we do both. Ideally we do natural gas and the broader deregulatory green agenda. We have to do at least one. Then this stuff is possible in the United States.\nDwarkesh Patel\n00:49:44\nBefore the conversation I was reading a good book about World War II industrial mobilization in the United States called\nFreedom's Forge\n. I’m thinking back on that period, especially in the context of reading Patrick Collison’s\nFast\nand the\nprogress study stuff\n. There’s this narrative out there that we had state capacity back then and people just got shit done but that now it's a clusterfuck.\nLeopold Aschenbrenner\n00:50:09\nIt wasn’t at all the case!\nDwarkesh Patel\n00:50:10\nIt was really interesting. You had people from the Detroit auto industry side, like\nWilliam Knudsen\n, who were running mobilization for the United States. They were extremely competent. At the same time you had\nlabor organization and agitation\n, which is very analogous to the climate change pledges and concerns we have today.\nThey would literally have these strikes, into 1941, costing millions of man-hours worth of time when we're trying to make tens of thousands of planes a month. They would just debilitate factories for trivial concessions from capital that were pennies on the dollar.\nThere were concerns that the auto companies were trying to use the pretext of a potential war to prevent paying labor the money it deserves. So with what climate change is today, you might think, \"ah, America's fucked. We're not going to be able to build this shit if you look at NEPA or something,” I didn't realize how debilitating labor was in World War II.\nLeopold Aschenbrenner\n00:51:18\nIt wasn’ just that. Before 1939, the American military was in total shambles. You read about it and it reads a little bit like\nthe German military today\n. Military expenditures were I think less than 2% of GDP. All the European countries had gone, even in peacetime, above 10% of GDP.\nIt was rapid mobilization starting from nothing. We were making no planes. There were no military contracts. Everything had been starved during the Great Depression. But there was this latent capacity. At some point the United States got its act together.\nThis applies the other way around too with China. Sometimes people count them out a little bit with the export controls and so on. They're\nable to make\n7-nanometer chips\nnow. There's a question of how many they could make. There's at least a possibility that they're going to mature that ability and make a lot of 7-nanometer chips.\nThere's a lot of latent industrial capacity in China. They are able to build a lot of power fast. Maybe that isn't activated for AI yet. At some point, the same way the United States and a lot of people in the US government are going to wake up, the CCP is going to wake up.\nDwarkesh Patel\n00:52:22\nCompanies realize that scaling is a thing. Obviously their whole plans are contingent on scaling. So they understand that in 2028 we're going to be building 10 GW data centers.\nAt that point, the people who can keep up are Big Tech, potentially at the edge of their capabilities, sovereign wealth fund-funded things, and also major countries like America and China. What's their plan? With the AI labs, what's their plan given this landscape? Do they not want the leverage of being in the United States?\nLeopold Aschenbrenner\n00:53:07\nThe Middle East does offer capital, but America has plenty of capital. We have trillion-dollar companies. What are these Middle Eastern states? They're kind of like trillion-dollar oil companies. We have trillion-dollar companies and very deep financial markets. Microsoft could issue hundreds of billions of dollars of bonds and they can pay for these clusters.\nAnother argument being made, which is worth taking seriously, is that if we don't work with the UAE or with these Middle Eastern countries, they're just going to go to China. They're going to build data centers and pour money into AI regardless. If we don't work with them, they'll just support China.\nThere's some merit to the argument in the sense that we should be doing benefit-sharing with them. On the road to AGI, there should be two tiers of coalitions. There should be a narrow coalition of democracies that's developing AGI. Then there should be a broader coalition of other countries, including dictatorships, and we should offer them some of the benefits of AI.\nIf the UAE wants to use AI products, run Meta recommendation engines, or run the last-generation models, that's fine. By default, they just wouldn't have had this seat at the AGI table. So they have some money, but a lot of people have money.\nThe only reason they're getting this seat at the AGI table and giving these dictators this leverage over this extremely important national security technology, is because we're getting them excited and offering it to them.\nDwarkesh Patel\n00:54:50\nWho specifically is doing this? Who are the companies who are going there to fundraise?\nLeopold Aschenbrenner\n00:54:55\nIt’s been\nreported\nthat\nSam Altman\nis trying to raise $7 trillion or whatever for a chip project. It's unclear how many of the clusters will be there, but definitely stuff is happening.\nThere’s another reason I'm a little suspicious of this argument that if the US doesn't work with them, they'll go to China. I've heard from multiple people — not from my time at OpenAI, and I haven't seen the memo — that at some point several years ago, OpenAI leadership had laid out a plan to fund and sell AGI by starting a bidding war between the governments of the United States, China, and Russia.\nIt's surprising to me that they're willing to sell AGI to the Chinese and Russian governments. There's also something that feels eerily familiar about starting this bidding war and then playing them off each other, saying, \"well, if you don't do this, China will do it.\"\nDwarkesh Patel\n00:55:47\nInteresting. That's pretty fucked up.\nSuppose you're right. We ended up in this place because, as one of our friends put it, the Middle East has billions or trillions of dollars up for persuasion like no other place in the world.\nLeopold Aschenbrenner\n00:56:11\nWith little accountability. There’s no Microsoft board. It's only the dictator.\nDwarkesh Patel\n00:56:15\nLet's say you're right, that you shouldn't have gotten them excited about AGI in the first place. Now we're in a place where they are excited about AGI and they're like, \"fuck, we want to have GPT-5 while you're going to be off building superintelligence. This\nAtoms for Peace\nthing doesn't work for us.\" If you're in this place, don't they already have the leverage?\nLeopold Aschenbrenner\n00:56:36\nThe UAE on its own is not competitive. They're already export-controlled. You're not supposed to ship Nvidia chips over there. It's not like they have any of the leading AI labs. They have money, but it's hard to just translate money into progress.\nDwarkesh Patel\n00:56:51\nBut I want to go back to other things you've been saying in laying out your vision. There's this almost industrial process of putting in the compute and algorithms, adding that up, and getting AGI on the other end. If it's something more like that, then the case for somebody being able to catch up rapidly seems more compelling than if it's some bespoke...\nLeopold Aschenbrenner\n00:57:00\nWell, if they can steal the algorithms and if they can steal the weights, that’s really important.\nDwarkesh Patel\n00:57:20\nHow easy would it be for an actor to steal the things that are not the trivial released things, like Scarlett Johansson's voice, but the RL things we're talking about, the unhobblings?\nLeopold Aschenbrenner\n00:57:32\nIt’s all extremely easy. They don’t make the claim that it’s hard.\nDeepMind\nput out their\nFrontier Safety Framework\nand they lay out security levels, zero to four. Four is resistant to state activity. They say, we're at level zero. Just recently, there was an indictment of\na guy who stole a bunch of really important AI code and went to China\nwith it. All he had to do to steal the code was copy it, put it into Apple Notes, and export it as a PDF. That got past their monitoring.\nGoogle has the best security of any of the AI labs probably, because they have the Google infrastructure. I would think of the security of a startup. What does security of a startup look like? It's not that good. It's easy to steal.\nDwarkesh Patel\n00:58:18\nEven if that's the case, a lot of your post is making the argument for why we are going to get the intelligence explosion. If we have somebody with the intuition of an\nAlec Radford\nto come up with all these ideas, that intuition is extremely valuable and you can scale that up.\nIf it's just intuition, then that's not going to be just in the code, right? Also because of export controls, these countries are going to have slightly different hardware. You're going to have to make different trade-offs and probably rewrite things to be compatible with that.\nIs it just a matter of getting the right pen drive and plugging it into the gigawatt data center next to the Three Gorges Dam and then you're off to the races?\nLeopold Aschenbrenner\n00:59:01\nThere are a few different things, right? One threat model is just them stealing the weights themselves. The weights one is particularly insane because they can just steal the literal end product — just make a replica of the atomic bomb — and then they're ready to go. That one is extremely important around the time we have AGI and superintelligence because China can build a big cluster by default. We'd have a big lead because we have the better scientists, but if we make the superintelligence and they just steal it, they're off to the races.\nWeights are a little bit less important right now because who cares if they steal the GPT-4 weights. We still have to get started on weight security now because if we think there’s AGI by 2027, this stuff is going to take a while. It's not just going to be like, \"oh, we do some access control.\" If you actually want to be resistant to Chinese espionage, it needs to be much more intense.\nThe thing that people aren't paying enough attention to is the secrets. The compute stuff is sexy, but people underrate the secrets. The half an order of magnitude a year is just by default, sort of algorithmic progress. That's huge. If we have a few years of lead, by default, that's a 10-30x, 100x bigger cluster, if we protect them.\nThere's this additional layer of the data wall. We have to get through the data wall. That means we actually have to figure out some sort of basic new paradigm. So it’s the “AlphaGo step two.” “AlphaGo step one” learns from human imitation. “AlphaGo step two” is the kind of self-play RL thing that everyone's working on right now. Maybe we're going to crack it. If China can't steal that, then they're stuck. If they can steal it, they're off to the races.\nDwarkesh Patel\n01:00:45\nWhatever that thing is, can I literally write it down on the back of a napkin? If it's that easy, then why is it so hard for them to figure it out? If it's more about the intuitions, then don't you just have to hire Alec Radford? What are you copying down?\nLeopold Aschenbrenner\n01:00:57\nThere are a few layers to this. At the top is the fundamental approach. On pre-training it might be unsupervised learning, next token prediction, training on the entire Internet. You actually get a lot of juice out of that already. That one's very quick to communicate.\nThen there's a lot of details that matter, and you were talking about this earlier. It's probably going to be somewhat obvious in retrospect, or there's going to be some not too complicated thing that'll work, but there's going to be a lot of details to get that.\nDwarkesh Patel\n01:01:29\nIf that's true, then again, why do we think that getting state-level security in these startups will prevent China from catching up? It’s just like, \"oh, we know some sort of self-play RL will be required to get past the data wall.\"\nIt's going to be solved by 2027, right? It's not that hard.\nLeopold Aschenbrenner\n01:01:49\nThe US, and the leading labs in the United States, have this huge lead. By default, China actually has some good LLMs because they're just using open source code, like Llama. People really underrate both the divergence on algorithmic progress and the lead the US would have by default because all this stuff was published until recently.\nLook at\nChinchilla Scaling laws\n,\nMoE\npapers,\ntransformers\n. All that stuff was published. That's why open source is good and why China can make some good models. Now, they're not publishing it anymore. If we actually kept it secret, it would be a huge edge.\nTo your point about tacit knowledge and Alec Radford, there's another layer at the bottom that is something about large-scale engineering work to make these big training runs work. That is a little bit more like tacit knowledge, but China will be able to figure that out. It's engineering schlep, and they're going to figure out how to do it.\nDwarkesh Patel\n01:02:40\nWhy can't they figure that out, but not how to get the RL thing working?\nLeopold Aschenbrenner\n01:02:45\nI don't know. Germany during World War II went down the wrong path with\nheavy water\n. There's an amazing anecdote in\nThe Making of the Atomic Bomb\nabout this.\nSecrecy was one of the most contentious issues early on.\nLeo Szilard\nreally thought a nuclear chain reaction and an atomic bomb were possible. He went around saying, \"this is going to be of enormous strategic and military importance.\" A lot of people didn't believe it or thought, \"maybe this is possible, but I'm going to act as though it's not, and science should be open.\"\nIn the early days, there had been some incorrect measurements made on graphite as a\nmoderator\n. Germany thought graphite wasn't going to work, so they had to do heavy water. But then\nEnrico Fermi\nmade new measurements indicating that graphite would work. This was really important.\nSzilard assaulted Fermi with another secrecy appeal and Fermi was pissed off, throwing a temper tantrum. He thought it was absurd, saying, \"come on, this is crazy.\" But Szilard persisted, and they roped in another guy,\nGeorge Pegram\n. In the end, Fermi didn't publish it.\nThat was just in time. Fermi not publishing meant that the Nazis didn't figure out graphite would work. They went down the path of heavy water, which was the wrong path. This is a key reason why\nthe German project\ndidn't work out. They were way behind.\nWe face a similar situation now. Are we just going to instantly leak how to get past the data wall and what the next paradigm is? Or are we not?\nDwarkesh Patel\n01:04:24\nThe reason this would matter is if being one year ahead would be a huge advantage. In the world where you deploy AI over time they're just going to catch up anyway.\nI interviewed Richard Rhodes, the guy who wrote\nThe Making of the Atomic Bomb\n.\nOne of the anecdotes he had was when the Soviets realized America had the bomb. Obviously, we dropped it in Japan.\nLavrentiy Beria\n— the guy who ran the\nNKVD\n, a famously ruthless and evil guy — goes to the Soviet scientist who was running\ntheir version of the Manhattan Project\n. He says, \"comrade, you will get us the American bomb.\" The guy says, \"well, listen, their implosion device actually is not optimal. We should make it a different way.\" Beria says, \"no, you will get us the American bomb, or your family will be camp dust.\"\nThe thing that's relevant about that anecdote is that the Soviets would have had a better bomb if they hadn't copied the American design, at least initially. That suggests something about history, not just for the Manhattan Project. There's often this pattern of parallel invention because the tech tree implies that a certain thing is next — in this case, a self-play RL — and people work on that and are going to figure it out around the same time. There's not going to be that much gap in who gets it first.\nFamously,\na bunch of people invented the light bulb around the same time\n. Is it the case that it might be true but the one year or six months makes the difference?\nLeopold Aschenbrenner\n01:05:56\nTwo years makes all the difference.\nDwarkesh Patel\n01:05:58\nI don't know if it'll be two years though.\nLeopold Aschenbrenner\n01:06:01\nIf we lock down the labs, we have much better scientists. We're way ahead. It would be two years. Even six months, a year, would make a huge difference. This gets back to the intelligence explosion dynamics. A year might be the difference between a system that's sort of human-level and a system that is vastly superhuman. It might be like five OOMs.\nLook at the current pace. Three years ago, on the\nmath benchmark\n— these are really difficult high school competition math problems — we were at a few percent, we couldn't solve anything. Now it's solved. That was at the normal pace of AI progress. You didn't have a billion superintelligent researchers.\nA year is a huge difference, particularly after superintelligence. Once this is applied to many elements of R&D, you get an industrial explosion with robots and other advanced technologies. A couple of years might yield decades worth of progress. Again, it’s like the technological lead the U.S. had in the first Gulf War, when the 20-30 years of technological lead proved totally decisive. It really matters.\nHere’s another reason it really matters. Suppose they steal the weights, suppose they steal the algorithms, and they're close on our tails. Suppose we still pull out ahead. We're a little bit faster and we're three months ahead.\nThe world in which we're really neck and neck, we only have a three-month lead, is incredibly dangerous. We're in this feverish struggle where if they get ahead, they get to dominate, maybe they get a decisive advantage. They're building clusters like crazy. They're willing to throw all caution to the wind. We have to keep up.\nThere are crazy new WMDs popping up. Then we're going to be in the situation where it's crazy new military technology, crazy new WMDs, deterrence, mutually assured destruction keeps changing every few weeks. It's a completely unstable, volatile situation that is incredibly dangerous.\nSo you have to look at it from the point of view that these technologies are dangerous, from the alignment point of view. It might be really important during the intelligence explosion to have a six-month wiggle room to be like, “look, we're going to dedicate more compute to alignment during this period because we have to get it right. We're feeling uneasy about how it's going.”\nOne of the most important inputs to whether we will destroy ourselves or whether we will get through this incredibly crazy period is whether we have that buffer.\n(01:08:20) – Geopolitical implications of AI\nDwarkesh Patel\n01:08:20\nBefore we go further, it's very much worth noting that almost nobody I talk to thinks about the geopolitical implications of AI. I have some object-level disagreements that we'll get into, things I want to iron out. I may not disagree in the end.\nThe basic premise is that if you keep scaling, if people realize that this is where intelligence is headed, it's not just going to be the same old world. It won't just be about what model we're deploying tomorrow or what the latest thing is. People on Twitter are like, \"oh, GPT-4 is going to shake your expectations\" or whatever.\nCOVID is really interesting because when March 2020 hit, it became clear to the world — presidents, CEOs, media, the average person — that there are other things happening in the world right now but the main thing we as a world are dealing with right now is COVID.\nLeopold Aschenbrenner\n01:09:21\nSoon it will be AGI. This is the quiet period.\nMaybe you want to go on vacation. Maybe now is the last time you can have some kids. My girlfriend sometimes complains when I’m off doing work that I don’t spend enough time with her. She threatens to replace me with GPT-6 or whatever. I'm like, “GPT-6 will also be too busy doing AI research.”\nDwarkesh Patel\n01:09:51\nWhy aren't other people talking about national security?\nLeopold Aschenbrenner\n01:09:56\nI made this mistake with COVID. In February of 2020, I thought it was going to sweep the world and all the hospitals would collapse. It would be crazy, and then it'd be over. A lot of people thought this kind of thing at the beginning of COVID. They shut down their office for a month or whatever.\nThe thing I just really didn't price in was societal reaction. Within weeks,\nCongress spent over 10% of GDP on COVID measures\n. The entire country was shut down. It was crazy. I didn't sufficiently price it in with COVID.\nWhy do people underrate it? Being in the trenches actually gives you a less clear picture of the trend lines. You don’t have to zoom out that much, only a few years.\nWhen you're in the trenches, you're trying to get the next model to work. There's always something that's hard. You might underrate algorithmic progress because you're like, \"ah, things are hard right now,\" or \"data wall\" or whatever. When you zoom out just a few years and count up how much algorithmic progress was made, it's enormous.\nPeople also just don’t think about this stuff. Smart people really underrate espionage. Part of the security issue is that people don't realize how intense state-level espionage can be. This\nIsraeli company had software that could just zero-click hack any iPhone\n. They just put in your number and it was a straight download of everything. The United States infiltrated an\nair-gapped\natomic weapons program. Wild.\nDwarkesh Patel\n01:11:28\nAre you talking about\nStuxnet\n?\nLeopold Aschenbrenner\n01:11:30\nYeah. Intelligence agencies have stockpiles of\nzero-days\n. When things get really hot, maybe we'll send special forces to go to the data center or something. China does this. They threaten people's families. They’re like, “if you don't cooperate, if you don't give us the intel…”\nThere's a good book along the lines of\nThe Gulag Archipelago\ncalled\nInside the Aquarium\n, which is by\na Soviet GRU defector\n.\nGRU\nwas military intelligence.\nIlya\nrecommended this book to me. When I read it, I was shocked at the intensity of state-level espionage.\nThe whole book was about how they go to these European countries and try and recruit people to get the technology. Here’s one anecdote. This eventual defector, he's being trained at the GRU spy academy. To graduate from the spy academy before being sent abroad, you had to pass a test to show that you can do this.\nThe test was recruiting a Soviet scientist in Moscow to give you information, like you would do in a foreign country. Of course, for whomever you recruited, the penalty for giving away secret information was death. So to graduate from the GRU spy academy, you had to condemn a countryman to death. States do this stuff.\nDwarkesh Patel\n01:12:58\nI started reading the book because you mentioned it in the series. I was wondering about the fact that you use this anecdote. Then you're like, \"a book recommended by Ilya.\" Is this some sort of Easter egg? We'll leave that as an exercise for the reader.\nLeopold Aschenbrenner\n01:13:16\nThe beatings will continue until morale improves.\nDwarkesh Patel\n01:13:23\nSuppose we live in a world where these secrets are locked down, but China realizes this progress is happening in America.\nLeopold Aschenbrenner\n01:13:39\nThe secrets probably won't be locked down.\nWe’re probably going to live in the bad world. It's going to be really bad.\nDwarkesh Patel\n01:13:46\nWhy are you so confident they won't be locked down?\nLeopold Aschenbrenner\n01:13:48\nI'm not confident they won't be locked down, but it's just not happening.\nDwarkesh Patel\n01:13:52\nLet’s say tomorrow, the lab leaders get the message. How hard is it? What do they have to do? Do they get more security guards? Do they air-gap? What do they do?\nLeopold Aschenbrenner\n01:14:03\nPeople have two reactions: \"we're already secure.\" We’re not.\nThen there's fatalism: \"it's impossible.\"\nYou need to stay ahead of the curve of how AGI-pilled the CCP is.\nRight now, you've got to be resistant to normal economic espionage. They're not. I probably wouldn't be talking about this stuff if the labs were. I wouldn't want to wake up the CCP more. But this stuff is really trivial for them to do right now.\nSo they're not resistant to that. It would be possible for a private company to be resistant to it. Both of us have friends in the quantitative trading world. Those secrets are shaped similarly where if I got on a call for an hour with somebody from a competitor firm, most of our alpha would be gone.\nDwarkesh Patel\n01:14:59\nYou're going to worry about that pretty soon.\nLeopold Aschenbrenner\n01:15:04\nAll the alpha could be gone but in fact, their alpha often persists for many years and decades. So this doesn't seem to happen. There's a lot you could do if you went from current startup security to good private sector security: hedge funds, the way Google treats customer data or whatever. That'd be good right now.\nThe issue is that basically the CCP will also get more AGI-pilled. At some point, we're going to face the full force of the Ministry of State Security. You're talking about smart people underrating espionage and the insane capabilities of states. This stuff is wild. There are papers about how you can find out the location of where you are in a video game map just from sounds. States can do a lot with electromagnetic emanations.\nAt some point, you have to be working from a\nSCIF\n. Your cluster needs to be air-gapped and basically be a military base. You need to have intense security clearance procedures for employees. All this shit is monitored. They basically have security guards. You can't use any other dependencies. It's all got to be intensely vetted. All your hardware has to be intensely vetted.\nIf they actually really face the full force of state-level espionage, this isn’t really the thing private companies can do empirically. Microsoft recently had\nexecutives' emails hacked by Russian hackers\n, and\ngovernment emails they've hosted hacked by government actors\n. Also, there's just a lot of stuff that only the people behind the security clearances know and only they deal with.\nTo actually resist the full force of espionage, you're going to need the government. We could do it by always being ahead of the curve. I think we're just going to always be behind the curve, unless we get a sort of government project.\nDwarkesh Patel\n01:16:57\nGoing back to the naive perspective, we're very much coming at this from, “there's going to be a race and the CCP, we must win.” Listen, I understand bad people are in charge of the Chinese government, with the CCP and everything.\nI want to step back to a sort of galactic perspective. Humanity is developing AGI. Do we want to come at this from the perspective of \"we need to beat China\"? To our superintelligent\nJupiter brain\ndescendants, China will be some distant memory that they have, America too.\nShouldn't it be more, as an initial approach, just going to them like, “listen, this is superintelligence. We come from a cooperative perspective.” Why immediately rush into it from a hawkish, competitive perspective?\nLeopold Aschenbrenner\n01:17:47\nA lot of the stuff I talk about in the series is primarily descriptive. On the China stuff,  in some ideal world, it's just all merry-go-round and cooperation. Again, people wake up to AGI. The issue in particular is, can we make a deal? Can we make an international treaty? It really relates to the stability of international arms control agreements.\nWe did very successful\narms control on nuclear weapons\nin the 1980s. The reason it was successful is because the new equilibrium was stable. You go down from 60,000 nukes to 10,000 nukes or whatever. When you have 10,000 nukes, breakout basically doesn't matter that much.\nSuppose the other guy now tried to make 20,000 nukes. Who cares? It's still mutually assured destruction. Suppose a rogue state went from zero nukes to one nuke. Who cares? We still have way more nukes than you. It's still not ideal for destabilization.\nIt'd be very different if the arms control agreement had been zero nukes. At zero nukes, you just need one rogue state to make one nuke and the whole thing is destabilized. Breakout is very easy. Your adversary state starts making nukes.\nWhen you're going to very low levels of arms or when you're in a very dynamic technological situation, arms control is really tough because breakout is easy. There are some other stories about this in the 1920s and 1930s. All the European states had disarmed.\nGermany did this kind of crash program to build the\nLuftwaffe\n. That was able to massively destabilize things because they were the first. They were able to pretty easily build a modern air force because the others didn't really have one. That really destabilized things.\nThe issue with AGI and superintelligence is the explosiveness of it. If you have an intelligence explosion, you're able to go from AGI to superintelligence. That superintelligence is decisive because you’ll developed some crazy WMD or you’ll have some super hacking ability that lets you completely deactivate the enemy arsenal. Suppose you're trying to put in a break. We're both going to cooperate. We're going to go slower on the cusp of AGI.\nThere is going to be such an enormous incentive to race ahead, to break out. We're just going to do the intelligence explosion. If we can get three months ahead, we win. That makes any sort of arms control agreement very unstable in a close situation.\nDwarkesh Patel\n01:20:15\nThat's really interesting. This is very analogous to a debate I had with\nRhodes on the podcast\nwhere he argued for nuclear disarmament. If some country tried to break out and started developing nuclear weapons, the six months you would get is enough to get international consensus and invade the country and prevent them from getting nukes. I thought that was not stable equilibrium.\nOn this, maybe it's a bit easier because you have AGI and so you can monitor the other person's cluster or something. You can see the data centers from space. You can see the energy draw they're getting. As you were saying, there are a lot of ways to get information from an environment if you're really dedicated. Also, unlike nukes, the data centers are fixed. Obviously, you have nukes in submarines, planes, bunkers, mountains, etc. You can have them so many different places. A 100 GW data center, we can blow that shit up if we're concerned. We can just use a cruise missile or something. That's very vulnerable.\nLeopold Aschenbrenner\n01:21:19\nThat gets to the insane vulnerability and the volatility of this period, post-superintelligence. You have the intelligence explosion. You have these vastly superhuman things on your cluster. You haven't done the industrial explosion yet. You don't have your robots yet. You haven't covered the desert in robot factories yet.\nThat is this crazy moment. Say the United States is ahead. The CCP is somewhat behind. There's actually an enormous incentive for first strike, if they can take out your data center. They know you're about to have this command, a decisive lead. They know if they can just take out this data center, then they can stop it. They might get desperate.\nWe're going to get into a position that's going to be pretty hard to defend early on. We're basically going to be in a position where we're protecting data centers with the threat of nuclear retaliation. Maybe it sounds kind of crazy.\nDwarkesh Patel\n01:22:10\nIs this the inverse of the\nEliezer\n…?\nLeopold Aschenbrenner\n01:22:14\nNuclear deterrence for data centers. This is Berlin in the late 1950s, early 1960s. Both Eisenhower and Kennedy multiple times made the threat of full-on nuclear war against the Soviets if they tried to\nencroach on West Berlin\n.\nIt's sort of insane. It's kind of insane that that went well. Basically, that's going to be the only option for the data centers. It's a terrible option. This whole scheme is terrible. Being in  neck and neck race at this point is terrible.\nI have some uncertainty on how easy that decisive advantage will be. I'm pretty confident that if you have superintelligence, you have two years, you have the robots, you're able to get that 30-year lead. Then you're in this Gulf War 1 situation. You have your millions or billions of mosquito-sized drones that can just take it out. There's even a possibility you can get a decisive advantage earlier.\nThere are these stories about colonization in the 1500s where\na few hundred Spaniards were able to topple the Aztec Empire\n, a couple of other empires as well. Each of these had a few million people. It was not a godlike technological advantage. It was some technological advantage. It was some amount of disease and cunning strategic play.\nThere's a possibility that even early on — when you haven't gone through the full industrial explosion yet — you have superintelligence, but you're able to manipulate the opposing generals, claiming you're allying with them. Then you have some crazy new bioweapons. Maybe there's even some way to pretty easily get a paradigm that deactivates enemy nukes. This stuff could get pretty wild.\nHere's what we should do. I really don't want this volatile period. A deal with China would be nice. It's going to be really tough if you're in this unstable equilibrium. We want to get in a position where it is clear that the United States, a coalition of democratic allies, will win. It is clear to the United States, it is clear to China. That will require having locked down the secrets, having built the 100 gigawatt cluster in the United States, having done the natural gas and doing what's necessary.\nWhen it is clear that the democratic coalition is well ahead, you go to China and offer them a deal. China will know we’re going to win. They're very scared of what's going to happen. We're going to know we're going to win, but we're also very scared of what's going to happen because we really want to avoid this kind of breakneck race right at the end. Things could really go awry.\nWe offer them a deal. There's an incentive to come to the table. There's a more stable arrangement you can do. It's an Atoms for Peace arrangement. We're like, \"look, we're going to respect you. We're not going to use superintelligence against you. You can do what you want. You're going to get your slice of the galaxy.\nWe're going to benefit-share with you. We're going to have some compute agreement where there's some ratio of compute that you're allowed to have, enforced with opposing AIs or whatever. We're just not going to do this volatile WMD arms race to the death.\nIt's a new world order that's US-led, democracy-led, but respects China and lets them do what they want.\nDwarkesh Patel\n01:25:11\nThere's so much there. On the galaxies thing, there’s a funny anecdote. I kind of want to tell it. We were at an event. I'm respecting\nChatham House rules\nhere. I'm not revealing anything about it. Leopold was talking to somebody influential. Afterwards, that person told the group, \"Leopold told me he's not going to spend any money on consumption until he's ready to buy galaxies.\"\nThe guy goes, \"I honestly don't know if he meant galaxies like the brand of private plane\nGalaxy\nor physical galaxies.\" There was an actual debate. He went away to the restroom. There was an actual debate among influential people about whether he meant Galaxys. Others who knew you better were like, \"no, he means galaxies.\"\nLeopold Aschenbrenner\n01:26:02\nI meant the galaxies. There are two ways to buy the galaxies. At some point, post-superintelligence, there’s some crazy...\nDwarkesh Patel\n01:26:13\nI'm laughing my ass off, not even saying anything. Wee were having this debate. Leopold comes back. Someone says, \"oh, Leopold, we're having this debate about whether you meant you want to buy the Galaxy, or you want to buy the other thing.\" Leopold assumes they must mean not the private plane Galaxy vs. the actual galaxy, but whether he wants to buy the property rights of the galaxy or actually just send out the probes right now.\nLeopold Aschenbrenner\n01:26:43\nExactly.\nDwarkesh Patel\n01:26:49\nAlright, back to China. There's a whole bunch of things I could ask about that plan and whether you’re going to get a credible promise to get some part of galaxies.\nLeopold Aschenbrenner\n01:27:02\nYou’ll have AIs to help you enforce stuff.\nDwarkesh Patel\n01:27:05\nSure, we’ll leave that aside. That’s a different rabbit hole. The thing I want to ask is...\nLeopold Aschenbrenner\n01:27:09\nThe only way this is possible is if we lock it down. If we don't lock it down, we are in this fever struggle. Greatest peril mankind will have ever seen.\nDwarkesh Patel\n01:27:19\nDuring this period, they don’t really understand how this AI governance is going to work, whether they’re going to check, whether we’re going to adjugate the galaxies. The data centers can't be built underground. They have to be above ground. Taiwan is right off the coast of China. The US needs the chips from there.\nWhy isn’t China just going to invade? Worst case scenario for them is the US wins the superintelligence, which we’re on track to do anyway. Wouldn't this instigate them to either invade Taiwan or blow up the data center in Arizona or something like that?\nLeopold Aschenbrenner\n01:27:52\nYou talked about the data center. You'd probably have to threaten nuclear retaliation to protect that. They might just blow it up. There are also ways they can do it without attribution.\nDwarkesh Patel\n01:28:00\nStuxnet.\nLeopold Aschenbrenner\n01:28:03\nStuxnet, yeah. We’ll talk about later, but we need to be working on a Stuxnet for the Chinese project. I talk about AGI by 2027 or whatever. On Taiwan, do you know about the terrible twenties?\nDwarkesh Patel\n01:28:23\nNo.\nLeopold Aschenbrenner\n01:28:25\nIn Taiwan watcher circles, people often talk about the late 2020s as the\nmaximum period of risk for Taiwan\n. Military modernization cycles and extreme fiscal tightening on the US military budget over the last decade or two have meant that we’re in a trough by the late twenties in terms of overall naval capacity.\nThat’s when China is saying they want to be ready. It’s already  kind of a parallel timeline there. Yeah, it looks appealing to invade Taiwan. Maybe not because of the remote cut off chips, which\ndeactivates\nthe machines. But imagine if during the Cold War, all of the world’s uranium deposits had been in Berlin. Berlin already almost caused a nuclear war multiple times. God help us all.\nDwarkesh Patel\n01:29:18\nLeslie Groves\nactually had a plan after the war that America would go around the world getting the rights to every single uranium deposit because they didn’t realize how much uranium there was in the world. They thought this was feasible. They didn’t realize, of course, that there were\nhuge deposits in the Soviet Union\nitself.\nLeopold Aschenbrenner\n01:29:37\nEast Germany, too. A lot of East German workers got screwed and got cancer.\nDwarkesh Patel\n01:29:42\nThe framing we’ve been assuming— I’m not sure I buy it yet—is that the United States has this leverage. This is our data center. China is the competitor right now. Obviously, that’s not the way things are progressing. Private companies control these AIs. They’re deploying them. It’s a market-based thing. Why will it be the case that the United States has this leverage or is doing this thing versus China doing this thing?\nLeopold Aschenbrenner\n01:30:11\nThere are descriptive and prescriptive claims, or normative and positive claims. The main thing I’m trying to say is, at these SF parties, people talk about AGI and always focus on private AI labs. I want to challenge that assumption.\nIt seems likely to me, for reasons we’ve discussed, that the national security state will get involved. There are many ways this could look: nationalization, a public-private partnership, a defense contractor-like relationship, or a government project that absorbs all the people. There’s a spectrum, but people vastly underrate the chances of this looking like a government project.\nWhen we have literal superintelligence on our cluster — with a billion superintelligent scientists who can hack everything and Stuxnet the Chinese data centers, and build robo armies — you really think it’ll be a private company ? The government would be like, \"oh, my God, what is going on?\"\n(01:31:23) – State-led vs. private-led AI\nDwarkesh Patel\n01:31:23\nSuppose there’s no China. Suppose there are countries like Iran and North Korea that theoretically could achieve superintelligence, but they’re not on our heels. In that world, are you advocating for a national project or do you prefer the private path forward?\nLeopold Aschenbrenner\n01:31:40\nTwo responses to this. One is, you still have Russia and other countries.\nYou need Russia-proof security. You can’t let Russia steal all your stuff. Their clusters may not be as big, but they can still make crazy bioweapons and mosquito-sized drone swarms.\nThe security component is a large part of the project because there’s no other way to prevent this from instantly proliferating to everyone. You still have to deal with Russia, Iran, and North Korea. Saudi and Iran will try to get it to screw each other. Pakistan and India will try to get it to screw each other. There’s enormous destabilization.\nStill, I agree with you. If AGI had emerged in 2005, during unparalleled American hegemony, there would have been more scope for less government involvement. But as we discussed, that would have been a unique moment in history. In almost all other moments in history, there would have been a great power competitor.\nDwarkesh Patel\n01:32:49\nLet’s get into this debate. My position is this. If you look at the people who were involved in the Manhattan Project, many of them regretted their participation. We can infer from this that we should start with a cautious approach to the nationalized\nASI\nproject.\nLeopold Aschenbrenner\n01:33:14\nDid they regret their participation because of the project or because of the technology itself? People will regret it, but it's about the nature of the technology, not the project.\nDwarkesh Patel\n01:33:24\nThey probably had a sense that different decisions would have been made if it wasn’t a concerted effort that everyone agreed to participate in. If it wasn’t in the context of a race to beat Germany and Japan, you might not develop it. That’s the technology part.\nLeopold Aschenbrenner\n01:33:40\nIt’s still going to be a weapon because of the destructive potential, the military potential. It’s not because of the project. It’s because of the technology. That will unfold regardless.\nImagine you go through the 20th century in a decade—\nDwarkesh Patel\n01:34:01\nLet’s run that example. Suppose the 20th century was run through in one decade.\nDo you think the technologies that happened during the 20th century shouldn’t have been privatized? Should it have been a more concerted, government-led project?\nLeopold Aschenbrenner\n01:34:21\nThere’s a history of dual-use technologies. AI will be dual-use in the same way. There will be lots of civilian uses of it. Like with nuclear energy, the government project developed the military angle of it and then worked with private companies. There was a flourishing of nuclear energy until the environmentalists stopped it.\nPlanes, like Boeing. Actually, the Manhattan Project wasn’t the biggest defense R&D project during World War II. It was the\nB-29 bomber\nbecause they needed a bomber with a long enough range to reach Japan to destroy their cities. Boeing made the\nB-47\n, and the\nB-52\nplane the US military uses today. They used that technology later on to build the\n707\n.\nDwarkesh Patel\n01:35:08\nWhat does \"later on\" mean in this context? I get what it means after a war to privatize. But if the government has ASI...\nLet me back up and explain my concern.\nYou have this institution in our society with a\nmonopoly on violence\n. We’re going to give it access to ASI that’s not broadly deployed. This maybe sounds silly, but we’re going to go through higher levels of intelligence. Private companies will be required by regulation to increase their security. They’ll still be private companies.\nThey’ll deploy this and release AGI. Now McDonald’s, JP Morgan, and some random startup will be more effective organizations because they have AGI workers. It’ll be like the\nIndustrial Revolution\n, where the benefits were widely diffused.\nBacking up, what is it we’re trying to do? Why do we want to win against China? We want to win because we don’t want a top-down authoritarian system to win. If the way to beat that is for the most important technology for humanity to be controlled by a top-down government, what’s the point?\nLet’s run our cards with privatization. That’s how we get to the classic liberal, market-based system we want for the ASIs.\nLeopold Aschenbrenner\n01:36:32\nAll right, there’s a lot to talk about here. I’ll start by looking at what the private world would look like. This is part of why there's no alternative. Then let’s look at what the government project looks like, what checks and balances look like, and so on.\nLet’s start with the private world. A lot of people talk about\nopen source\n. There’s a misconception that AGI development will be a beautiful, decentralized thing, a giddy community of coders collaborating. That’s not how it’s going to look. It’s a $100 billion or trillion-dollar cluster. Not many people will have it.\nRight now, open source is good because people use the stuff that’s published. They use the published algorithms, or, like\nMistral\n, they leave DeepMind, take all the secrets, and replicate it.\nThat’s not going to continue. People also say stuff like, “10^26 flops will be in my phone.” No, it won’t.\nMoore’s Law\nis really slow. AI chips are getting better but the $100 billion computer won’t cost $1,000 within your lifetime. So it’s going to be like two or three big players in the private world.\nYou talk about the enormous power that superintelligence and the government will have. It’s pretty plausible that in the alternative world one AI company will have that power. Say OpenAI has a six-month lead. You’re talking about the most powerful weapon ever. You’re making a radical bet on a private company CEO as the benevolent dictator.\nDwarkesh Patel\n01:38:17\nNot necessarily. Like any other thing that’s privatized, we don’t count on them being benevolent. Think of someone who manufactures industrial fertilizer. This person with this factory, if they went back to an ancient civilization, they could blow up Rome. They could probably blow up Washington, DC.\nLeopold Aschenbrenner\n01:38:31\nIndeed.\nDwarkesh Patel\n01:38:36\nIn your series, you talk about Tyler Cowen’s phrase of “\nmuddling through\n.” Even with privatization, people underrate that there are a lot of private actors who control vital resources like the water supply.\nWe can count on cooperation and market-based incentives to maintain a balance of power. Sure, things are proceeding really fast.\nWe have a lot of historical evidence that this works best.\nLeopold Aschenbrenner\n01:39:02\nWhat do we do with nukes, right? We don't keep nukes in check by beefing up the Second Amendment so each state has its own nuclear arsenal. Dario and Sam don’t have their own little arsenal.\nNo, it’s institutions, constitutions, laws, and courts. I’m not sure this balance of power analogy holds. The government having the biggest guns was an enormous civilizational achievement, like\nLandfrieden\nin the\nHoly Roman Empire\n. If someone from the neighboring town committed a crime, you didn’t start a battle between the towns. You took it to a court of the Holy Roman Empire. They decided it. It’s a big achievement.\nThe key differences with the analogy about the industrial fertilizer are speed and offense-defense balance issues. It’s like compressing the 20th century into a few years. That is incredibly scary because of the rapid advancement in destructive technology and military advancements.\nYou'd go from bayonets and horses to tank armies and fighter jets in a couple of years. In just a few more years you’d have nukes, ICBMs, and stealth. That speed creates an incredibly volatile and dangerous period. We have to make it through that, which will be incredibly challenging.\nThat’s where a government project is necessary. If we can make it through that, the situation stabilizes. We don’t face this imminent national security threat. Yes, there were WMDs that developed, but we’ve managed to create a stable offense-defense balance.\nBioweapons are a huge issue initially. An attacker can create 1000 different synthetic viruses and spread them. It’s hard to defend against each. Maybe at some point, you figure out a universal defense against every possible virus, then you’re in a stable situation again on the offense-defense balance. Or like with planes,  you restrict certain capabilities that the private sector isn’t allowed to have, then you can let the civilian uses run free.\nDwarkesh Patel\n01:41:20\nI’m skeptical of this.\nLeopold Aschenbrenner\n01:41:23\nThis is the other important thing. I talked about one company having all this power. It is unprecedented because the industrial fertilizer guy cannot overthrow the US government. It’s quite plausible that the AI company with superintelligence can.\nDwarkesh Patel\n01:41:41\nThere would be multiple AI companies, right? I buy that one of them could be ahead.\nLeopold Aschenbrenner\n01:41:44\nIt’s not obvious that it’ll be multiple. If there’s a six-month lead, maybe there are two or three.\nDwarkesh Patel\n01:41:49\nI agree.\nLeopold Aschenbrenner\n01:41:50\nIf there are two or three, then it’s a crazy race between these companies.\nDemis\nand Sam would be like, \"I don’t want to let the other one win.\" They’re both developing their nuclear arsenals and robots.\nCome on. The government is not going to let these people do that. Is Dario going to be the one developing super hacking Stuxnet and deploying it against the Chinese data center?\nThe other issue is that if it’s two or three, it won’t just be two or three. It’ll be China, Russia, and North Korea too. In the private lab world, there’s no way they’ll have good enough security.\nDwarkesh Patel\n01:42:23\nWe’re also assuming that if you nationalize it, especially in a world where this stuff is priced in by the CCP, you’ve got it nailed down. I’m not sure why we would expect that.\nLeopold Aschenbrenner\n01:42:36\nThe government’s the only one who does this stuff.\nDwarkesh Patel\n01:42:40\nIf we don’t trust Sam or Dario to be benevolent dictators…\nLeopold Aschenbrenner\n01:42:44\nJust corporate governance in general.\nDwarkesh Patel\n01:42:47\nBecause you can cause a coup, the same capabilities are going to be true of the government project, right? The modal president in 2025, Donald Trump, will be the person versus you not trusting Sam or Dario to have these capabilities. I agree that if Sam and Dario have a one-year lead on ASI, in that world I’m concerned about privatization.\nIn that exact same world, I’m very concerned about Donald Trump having the capability. Potentially, if the takeoff is slower than anticipated, I prefer the private companies in that world. In no part of this matrix is it obviously true that the government-led project is better.\nLeopold Aschenbrenner\n01:43:31\nLet’s talk about the government project and checks and balances.\nIn some sense, my argument is a\nBurkean\none. American checks and balances have held for over 200 years through crazy technological revolutions. The US military could kill every civilian in the United States.\nDwarkesh Patel\n01:43:46\nYou’re going to make that argument. The private-public balance of power has held for hundreds of years.\nLeopold Aschenbrenner\n01:43:51\nBut, why has it held? It’s because the government has had the biggest guns. Never before has a single CEO or a random nonprofit board had the ability to launch nukes.\nWhat is the track record of government checks and balances versus the track record of the private company checks and balances? Well the AI lab's first stress test went really badly.\nEven worse in the private company world, it’s two private companies and the CCP. They’ll just instantly have all the tech. They probably won’t have good enough internal control. It’s not just the random CEO, but rogue employees who can use these superintelligences to do whatever they want.\nDwarkesh Patel\n01:44:30\nThis won’t be true of the government? Rogue employees won’t exist on the project?\nLeopold Aschenbrenner\n01:44:33\nThe government has actual decades of experience and actually cares about this stuff. They deal with nukes and really powerful technology. This is the stuff that the national security state cares about.\nLet's talk about government checks and balances a little bit. What are checks and balances in the government world? First, it’s important to have some international coalition. I talked about these two tiers before. The inner tiers are modeled on the\nQuebec Agreement\n, Churchill and Roosevelt agreeing to pool efforts on nukes but not using them against each other, or anyone else without consent.\nBring in the UK with DeepMind, Southeast Asian states with the chip supply chain, and more NATO allies with talent and industrial resources. You have those checks and balances with more international countries at the table.\nSeparately, you have the second tier of coalitions, the Atoms for Peace thing. You go to countries including the UAE and make a deal similar to the\nNPT\n. They’re not allowed to do crazy military stuff, but we’ll share civilian applications. We’ll help them and share the benefits, creating a new post-superintelligence world order.\nUS checks and balances: Congress will have to be involved to appropriate trillions of dollars. Ideally, Congress needs to confirm whoever’s running this. You have Congress, different factions of the government, and the courts. I expect the First Amendment to remain really important.\nThis sounds crazy to people, but these institutions have withstood the test of time in a powerful way. This is why alignment is important. You program AIs to follow the constitution. The military works because generals are\nnot allowed to follow unlawful\nor\nunconstitutional\norders. You have the same thing for the AIs.\nDwarkesh Patel\n01:46:33\nSo what’s wrong with this argument. Maybe you have a point in a world with an extremely fast takeoff, one year from AGI to ASI.\nLeopold Aschenbrenner\n01:46:41\nThen you have the years after ASI where you have this extraordinary explosion and technological progress.\nDwarkesh Patel\n01:46:48\nMaybe you have a point. We don’t know. You have arguments for why that’s a more likely world, but maybe that’s not the world we live in.\nIn the other world, I’m very much on the side of ensuring these things are privately held. When you nationalize, that’s a one-way function. You can’t go back.\nWhy not wait until we have more evidence on which world we live in? Rushing nationalization might be a bad idea while we’re uncertain. I’ll let you respond to that first.\nLeopold Aschenbrenner\n01:47:18\nI don’t expect us to nationalize tomorrow. If anything I expect it to be like COVID,where it’s kind of too late. Ideally, you nationalize early enough to lock stuff down. It’ll probably be chaotic. You’ll be trying to do a crash program to lock stuff down. It’ll be kind of late. It’ll be clear what’s happening. We’re not going to nationalize when it’s not clear what’s happening.\nDwarkesh Patel\n01:47:37\nThe argument that these institutions have held up historically so well is flawed. They’ve actually almost broken a bunch of times.\nLeopold Aschenbrenner\n01:47:44\nThey’ve held up. They didn’t break the first time they were tested.\nDwarkesh Patel\n01:47:46\nThis is similar to the argument that some people make about nuclear war: we’ve had nukes for 80 years and have been fine, so the risk must be low. The answer to that is no. The risk is really high. We’ve avoided it because people have made a lot of effort to prevent it. Giving the government ASI without knowing the implications isn’t making that effort.\nLook at the base rate. America is very exceptional, not just in terms of avoiding dictatorship. Every other country in history has had a complete drawdown of wealth because of war, revolution, etc. America is very unique in not having had that.\nWe have to think of the historical base rate. We haven’t thought about great power competition in the last 80 years, but it’s significant. Dictatorship is the default state of mankind. Relying on institutions in an ASI world is fundamentally different. Right now, if the government tried to overthrow, it’s much harder without ASI. There are people with AK-47s and AR-15s, making it harder.\nLeopold Aschenbrenner\n01:48:58\nThe government could crush the AR-15s.\nDwarkesh Patel\n01:49:00\nNo, it would actually be pretty hard. It’s the reason why Vietnam and Afghanistan were so hard.\nLeopold Aschenbrenner\n01:49:03\nThey could just nuke the whole country.\nDwarkesh Patel\n01:49:05\nI agree.\nLeopold Aschenbrenner\n01:49:07\nThey could. It’s similar to the ASI.\nDwarkesh Patel\n01:49:10\nIt’s just easier if you have what you were talking.\nLeopold Aschenbrenner\n01:49:14\nNo, there are institutions, constitutions, legal restraints, courts, and checks and balances. The crazy bet is the bet on a private company CEO.\nDwarkesh Patel\n01:49:21\nIsn’t the same thing true of nukes where we have institutional agreements about non-proliferation? We’re still very concerned about those being broken and someone getting nukes. We stay up at night worrying about that situation.\nLeopold Aschenbrenner\n01:49:31\nIt’s a precarious situation. ASI will a precarious situation as well. Given how precarious nukes are, we’ve done pretty well.\nDwarkesh Patel\n01:49:40\nWhat does privatization in this world even mean?\nWhat happens after?\nLeopold Aschenbrenner\n01:49:44\nWe’re talking about whether the government project is good or not. I have very mixed feelings about this as well.\nMy primary argument is that if you’re at the point where this thing has vastly superhuman capabilities — it can develop crazy\nbioweapons targeted to kill everyone but the Han Chinese\n, it can wipe out entire countries, it can build robo armies and drone swarms with mosquito-sized drones — the US national security state will be intimately involved.\nThe government project will look like a joint venture between cloud providers, labs, and the government. There is no world in which the government isn’t involved in this crazy period. At the very least, intelligence agencies need to run security for these labs. They’re already controlling access to everything.\nIf we’re in a volatile international situation, initial applications will focus on stabilizing it. It’ll suck. It’s not what I want to use ASI for. Somehow we need to prevent proliferation of new WMDs, and maintain mutually assured destruction with North Korea, Russia, and China.\nThere’s a broader spectrum than you’re acknowledging. In a world with private labs, there will be heavy government involvement. What we’re debating is the form of government involvement, but it will look more like the national security state than a startup, which is what it is right now.\nDwarkesh Patel\n01:51:30\nSomething like that makes sense. I’d be very worried if it’s like the Manhattan Project, where it’s directly part of the US military. If it’s more like needing to talk to\nJake Sullivan\nbefore running the next training line...\nLeopold Aschenbrenner\n01:51:44\nIs Lockheed Martin’s\nSkunk Works\npart of the US military? They call the shots.\nDwarkesh Patel\n01:51:48\nI don’t think that’s great and I think that’s bad if it happens with ASI. What’s the scenario?\nLeopold Aschenbrenner\n01:51:55\nWhat’s the alternative? What’s the alternative?\nDwarkesh Patel\n01:51:57\nIt’s closer to my end of the spectrum. You talk to Jake Sullivan before you can launch the next training cluster, but many companies are still going for it, and the government is involved in security.\nLeopold Aschenbrenner\n01:52:13\nIs Dario launching the Stuxnet attack?\nDwarkesh Patel\n01:52:16\nWhat do you mean by launching?\nLeopold Aschenbrenner\n01:52:20\nDario is deactivating the Chinese data centers?\nDwarkesh Patel\n01:52:22\nThis is similar to the story you could tell about Big Tech right now.\nSatya\n, if he wanted to, could get his engineers to find zero days in Windows and infiltrate the president’s computer. Right now, Satya could do that.\nLeopold Aschenbrenner\n01:52:41\nThey’d be shut down.\nDwarkesh Patel\n01:52:42\nWhat do you mean?\nLeopold Aschenbrenner\n01:52:43\nThe government wouldn’t let them do that.\nDwarkesh Patel\n01:52:44\nThere’s a story where they could pull off a coup.\nLeopold Aschenbrenner\n01:52:49\nThey could not pull off a coup.\nDwarkesh Patel\n01:52:52\nFine, I agree. What’s wrong with a scenario where multiple companies are going for it?\nThe AI is still broadly deployed. Alignment works. The system-level prompt is that it can’t help people make bioweapons or something. It’s still broadly deployed.\nLeopold Aschenbrenner\n01:53:14\nI expect AIs to be broadly deployed.\nDwarkesh Patel\n01:53:16\nEven if it’s a government project?\nLeopold Aschenbrenner\n01:53:18\nYeah, I think the Metas of the world open-sourcing their AIs that are two years behind is super valuable. There will be some question of whether the offense-defense balance is fine, and open-sourcing two-year-old AIs is fine. Or there are restrictions on the most extreme dual-use capabilities, like not letting private companies sell crazy weapons.\nThat’s great and will help with diffusion. After the government project, there will be an initial tense period. Hopefully, it stabilizes. Then, like Boeing, they’ll go out and do all the flourishing civilian applications like nuclear energy. The civilian applications will have their day.\nDwarkesh Patel\n01:53:57\nHow does that proceed? Because in the other world, there are existing stocks of capital that are worth a lot.\nLeopold Aschenbrenner\n01:54:02\nThere will still be Google clusters.\nDwarkesh Patel\n01:54:04\nSo Google, because they got the contract from the government, will control the ASI? But why are they trading with anybody else?\nLeopold Aschenbrenner\n01:54:11\nIt’ll be the same companies that would be doing it anyway. In this world, they’re just contracting with the government or are\nDPA\n’d so all their compute goes to the government. In some sense it’s very natural.\nDwarkesh Patel\n01:54:22\nAfter you get the ASI and we’re building the robot armies and fusion reactors…\nLeopold Aschenbrenner\n01:54:33\nOnly the government will get to build robot armies.\nDwarkesh Patel\n01:54:35\nNow I’m worried. Or like the fusion reactors and stuff.\nLeopold Aschenbrenner\n01:54:38\nThat’s what we do with nukes today.\nDwarkesh Patel\n01:54:40\nIf you already have the robot armies and everything, the existing society doesn’t have some leverage where it makes sense for the government to—\nLeopold Aschenbrenner\n01:54:45\nThey don’t have that today.\nDwarkesh Patel\n01:54:46\nThey do, in the sense that they have a lot of capital that the government wants. There are other things as well. Why was Boeing privatized after WWII?\nLeopold Aschenbrenner\n01:54:54\nThe government has the biggest guns. The way we regulate is through institutions, constitutions, legal restraints, courts, etc.\nDwarkesh Patel\n01:54:57\nTell me what privatization looks like in the ASI world afterwards.\nLeopold Aschenbrenner\n01:55:00\nAfterwards, it’s like the Boeing example.\nDwarkesh Patel\n01:55:03\nWho gets it?\nLeopold Aschenbrenner\n01:55:04\nGoogle and Microsoft, the AI labs—\nDwarkesh Patel\n01:55:05\nWho are they selling it to? They already have robot factories. Why are they selling it to us? They don’t need anything from us. This is chump change in the ASI world because we didn’t get the ASI broadly deployed throughout this takeoff.\nWe don’t have the robots, the fusion reactors, or the advanced decades of science you’re talking about. What are they trading with us for?\nLeopold Aschenbrenner\n01:55:26\nTrading with whom for?\nDwarkesh Patel\n01:55:29\nFor everybody who was not part of the project. They’ve got technology that’s decades ahead.\nLeopold Aschenbrenner\n01:55:33\nThat’s a whole other issue of how economic distribution works. I don’t know. That’ll be rough. I’m just saying, I don’t see the alternative. The alternative is overturning a 500-year civilizational achievement of Landfrieden. You basically instantly leak the stuff to the CCP.\nEither you barely scrape ahead, but you’re in a fever struggle, proliferating crazy WMDs. It’s enormously dangerous for alignment because you’re in a crazy race at the end, and you don’t have the ability to take six months to get alignment right. The alternative is not bundling efforts to win the race against authoritarian powers.\nI don’t like it. I wish we used the ASI to cure diseases and do all the good in the world. But it’s my prediction that in the end game, what’s at stake is not just cool products but whether liberal democracy survives, whether the CCP survives.\nWhat will the world order for the next century be? When that is at stake, forces will be activated that are way beyond what we’re talking about now. In this crazy race at the end, the national security implications will be the most important.\nTo go back to World War II, nuclear energy had its day, but in the initial period when the technology was first discovered, you had to stabilize the situation. You had to get nukes and do it right. Then the civilian applications had their day.\nDwarkesh Patel\n01:57:12\nI agree that nuclear energy is a thing that happened later on and is dual-use. But it’s something that happened literally a decade after nuclear weapons were developed.\nLeopold Aschenbrenner\n01:57:21\nRight because everything took a long time.\nDwarkesh Patel\n01:57:23\nWhereas with AI, all the applications are immediately unlocked. This is closer to the analogy people make about AGI. Assume your society had 100 million more\nJohn von Neumanns\n.\nIf that literally happened, if tomorrow you had 100 million more of them, I don’t think the approach would be that we have to worry about some of them converting to ISIS or “what if a bunch are born in China?” I don’t think we’d be talking about nationalizing all the John von Neumanns.\nI think it’d generally be a good thing. I’d be concerned about one power getting all the John von Neumanns.\nLeopold Aschenbrenner\n01:57:57\nThe issue is bottling up, in a short period of time, this enormous unfolding of technological progress, an industrial explosion. We do worry about 100 million John von Neumanns.\nWhy do we worry about the rise of China? It’s one billion people who can do a lot of industry and technology. This is like the rise of China multiplied by 100. It’s not just one billion people, but a billion super-intelligent beings.\nPlus, it comes all in a very short period.\nDwarkesh Patel\n01:58:28\nPractically, if the goal is to beat China, part of that is protecting ourselves.\nLeopold Aschenbrenner\n01:58:34\nBeating China is just one of the goals.\nWe also want to manage this incredibly crazy, scary period.\nDwarkesh Patel\n01:58:40\nRight. Part of that is making sure we’re not leaking algorithmic secrets to them.\nLeopold Aschenbrenner\n01:58:45\nBuilding the trillion-dollar cluster.\nDwarkesh Patel\n01:58:47\nThat’s right.\nBut isn’t your point that Microsoft can issue corporate bonds?\nLeopold Aschenbrenner\n01:58:52\nMicrosoft can do hundreds of billions of dollars. The trillion-dollar cluster is closer to a national effort.\nDwarkesh Patel\n01:58:57\nI thought your earlier point was that American capital markets are deep and good.\nLeopold Aschenbrenner\n01:59:02\nThey’re pretty good. The trillion-dollar cluster is possible privately, but it’s going to be tough.\nDwarkesh Patel\n01:59:06\nAt this point, we have AGI that’s rapidly accelerating productivity.\nLeopold Aschenbrenner\n01:59:10\nThe trillion-dollar cluster will be planned before the AGI. You get the AGI on the 10 GW cluster. Maybe have one more year of final unhobbling to fully unlock it. Then you have the intelligence explosion.\nMeanwhile, the trillion-dollar cluster is almost finished. You run your superintelligence on it. You also have hundreds of millions of GPUs on inference clusters everywhere.\nDwarkesh Patel\n01:59:35\nIn this world, I think private companies have their capital and can raise capital.\nLeopold Aschenbrenner\n01:59:40\nYou will need the government to do it fast.\nDwarkesh Patel\n01:59:43\nWe know private companies are on track to do this. In China, if they’re unhindered by climate change or whatever—\nLeopold Aschenbrenner\n01:59:53\nThat’s part of what I’m saying.\nDwarkesh Patel\n01:59:56\nIf it really matters that we beat China…\nThere will be all sorts of practical difficulties. Will the AI researchers actually join the AI effort? If they do, there will be at least three different teams currently doing pre-training in different companies.\nWho decides at some  point that you’re going to have to\nYOLO\nthe hyperparameters. Who decides that? Merging extremely complicated research and development processes across very different organizations is somehow supposed to speed up America against the Chinese?\nLeopold Aschenbrenner\n02:00:34\nBrain and DeepMind merged. It was a little messy, but it was fine.\nDwarkesh Patel\n02:00:36\nIt was pretty messy. It was also the same company and much earlier on in the process.\nLeopold Aschenbrenner\n02:00:40\nPretty similar, right? Different codebases, lots of different infrastructure and teams. It wasn’t the smoothest process, but DeepMind is doing very well.\nDwarkesh Patel\n02:00:49\nYou give the example of COVID. In the COVID example, we woke up to it, maybe it was late, but then we deployed all this money. The COVID response from the government was a clusterfuck. I agree that\nWarp Speed\nwas enabled by the government, but it was literally just giving permission that you can actually—\nLeopold Aschenbrenner\n02:01:08\nIt was also making big advance market commitments.\nDwarkesh Patel\n02:01:10\nI agree. But fundamentally, it was a private sector-led effort. That was the only part of the COVID response that worked.\nLeopold Aschenbrenner\n02:01:17\nThe project will look closer to Operation Warp Speed. You’ll have all the companies involved in the government project. I’m not convinced that merging is that difficult. You run pre-training on GPUs with one codebase, then do the secondary step on the other codebase with TPUs. It’s fine.\nOn whether people will sign up for it, they wouldn’t sign up for it today. It would seem crazy to people.\nBut this is part of the secrecy thing. People gather at parties and… you know this. I don’t think anyone has really gotten up in front of these people and said, \"look, what you’re building is the most important thing for the national security of the United States, for the future of the free world and whether we have another century ahead of it. This is really important for your country and democracy. Don’t talk about the secrets.\" It’s not just about DeepMind or whatever. It’s about these really important things.\nWe’re talking about the Manhattan Project. It was really contentious initially, but at some point it became clear that this was coming. There was an exigency on the military national security front. A lot of people will come around.\nOn whether it’ll be competent, a lot of this stuff is more predictive. This is reasonably likely, and not enough people are thinking about it. A lot of people think about AI lab politics but nobody has a plan for the grand project.\nDwarkesh Patel\n02:02:47\nShould they be more pessimistic about it? We don’t have a plan for it, and we need to act soon because AGI is upon us. The only competent technical institutions capable of making AI right now are private.\nLeopold Aschenbrenner\n02:02:59\nCompanies will play that leading role. It’ll be a partnership.\nWe talked about World War II and American unpreparedness. The beginning of World War II was complete shambles. America has a very deep bench of incredibly competent managerial talent. There are a lot of dedicated people. An Operation Warp Speed-like public-private partnership is what I imagine it would look like.\nDwarkesh Patel\n02:03:26\nRecruiting talent is an interesting question. For the Manhattan Project, you initially had to convince people to beat the Nazis and get on board. Many of them regretted how much they accelerated the bomb. This is generally a thing with war.\nLeopold Aschenbrenner\n02:03:48\nI think they were wrong to regret it.\nDwarkesh Patel\n02:03:51\nWhy?\nLeopold Aschenbrenner\n02:03:54\nWhat’s the reason for regretting it?\nDwarkesh Patel\n02:03:56\nThey way nuclear weapons were developed after the war was explosive because there was a precedent that you can use nuclear weapons. Then because of the race that was set up, you immediately go to the\nH-bomb\n.\nLeopold Aschenbrenner\n02:04:11\nThis is related to my view on AI and maybe where we disagree. That was inevitable. There was a world war, then a cold war. Of course the military angle would be pursued with ferocious intensity. There’s no world in which we all decide not to build nukes. Also, nukes went really well. That could have gone terribly.\nIt’s not physically possible to have something like pocket nukes for everybody, where WMDs proliferated and were fully democratized. The US led on nukes and built a new world order, with a few great powers and a non-proliferation regime for nukes. It was a partnership and a deal: no military application of nuclear technology, but help with civilian technology. They enforced safety norms on the rest of the world. That worked and could have gone much worse.\nNot to mention, I say this in the piece but the A-bomb in Hiroshima was just like firebombing. What changed the game was the H-bombs and\nICBM\ns. That’s when it went to a whole new level.\nDwarkesh Patel\n02:05:27\nWhen you say we will tell people that we need to pursue this project for the free world to survive, it sounds similar to World War II. World War II is a sad story, not only because it happened, but the victory was sad in the sense that Britain went in to protect Poland.\nAt the end, the USSR, which as your family knows is incredibly brutal, ends up occupying half of Europe. The idea of protecting the free world by rushing AI might end up with an American AI Leviathan. We might look back on this with the same twisted irony as Britain going into World War II to protect Poland.\nLeopold Aschenbrenner\n02:06:22\nThere will be a lot of unfortunate things that happen. I'm just hoping we make it through. The pitch won’t only be about the race. The race will be a backdrop. It's important that democracy shapes this technology. We can't leak this stuff to North Korea.\nSafety, including alignment and the creation of new WMDs, is also just  important. I'm not convinced there's another path. Say we have a breakneck race internationally, instantly leaking all this stuff, including the weights, with a commercial race with Demis, Dario, and Sam all wanting to be first. It's incredibly rough for safety.\nSafety regulation, as people talk about it, is like\nNIST\ninvolving years of bureaucracy and expert consensus.\nDwarkesh Patel\n02:07:13\nIsn’t that what’s going to happen with the project as well?\nLeopold Aschenbrenner\n02:07:15\nAlignment during the intelligence explosion is not a years-long bureaucratic process. It's more like a war, with a fog of war. Is it safe to do the next OOM? We're three OOMs into the intelligence explosion, and we don't fully understand what's happening.\nOur generalization-scaling curves don’t look great, some automated AI researchers say it's fine, but we don't quite trust them. AIs might start doing problematic things, but we hammer it out, and then it's fine. Should we go ahead? Should we take another six months?\nMeanwhile, China might steal the weights or deploy a robot army. It's a crazy situation, relying more on a sane chain of command than a deliberative regulatory scheme. Although I wish we could do that more deliberative regulatory scheme.\nThis is the thing with private companies too. Private companies claim they'll do safety, but it’s rough in a commercial race, especially for startups. Startups are startups. They aren't fit to handle WMDs.\nDwarkesh Patel\n02:08:35\nI’m coming closer to your position but…\nLet’s talk about the\nresponsible scaling policies\n. I was told by people advancing this idea — because they know I’m a libertarian-type person and the way they approached me was like this — that the way to think about it was that it's fundamentally a way to protect market-based development of AGI. If you didn’t have this, there would be misuse and lead to nationalization. The RSPs are a way to ensure a market-based order with safeguards to prevent things from going off the rails.\nIt seems like your story is self-consistent. I know this was never your position, so I’m not looping you into this. But it's almost like a\nmotte-and-bailey\nargument.\nLeopold Aschenbrenner\n02:09:36\nHere’s what I think about RSP-type stuff and current safety regulations. They’re important for helping us figure out what world we’re in and flashing the warning signs when we’re close.\nThe story we’ve been telling is what I think the modal version of this decade is. There are many ways it could be wrong. We should talk about the data wall more. There’s a world where this stuff stagnates or we don’t have AGI.\nThe RSPs preserve optionality. Let’s see how things go, but we need to be prepared if the red lights start flashing. If we get the automated AI researcher, then it’s crunch time.\nDwarkesh Patel\n02:10:12\nI can be on the same page with that and have a strong prior on pursuing a market-based way. Unless you’re right about what the intelligence explosion looks like, don’t move yet. But in that world where it really does seem like Alec Radford can be automated, and that's the only bottleneck to getting to ASI…\nOkay I think we can leave it at that. I’m somewhat on the way there.\nLeopold Aschenbrenner\n02:10:42\nI hope it goes well. It's going to be very stressful. Right now is the chill time. Enjoy your vacation while it lasts.\nDwarkesh Patel\n02:10:53\nIt’s funny to look out over this. This is San Francisco.\nLeopold Aschenbrenner\n02:11:00\nYeah OpenAI is right there. Anthropic is there. You guys have this enormous power over how it’s going to go for the next couple of years, and that power is depreciating.\nDwarkesh Patel\n02:11:09\nWho's \"you guys\"?\nLeopold Aschenbrenner\n02:11:10\nPeople at labs.\nIt's a crazy world you're talking about. You mention that maybe they'll nationalize too soon. Almost nobody sees what's happening. This is what I find stressful about all this.\nMaybe I'm wrong, but if I'm right, we’re in this crazy situation where only a few hundred guys are paying attention. It’s daunting.\nDwarkesh Patel\n02:11:36\nI went to Washington a few months ago. I was talking to people doing AI policy stuff there. I asked how likely they think nationalization is. They said it's really hard to nationalize stuff. It’s been a long time since it's been done. There are specific procedural constraints on what kinds of things can be nationalized.\nThen I asked about ASI. Because of constraints like the\nDefense Production Act\n, that won’t be nationalized? The Supreme Court would overturn that? They were like, “yeah I guess that would be nationalized.”\nLeopold Aschenbrenner\n02:12:13\nThat’s the short summary of my post or my view on the project.\n(02:12:23) – Becoming Valedictorian of Columbia at 19\nDwarkesh Patel\n02:12:23\nBefore we go further on the AI stuff, let’s back up.\nWe began the conversation, and I think people will be confused. You graduated valedictorian of Columbia when you were 19. So, you got to college when you were 15.\nYou were in Germany then, and you got to college at 15.\nLeopold Aschenbrenner\n02:12:37\nYeah.\nDwarkesh Patel\n02:12:39\nHow the fuck did that happen?\nLeopold Aschenbrenner\n02:12:41\nI really wanted out of Germany. I went to a German public school. It was not a good environment for me.\nDwarkesh Patel\n02:12:52\nIn what sense? No peers?\nLeopold Aschenbrenner\n02:12:55\nThere’s also a particular German cultural sense. In the US, there are amazing high schools and an appreciation of excellence. In Germany, there’s a\ntall poppy syndrome\n. If you're the curious kid in class wanting to learn more, instead of the teacher encouraging you, they resent you and try to crush you.\nThere are also no elite universities for undergraduates, which is kind of crazy. The meritocracy was crushed in Germany at some point. There's also an incredible sense of complacency across the board. It always puzzles me but even going to a US college was seen as a radical act. It doesn’t seem radical to anyone here because it’s the obvious thing to do. You can go to Columbia and get a better education.\nIt’s wild to me because this is where stuff is happening and you can get a better education but people in Germany don’t do it. I skipped a few grades, and it seemed normal to me at the time to go to college at 15 and come to America. One of my sisters is turning 15 now, and when I look at her, I understand why my mother was worried.\nDwarkesh Patel\n02:14:27\nSo you were presumably the only 15-year-old.\nWas it normal for you to be a 15-year-old in college? What were the initial years like?\nLeopold Aschenbrenner\n02:14:34\nAgain, it felt so normal at the time. Now I understand why my mother was worried. I worked on my parents for a while and eventually persuaded them. It felt very normal at the time.\nIt was great. I really liked college. It came at the right time for me. I really appreciated the liberal arts education, the\ncore curriculum\n, and reading core works of political philosophy and literature.\nDwarkesh Patel\n02:15:02\nYou did what? Econ?\nLeopold Aschenbrenner\n02:15:05\nMy majors were math, statistics, and economics, but Columbia has a pretty heavy core curriculum and liberal arts education. Honestly, I shouldn’t have done all the majors. The best courses were those with amazing professors in some history classes. That’s what I would recommend people spend their time on in college.\nDwarkesh Patel\n02:15:26\nWas there one professor or class that stood out?\nLeopold Aschenbrenner\n02:15:29\nA few.\nRichard Betts\n'\nclass on war, peace, and strategy\n.\nAdam Tooze\nwas fantastic and has written\nvery riveting books\n. You should have him on the podcast, by the way.\nDwarkesh Patel\n02:15:43\nI’ve tried. I think you tried for me.\nLeopold Aschenbrenner\n02:15:45\nYou’ve got to get him on the pod. It’d be so good.\nDwarkesh Patel\n02:15:49\nRecently, we were talking to Tyler Cowen. He said when he first encountered you, it was through your\npaper on economic growth and existential risk\n. He said, “when I read it, I couldn’t believe that a 17-year-old had written it. If this were an MIT dissertation, I’d be impressed.” You’re a junior and you’re writing novel economic papers? Why did you get interested in this, and what was the process to get into that?\nLeopold Aschenbrenner\n02:16:30\nI just get interested in things. It feels natural to me. I get excited about something, read about it, and immerse myself. I can learn and understand information quickly.\nRegarding the paper, moments of peak productivity matter more than average productivity, at least for the way at work.  Some jobs, like CEO, require consistent productivity. I have periods of a couple months where there’s effervescence and other times, I'm computing stuff in the background. Writing the series was similar. You write it and it’s really flowing. That’s what ends up mattering.\nDwarkesh Patel\n02:17:14\nEven for CEOs, peak productivity might be very important. One of our friends in a group chat, following Chatham House rules, pointed out how many famous CEOs and founders have been bipolar or manic. The call option on your productivity is the most important thing, and you get it by increasing volatility through being bipolar. That’s interesting.\nYou got interested in economics first. Why economics? You could read about anything. You kind of got a slow start on ML. You wasted all these years on econ. There's an alternative world where you’re on the superalignment team at 17 instead of 21 or whatever it was.\nLeopold Aschenbrenner\n02:18:04\nIn some sense, I’m still doing economics. I’m looking at straight lines on a graph,\nlog-log plots\n, figuring out trends, and thinking about feedback loops, equilibrium, and arms control dynamics. It’s a way of thinking that I find very useful.\nDario and Ilya seeing scaling early is, in some sense, a very economic way of thinking. It’s also related to empirical physics. Many of them are physicists. Economists often can’t code well enough, which is their issue, but it's that way of thinking.\nI also thought a lot of core ideas in economics were beautiful. In some sense, I feel a little duped because econ academia is kind of decadent now. The paper I wrote is long, 100 pages of math, but the core takeaway can be explained in 30 seconds and it makes sense and you don’t really need the math. The best pieces of economics are like that.\nYou do the work to uncover insights that weren’t obvious to you before. Once you’ve done the work, some sort of mechanism falls out of it that makes a lot of  crisp, intuitive sense and explains facts about the world. You can then use it in arguments. Econ 101 is great like this. A lot of econ in the fifties and sixties was like this.\nChad Jones\n' papers are often like this. I really like his papers for this.\nWhy didn’t I ultimately pursue econ academia? There were several reasons, one of them being Tyler Cowen. He took me aside and said, \"I think you’re one of the top young economists I’ve ever met, but you should probably not go to grad school.\"\nDwarkesh Patel\n02:19:50\nOh, interesting. Really? I didn’t realize that.\nLeopold Aschenbrenner\n02:19:53\nYeah, it was good because he kind of introduced me to the Twitter weirdos. I think the takeaway from that was that I have to move out west one more time.\nDwarkesh Patel\n02:20:03\nWait Tyler introduced you to the Twitter weirdos?\nLeopold Aschenbrenner\n02:20:05\nA little bit. Or just kind of the broader culture?\nDwarkesh Patel\n02:20:08\nA 60-year-old economist introduced you to Twitter?\nLeopold Aschenbrenner\n02:20:12\nWell, I had been in Germany, completely on the periphery, and then moved to a US elite institution. I got a sense of meritocratic elite US society. Basically, there was a directory. To find the true American spirit I had to come out here.\nThe other reason I didn’t become an economist, or at least pursue econ academia, is that econ academia has become a bit decadent. Maybe it's just that ideas are getting harder to find, or that all the beautiful, simple things have been discovered.\nBut what are econ papers these days? They’re often 200 pages of empirical analyses on things like how buying 100,000 more textbooks in Wisconsin affects educational outcomes. I'm happy that work happens. It’s important work but it doesn't uncover fundamental insights and mechanisms in society.\nEven the theory work often involves really complicated models and the model spits out something like, “Fed does X, then Y happens” and you have no idea why that happened. There’s a gazillion parameters and they’re all calibrated in some way and it’s a computer simulation and you have no idea about the validity. The most important insights are the ones where you have to do a lot of work to get them but then there’s this crisp intuition.\nDwarkesh Patel\n02:21:26\nThe P versus NP of…\nLeopold Aschenbrenner\n02:21:27\nSure, yeah.\nDwarkesh Patel\n02:21:30\nThat’s really interesting. Going back to your time in college, you say that peak productivity explains this paper and things. But being valedictorian, getting straight A’s, is very much an average productivity phenomenon.\nLeopold Aschenbrenner\n02:21:48\nThere’s one award for the highest GPA, which I won, but the valedictorian is selected by the faculty from among those with the highest GPA.\nDwarkesh Patel\n02:21:55\nSo it's not just peak productivity.\nLeopold Aschenbrenner\n02:21:58\nI generally just love this stuff. I was curious, found it really interesting, and enjoyed learning about it. It made sense to me, and it felt very natural.\nOne of my faults is that I’m not that good at eating glass. Some people are very good at it. The moments of peak productivity come when I’m excited and engaged and love it. If you take the right courses, that’s what you get in college.\nDwarkesh Patel\n02:22:30\nIt’s like\nBruce Banner’s quote\nin\nThe Avengers\n: \"I’m always angry.\" I’m always excited and curious. That’s why I’m always at peak productivity.\nBy the way, when you were in college, I was also in college. Despite being a year younger than me, you were ahead of me by at least two years or something. We met around this time through the Tyler Cowen universe. It’s very insane how small the world is. Did I reach out to you? I must have.\nLeopold Aschenbrenner\n02:23:07\nI’m not sure.\nDwarkesh Patel\n02:23:09\nWhen I had a couple of videos with a few hundred views.\nLeopold Aschenbrenner\n02:23:11\nIt’s a small world. This is the crazy thing about the AI world. It’s the same few people at the parties running the models at DeepMind, OpenAI, and Anthropic. Some of our friends, now successful in their careers, met many of the people who are now successful in Silicon Valley before their twenties or in their early twenties.\nWhy is it a small world? There’s some amount of agency. I think in a funny way, this is what I took away from my Germany experience. It was crushing. I didn’t like it. Skipping grades and moving to the US were unusual moves.\nJust trying to do it, and then seeing it work, reinforced the idea that you don’t have to conform to the Overton window. You can try to do what seems right to you, even if most people are wrong. That was a valuable and formative early experience.\nDwarkesh Patel\n02:24:33\nAfter college, what did you do?\nLeopold Aschenbrenner\n02:24:36\nI did econ research for a bit, at Oxford and other places, and then I worked at\nFuture Fund\n.\nDwarkesh Patel\n02:24:41\nTell me about it.\nLeopold Aschenbrenner\n02:24:46\nFuture Fund was a foundation funded by\nSam Bankman-Fried\nbut we were our own thing. We were based in the Bay Area. At the time, in early 2022, it was an incredibly exciting opportunity. It was basically a startup foundation, which doesn’t come along often. We thought we would be able to give away billions of dollars and remake how philanthropy is done from first principles.\nWe thought we’d have significant impact, focusing on causes like biosecurity, AI, and finding exceptional talent to work on hard problems. A lot of the work we did was exciting. Academics, who would usually take six months, would send us emails saying, \"this is great. This is so quick and straightforward.\" I often find that with a little encouragement and empowerment, by removing excuses and making the process easy, you can get people to do great things.\nDwarkesh Patel\n02:25:49\nFor context, not only were you guys planning on deploying billions of dollars, but it was a team of four people. So you, at 18, were on a team of four people that was in charge of deploying billions of dollars.\nLeopold Aschenbrenner\n02:26:07\nThat was sort of the heyday. Then in November 2022, it was\nrevealed that Sam was a giant fraud\n, and from one day to the next, the whole thing collapsed. It was really tough. It was devastating for the people who had their money in FTX. Closer to home, we wanted to help all the grantees do amazing projects but they ended up suddenly saddled with a giant problem.\nPersonally, it was difficult because it was a startup. I had worked 70-hour weeks every week for almost a year to build it up. We were a tiny team, and then from one day to the next, it was all gone and associated with a giant fraud. That was incredibly tough.\nDwarkesh Patel\n02:27:04\nWere there any early signs about SBF?\nLeopold Aschenbrenner\n02:27:10\nObviously, I didn’t know he was a fraud. If I had, I would have never worked there. We were a separate entity and didn’t work with the business. I do think there are some takeaways for me.\nI, and people in general, had this tendency to give successful CEOs a pass on their behavior because they’re successful. You think that’s just a successful CEO thing. I didn’t know Sam Bankman-Fried was a fraud.\nI knew he was extremely risk-taking, narcissistic, and didn’t tolerate disagreement well. By the end, he and I didn’t get along because I pointed out that some biosecurity grants weren’t cost effective but he liked them because they were cool and flashy. He was unhappy about that.\nSo I knew his character. I realized that it’s really worth paying attention to people’s characters, including people you work for and successful CEOs. That can save you a lot of pain down the line.\nDwarkesh Patel\n02:28:26\nAfter FTX imploded and you were out, you went to OpenAI. The superalignment team had just started. You were part of the initial team.\nWhat was the original idea? What compelled you to join?\nLeopold Aschenbrenner\n02:28:47\nThe alignment teams at OpenAI and other labs had done basic research and developed RLHF. reinforcement learning from human feedback. That ended up being a really successful technique for controlling current AI models.\nOur task was to find the successor to RLHF. The reason we need that is that RLHF probably won’t scale to superhuman systems. RLHF relies on human raters giving feedback, but superintelligent models will produce complex outputs beyond human comprehension. It’ll be like a million lines of complex code and you won’t know at all what’s going on anymore.\nHow do you steer and control these systems? How do you add side constraints? I joined because I thought this was an important and solvable problem. I still do and even more so. I think there’s a lot of promising ML research on aligning superhuman systems, which we can discuss more later.\nDwarkesh Patel\n02:30:01\nIt was so solvable, you solved it in a year. It’s all over now.\nLeopold Aschenbrenner\n02:30:07\nOpenAI wanted to do a really ambitious effort on alignment. Ilya was backing it. I liked a lot of the people there. I was really excited. There are always people making hay about alignment. I appreciate people highlighting the importance of the problem and I was just really into trying to solve it. I wanted to do the ambitious effort, like an Operation Warp Speed for solving alignment. It seemed like an amazing opportunity to do it.\n(02:30:35) – What happened at OpenAI\nDwarkesh Patel\n02:30:35\nNow the team basically doesn't exist. The heads of it,\nJan\nand Ilya, have left.\nThat’s been the news of last week\n. What happened? Why did the team break down?\nLeopold Aschenbrenner\n02:30:48\nOpenAI decided to take things in a different direction.\nDwarkesh Patel\n02:30:53\nMeaning what? That superalignment isn’t the best way to frame it?\nLeopold Aschenbrenner\n02:30:59\nNo, obviously after the November board events there were personnel changes. Ilya leaving was incredibly tragic for OpenAI. There was some reprioritization. There’s been reporting on the superalignment compute commitment, the\n20% compute commitment\n, which was how a lot of people were recruited. There was a decision to not keep that commitment and go in a different direction.\nDwarkesh Patel\n02:31:26\nNow Jan and Ilya have left, and the team itself has dissolved. You were the first person who left or was forced to leave.\nThe Information\nreported\nthat you were fired for leaking. What happened? Is this accurate?\nLeopold Aschenbrenner\n02:31:43\nWhy don’t I tell you what they claim I leaked, and you can tell me what you think. OpenAI claimed to employees that I was fired for leaking. I and others have pushed them to say what the leak was. Here’s their response in full: Sometime last year, I had written a brainstorming document on preparedness, safety, and security measures needed in the future on the path to AGI. I shared that with three external researchers for feedback. That’s the leak.\nFor context, it was totally normal at OpenAI at the time to share safety ideas with external researchers for feedback. It happened all the time. The doc had my ideas. Before I shared it, I reviewed it for anything sensitive. The internal version had a reference to a future cluster, which I redacted for the external copy. There was a link to some internal slides, but that was a dead link for the external people. The slides weren’t shared with them.\nWhen I pressed them to specify what confidential information was in this document. They came back with a line about planning for AGI by 2027-2028 and not setting timelines for preparedness.\nI wrote this doc a couple of months after the\nsuperalignment announcement\n. We had put out a four-year planning horizon. I didn’t think that planning horizon was sensitive. It’s the sort of thing Sam says publicly all the time. I think Jan mentioned it on a podcast a couple of weeks ago. So, that’s it.\nDwarkesh Patel\n02:33:20\nThat’s it? That sounds pretty thin if the cause was leaking. Was there anything else to it?\nLeopold Aschenbrenner\n02:33:28\nThat was the leaking claim. Let me explain more about what happened during the firing. Last year, I wrote an internal memo about OpenAI's security, which I thought was egregiously insufficient to protect against the theft of model weights or key algorithmic secrets from foreign actors. I shared this memo with a few colleagues and a couple of members of leadership, who mostly said it was helpful.\nA few weeks later, a major security incident occurred\n1\n. That prompted me to share the memo with a couple of board members. Days later, it was made very clear to me that leadership was very unhappy I had shared this memo with the board. Apparently, the board hassled leadership about security.\nI got an official HR warning for sharing the memo with the board. The HR person told me it was racist to worry about CCP espionage and that it was unconstructive. I probably wasn’t at my most diplomatic and could have been more politically savvy. I thought it was a really important issue. The security incident made me very worried.\nThe reason I bring this up is that when I was fired, it was very made explicit that the security memo was a major reason for my being fired. They said, \"the reason this is a firing and not a warning is because of the security memo.\"\nDwarkesh Patel\n02:34:59\nYou sharing it with the board?\nLeopold Aschenbrenner\n02:35:01\nThe warning I’d gotten for the security memo.\nWhat might also be helpful context is the kinds of questions they asked me when they fired me. A bit over a month ago, I was pulled aside for a chat with a lawyer that quickly turned adversarial. The questions were about my views on AI progress, on AGI, the appropriate level of security for AGI, whether the government should be involved in AGI, whether I and the superalignment team were loyal to the company, and what I was up to during the OpenAI board events. They then talked to a couple of my colleagues and came back and told me I was fired. They’d gone through all of my digital artifacts from my time at OpenAI, and that’s when they found the leak.\nThe main claim they made was this leaking allegation. That’s what they told employees. The security memo was another thing. There were a couple of other allegations they threw in. One thing they said was that I was unforthcoming during the investigation because I didn’t initially remember who I had shared the preparedness brainstorming document with, only that I had talked to some external researchers about these ideas.\nThe document was over six months old, I’d spent a day on it. It was a Google Doc I shared with my OpenAI email. It wasn’t a screenshot or anything I was trying to hide. It simply didn’t stick because it was such a non-issue. They also claimed I was engaging on policy in a way they didn’t like. They cited there that I had spoken to a couple of external researchers, including someone at a think tank, about my view that AGI would become a government project, as we just discussed.\nIn fact, I was speaking with lots of people in the field about that view at the time. I thought it was a really important thing to think about. So they found a DM I had written to a friendly colleague, five or six months earlier, and they cited that too. I had thought it was well within OpenAI norms to discuss high-level issues about the future of AGI with external people in the field.\nThat’s what they allege happened. I’ve spoken to a few dozen former colleagues about this since. The universal reaction has been, \"that’s insane.\" I was surprised as well. I had been promoted just a few months before. Ilya’s comment for the promotion case at the time was something like, \"Leopold’s amazing. We’re lucky to have him.\"\nThe thing I understand, and in some sense it’s reasonable, is that I ruffled some feathers and was probably annoying at times with the security stuff. I repeatedly raised that, maybe not always in the most diplomatic way. I didn’t sign the\nemployee letter\nduring the board events, despite pressure to do so.\nDwarkesh Patel\n02:38:03\nYou were one of like eight people or something?\nLeopold Aschenbrenner\n02:38:05\nNot that many people. I think the two most senior people who didn’t sign were\nAndrej\nand Jan, who have both since left.\nOn the letter, by Monday morning when it was circulating, I thought it was probably appropriate for the board to resign because they had lost too much credibility and trust with the employees.\nBut I thought the letter had issues. It didn’t call for an independent board, which is a basic of corporate governance. In other discussions, I pressed leadership for OpenAI to abide by its public commitments. I raised tough questions about whether it was consistent with the OpenAI mission and the national interest to partner with authoritarian dictatorships to build the core infrastructure for AGI.\nIt’s a free country. That’s what I love about it. We talked about it. They have no obligation to keep me on staff. It would have been reasonable for them to come to me and say, \"we’re taking the company in a different direction. We disagree with your point of view. We don’t trust you to toe the company line anymore. Thank you so much for your work at OpenAI, but it’s time to part ways.\"\nThat would have made sense. We had started diverging on important issues. I came in very excited and aligned with OpenAI, but that changed over time. That would have been a very amicable way to part ways. It’s a shame how it went down.\nAll that being said, I really want to emphasize that there are a lot of incredible people at OpenAI, and it was an incredible privilege to work with them. Overall, I’m extremely grateful for my time there.\nDwarkesh Patel\n02:40:01\nNow there’s been reporting about\nan NDA\nthat former employees have to sign to access their vested equity. Did you sign such an NDA?\nLeopold Aschenbrenner\n02:40:16\nNo. My situation was a little different because I was right before my cliff. They still offered me the equity, but I didn’t want to sign. Freedom is priceless\nDwarkesh Patel\n02:40:28\nHow much was the equity?\nLeopold Aschenbrenner\n02:40:30\nClose to a million dollars.\nDwarkesh Patel\n02:40:32\nSo it was definitely something you and others were aware of. OpenAI explicitly offered you a choice. Presumably, the person on OpenAI staff knew they were offering equity but required signing an NDA that prevents making statements about AGI and OpenAI, like the ones you’re making on this podcast.\nLeopold Aschenbrenner\n02:40:57\nI don’t know the whole situation. I certainly think conditioning vested equity on signing an NDA is pretty rough. It might be different if it’s a severance agreement.\nDwarkesh Patel\n02:41:05\nRight, but an OpenAI employee who had signed it presumably couldn’t give the podcast you’re giving today.\nLeopold Aschenbrenner\n02:41:11\nQuite possibly not. I don’t know.\nDwarkesh Patel\n02:41:15\nThe board thing is really tough. Analyzing the situation here, if you were trying to defend them, you might say, \"well, listen you were just going outside the regular chain of command.\" There might be a point there.\nAlthough the idea that HR thinks you’re supposed to have an adversarial relationship with the board is odd. You’re giving the board relevant information about whether OpenAI is fulfilling its mission and how it can improve. That seems important since the board is supposed to ensure OpenAI follows its mission. Them treating that as part of the leak, as if the board were an external actor…\nLeopold Aschenbrenner\n02:42:00\nTo be clear, the leak allegation was just about that document I shared for feedback. This is a separate issue they cited. They said I wouldn’t have been fired if not for the security memo.\nDwarkesh Patel\n02:42:09\nThey said you wouldn’t have been fired for it.\nLeopold Aschenbrenner\n02:42:11\nThey said the reason this is a firing and not a warning is because of the warning I had gotten for the security memo.\nDwarkesh Patel\n02:42:16\nBefore you left, the incidents with the board happened. Sam was\nfired\nand then\nrehired\nas CEO, and now he’s on the board. Ilya and Jan, who were the heads of the superalignment team, have left. Ilya, was a co-founder of OpenAI and the most significant member of OpenAI from a research perspective. There has been a lot of personnel drama over the last few months regarding superalignment and just generally with the OpenAI personnel drama. What’s going on?\nLeopold Aschenbrenner\n02:42:49\nThere’s a lot of drama. Why is there so much drama?\nThere would be much less drama if all OpenAI claimed to be was building ChatGPT or business software. A lot of the drama comes from OpenAI really believing they’re building AGI. That isn’t just a marketing claim. There’s a report that Sam is raising $7 trillion for chips. That only makes sense if you really believe in AGI.\nWhat gets people is the cognitive dissonance between believing in AGI and not taking some of the other implications seriously. This technology will be incredibly powerful, both for good and bad. That implicates national security issues. Are you protecting the secrets from the CCP? Does America control the core AGI infrastructure or does a Middle Eastern dictator control it?\nThe thing that really gets people is the tendency to make commitments and say they take these issues seriously, but then frequently not follow. For instance, as mentioned, there was a commitment around superalignment compute, dedicating 20% of compute for long-term safety research.\nYou and I could have a totally reasonable debate about the appropriate level of compute for superalignment. That’s not really the issue. The issue is that the commitment was made and it was used to recruit people. It was very public.\nIt was made because there was a recognition that there would always be something more urgent than long-term safety research, like a new product. In the end, they just didn’t keep the commitment. There was always something more urgent than long-term safety research.\nAnother example is when I raised security issues. They would tell me security is our number one priority. Invariably, when it came time to invest serious resources or make trade-offs to take basic measures, security was not prioritized. The cognitive dissonance and unreliability cause a lot of the drama.\n(02:45:11) – Accelerating AI research progress\nDwarkesh Patel\n02:45:11\nLet’s zoom out and talk about a big part of the story. A big motivation for the way we must proceed with regards to geopolitics is that once you have AGI, you soon proceed to ASI, or superintelligence. You have these AGIs functioning as researchers into further AI progress and within a matter of years, maybe less, you reach superintelligence. From there, according to your story, you do all this research and development into robotics, pocket nukes, and other crazy shit.\nI’m skeptical of this story for many reasons. At a high level, it’s not clear to me that this input-output model of research is how things actually happen in research. We can look at the economy as a whole. Patrick Collison and others have pointed out that, compared to 100 years ago, we have 100x more researchers in the world. Yet progress isn’t happening 100 times faster. It's clearly not as simple as pumping in more researchers to get higher research output. I don't see why it would be different for AI researchers.\nLeopold Aschenbrenner\n02:46:31\nThis is getting into good stuff. This is the classic disagreement I have with Patrick and others. Obviously, inputs matter. The United States produces a lot more scientific and technological progress than Liechtenstein or Switzerland.\nSay you made Patrick Collison dictator of Liechtenstein or Switzerland and he implemented his utopia of ideal institutions. Keep the talent pool fixed. He’s not able to do some crazy high-skilled immigration thing or genetic breeding scheme. You keep the talent pool fixed with amazing institutions. Even then, even if Patrick Collison were the dictator, Switzerland still wouldn’t be able to outcompete the United States in scientific and technological progress. Magnitudes matter.\nDwarkesh Patel\n02:47:19\nI'm not sure I agree with this. There are many examples in history where small groups of people,\nBell Labs\nor Skunk Works, have made significant progress. OpenAI has a couple hundred researchers.\nLeopold Aschenbrenner\n02:47:33\nHighly selected though.\nDwarkesh Patel\n02:47:35\nThat’s why Patrick Collison as a dictator would do a good job of this.\nLeopold Aschenbrenner\n02:47:39\nWell, yes, if he can highly select all the best AI researchers in the world, he might only need a few hundred. But that’s the talent pool. You have the 300 best AI researchers in the world.\nDwarkesh Patel\n02:47:48\nBut from 100 years ago to now, the population has increased massively. You would expect the density of talent to have increased, considering that things like malnutrition and poverty which affected past talent are no longer as debilitating to the same level.\nLeopold Aschenbrenner\n02:48:06\nI don’t know if it’s 100x. It’s probably at least 10x. Some people think ideas haven’t gotten much harder to find, so why would we need this 10x increase in research effort? To me, this is a very natural story. Why is it natural? It’s a straight line on a log-log plot. It’s a deep learning researcher’s dream.\nWhat is this log-log plot? On the x-axis you have log cumulative research effort. On the y-axis you have log GDP, OOMs of algorithmic progress, log transistors per square inch, log price for a gigawatt of solar energy. It’s extremely natural for that to be a straight line. It’s classic. Initially, things are easy, but you need logarithmic increments of cumulative research effort to find the next big thing. This is a natural story.\nOne objection people make is, “isn’t it suspicious that we increased research effort 10x and ideas also got 10x harder to find, perfectly equilibrating?” I say it’s just equilibrium—it’s in a endogenous equilibrium. Isn’t it a coincidence that supply equals demand and the market clears? It’s the same here. The difficulty of finding new ideas depends on how much progress has been made.\nThe overall growth rate is a function of how much ideas have gotten harder to find in ratio to how much research effort has increased. This story is fairly natural, and you see it not just economy-wide but also in the experience curve for various technologies.\nIt’s plausible that institutions have worsened by some factor. Obviously, there’s some sort of exponent of diminishing returns on adding more people. Serial time is better than just parallelizing. Still, clearly inputs clearly matter.\nDwarkesh Patel\n02:50:06\nI agree, but if the coefficient, of how fast they diminish as you grow the input, is high enough, then in the abstract the fact that inputs matter isn’t that relevant.\nWe’re talking at a very high level, but let’s take it down to the concrete. OpenAI has a staff of at most a few hundred directly involved in algorithmic progress for future models. Let’s say you could really arbitrarily scale this number for faster algorithmic progress and better AI. It’s not clear why OpenAI doesn’t just go hire every person with a 150 IQ, of which there are hundreds of thousands in the world.\nMy story is that there are transaction costs to managing all these people. They don’t just go away if you have a bunch of AIs. These tasks aren’t easy to parallelize. I’m not sure how you would explain the fact that OpenAI doesn’t go on a recruiting binge of every genius in the world?\nLeopold Aschenbrenner\n02:51:12\nLet’s talk about the OpenAI example and the automated AI researchers. Look at the inflation of\nAI researcher salaries\nover the last yea. It’s gone up by 4x or 5x. They’re clearly trying to recruit the best AI researchers in the world and they do find them. My response would be that almost all of these 150 IQ people wouldn’t just be good AI researchers if you hired them tomorrow. They wouldn’t be Alec Radford.\nDwarkesh Patel\n02:51:41\nThey’re willing to make investments that take years to pan out. The data centers they’re buying now will come online in 2026. Some of them won’t work out, some won’t have traits we like. But why wouldn’t they make the investment to turn these 150 IQ people into amazing AI researchers by 2026?\nLeopold Aschenbrenner\n02:51:58\nSometimes this does happen. Smart physicists have been really good at AI research, like all the Anthropic co-founders.\nDwarkesh Patel\n02:52:03\nBut for example, Dario said on the\npodcast\nthat they have a careful policy of being extremely selective and not hiring arbitrarily.\nLeopold Aschenbrenner\n02:52:11\nTraining is not as easily scalable. Training is really hard. If you hired 100,000 people, it would be really hard to train them all. You wouldn’t be doing any AI research. There are huge costs to bringing on new people and training them.\nThis is very different with AIs. It’s important to talk about the advantages AIs will have. What does it take to be an Alec Radford? You need to be a really good engineer. AIs will be amazing engineers and coders. You can train them to do that. They also need to have good research intuitions and a really good understanding of deep learning.\nAlec Radford, or people like him, has acquired this over years of being deeply immersed in deep learning and having tried lots of things himself and failed. AIs will be able to read every research paper ever written, learn from every experiment ever run at the lab, and gain intuition from all of this. They’ll be able to learn in parallel from each other's experiments and experiences.\nThere’s also a cultural acclimation aspect. If you hire someone new, there’s politicking, and maybe they don’t fit in well. With AIs, you just make replicas. There's a motivation aspect as well. If I could duplicate Alec Radford, and before I run every experiment, have him spend a decade’s worth of human time double-checking code and thinking really carefully about it, he wouldn’t care and he wouldn't be motivated. With AIs, you can have 100 million of them focused on making sure the code is correct with no bugs.\nThe idea of 100 million human-equivalent AI researchers is just a way to visualize it. You might not have literally 100 million copies. There’s tradeoffs you can make between serial speed and parallel. You might run them at 10x or 100x serial speed, resulting in fewer tokens overall because of inherent trade-offs. You might have 100,000 AIs running at 100x human speed. They can coordinate by sharing latent space and attending to each other's context. There’s a huge range of possibilities for what you can do.\nAnother illustration is that by 2027 or 2028, with automated AI researchers, you'll be able to generate an entire Internet’s worth of tokens every day. It’s clearly a huge amount of intellectual work that you can do.\nDwarkesh Patel\n02:54:44\nHere’s an analogy. Today, we generate more patents in a year than during the actual physics revolution in the early 20th century. Are we making more physics progress in a year today than we did in half a century back then? Generating all these tokens doesn't necessarily equate to generating as much codified knowledge in the initial creation of the Internet.\nLeopold Aschenbrenner\n02:55:09\nInternet tokens are usually final output. We talked about the unhobbling. I think of a GPN token as one token of my internal monologue. That’s how I do this math on human equivalents. It's like 100 tokens a minute and then humans working for X hours. What is the equivalent there?\nDwarkesh Patel\n02:55:28\nThis goes back to something from earlier. Why haven’t we seen huge revenues from AI yet? People often ask this question. If you took GPT-4 back ten years, people would think it would automate half the jobs. There’s a\nmodus ponens\n,\nmodus tollens\nhere. Part of the explanation is that we’re on the verge and we just need to do these unhobblings. Part of that is probably true. But there is another lesson to learn there. Just looking at a set of abilities at face value, there are likely more hobblings behind the scenes. The same will be true of AGIs running as AI researchers.\nLeopold Aschenbrenner\n02:56:09\nI basically agree with a lot of what you said. My story here is that there’s going to be a long tail. Maybe by 2026 or 202, you’ll have the proto-automated engineer that’s really good at engineering. It doesn’t yet have the research intuition. You don’t quite know how to put them to work.\nEven so, the underlying pace of AI progress is already so fast. In just three years, we've gone from AI not being able to do any kind of math at all to now crushing these math competitions. So, you might have the initial automated research engineer by 2026 or 2027, which speeds you up by 2x. You go through a lot more progress in that year. By the end of the year, you’ve figured out the remaining unhobblings and you've got a smarter model.\nMaybe it’s two years but then maybe that model can automate 100% of the research. They don’t need to be doing everything. They don’t need to make coffee or deal with tacit knowledge in other fields. AI researchers at AI labs really know the job of an AI researcher. There are lots of clear metrics. It's all virtual. There’s code. There are things you can develop and train for.\nDwarkesh Patel\n02:57:11\nAnother thing is how do you actually manage a million AI researchers? Humans’ comparative ability, and we’ve been especially trained for it, is to work in teams. We’ve been learning for thousands of years about how we work together in groups. Despite this, management is a clusterfuck. Most companies are poorly managed. It's really hard to do this stuff.\nFor AIs, we talk about AGI, but for it will be some bespoke set of abilities some of which will be higher than human level and some at human level. It will be some bundle and you’ll need to figure out how to put these bundles together with their human overseers and equipment. I’m just very skeptical of the idea that as soon as you get the bundle, you can just shove millions of them together and manage them.\nAny other technological revolution in history has been much more piecemeal than you'd expect on paper. What is the industrial revolution? We dug up coal to power steam engines, used steam engines to run railroads, which helped us get more coal. There’s sort of a\nFactorio\nstory you can tell where in six hours you can be pumping out thousands of times more coal. In real life, it takes centuries.\nFor example, with electrification, there's a\nfamous study\nshowing how it took decades after electricity to switch from the pulley and water wheel-based system for steam engines to one that works with more spread-out electrical motors. This will be the same kind of thing. It might take decades to actually get millions of AI researchers to work together effectively.\nLeopold Aschenbrenner\n02:59:05\nThis is great. A few responses to that. I totally agree with the real-world bottlenecks idea. It's easy to underrate these constraints. Basically, we’re automating labor and exploiting technology, but there are still many other bottlenecks in the world.\nThat’s why the story starts narrowly where there aren’t these bottlenecks and then expands to broader areas over time. This is part of why I think initially it’s an AI research explosion. AI research doesn’t run into these real-world bottlenecks. It doesn’t require plowing a field or digging up coal. It’s just doing AI research.\nDwarkesh Patel\n02:59:42\nI love how in your model, AI research isn’t complicated. It’s like flipping a burger. It’s just AI research.\nLeopold Aschenbrenner\n02:59:51\nPeople make these arguments like, “AGI won’t do anything because it can’t flip a burger.” Yeah it won’t be able to flip a burger, but it’ll be able to do algorithmic progress. Once it achieves that, it can figure out how to create a robot that flips burgers. The quantities we’re talking about are a lower bound. We can definitely run 100 million of these.\nOne of the first things we’ll figure out is how to translate quantity into quality. Even at the baseline rate of progress, you’re quickly getting smarter and smarter systems. It took four years to go from preschooler to high schooler. Pretty quickly, there are probably some simple algorithmic changes you find if you have a hundred Alec Radfors instead of one. You don’t even need a hundred million. We’ll soon have systems that are even smarter and capable of creative, complicated behavior we don’t understand.\nMaybe there’s some way to use all this test time compute in a more unified way than all these parallel copies. They won’t just be quantitatively superhuman. They’ll pretty quickly become qualitatively superhuman. It’s like a high school student trying to understand standard physics versus a super-smart professor who gets quantum physics. You quickly enter that regime just given the underlying pace of AI progress but even more quickly with the accelerated force of automated AI research.\nDwarkesh Patel\n03:01:21\nI agree that over time you would get there. I'm not denying that ASI is possible. I’m just questioning how this happens in a year.\nLeopold Aschenbrenner\n03:01:40\nThe story is a bit more continuous. By 2025 or 2026, you’ll already have models as good as a college graduate. I don’t know where all the unhobbling is going to be but even it’s possible that you have a proto-automated engineer.\nThere’s a bit of an AGI smear that there are unhobblings missing. There’s ways of connecting them that are missing. There’s some level of intelligence you’re missing. At some point you are going to get this thing that is 100% automated Alec Radford and once you have that, things really take off.\nDwarkesh Patel\n03:02:06\nLet’s go back to the unhobbling.\nWe’re going to get a bunch of models by the end of the year. Suppose we didn’t get some capacity by the end of the year. Is there some such capacity which us lacking would suggest that AI progress will take longer than you are projecting?\nLeopold Aschenbrenner\n03:02:22\nThere are two key things: the unhobbling and the data wall. Let's talk about the data wall for a moment. Even though we’re seeing crazy AI progress, the data wall is actually underrated. There's a real scenario where we stagnate because we've been riding this tailwind of easily bootstrapping unsupervised learning.\nIt learns these amazing world models. You just buy more compute, make simple efficiency changes, and get big gains. All of the big gains in efficiency have been pretty dumb things. You add a\nnormalization layer\n. You fix\nscaling laws\n. These have already been huge things, let alone obvious ways in which these models aren’t good yet.\nThe data wall is a big deal. For instance,\nCommon Crawl\nonline is about 30 trillion tokens.\nLlama-3\nwas trained on 15 trillion tokens. We're already using all of the data. You can get somewhat further by repeating data, but\nan academic paper by Boaz Barak that does scaling laws for this\n. It says that after about 16 repetitions, the returns basically go to zero.\nLlama-3 is already at the limit of data. Maybe we can get 10x more by repeating data. At most that’s a 100x model than GPT-4, a 100x effective compute from GPT-4. That’s not that much. If you do half an OOM of compute and half an OOM of algorithmic progress each year, that's like two years from GPT-4. GPT-4 finished pre-training in 2022, so it’s 2024. We won’t quite know by the end of the year but by 2025 and 2026 we’ll get a sense of if we’re cracking the data wall.\nDwarkesh Patel\n03:04:13\nSuppose we had three OOMs less data in Common Crawl on the Internet than we happen to have now. For decades, with the Internet and other things, the stock of data humanity has has been rapidly increasing. Is it your view that, for contingent reasons, we just happen to have enough data to train models just powerful enough, like GPT-4.5, to kick off the self-play RL loop?\nOr is it just that if it had been 3 OOMs higher, then progress would have been slightly faster? In that world, we would have been looking back, thinking it would have been hard to kick off the RL explosion with just GPT-4.5. We would have figured it out eventually.\nIn this world, we would have gotten to GPT-3 and then had to kick off some sort of RL explosion. We would have still figured it out. Did we just luck out on the amount of data we happen to have in the world?\nLeopold Aschenbrenner\n03:05:09\n3 OOMs less data is pretty rough. That would mean 6 OOMs less compute model and Chinchilla scaling laws. That’s basically capping out at something barely better than GPT-2. That would be really rough.\nYou make an interesting point about contingency. If we consider the human trajectory analogy, a preschooler model can't learn from itself. An elementary school model can't learn from itself. Maybe GPT-4 is like a smart high schooler that can start learning from itself. Ideally, you want a somewhat better model that can truly learn by itself. 1 OOM less data would make me more iffy, but it might still be doable. It would feel chiller if we had 1 or two OOMs of more data.\nDwarkesh Patel\n03:05:56\nIt would be an interesting exercise to get probability distributions of AGI contingent across OOMs of data.\nLeopold Aschenbrenner\n03:06:01\nYeah, I agree.\nDwarkesh Patel\n03:06:02\nThe thing that makes me skeptical of this story is that it totally makes sense why pre-training works so well. With these other things, there are stories of why they ought to work in principle.. Humans can learn this way and so on. Maybe they're true.\nI worry that a lot of this case is based on first principles evaluation of how learning happens. Maybe fundamentally, we don't understand how humans learn. Maybe there's some key thing we're missing.\nOn sample efficiency, you say the fact that these things are way less sample efficient than humans in learning suggests there's a lot of room for improvement.\nAnother perspective is that we are just on the wrong path altogether. That’s why they’re so sample inefficient when it comes to pre-training.\nThere are a lot of first principles arguments stacked on top of each other where you get these unhobblings, then you get to AGI. Then because of these reasons why you can stack all these things on top of each other and you get to ASI. I'm worried that there are too many steps of this sort of first principles thinking.\nLeopold Aschenbrenner\n03:07:11\nWe'll see. On sample efficiency, it’s sort of first principles but there's this clear missing middle. People hadn't been trying. Now people are really trying. Again, often in deep learning something like the obvious thing works and there are a lot of details to get right. It might take some time, but now people are really trying. We will get a lot of signal in the next couple of years on unhobbling.\nWhat is the signal on unhobbling that would be interesting? The question is basically, are you making progress on test time compute? Is this thing able to think longer horizon than just a couple hundred tokens? That was unlocked by chain-of-thought.\nDwarkesh Patel\n03:07:55\nOn that point in particular, many people who have longer timelines have come on the podcast and made the point that the way to train this long horizon RL, it's not…\nEarlier we were talking about how they can think for five minutes, but not for longer. It's not because they can't physically output an hour's worth of tokens.\nLeopold Aschenbrenner\n03:08:17\nEven Gemini has a million in context, and the million of context is actually great for consumption. It solves one important hobbling, which is the onboarding problem. A new coworker in your first five minutes, like a new smart high school intern, is not useful at all.\nA month in, they’re much more useful because they've looked at the\nmonorepo\n, understand how the code works, and they've read your internal docs. Being able to put that in context solves this onboarding problem. They're not good at the production of a million tokens yet.\nDwarkesh Patel\n03:08:46\nOn the production of a million tokens, there's no public evidence that there's some easy loss function where you can...\nLeopold Aschenbrenner\n03:08:55\nGPT-4 has gotten a lot better since launch. The GPT-4 gains since launch are a huge indicator.\nYou\ntalked about this\nwith John Schulman on the podcast. John said this was mostly post-training gains. If you look at the\nLMSys\nscores, it's like 100 Elo or something. It's a bigger gap than between\nClaude 3 Opus\nand\nClaude 3 Haiku\n. The price difference between those is 60x.\nDwarkesh Patel\n03:09:16\nBut it’s not more agentic. It's better in the same chat.\nLeopold Aschenbrenner\n03:09:19\nIt’s  much better about math. It went from 40% to 70%. That indicates that clearly there's stuff to be done on hobbling. The interesting question is, this time a year from now, is there a model that is able to think for a few thousand tokens coherently, cohesively, identically? Again, I'd probably feel better if we had 1–2 OOMs more data because the scaling just gives you this tailwind.\nWith tools, when you talk to people who try to make things work with tools, GPT-4 is really when tools start to work. You can kind of make them work with GPT-3.5, but it's just really tough. Having GPT-4, you can help it learn tools in a much easier way. So it’d be great to have just a bit more tailwind from scaling. I don't know if it'll work, but it's a key question.\nDwarkesh Patel\n03:10:12\nIt's a good place to sort of close that part where we know what the crux is and what evidence of that would look like.\nLet’s talk about AGI to superintelligence. Maybe it's the case that the gains are really easy right now and you can just sort of let loose. Give Alec Radford a compute budget and he’ll comes out the other end with something that is an additive change as part of the code.\nHow many other domains in the world are like this, where you think you could get the equivalent of in one year? You just throw enough intelligence across multiple instances and you come out the other end with something that is remarkably decades, centuries ahead? You start off with no flight, and then you have the\nWright brothers\n. You have a million instances of GPT-6, and you come out the other end with\nStarlink\n? Is that your model of how things work?\nLeopold Aschenbrenner\n03:11:17\nYou're exaggerating the timelines a little bit, but I think a decade's worth of progress in a year or something is a reasonable prompt. This is where the automated AI researcher comes in. It gives you this enormous tailwind on all the other stuff.\nYou automate AI research with your automated Alec Radfords. You come out the other end. You've done another five OOMs. You have a thing that is vastly smarter. Not only is it vastly smarter, you've been able to make it good at everything else. You're solving robotics.\nThe robots are important because for a lot of other things, you do actually need to try things in the physical world. Maybe you can do a lot in simulation. Those are the really quick worlds. I don't know if you saw the last\nNvidia GTC\nand it was all about the\ndigital twins\nhaving all your manufacturing processes in simulation. Again, if you have these superintelligent cognitive workers, can they just make simulations of everything, off the float style, and make a lot of progress there?\nYou're just going to get the robots. I agree there are a lot of real-world bottlenecks. It's quite possible that we're going to have crazy drone swarms, but also lawyers and doctors still need to be humans because of regulation. You kind of start narrowly, you broaden, and there are worlds in which you let them loose. Again, because of these competitive pressures, we will have to let them loose to some degree on various national security applications. Rapid progress is quite possible.\nIn the explosion after, there are basically two components. The\nA\nin the\nproduction function\n, the growth of technology, has massively accelerated. Now you have a billion superintelligent scientists and engineers and technicians, superbly competent at everything.\nYou also just automated labor. Even without the whole technological explosion thing, you have this industrial explosion, at least if you let them loose, because you can cover Nevada and you start with one robot factory producing more robots. It's this cumulative process because you've taken labor out of the equation.\nDwarkesh Patel\n03:13:21\nThat's super interesting.\nAlthough when you increase the\nK\nor the\nL\nwithout increasing the\nA\n, you can look at examples like the Soviet Union or China. They rapidly increased inputs, which did have a geopolitically game-changing effect. It is remarkable to see the transformation of cities like Shanghai over just decades.\nLeopold Aschenbrenner\n03:13:43\nThey throw out these crazy cities in like a decade.\nDwarkesh Patel\n03:13:46\nPeople talk about 30% growth rates from AI. The closest thing—\nLeopold Aschenbrenner\n03:13:49\nLook at the\nAsian Tigers\nat 10%. It's totally possible.\nDwarkesh Patel\n03:13:50\nBut without productivity gains, it's not like the Industrial Revolution. From a perspective of outside the system, your goods become much cheaper and you can manufacture more things. But it's not a sign that the next century is rapidly approaching.\nLeopold Aschenbrenner\n03:14:07\nBoth are important. The other thing I'll say is that with all of this stuff, the magnitudes are really, really important. We talked about a 10x in research effort, or maybe 10-30x over a decade. Even without any kind of self-improvement type loops — even in the sort of GPT-4 to AGI story — we're talking about an OOM of effective compute increase a year.\nIt’s half an OOM of compute, half an OOM of algorithmic progress that sort of translates into effective compute. You're basically doing 10x a year on your labor force. It's a radically different world if you're doing a 10x or 30x in a century versus a 10x a year on your labor force. The magnitudes really matter.\nIt also really matters in the intelligence explosion scenario, just the automated AI research part. One story you could tell there is that ideas get harder to find. Algorithmic progress is going to get harder. Right now, you have the easy wins, but in like four or five years, there will be fewer easy wins. So the sort of automated AI researchers are going to be necessary to just keep it going, because it's gotten harder. That's sort of a really weird knife-edge assumption economics.\nDwarkesh Patel\n03:15:09\nIsn't that the equilibrium story you were just telling about why the economy as a whole has 2% economic growth? You just proceed on the equilibrium. I guess you're saying by the time—\nLeopold Aschenbrenner\n03:15:17\nThe result of the equilibrium here is that it’s way faster. AIt depends on the sort of exponents. Suppose you need to 10x the effective research effort in AI research in the last four or five years to keep the pace of progress. We're not just getting a 10x, you're getting 1,000,000x or 100,000x. The magnitudes really matter.\nOne way to think about this is that you have two exponentials. You have your normal economy that's growing at 2% a year, and you have your AI economy growing at 10x a year. It's starting out really small. It's way faster and it's going to overtake eventually. You can just do the simple revenue extrapolation if you think your AI economy has some growth rate. It's a very simplistic way, but there's this 10x a year process.\nYou're going to transition the whole economy, as it broadens, from the 2% a year to the much faster growing process. That's very consistent with historical stories. There's this long-run hyperbolic trend. It manifested in the change in growth mode in the Industrial Revolution, but there's just this long-run hyperbolic trend. Now you have another change in growth mode.\nDwarkesh Patel\n03:16:35\nThat was one of the questions I asked Tyler\nwhen I had him on the podcast\n. The fact that, after 1776, you went from a regime of negligible economic growth to 2% is really interesting. From the perspective of somebody in the Middle Ages or before, 2% is the equivalent of like 10%. I guess you're projecting even higher for the AI economy.\nLeopold Aschenbrenner\n03:16:59\nIt depends. Again, with all this stuff I have a lot of uncertainty. A lot of the time I'm trying to tell the modal story because it's important to be concrete and visceral about it.\nI have a lot of uncertainty over how the 2030s play out. The thing I know is that it's going to be fucking crazy. As for exactly where the bottlenecks are and so on…\nDwarkesh Patel\n03:17:20\nLet's talk through the numbers here. You mentioned hundreds of millions of AI researchers. Right now,\nGPT-4o is like $15\nfor a million tokens outputted. A human thinks at 150 tokens a minute or something. If you do the math on that, for an hour's worth of human output, it's like $0.10 or something.\nLeopold Aschenbrenner\n03:17:48\nIt's cheaper than a human worker. It can't do the job yet.\nDwarkesh Patel\n03:17:51\nThat's right. But by the time you're talking about models that are trained on the 10 GW cluster, then you have something that is four OOMs more expensive via inference, something like three OOMs. That's like $100/hour of labor. Now you're having hundreds of millions of such laborers. Is there enough compute to do this kind of labor with the model that is 1000 times bigger?\nLeopold Aschenbrenner\n03:18:17\nGreat question. I actually don't think inference costs for frontier models are necessarily going to go up that much.\nDwarkesh Patel\n03:18:24\nBut isn't the test time sort of thing that it will go up even higher?\nLeopold Aschenbrenner\n03:18:28\nWe're just doing per token. Suppose each model token was the same as a human token thing at 100 tokens a minute. It'll use more, but the token calculation is already pricing that in. The question is per token pricing. GPT-3 when it launched was actually more expensive than GPT-4 now. Over vast increases in capability gains, inference cost has remained constant. That's sort of wild, and it's worth appreciating. It gestures at an underlying pace of algorithmic progress.\nThere's a more theoretically grounded way to explain why inference costs would stay constant. On Chinchilla scaling laws, half of the additional compute you allocate to bigger models and half of it you allocate to more data. If we go with the basic story of 0.5 OOM/year more compute and 0.5 OOM/year of algorithmic progress you're saving 0.5 OOM/year. That would exactly compensate for making the model bigger.\nThe caveat is that obviously not all training efficiencies are also inference efficiencies. A bunch of the time they are. Separately, you can find inference efficiencies. Given this historical trend and baseline theoretical reason, it's not a crazy baseline assumption that the frontier models are not necessarily going to get more expensive, per token.\nDwarkesh Patel\n03:19:52\nReally? Okay, that's wild.\nLeopold Aschenbrenner\n03:19:55\nWe'll see. Even if they get 10x more expensive, then you have 10 million instead of 100 million. It's not really—\nDwarkesh Patel\n03:20:04\nBut part of the intelligence explosion is that each of them has to run experiments that are GPT-4 sized. As a result, that takes up a lot of compute.\nThen you need to consolidate the results of experiments. What is the synthesized weight?\nLeopold Aschenbrenner\n03:20:20\nYou have much bigger inference compute anyway than your training. But the experiment compute is a constraint.\nDwarkesh Patel\n03:20:25\nLet’s go back to a more fundamental thing we're talking about here. In the series you say we should denominate the probability of getting to AGI in terms of OOMs of effective compute. Effective here accounts for the fact that there's a compute multiplier if you have a better algorithm.\nI'm not sure that it makes sense to be confident that this is a sensible way to project progress. It might be, but I have a lot of uncertainty about it.\nIt seems similar to somebody trying to project when we're going to get to the moon. They're looking at the\nApollo program\nin the 1950s or something. They're like, \"we have some amount of effective jet fuel and if we get more efficient engines, then we have more effective jet fuel. So we're going to determine the probability of getting to the moon based on the amount of effective jet fuel we have.\" I don't deny that jet fuel is important to launch rockets, but that seems like an odd way to denominate when you're going to get to the moon.\nLeopold Aschenbrenner\n03:21:36\nI don't know how rocket science works, but I didn't get the impression that there's some clear scaling behavior with the amount of jet fuel. First of all, the scaling laws in AI have just held. A friend of mine pointed this out and it's a great point. If you look at the\noriginal Kaplan scaling laws paper\n— it went from 10^-9 to 10 petaflop days — and then concatenate additional compute from there to GPT-4, assuming some algorithmic progress, the scaling laws have held probably over 15 OOMs. It’s a rough calculation so it’s maybe even more. They’ve held for a lot of OOMs.\nDwarkesh Patel\n03:22:14\nThey held for the specific loss function they're trained on, which is training the next token. Whereas the progress you are forecasting, we specifically know that that scaling can’t work because of the data wall. There's some new thing that has to happen, and I'm not sure whether you can extrapolate that same scaling curve to tell us whether these hobblings will also be fixed. Is this not on the same graph?\nLeopold Aschenbrenner\n03:22:39\nThe hobblings are just a separate thing.\nThere’s a few things here. On effective compute scaling, people center the scaling laws because they’re easy to explain. Why does scaling matter?\nThe scaling laws came way after people, at least like Dario and Ilya, realized that scaling mattered. There's this great quote from Dario on your\npodcast\n. The models just want to learn. You make them bigger and they learn more. That’s more important than the sort of loss curve.\nThat just applied across domains. You can look at this in benchmarks. Again, the headwind is the data wall. I’m bracketing that and talking about that separately.\nThe other thing is unhobblings. If you just put them on the effective compute graph, these unhobblings would be huge.\nDwarkesh Patel\n03:23:32\nWhat does it even mean? What is on the y-axis here?\nLeopold Aschenbrenner\n03:23:36\nSay MLPR on this benchmark or whatever. We mentioned the LMSys differences, RLHF which is as good as 100x, chain-of-thought. Just going from this prompting change, a simple algorithmic change can be like 10x effective compute increases on math benchmarks. This is useful to illustrate that unhobblings are large, but they're slightly separate things.\nAt a per token level, GPT-4 is not that far away from a token of my internal monologue. Even 3.5 to 4 took us from the bottom of the human range to the top of the human range on a lot of high school tests. It's a few more 3.5 to 4 jumps per token basis, per token intelligence. Then you've got to unlock the test time, solve the onboarding problem, make it use a computer, and then you're getting real close. The story might be wrong, but it is strikingly plausible.\nThe other thing I'll say is on the 2027 timeline, I do think it’s unlikely, but I do think there's worlds where there are AGI next year. That's basically if the test time compute overhang is really easy to crack. If it's really easy to crack, then you do like four OOMs of test time compute from a few hundred tokens to a few million tokens quickly. Then again, maybe it only takes one or two jumps equivalent equivalent to GPT-3.5 to 4, per token. One or two of those jumps per token plus test time compute and you basically have the proto automated engineer.\nDwarkesh Patel\n03:25:03\nI'm reminded of\nSteven Pinker\n’s book,\nThe Better Angels of Our Nature\n. It talks about the secular decline in violence and war and everything. You can just plot the line from the end of World War Two. In fact from before World War Two, and then these are just aberrations. Basically as soon as it happens you get Ukraine, Gaza, etc.\nLeopold Aschenbrenner\n03:25:32\nImpending ASI increasing crazy global conflict. ASI and crazy new WMDs.\nDwarkesh Patel\n03:25:39\nThis is a thing that happens in history where you see a straight line then as soon as you make that prediction… Who is that famous author?\nLeopold Aschenbrenner\n03:25:48\nAgain, people have been predicting deep learning will hit a wall every year. Maybe one year they're right. But it's gone a long way and it hasn't hit a wall. You don't have that much more to go.\n(03:25:58) – Alignment\nDwarkesh Patel\n03:25:58\nThis is a plausible story and let's just run with it and see what it implies.\nIn your series, you talk about alignment not from the perspective of “this is some doomer scheme to get the 0.01% of the probability distribution where things don't go off the rails.” It's more about just controlling the systems and making sure they do what we intend them to do.\nIf that's the case, we're going to be in this sort of geopolitical conflict with China. What we're worried about is them making the CCP bots that go out and take the red flag of Mao across the galaxies. Shouldn't we then be worried about alignment as something that, in the wrong hands, enables brainwashing, and dictatorial control?\nThis seems like a worrying thing. This should be part of the sort of algorithmic secrets we keep hidden. The secret of how to align these models, because that's also something the CCP can use to control their models.\nLeopold Aschenbrenner\n03:27:04\nIn the world where you get the democratic coalition, yeah. Also, alignment is often dual use.\nThe alignment team developed RLHF and it was great. It was a big win for alignment, but it also obviously makes these models useful. So alignment enables the CCP bots. Alignment also is what you need to get the US AIs to follow the Constitution, disobey unlawful orders, and respect separation of powers and checks and balances. You need alignment for whatever you want to do. It's just the underlying technique.\nDwarkesh Patel\n03:27:37\nTell me what you make of this take. I've been struggling with this a little bit.\nFundamentally, there's many different ways the future could go. There's one path that’s the Eliezer type: crazy AIs with nanobots take the future and turn everything into gray goo or paperclips.\nThe more you solve alignment, the more that path of the decision tree is circumscribed. The more you solve alignment, the more it’sjust different humans and the visions they have. Of course, we know from history that things don't turn out the way you expect. It's not like you can decide the future.\nLeopold Aschenbrenner\n03:28:10\nThat’s part of the beauty of it.  You want these mechanisms like error correction—\nDwarkesh Patel\n03:28:14\nBut from the perspective of anybody who's looking at the system it'll be like, “I can control where this thing is going to end up.” So the more you solve alignment — the more you circumscribe the different futures that are the result of AI will — the more that accentuates the conflict between humans and their visions of the future. The world where alignment is solved is the one in which you have the most sort of human conflict over where to take AI.\nLeopold Aschenbrenner\n03:28:42\nBy removing the worlds in which the AIs take over, the remaining worlds are the ones where the humans decide what happens. As we talked about, there are a whole lot of worlds there and how that could go.\nDwarkesh Patel\n03:28:53\nYou think about alignment as just controlling these things. Just think a little forward. There are worlds in which hopefully human descendants, or some version of that in the future, merge with superintelligences. They have the rules of their own but they're in some sort of law and market-based order. I worry because you’ll have things that are conscious and should be treated with rights. I’m thinking about what these alignment schemes actually are.\nYou read these books about what actually happened during the\nCultural Revolution\n, what happened when Stalin took over Russia. You have very strong monitoring from different instances where everybody's tasked with watching each other. You have brainwashing. You have\nred teaming\nlike the spy stuff you were talking about where you try to convince somebody you're a defector and you see if they defect with you. If they do, then you realize they're an enemy.\nMaybe I'm stretching the analogy too far but the ease with which these alignment techniques actually map onto something you could have read about during Mao's Cultural Revolution is a little bit troubling.\nLeopold Aschenbrenner\n03:30:06\nSentient AI is a whole other topic. I don't know if we want to talk about it. I agree that it's going to be very important how we treat them. In terms of what you're actually programming these systems to do, again alignment is just a technical solution. It enables the CCP bots\nTalking about checks and balances, the model is sort of like the\nFederal Reserve\nor Supreme Court justices. There's a funny way in which they're kind of this very dedicated order. It's amazing. They're actually quite high quality. They're really smart people who truly believe in and love the Constitution. They believe in their principles.\nThey have different persuasions, but they have very sincere debates about what is the meaning of the Constitution and what is the best actuation of these principles. By the way, I recommend\nSCOTUS oral arguments\nas the best podcast when I run out of high quality content on the Internet.\nThere's going to be a process of figuring out what the Constitution should be. This Constitution has worked for a long time. You start with that. Maybe eventually things change enough that you want edits to that. For example, on the checks and balances, they really love the Constitution. They believe in it and and they take it seriously.\nAt some point you are going to have AI police and AI military it’ll be important to ensure that they believe in the Constitution the way that a Supreme Court justice does or the way that a Federal Reserve official takes their job really seriously.\nThe other important thing is that a bunch of different factions need their own AIs. It's really important that each political party gets to have their own superintelligence. You might totally disagree with their values, but it's important that they get to have their own kind of superintelligence. It’s important that these classical liberal processes play out, including different people of different persuasions and so on. The AI advisors might not make them wise. They might not follow the advice or whatever, but it's important.\nDwarkesh Patel\n03:32:08\nYou seem pretty optimistic about alignment. Let’s get to the source of the optimism. You laid out different worlds in which we could get AI. There's one that you think has a low probability of happening next year, where GPT-5 plus scaffolding plus unhobblings gets you to AGI. There are also scenarios where it takes much longer.\nGPT-4 seems pretty aligned in the sense that I don't expect it to go off the rails. Maybe with scaffolding, things might change. It looks pretty good, and maybe you will keep turning the cranks, and one of them gets you to ASI.\nIs there any point at which the sharp left turn happens? Do you think it's plausible that when they start acting more like agents, this is something to worry about? Is there anything qualitative that you expect to change with regards to the alignment perspective?\nLeopold Aschenbrenner\n03:33:07\nI don't know if I believe in this concept of a sharp left turn, but there are important qualitative changes that happen between now and somewhat superhuman systems early on in the intelligence explosion. There are also important qualitative changes that occur from early in the intelligence explosion to true superintelligence in all its power and might.\nLet's talk about both of those. The first part of the problem is one we're going to have to solve ourselves. We have to align the initial AI and the intelligence explosion, the sort of automated Alec Radford. There are two important things that change from GPT-4. If you believe the story on synthetic data RL, self-play, to get past the data wall, and if you believe this unhobbling story, at the end you're going to have things that are agents. They’ll do long-term planning. They have long horizons, which is a prerequisite to being able to do automated AI research.\nPre-training is alignment-neutral in the sense that it has good representations  and representations of doing bad things, but it's not scheming against you. Misalignment can arise once you're doing more long-horizon training. For example, if you're training an AI to make money using reinforcement learning, it might learn to commit fraud, lie, deceive, or seek power simply because those are successful strategies in the real world. With RL, it explores, maybe it tries to hack and then it gets some money. If that’s successful, that gets reward and that’s just reinforced. There’s more serious misalignments, like misaligned long-term goals, that necessarily have to be able to arise if you’re able to get long-horizon systems.\nLet’s swap. What you want to do in that situation is add side constraints, like \"don't lie,\" \"don't deceive,\" or \"don't commit fraud.\" How do you add those side constraints? The basic idea you might have is RLHF. You have this goal of making money, but you're watching what it's doing. If it starts trying to lie, deceive, commit fraud, or break the law, you give it a thumbs down and anti-reinforce that behavior.\nThe critical issue that arises is that these AI systems are becoming superhuman and will be able to do things that are too complex for humans to evaluate. Even early on in the intelligence explosion, the automated AI researchers and engineers might write millions, billions, or trillions of lines of complicated code. You won't understand what they're doing anymore. In those millions of lines of code, you don't know if it's hacking, exfiltrating itself, or trying to go for the nukes.\nYou don’t know anymore. Thumbs up, thumbs down pure RLHF doesn't fully work anymore in this scenario. There’s a hard technical problem of what do you do post-RLHF but it’s a solvable problem. There’s various things I’m bullish on. There’s ways in which deep learning has shaped out favorably.\nThe second part of the picture is going from your initial systems in the intelligence explosion to superintelligence, many OOMs of improvement. By the end of it, you have a thing that's vastly smarter than humans. The intelligence explosion is really scary from an alignment point of view. If you have this rapid intelligence explosion in less than a year or two, you're going from systems where failure would be bad but not catastrophic to a world where if something goes awry, the AI could exfiltrate itself, start hacking the military, and do really bad things.\nIn less than a year, you're going from a world where the AI is some descendant of current systems that you understand and has good properties. It becomes something that potentially has a very alien and different architecture after having gone through another decade of ML advances. One salient example is legible and faithful chain-of-thought. A lot of the time when we're talking about these things, we're talking about how it has tokens of thinking and then uses many tokens of thinking. Maybe we bootstrap ourselves by pre-training it to learn to think in English, and then we do something else on top so it can do longer chains of thought.\nIt's very plausible to me that for the initial automated alignment researchers, we don't need to do any complicated\nmechanistic interpretability\n. You can just read what they're thinking, which is a huge advantage. However, it's very likely not the most efficient way to do it. There's probably some way to have a recurrent architecture with all internal states. That's a much more efficient way to do it.\nThat's what you get by the end of the year. You're going in this year from RLHF++ to something that's vastly superhuman. To us, it might be like an expert in the field compared to an elementary or middle school student. It’s an incredibly hairy period for alignment. The thing you do have is the automated AI researchers. You can use them to also do alignment.\nDwarkesh Patel\n03:38:18\nIn this world, why are we optimistic that the project is being run by the right people? Here's something to think about. OpenAI starts off with people who are very explicitly thinking about exactly these kinds of things.\nLeopold Aschenbrenner\n03:38:38\nYes but are they still there?\nDwarkesh Patel\n03:31:48\nNo, but here’s the thing. Even with the current leadership, you can find them in interviews and blog posts talking about it. You talked about what happens and it’s not just you. Jan talked about it in his\nTweet thread\n. There is some trade-off that has to be made with doing a flashy release this week and not next week because Google I/O is next week or whatever. The trade-off is made in favor of the more careless decision.\nThe government, the national security advisor or the military or whatever, is much less familiar with this kind of discourse. They’re not like, “I'm worried the chain-of-thought is unfaithful. How do we think about the features that are represented here?” Why should we be optimistic that a project run by people like that will be thoughtful about these kinds of considerations?\nLeopold Aschenbrenner\n03:39:42\nThey might not be. Here’s a few thoughts. First of all, the private world is extremely tough for alignment even if they nominally care. There’s a couple of reasons. You have the race between the commercial labs. You don't have any headroom there to be like, aAh, actually we're going to hold back for three months, get this right. We're going to dedicate 90% of our compute to automated alignment research instead of just pushing the next OOM.\"\nThe other thing is that in the private world, China has stolen your weights. China has your secrets. They're right on your tails. You're in this fever struggle. There’s no room at all for maneuver. It's absolutely essential to get alignment right. To get it during this intelligence explosion, to get it right, you need to have that room to maneuver and you need to have that clear lead. Again, maybe you've made the deal or whatever, but you're in an incredibly tough spot if you don't have this clear lead.\nSo the private world is kind of rough there. Whether people will take it seriously… I have some faith normal mechanisms of a liberal society.\nWwe don't fully know yet if alignment is an issue. The science will develop.  We're going to get better measurements of alignment. The case will be clear and obvious.\nI worry that there's worlds where evidence is ambiguous. A lot of the most scary intelligence explosion scenarios are worlds in which evidence is ambiguous. But again, if evidence is ambiguous, then those are the worlds in which you really want the safety margins. Those are also the worlds in which running the intelligence explosion is sort of like running a war. The evidence is ambiguous. You have to make these really tough trade-offs. You better have a really good chain of command for that where they’re not just YOLOing it.\n(03:41:26) – On Germany, and understanding foreign perspectives\nDwarkesh Patel\n03:41:26\nLet’s talk a little about Germany. We’re making the analogy to World War II. You made this. really interesting point many hours ago. Throughout history, World War Two is not unique at least when you think in proportion to the size of the population. Let’s look at these other sorts of catastrophes where a substantial proportion of the population has been killed off.\nAfter that, the nation recovers and they get back to their heights. What's interesting after World War Two is that Germany especially, and maybe Europe as a whole, they experienced vast economic growth in the immediate aftermath because of catch-up growth.\nWe're not talking about Germany potentially launching an intelligence explosion and them getting a seat at the AI table. We were talking about Iran and North Korea and Russia. We didn't talk about Germany.\nLeopold Aschenbrenner\n03:42:33\nBecause they're allies.\nDwarkesh Patel\n03:42:35\nSo what happened? We had World War Two and now it didn't come back to the Seven Years' War or something.\nLeopold Aschenbrenner\n03:42:42\nI'm generally very bearish on Germany. In this context, you're underrating a little bit. It's probably still one of the top five most important countries in the world. Europe overall still has a GDP that's close to the United States in size. There are things that Germany is actually kind of good at, like state capacity. The roads are good and they're clean and they're well-maintained.\nIn some sense, a lot of this is the flip side of things that are bad about Germany. In the US, there's a bit more of a Wild West feeling. It includes the kind of crazy bursts of creativity. It includes political candidates. There's a much broader spectrum. Both Obama and Trump are politicians you just wouldn't see in the much more confined kind of German political debate. I wrote a blog post at some point about this, “\nEurope’s Political Stupor\n.”\nThere's this punctilious sort of rule-following that is good in terms of keeping your state capacity functioning. But that is also a very constrained view of the world in some sense. After World War Two, there's a real backlash against anything elite. There are no elite high schools or elite colleges. Excellence isn't cherished.\nDwarkesh Patel\n03:44:11\nWhy is that the logical, intellectual thing to rebel against if you're trying to overcorrect from the Nazis?\nWas it because the Nazis were very much into elitism? I don't understand why that's a logical sort of reaction.\nLeopold Aschenbrenner\n03:44:26\nMaybe it was a counter reaction against the whole Aryan race and that sort of thing. Look at the end of World War I versus the end of World War II for Germany. A common narrative is that the\nPeace of Versailles\nwas too strict on Germany. The peace imposed after World War Two was much more strict.\nThe whole country was destroyed. In most of the major cities, over half of the housing stock had been destroyed. In some birth cohorts, something like 40% of the men had died, Almost 20 million people displaced. It was huge and crazy.\nDwarkesh Patel\n03:45:09\nAnd the borders are way smaller than the Versailles borders.\nLeopold Aschenbrenner\n03:45:11\nYeah, exactly. There’s also a complete imposition of a new political system on both sides. But in some sense, that worked out better than the post-World War I peace where there was this resurgence of German nationalism. In some sense, it's unclear if you want to wake the sleeping beast. At this point, it's gotten a bit too sleepy.\nDwarkesh Patel\n03:45:39\nIt's an interesting point about how we underrate the American political system. I've been making the same correction myself. There was this book written by a Chinese economist called\nChina's World View\n.\nOverall, I wasn't a big fan, but it made a really interesting point, which was the way in which candidates rise up through the Chinese hierarchy for politics and administration. In some sense, it selects so that you're not going to get some\nMarjorie Taylor Greene\nor someone like that.\nLeopold Aschenbrenner\n03:46:11\nDon't get that in Germany either.\nDwarkesh Patel\n03:46:13\nBut he explicitly made the point in the book that it also means they’re never going to get a\nHenry Kissinger\nor Barack Obama in China. By the time they end up in charge of the\nPolitburo\n, they'll be some 60-year-old bureaucrat who's never ruffled any feathers.\nLeopold Aschenbrenner\n03:46:28\nThere's something really important about the very raucous political debate in the US. In general in America lots of people live in their own world. We live in this kind of bizarre little bubble in San Francisco and people. But that's important for the evolution of ideas, error correction, that sort of thing.\nThere are other ways in which the German system is more functional.\nThere are also major mistakes, like with\ndefense spending\n. Russia invades Ukraine and they’re like, \"wow, what did we do?\"\nDwarkesh Patel\n03:47:06\nThat's a really good point.\nThe main issue is that everybody agrees.\nLeopold Aschenbrenner\n03:47:09\nExactly. There was no debate about it. It’s a consensus\nBlob\nkind of thing.\nOn the China point, I have this experience of reading German newspapers and I would understand the German debate and the state of mind much more poorly without it from just afar. It is interesting just how impenetrable China is to me. It's a billion people.\nAlmost everything else is really globalized. You have a globalized Internet. I kind of have a sense what's happening in the UK. Even if I didn't read German newspapers, I would have a sense of what's happening in Germany. But I really don't feel like I have a sense of what is the state of mind, what is the state of political debate, of an average Chinese person or an average Chinese elite.\nI find that distance kind of worrying. There are some people who do this and they do really\ngreat work\nwhere they go through the party documents and the party speeches. It seems to require a lot of interpretive ability. There are very specific words in Mandarin that mean one connotation, not the other connotation. It's interesting given how globalized everything is. Now we have basically perfect translation machines and it's still so impenetrable.\nDwarkesh Patel\n03:48:22\nThat's really interesting. I'm sort of ashamed almost that I haven't done this yet. Many months ago when\nAlexey interviewed me\non his YouTube channel, I said, \"I'm meaning to go to China to actually see for myself what's going on.\" By the way, if anybody listening has a lot of context on China and if I went to China, could introduce me to people, please email me.\nLeopold Aschenbrenner\n03:48:44\nYou have to do some pods and find some of the Chinese AI researchers. It’d be so good.\nI don't know if they can speak freely.\nDwarkesh Patel\n03:48:54\nSo they have these papers and on the paper they'll say who's a co-author. I was thinking of just cold emailing everybody, like, \"Here's my Calendly. Let's just talk.” I just want to see what the vibe is. Even if they don't tell me anything, I'm just like, “what kind of person is this? How westernized are they?\"\nI just remembered that, in fact, ByteDance, according to mutual friends we have at Google, cold emailed every single person on the Gemini paper and said, \"if you come work for ByteDance, we'll make you a L8 engineer. You'll report directly to the CTO.\"\nLeopold Aschenbrenner\n03:49:32\nThat's how the secrets go over.\nDwarkesh Patel\n03:49:34\nI meant to ask this earlier. If there's only 100 or so people, maybe less, who are working on the key algorithmic secrets. If they hired one such person, is all the alpha that these labs have gone?\nLeopold Aschenbrenner\n03:49:55\nIf this person was intentional about it, they could get a lot. Actually, they could probably just exfiltrate the code. They could get a lot of the key ideas. Again up until recently, stuff was published but they could get a lot of the key ideas if they tried. There are a lot of people who don't actually look around to see what the other teams are doing. But you kind of can. They could. It's scary.\nDwarkesh Patel\n03:50:08\nThe project makes more sense there, where you can't just recruit a Manhattan Project engineer.\nLeopold Aschenbrenner\n03:50:15\nThese are secrets that can be used for probably every training run in the future. Maybe they’re the key to the data wall without which they can’t go on. They're going to give multipliers on compute worth hundreds of billions, trillions of dollars. All it takes is China to offer $100 million to somebody and say “come work for us.” I'm really uncertain on how seriously China is taking AGI right now.\nOne anecdote was related to me by another researcher in the field. They were at a conference with somebody, a Chinese AI researcher. He was talking to him and he was like, \"I think it's really good that you're here. We have to have international coordination and stuff.\" Apparently this guy said that \"I'm the most senior person that they're going to let leave the country to come to things like this.\"\nDwarkesh Patel\n03:51:07\nWhat's the takeaway?\nLeopold Aschenbrenner\n03:51:09\nThey're not letting really senior AI researchers leave the country. It’s a kind of classic\nEastern Bloc\nmove.\nI don't know if this is true, but it’s what I heard.\nDwarkesh Patel\n03:51:19\nLet’s go back to the point you made earlier about being exposed to German newspapers. Earlier you mentioned you were interested in economics and law and national security. The variety in intellectual diet has exposed you to thinking about the geopolitical question here in ways that that others talking about AI aren’t.\nThis is the first episode I've done about this where we've talked about things like this. Now that I think about it, that’s weird given that this is an obvious thing in retrospect. I should have been thinking about it. That's one thing we've been missing.\nWhat are you missing, not just in national security? What perspective are you probably underexposed to as a result? I guess you mentioned China as one.\nLeopold Aschenbrenner\n03:52:00\nThe China one is an important one. Another one would be a sort of very Tyler Cowen-esque take. You're not exposed to how a normal person in America will use AI. That kind of thing will be a bottleneck to the diffusion of these things. I'm overrating the revenue because I assume everyone is adopting it. But Joe Schmo engineer at a company, will they be able to integrate it? Also what’s the reaction to it? This was a question hours ago. Won’t people rebel against this? Will they not want to do the project? I don't know. Maybe they will.\nDwarkesh Patel\n03:52:41\nHere's a political reaction that I didn't anticipate.\nI already told you about this, but I'm just going to tell the story again.\nTucker Carlson\nwas recently on a\nJoe Rogan episode\n. They start talking about World War II.\nTucker says, \"well, listen, I'm going to say something that my fellow conservatives won't like, but I think nuclear weapons are immoral. I think it was obviously immoral that we use them on Nagasaki and Hiroshima.\"\nThen he says, \"In fact, nuclear weapons are always immoral, except when we would use them on data centers. In fact, it would be immoral not to use them on data centers, because, look, these people in Silicon Valley, these fucking nerds, are making superintelligence, and they say that it could enslave humanity. We made machines to serve humanity, not to enslave humanity. And they're just going on and making these machines. And so we should, of course, be nuking the data centers.\" That is definitely not a political reaction in 2024 I was expecting.\nIt's going to be crazy.\nDwarkesh Patel\n03:53:50\nThe thing we learned with COVID is also that the left-right reactions that you’d anticipate just based on hunches—\nLeopold Aschenbrenner\n03:53:58\nIt completely flipped multiple times. Initially the right was on it and the left was like, \"this is racist.\" Then it flipped. The left was really into the lockdowns. The whole thing also is just so blunt and crude.\nProbably in general, people like to make sort of complicated technocratic AI policy proposals. If things go kind of fairly rapidly on the path to AGI, there might not actually be that much space for complicated, clever proposals. It might just be much cruder reactions.\nDwarkesh Patel\n03:54:33\nYou mentioned spies and national security getting involved and everything. You can talk about that in the abstract, but now that we're living in San Francisco we know many of the people who are doing the top AI research. It’s also a little scary to think about people I personally know and am friends with. It's not unfeasible if they have secrets in their head that are worth $100 billion or something that you might see kidnapping, assassination, sabotage.\nLeopold Aschenbrenner\n03:55:01\nOh, their family. It's really bad. To the point on security, right now it’s really foreign. At some point, as it becomes really serious, you're going to want the security guards.\nDwarkesh Patel\n03:55:16\nPresumably, you have thought about the fact that people in China will be listening to this and reading your series.\nSomehow you made the trade-off that it's better to let the whole world know, including China, and wake them up to AGI than to stay silent. Part of the thing you're worried about is China waking up to AGI. I’m curious about that. Walk me through how you've thought about that trade-off.\nLeopold Aschenbrenner\n03:55:44\nThis is a tough trade-off. I thought about this a bunch. People in the PRC will read this.\nTo some extent the cat is out of the bag. AGI is a thing people are thinking about very seriously. That’s not new anymore. A lot of these takes are kind of old or I had similar views a year ago. I might not have written it up a year ago, in part because I didn’t think the cat wasn't out of the bag enough then.\nTo be able to manage this challenge, much broader swaths of society will need to wake up. If we're going to get the project, we actually need a broad bipartisan understanding of the challenges facing us. It's a tough trade-off. The need to wake up people in the United States, in the Western world, in the democratic coalition, is ultimately imperative. My hope is more people here will read it than in the PRC.\nPeople sometimes underrate the importance of just kind of writing it up and laying out the strategic picture. You have done actually a great service to mankind in some sense with your podcast. It's overall been good.\n(03:57:04) – Dwarkesh’s immigration story and path to the podcast\nLeopold Aschenbrenner\n03:57:04\nBy the way, on the topic of Germany. We were talking at some point about immigration stories. You have an interesting story you haven't told, butI think you should tell it\nDwarkesh Patel\n03:57:17\nSo a couple years ago, I was in college and I was 20. I was about to turn 21.\nLeopold Aschenbrenner\n03:57:25\nYou came from India when you were really young, right?\nDwarkesh Patel\n03:57:28\nUntil I was eight or nine, I lived in India. Then we moved around all over the place. Because of the\nbacklog for Indians\nwe’d been in the queue for decades.\nLeopold Aschenbrenner\n03:57:44\nEven though you came at eight, you're still on the\nH-1B\n.\nDwarkesh Patel\n03:57:47\nWhen you're 21 you get kicked off the queue and you have to restart the process. My dad's a doctor and I'm on his H-1B as a dependent. But when you're 21, you get kicked off.\nSo I'm 20 and it just kind of dawns on me that this is my situation.\nLeopold Aschenbrenner\n03:58:00\nYou’re completely screwed.\nDwarkesh Patel\n03:58:01\nI also had the experience with my dad.\nWe moved all around the country. They have to prove, him being a doctor, that you can't get native talent.\nLeopold Aschenbrenner\n03:58:11\nAnd you can’t start a startup or anything.\nEven getting the H-1B for you would have been a 20% lottery, if you're lucky.\nDwarkesh Patel\n03:58:19\nPlus they had to prove that they can't get native talent, which meant that we lived in North Dakota for three years, West Virginia for three years, Maryland, West Texas.\nSo it dawned on me that this is my situation as I turn 21. I'll be on this lottery. Even if I get the lottery, I'll be a fucking code monkey for the rest of my life, because this thing isn't going to let up.\nLeopold Aschenbrenner\n03:58:37\nYeah. Can't do a startup.\nDwarkesh Patel\n03:58:38\nExactly. At the same time, I had been reading for the last year and was super obsessed with\nPaul Graham essays\n. My plan at the time was to make a startup or something. I was super excited about that.\nIt just occurred to me that I couldn't do this. That just wasn’t in the cards for me. I was kind of depressed about it. I remember I was in a daze through finals because it had just occurred to me. I was really anxious about it.\nI remember thinking to myself at the time that if somehow I ended up getting my green card before I turned 21, there's no fucking way I'm becoming a code monkey. The feeling of dread that I have is this realization that I'm just going to have to be a code monkey. I realized that's my default path. If I hadn't made a proactive effort not to do that, I would have graduated college as a computer science student. I would have just done that. That's the thing I was super scared about. That was an important realization for me.\nAnyway, COVID happened. Because of that, since there weren't any foreigners coming, the backlog got fast-tracked and by the skin of my teeth, like a few months before I turned 21, I ended up getting a green card for crazy, extremely contingent reasons.\nBecause I got a green card, I could—\nLeopold Aschenbrenner\n03:59:56\nThe whole podcast.\nDwarkesh Patel\n03:59:57\nI graduated college and I was bumming around. I graduated a semester early. I'm going to do this podcast and see what happens? If I didn’t have a green card, I mean the best case scenario—\nLeopold Aschenbrenner\n04:00:08\nIt’s such a cultural artifact. What is the impact of immigration reform?\nWhat is the impact of clearing 50,000 green cards in the backlog? You're such an amazing example how all of this is only possible contingent on that. It's just incredibly tragic that this is so dysfunctional.\nDwarkesh Patel\n04:00:30\nIt's insane.\nLeopold Aschenbrenner\n04:00:34\nI'm glad you did it. I'm glad you kind of tried the unusual path.\nDwarkesh Patel\n04:00:39\nI could only do it because I was extremely fortunate to get the green card. I had a little bit of saved up money. I got a small grant out of college, thanks to the Future Fund, to do this for like six months. It turned out really well. At each time, I was like, \"oh, okay, podcast. Come on. I wasted a few months on this. Let's now go do something real.\" Something big would happen at each moment.\nLeopold Aschenbrenner\n04:01:09\nYou kept with it.\nDwarkesh Patel\n04:01:11\nThere would always be something the moment I'm about to quit the podcast. Jeff Bezos would\nsay something nice about me on Twitter.\nThe\nIlya episode\ngets like half a million views. Now this is my career. Looking back on it though, it was incredibly contingent that things worked out the right way.\nLeopold Aschenbrenner\n04:01:27\nIf the AGI stuff goes down, it'll be how most of the people who kind of end up feeling AGI first heard about it.\nDwarkesh Patel\n04:01:41\nYou're also very linked with the story in many ways. I got like a $20,000 grant from Future Fund right out of college and that sustained me for six months or something. Without that…\nLeopold Aschenbrenner\n04:01:57\nTiny grant. It was kind of crazy. It goes to show how far small grants can go. Emergent Ventures, too.\nDwarkesh Patel\n04:11:32\nExactly. Emergent Ventures. The last year I've been in San Francisco, we've just been in close contact the entire time and just bouncing ideas back and forth. People would be surprised by how much of the alpha I have I got from you, Sholto, Trenton and a couple others.\nLeopold Aschenbrenner\n04:02:26\nIt’s been an absolute pleasure.\nDwarkesh Patel\n04:02:27\nLikewise, it's been super fun. Here are some random questions for you. If you could convert to Mormonism and you could really believe it, would you do it? Would you push the button?\nLeopold Aschenbrenner\n04:02:40\nBefore I answer that question, one observation about the Mormons. There's\nan article\nthat actually made a big impact on me. It was about the Mormons, by McKay Coppins in\nThe Atlantic\n. He even interviewed Mitt Romney in it.\nThe thing he talked about was how the experience of growing up different, growing up very unusual, especially if you grew up Mormon outside of Utah. You’re the only person who doesn't drink caffeine, you don't drink alcohol, you're kind of weird. That got people prepared for being willing to be outside of the norm later on.\nMitt Romney was willing to take stands alone in his party because he believed what he believed was true. Probably not in the same way, but I feel a little bit like this from having grown up in Germany, having been kind of an outsider or something.\nGrowing up as an outsider gives you unusual strength later on to be willing to say what you think. So that is one thing I really appreciate about the Mormons, at least the ones that grow up outside of Utah.\nThe other thing is the fertility rates. They're good. They're important. They're going down as well. This is the thing that really clinched the fertility decline story for me. Even the Mormons.\nDwarkesh Patel\n04:03:57\nYou're like, \"oh, this is like a good start. Mormons will replace everybody.\"\nLeopold Aschenbrenner\n04:04:00\nI don't know if it's good, but at least some people will maintain high fertility rates. But no, even the Mormons... Once these religious subgroups that have high fertility rates grow big enough, they become too close in contact with normal society and become normalized. Their fertility rates drop from maybe like four to two in the course of 10-20 years.\nPeople point to the Amish or whatever, but it's probably just not scalable. If you grow big enough, then there's just this overwhelming force of modernity that gets you.\nNo, if I could convert to Mormonism... Look, I think there's something... I don't believe it, right? If I believed it, I obviously would convert to Mormonism, because you got to convert.\nDwarkesh Patel\n04:04:41\nBut you can choose a world in which you do believe it.\nLeopold Aschenbrenner\n04:04:46\nThere's something really valuable in believing in something greater than yourself and having a certain amount of faith.\nDwarkesh Patel\n04:04:55\nYou do, right? That's what your series is.\nLeopold Aschenbrenner\n04:04:59\nIt’s valuable to feel some sort of duty to something greater than yourself. Maybe my version of this is somewhat different. I feel some sort of duty to the historical weight on how this might play out. I feel some sort of duty to make that go well. I feel some sort of duty to our country, to the national security of the United States. We can be a force for a lot of good.\nDwarkesh Patel\n04:05:28\nGoing back to OpenAI, there’s something that's especially impressive about that is. There are people at the company who have — through years and decades of building up savings from working in tech — probably tens of millions of dollars liquid and more than that in terms of their equity. Many people were concerned about the clusters and the Middle East and the secrets leaking to China and all these things.\nThe person who actually made a hassle about it — hassling people is so underrated — is the 22-year-old who has less than a year at the company, who doesn't have savings built up, who isn't a solidified member of the company.\nLeopold Aschenbrenner\n04:06:24\nMaybe it's me being naive and not knowing how big companies work. Sometimes I'm a bit of a speech\ndeontologist\n. I kind of believe in saying what you think. Sometimes friends tell me I should be more of a speech\nconsequentialist\n.\nDwarkesh Patel\n04:06:39\nI mean I think about the amount of people who, when they have the opportunity to talk to the person, will just bring up the thing. I've been with you in multiple contexts. I guess I shouldn't reveal who the person is or what the context was.\nI've just been very impressed that the dinner begins and by the end, somebody who has a major voice in how things go is seriously thinking about a worldview they would have found incredibly alien before the dinner. I've been impressed that you just give them the spiel and hassle them.\nLeopold Aschenbrenner\n04:07:16\nI just feel this stuff pretty viscerally now. There was a time when I thought about this stuff a lot, but it was kind of like econ models and these theoretical abstractions. You talk about human brain size or whatever.\nSince at least last year, I feel like I can see it. I feel it. I can sort of see the cluster that AGI can be trained on. I can see the kind of rough combination of algorithms and the people that will be involved and how this is going to play out. Look, we'll see how it plays out. There are many ways this could be wrong. There are many ways it could go, but this could get very real.\n(04:07:58) – Launching an AGI hedge fund\nDwarkesh Patel\n04:07:58\nShould we talk about what you're up to next?\nLeopold Aschenbrenner\n04:07:59\nSure, yeah.\nDwarkesh Patel\n04:08:01\nYou're starting an investment firm with anchor investments from Nat Friedman, Daniel Gross, Patrick Collison, John Collison. First of all, why is this the thing to do if you believe AGI is coming in a few years? Why the investment firm?\nLeopold Aschenbrenner\n04:08:18\nGood question. Fair question. A couple of things. We talked about this earlier, but the screen doesn't go blank when AGI intelligence happens. People really underrate the decade after you have the intelligence explosion. That's maybe the most wild period. The decade after is also going to be wild.\nThis combination of human institutions with superintelligence and crazy geopolitical things going on. You have this broadening of this explosive growth. Basically, it's going to be a really important period. Capital will really matter. Eventually we're going to go to the stars, going to go to the galaxies.\nPart of the answer is just that done right, there's a lot of money to be made. If AGI were priced in tomorrow, you could maybe make 100x. Probably you can make even way more than that because of the sequencing and capital matters.\nThe other reason is just some amount of freedom and independence. There are some people who are very smart about this AI stuff and who see it coming. Almost all of them are constrained in various ways. They're in the labs, they're in some other position where they can't really talk about this stuff.\nI've really admired the thing you've done. It's really important that there are voices of reason on this stuff publicly or people who are in positions to kind of advise important actors and so on.\nBasically, this investment firm will be kind of like a brain trust on AI. It's going to be all about situational awareness. We're going to have the best situational awareness in the business. We're going to have way more situational awareness than any of the people who manage money in New York. We're definitely going to do great on investing, but it's the same sort of situational awareness that is going to be important for understanding what's happening, being a voice of reason publicly, and being able to be in a position to advise.\nDwarkesh Patel\n04:10:09\nThe\nbook\nabout Peter Thiel, they had an interesting quote about his hedge fund. It got terrible returns. So this isn't the example...\nLeopold Aschenbrenner\n04:10:17\nIt blew up. That’s sort of the bear case. It’s too theoretical.\nDwarkesh Patel\n04:10:22\nThey had an interesting quote that it's basically a think tank inside of a hedge fund.\nLeopold Aschenbrenner\n04:10:28\nThat’s what I’m going to try to build.\nDwarkesh Patel\n04:10:30\nPresumably you've thought about the ways in which these kinds of things can blow up. There's a lot of interesting business history books about people who got the thesis right but timed it wrong. They buy into the idea that the Internet's going to be a big deal. They\nsell at the wrong time and buy at the wrong time\nduring the\ndot-com boom\n. They miss out on the gains even though they're right about the. What is the trick to preventing that kind of thing?\nLeopold Aschenbrenner\n04:10:58\nObviously, not blowing up is task number one and two. This investment firm is going to just be betting on AGI. We’re going to be betting on AGI and superintelligence before the decade is out, taking that seriously, making the bets you would make if you took that seriously. If that's wrong, the firm is not going to do that well.\nThe thing you have to be resistant to is you have to be able to resist one or a couple or a few individual calls. AI stagnates for a year because of the data wall, or you got the call wrong on when revenue would go up. That's pretty critical. You have to get the timing right. The sequence of bets on the way to AGI is actually pretty critical. People underrate it.\nWhere does the story start? Obviously, the only bet over the last year was Nvidia. It's obvious now, very few people did it. This is also a classic debate I and a friend had with another colleague of ours. This colleague was really into\nTSMC\n. He was just kind of like, \"well, these fabs are going to be so valuable. With Nvidia, there's just a lot of idiosyncratic risk, right? Maybe somebody else will make better GPUs.\" That was basically right.\nBut only Nvidia had the AI\nbeta\n, because only Nvidia was kind of like large fraction AI. The next few doublings would just meaningfully explode their revenue, whereas TSMC was a couple percent AI. Even though there's going to be a few doublings of AI, it was not going to make that big of an impact. The only place to find the AI beta, basically was Nvidia for a while.\nNow it's broadening. Now TSMC is like 20% AI by 2027 or something. That’s what they're saying. When we're doubling, it'll be kind of like a large fraction of what they're doing. There's a whole stack. There's people making memory and coops and power. Utilities companies are starting to get excited about AI. They're like, \"power production in the United States will grow not 2.5%, but 5% over the next five years.\" I'm like, \"no, it'll grow more.”\nAt some point, a Google or something becomes interesting. People are excited about them with AI because it's like, \"oh, AI revenue will be $10 billion or tens of billions.\" I don't really care about them before then. I care about it once you get the AI beta. At some point Google will get $100 billion of revenue from AI. Probably their stock will explode. They're going to become a $5 trillion, $10 trillion company anyway.\nThe timing there is very important. You have to get the timing right. You have to get the sequence right. At some point, actually, there's going to be real headwind to equities from real interest rates. In these sorts of explosive growth worlds, you would expect real interest rates to go up a lot. On the supply side it’ll be around the demand for money because people are going to be making these crazy investments, initially in clusters and then in the robo factories or whatever. They're going to be borrowing like crazy. They want all this capital, high ROI.\nOn the consumer saving side, to give up all this capital, it’ll be the\nEuler equation\n, standard intertemporal transfer trade-off of consumption.\nDwarkesh Patel\n04:14:06\nVery standard.\nLeopold Aschenbrenner\n04:14:09\nSome of our friends have a paper on this. Basically, if consumers expect real growth rates to be higher, interest rates are going to be higher because they're less willing to give up consumption today for consumption in the future.\nAt some point real interest rates will go up. Higher growth rate expectations mean equities go down because the interest rate effect outweighs the growth rate effect.\nAt some point there's the big bond short. You got to get that right. You got to get it right on nationalization. There's this whole sequence of things.\nDwarkesh Patel\n04:14:45\nAnd the unknown unknowns.\nLeopold Aschenbrenner\n04:14:46\nUnknown unknowns, yeah. You've got to be really, really careful about your overall risk positioning. If you expect these crazy events to play out, there's going to be crazy things you didn't foresee.\nYou do also want to make the bets that are tailored to your scenarios in the sense of you want to find bets that are bets on the tails. I don't think anyone is expecting interest rates to go above 10%, real interest rates. There's at least a serious chance of that before the decade is out. Maybe there's some cheap insurance you can buy on that.\nDwarkesh Patel\n04:15:18\nVery silly question. In these worlds, are financial markets where you make these kinds of bets going to be respected? Is my Fidelity account going to mean anything when we have 50% economic growth? Who’s like, “we have to respect his property rights”?\nLeopold Aschenbrenner\n04:15:34\nThat’s pretty deep into it, the bond short, the 50% growth. That's pretty deep into it. Again, there's this whole sequence of things. I think property rights will be respected. At some point, there's going to be figuring out the property rights for the galaxies. That'll be interesting.\nDwarkesh Patel\n04:15:53\nThat will be interesting. Going back to your strategy about how important the 2030s will be for how the rest of the future goes, you want to be in a position of influence by that point because of capital.\nAs far as I know, there's probably a whole bunch of literature on this, I'm just riffing. The landed gentry before the beginning of the Industrial Revolution, I'm not sure if they were able to leverage their position in a sort of\nGeorgist\nor\nPiketty\n-type sense, in order to accrue the returns that were realized through the Industrial Revolution. I don't know what happened. At some point, they just weren't the landed gentry.\nI'd be concerned that even if you make great investment calls, you'll be like the guy who owned a lot of farmland before the Industrial Revolution. The guy who's actually going to make a bunch of money is the one with the steam engine. Even he doesn't make that much money because most of the benefits are widely diffused and so forth.\nLeopold Aschenbrenner\n04:16:59\nThe analog is you sell your land and you put it all in the people who are building the new industries. The real depreciating asset for me is human capital. I was valedictorian of Columbia. The thing that made you special is you're smart. In four years, it might not matter because it's automatable.\nA friend joked that the investment firm is perfectly hedged for me. Either AGI happens this decade and my human capital depreciates, but I turn it into financial capital, or no AGI happens and the firm doesn’t do well, but I’m still in my twenties and smart.\nDwarkesh Patel\n04:17:44\nExcellent. What’s your story for why AGI hasn’t been priced in? Financial markets are supposed to be very efficient, so it’s hard to get an edge. Naively, you might say, “I’ve looked at these scaling curves, and they imply we’ll be buying much more compute and energy than analysts realize.”\nShouldn’t those analysts be broke by now? What’s going on?\nLeopold Aschenbrenner\n04:18:10\nI used to believe in the\nEMH\nguy as an economist. But now, I think there are groups of smart people, like those in San Francisco, who have alpha over the rest of society in seeing the future.\nIt’s like with COVID. A similar group of people saw it coming and called it completely corrected. They shorted the market and did really well. Why isn’t AGI priced in? It’s like asking why the government hasn’t nationalized the labs yet. Society hasn’t priced it in yet. It hasn’t completely diffused. I might be wrong but not many people take these ideas seriously.\n(04:19:14) – Lessons from WWII\nDwarkesh Patel\n04:19:14\nThere are a couple of other ideas I was playing around with that we haven’t gotten to talk about yet. One’s systems competition. One of my favorite books about World War II is\nVictor Davis Hanson’s summary of everything\n. He explains why the Allies made better decisions than the Axis.\nLeopold Aschenbrenner\n04:19:39\nWhy did they?\nDwarkesh Patel\n04:19:40\nThere were decisions the Axis made that were pretty good, like\nblitzkrieg\n.\nLeopold Aschenbrenner\n04:19:45\nThat was sort of by accident though.\nDwarkesh Patel\n04:19:47\nIn what sense? That they just had the infrastructure left over?\nLeopold Aschenbrenner\n04:19:49\nMy read of it is that blitzkrieg wasn’t an ingenious strategy. Their hand was forced. This is the very Adam Tooze-ian story of World War II. There’s the concept of a long war versus a short war, which is important. Germany realized that if they were in a long war, including the United States, they would not be able to compete industrially. Their only path to victory was to make it a short war. That worked much more spectacularly than they thought, allowing them to take over France and much of Europe.\nThe decision to invade the Soviet Union was related to the western front because they needed resources like oil. Auschwitz was actually a giant chemical plant to produce synthetic oil and other materials. It was the largest industrial project in Nazi Germany. They thought, “we crushed them in World War I, it’ll be easy. We’ll invade, get the resources, and then fight on the western front.” Even during the invasion of the Soviet Union, even though a large number of the deaths happened there, a large fraction of German industrial production—planes, naval forces, and so on—was directed towards the western front and the western allies.\nBy the way, this concept of a long war versus a short war is interesting, especially when thinking about the China competition. I worry about the decline of latent American industrial capacity.\nChina builds like 200 times more ships than we do right now\n.\nMaybe we have superiority in the non-AI world in military materiel and can win a short war or defend Taiwan. If it drags on, China might be better able to mobilize industrial resources in a way we can’t anymore. This is also relevant to AI. If building AGI requires a trillion-dollar cluster instead of a $100 billion cluster, or even if it’s on the $100 billion cluster, it really matters if you can do an order of magnitude more compute for your superintelligence. Maybe right now they’re behind, but they have the raw latent industrial capacity to outbuild us.\nThat matters both in the run-up to AGI and afterward. You have the superintelligence on your cluster, and then it’s time to expand the explosive growth. Will we let the robo-factories run wild? Maybe not, but maybe China will. How many drones will we produce? There’s an industrial explosion that I worry about.\nDwarkesh Patel\n04:22:41\nYou’ve got to be one of the few people in the world who is both concerned about alignment but also wants to ensure we let the robo-factories proceed once we get ASI to beat out China. That’s very interesting.\nLeopold Aschenbrenner\n04:22:54\nIt’s all part of the picture.\nDwarkesh Patel\n04:22:59\nSpeaking of ASIs and the robot factories and robo armies. One of the interesting things is the question of what you do with industrial-scale intelligence. Obviously, it’s not chatbots. It’s very hard to predict.\nThe history of oil is very interesting. In the 1860s, we figured out how to refine oil. A\ngeologist\ndiscovered it, and then Standard Oil got started. There was a huge boom, changing American politics. Legislators were bought out by oil interests. Presidents were elected based on divisions about oil and breaking them up.\nAll this happened before the car was invented. The light bulb was invented 50 years after oil refining was discovered. Most of Standard Oil’s history is before the car is invented. It was just kerosene lamps just used for lighting.\nLeopold Aschenbrenner\n04:24:06\nSo they thought oil would just no longer be relevant?\nDwarkesh Patel\n04:24:09\nYeah. There was a concern that Standard Oil would go bankrupt when the light bulb was invented. You realize there's an immense amount of compressed energy here. You're going to have billions of gallons of this stuff a year.\nIt’s hard to predict in advance what you can do with that.\nLater on, it turns out it’s used for transportation and cars.\nWith intelligence, maybe one answer is the intelligence explosion. But even after that, you have all these ASIs and enough compute, especially the compute they'll build to run—\nLeopold Aschenbrenner\n04:24:48\nHundreds of millions of GPUs will hum.\nDwarkesh Patel\n04:24:50\nWhat are we doing with that? It’s very hard to predict in advance. It’ll be very interesting to figure out what the Jupiter brains will be doing.\nSo there’s situational awareness of where things stand now, and we’ve gotten a good dose of that. A lot of what we’re talking about now couldn’t have been predicted many years back. Part of your worldview implies that things will accelerate because of AI.\nMany unpredictable factors will become evident over time, like how people, the political system, and foreign adversaries will react. Situational awareness isn’t just knowing where things stand now, but being in a position to react appropriately to new information and to change your worldview and recommendations accordingly.\nWhat is the appropriate way to think about situational awareness as a continuous process rather than as a one-time realization?\nLeopold Aschenbrenner\n04:25:58\nThis is great. There’s a sort of mental flexibility and willingness to change your mind that’s really important. This is how a lot of brains have been broken in the AGI debate. The doomers were prescient about AGI a decade ago, but they haven’t updated on the empirical realities of deep learning. Their proposals are naive and unworkable. It doesn’t really make sense.\nSome people come in with a predefined ideology, like\ne/accs\n. They like to shitpost about technology but they’re not actually thinking it through. You have stagnationists who think this stuff is just chatbots and not risky or those not considering the immense national security implications.\nThere’s a risk of calcification of worldview when you publicly articulate a position and cling to it despite evidence against it. So I want to give a big disclaimer. It’s valuable to paint a concrete and visceral picture. This is currently my best guess on how this decade will go. If it goes anything like this, it’ll be wild. Given the rapid pace of progress, we’re going to keep getting a lot more information and it’s important to keep your head on straight.\nI feel like the most important thing here is that. This relates to some of the stuff we talked about the world being surprisingly small. I used to think important things were being handled by capable people in government and AI labs.\nFrom personal experience, and seeing how Covid was managed, I realized that not everyone is on it. There’s not somebody else who’s on it and making sure this goes well. What really matters is that good people take these issues as seriously as they deserve, have situational awareness, are willing to change their minds, and face reality head-on. I’m counting on those good people.\nDwarkesh Patel\n04:48:26\nAll right, that’s a great place to close.\nLeopold Aschenbrenner\n04:28:29\nThanks so much Dwarkesh. Absolute joy.\nDwarkesh Patel\n04:28:31\nThis was excellent.\n(04:29:08) – Coda: Frederick the Great\nLeopold Aschenbrenner\n04:28:32\nThe funny thing is a lot of this German history stuff we’ve talked about isn’t actually what I learned in Germany. It’s stuff I learned after. I would go back to Germany over Christmas or whatever and suddenly understand the street names—\nGneisenau\n,\nScharnhorst\n, all these Prussian military reformers. And you finally understand\nSanssouci\n, and you realize it was for\nFrederick\n.\nFrederick the Great is a really interesting figure. He was kind of a gay lover of the arts. He hated speaking German, only wanted to speak French. He played the flute, and composed. He had all the great artists of his day over at Sanssouci. He had a really tough upbringing with a very stern Prussian military father. When Frederick was about 17, he had a male lover. His father imprisoned his son and hanged his lover in front of him.\nDespite this, Frederick the Great became one of the most successful Prussian conquerors. He got\nSilesia\n, won the Seven Years' War, and was an amazing military strategist. His brilliance lay in flanking the army, which was revolutionary at the time. They almost lost the Seven Years' War, but the Russian tsar changed and turned out to be a Prussia stan, which allowed Frederick to regroup and succeed. He’s a bizarre but interesting figure in German history.\n1\nThe previous linked item here was not the major security incident Leopold was referring to.",
  "metadata": {
    "title": "Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history",
    "description": "The trillion dollar cluster...",
    "keywords": null,
    "author": "Dwarkesh Patel",
    "og:type": "article",
    "og:title": "Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history",
    "og:description": "The trillion dollar cluster...",
    "og:image": "https://substackcdn.com/image/fetch/$s_!RTpY!,w_1200,h_600,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-video.s3.amazonaws.com%2Fvideo_upload%2Fpost%2F145136502%2F7f960bf1-958e-4ddf-a6e9-253ae7a53320%2Ftranscoded-00001.png",
    "og:url": "https://www.dwarkesh.com/p/leopold-aschenbrenner",
    "twitter:player": "https://www.dwarkesh.com/embed/podcast/leopold-aschenbrenner?autoplay=1",
    "twitter:player:width": "1",
    "twitter:player:height": "1",
    "twitter:text:player_width": "1",
    "twitter:text:player_height": "1",
    "twitter:title": "Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history",
    "twitter:description": "The trillion dollar cluster...",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!BY0E!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fdwarkesh.substack.com%2Fapi%2Fv1%2Fpost_preview%2F145136502%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2026-10-01T20:12:10.041Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=zdbVtZIn9IM",
      "src": "https://youtu.be/zdbVtZIn9IM",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/cfae7be5aa63/videos/repair-zdbVtZIn9IM.mp4",
      "bytes": 2339798726,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "16326.341000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=_Qq6dQwLh1s",
      "src": "https://youtu.be/_Qq6dQwLh1s?t=47",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/cfae7be5aa63/videos/repair-_Qq6dQwLh1s.mp4",
      "bytes": 17349998,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "82.141000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=qlJ2Zpyx31c",
      "src": "https://www.youtube.com/watch?v=qlJ2Zpyx31c",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/cfae7be5aa63/videos/repair-qlJ2Zpyx31c.mp4",
      "bytes": 625634897,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "4630.881000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=DfTU5LA_kw8",
      "src": "https://www.youtube.com/watch?v=DfTU5LA_kw8",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/cfae7be5aa63/videos/repair-DfTU5LA_kw8.mp4",
      "bytes": 1565335098,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "11277.321000",
      "reused": false
    }
  ],
  "links": [
    {
      "href": "https://situational-awareness.ai/leopold-aschenbrenner/",
      "label": "Leopold Aschenbrenner",
      "type": "a"
    },
    {
      "href": "https://situational-awareness.ai/",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://youtu.be/zdbVtZIn9IM",
      "label": "YouTube",
      "type": "a"
    },
    {
      "href": "https://podcasts.apple.com/us/podcast/leopold-aschenbrenner-china-us-super-intelligence-race/id1516093381?i=1000657821539",
      "label": "Apple Podcasts",
      "type": "a"
    },
    {
      "href": "https://open.spotify.com/episode/5NQFPblNw8ewxKolIDpiYN?si=6NaTHAugT2SxZrspW3lziw",
      "label": "Spotify",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/transcript",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://www.youtube-nocookie.com/embed/zdbVtZIn9IM?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://twitter.com/dwarkesh_sp",
      "label": "me on Twitter",
      "type": "a"
    },
    {
      "href": "https://x.com/leopoldasch",
      "label": "Leopold on Twitter",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/the-trillion-dollar-cluster-and-unhobbling",
      "label": "The trillion-dollar cluster and unhobbling",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/ai-the-return-of-history",
      "label": "AI 2028: The return of history",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/espionage-and-american-ai-superiority",
      "label": "Espionage & American AI superiority",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/geopolitical-implications-of-ai",
      "label": "Geopolitical implications of AI",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/state-led-vs-private-led-ai",
      "label": "State-led vs. private-led AI",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/becoming-valedictorian-of-columbia-at",
      "label": "Becoming Valedictorian of Columbia at 19",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/what-happened-at-openai",
      "label": "What happened at OpenAI",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/accelerating-ai-research-progress",
      "label": "Accelerating AI research progress",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/alignment",
      "label": "Alignment",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/on-germany-and-understanding-foreign-perspectives",
      "label": "On Germany, and understanding foreign perspectives",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/dwarkeshs-immigration-story-and-path-to-the-podcast",
      "label": "Dwarkesh’s immigration story and path to the podcast",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/launching-an-agi-hedge-fund",
      "label": "Launching an AGI hedge fund",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/lessons-from-wwii",
      "label": "Lessons from WWII",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/i/145136502/coda-frederick-the-great",
      "label": "Coda: Frederick the Great",
      "type": "a"
    },
    {
      "href": "https://bit.ly/4aVllm4",
      "label": "Teddy Kim",
      "type": "a"
    },
    {
      "href": "https://www.forourposterity.com/",
      "label": "Leopold Aschenbrenner",
      "type": "a"
    },
    {
      "href": "https://openai.com/superalignment/",
      "label": "OpenAI superalignment",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/p/patrick-collison",
      "label": "Patrick",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/John_Collison",
      "label": "John Collison",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Daniel_Gross_(entrepreneur)",
      "label": "Daniel Gross",
      "type": "a"
    },
    {
      "href": "https://nat.org/",
      "label": "Nat Friedman",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/p/sholto-douglas-trenton-bricken",
      "label": "Sholto and Trenton episode",
      "type": "a"
    },
    {
      "href": "https://openai.com/index/chatgpt/",
      "label": "ChatGPT",
      "type": "a"
    },
    {
      "href": "https://finance.yahoo.com/news/nvidia-stock-explodes-after-guidance-for-the-ages-what-wall-street-is-saying-120105455.html",
      "label": "first blockbuster earnings call",
      "type": "a"
    },
    {
      "href": "https://openai.com/index/gpt-4-research/",
      "label": "GPT-4",
      "type": "a"
    },
    {
      "href": "https://blogs.nvidia.com/blog/what-is-a-pretrained-ai-model/",
      "label": "pre-training",
      "type": "a"
    },
    {
      "href": "https://www.semianalysis.com/p/gpt-4-architecture-infrastructure",
      "label": "SemiAnalysis",
      "type": "a"
    },
    {
      "href": "https://www.nvidia.com/en-us/data-center/a100/",
      "label": "A100s",
      "type": "a"
    },
    {
      "href": "https://www.nvidia.com/en-us/data-center/h100/",
      "label": "H100",
      "type": "a"
    },
    {
      "href": "https://hazelcast.com/glossary/machine-learning-inference/",
      "label": "inference",
      "type": "a"
    },
    {
      "href": "https://www.eia.gov/energyexplained/electricity/electricity-in-the-us.php",
      "label": "barely grown for decades",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/p/mark-zuckerberg",
      "label": "Zuck on the podcast",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Three_Gorges_Dam",
      "label": "Three Gorges Dam",
      "type": "a"
    },
    {
      "href": "https://www.costar.com/article/1471314418/amazon-pays-650-million-for-nuclear-powered-data-center-in-pennsylvania",
      "label": "According to public reports",
      "type": "a"
    },
    {
      "href": "https://www.theinformation.com/articles/microsoft-and-openai-plot-100-billion-stargate-ai-supercomputer",
      "label": "The Information",
      "type": "a"
    },
    {
      "href": "https://www.cnbc.com/2023/12/08/amd-now-sees-a-400-billion-market-for-ai-chips-why-thats-good-news-for-nvidia.html",
      "label": "AMD forecasted a $400 billion AI accelerator market by 2027",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Overton_window",
      "label": "Overton window",
      "type": "a"
    },
    {
      "href": "https://copilot.microsoft.com/",
      "label": "Copilot",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Artificial_general_intelligence",
      "label": "AGI",
      "type": "a"
    },
    {
      "href": "https://paulgraham.com/schlep.html",
      "label": "schlep",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/p/tyler-cowen-3",
      "label": "Tyler Cowen",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/p/john-schulman",
      "label": "last episode",
      "type": "a"
    },
    {
      "href": "http://joschu.net/",
      "label": "John Schulman",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.03300",
      "label": "MMLU",
      "type": "a"
    },
    {
      "href": "https://openai.com/index/hello-gpt-4o/",
      "label": "GPT-4o",
      "type": "a"
    },
    {
      "href": "https://www.lesswrong.com/posts/75dnjiD8kv2khe9eQ/measuring-hardware-overhang",
      "label": "test time compute overhang",
      "type": "a"
    },
    {
      "href": "https://techpolicyinstitute.org/publications/artificial-intelligence/from-tokens-to-context-windows-simplifying-ai-jargon/",
      "label": "tokens",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2201.11903",
      "label": "chain-of-thought",
      "type": "a"
    },
    {
      "href": "https://campus.datacamp.com/courses/chatgpt-prompt-engineering-for-developers/advanced-prompt-engineering-strategies?ex=1",
      "label": "shotgun",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2104.03113",
      "label": "great paper",
      "type": "a"
    },
    {
      "href": "https://deepmind.google/technologies/alphago/",
      "label": "AlphaGo",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/High_availability",
      "label": "nines of reliability",
      "type": "a"
    },
    {
      "href": "https://thedecisionlab.com/reference-guide/philosophy/system-1-and-system-2-thinking",
      "label": "System 2",
      "type": "a"
    },
    {
      "href": "https://www.datacamp.com/tutorial/loss-function-in-machine-learning",
      "label": "loss function",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/G_factor_(psychometrics)",
      "label": "general intelligence",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Deep_learning",
      "label": "deep learning",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/GPT-2",
      "label": "GPT-2",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback",
      "label": "RLHF",
      "type": "a"
    },
    {
      "href": "https://openai.com/index/instruction-following/",
      "label": "original InstructGPT paper",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Robotics",
      "label": "robotics",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Unsupervised_learning",
      "label": "unsupervised learning",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Reinforcement_learning",
      "label": "RL",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Self-play",
      "label": "self-play",
      "type": "a"
    },
    {
      "href": "https://www.wsj.com/tech/ai/ai-training-data-synthetic-openai-anthropic-9230f8d8",
      "label": "data wall",
      "type": "a"
    },
    {
      "href": "https://knowyourmeme.com/memes/wordcel-shape-rotator-mathcel",
      "label": "wordcels",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Synthetic_data",
      "label": "synthetic data",
      "type": "a"
    },
    {
      "href": "https://www.hopsworks.ai/dictionary/in-context-learning-icl",
      "label": "in-context learning",
      "type": "a"
    },
    {
      "href": "https://ai.stackexchange.com/questions/5246/what-is-sample-efficiency-and-how-can-importance-sampling-be-used-to-achieve-it",
      "label": "sample efficient",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2403.05530",
      "label": "Gemini paper",
      "type": "a"
    },
    {
      "href": "https://deepai.org/machine-learning-glossary-and-terms/weight-artificial-neural-network",
      "label": "weights",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/p/dario-amodei",
      "label": "Dario",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/G-force",
      "label": "g-forces",
      "type": "a"
    },
    {
      "href": "https://learnprompting.org/blog/2024/2/4/gpt_wrappers",
      "label": "GPT wrapper",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Xi_Jinping",
      "label": "Xi Jinping",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Chinese_Communist_Party",
      "label": "CCP",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Superintelligence",
      "label": "superintelligence",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Technological_singularity",
      "label": "intelligence explosion",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Machine_learning",
      "label": "ML",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Gulf_War",
      "label": "first Gulf War",
      "type": "a"
    },
    {
      "href": "https://www.history.com/news/tanks-abrams-persian-gulf-war",
      "label": "better sensors on their tanks",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Ministry_of_State_Security_(China)",
      "label": "Ministry of State Security",
      "type": "a"
    },
    {
      "href": "https://www.eia.gov/todayinenergy/detail.php?id=53959",
      "label": "They added as much power in the last decade as an entire US electric grid",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Smartglasses",
      "label": "glasses",
      "type": "a"
    },
    {
      "href": "https://www.meta.com/smart-glasses/",
      "label": "Meta's Ray-Bans",
      "type": "a"
    },
    {
      "href": "https://llama.meta.com/",
      "label": "Llama",
      "type": "a"
    },
    {
      "href": "https://finance.yahoo.com/news/tsmc-complains-t-enough-skilled-100125351.html",
      "label": "their constraint is skilled workers",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Bill_de_Blasio",
      "label": "mayor of New York",
      "type": "a"
    },
    {
      "href": "https://ny.eater.com/2020/3/11/21175497/coronavirus-nyc-restaurants-safe-dine-out",
      "label": "\"go out to the shows,\"",
      "type": "a"
    },
    {
      "href": "https://www.nyc.gov/office-of-the-mayor/news/079-20/mayor-de-blasio-speaker-johnson-queens-chamber-commerce-encourage-new-yorkers-visit",
      "label": "\"this is just Asian racism.\"",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Manhattan_Project",
      "label": "Manhattan Project",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Military_production_during_World_War_II",
      "label": "50% of GDP went to war production",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Seven_Years%27_War",
      "label": "Seven Years' War",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Thirty_Years%27_War",
      "label": "Thirty Years' War",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/End_of_history",
      "label": "history",
      "type": "a"
    },
    {
      "href": "https://amzn.to/4e0jhvp",
      "label": "The Gulag Archipelago",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Aleksandr_Solzhenitsyn",
      "label": "Solzhenitsyn",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Mikhail_Gorbachev",
      "label": "Gorbachev",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Pluralism",
      "label": "pluralism",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/East_Germany",
      "label": "East",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/West_Germany",
      "label": "West",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Fall_of_the_Berlin_Wall",
      "label": "Wall fell",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Bombing_of_Dresden",
      "label": "firebombing of Dresden",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/East_German_uprising_of_1953",
      "label": "popular uprising in 1954",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Iron_Curtain",
      "label": "Iron Curtain",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Stasi",
      "label": "Stasi",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Removal_of_Sam_Altman_from_OpenAI",
      "label": "November event",
      "type": "a"
    },
    {
      "href": "https://letterboxd.com/film/oppenheimer-2023/",
      "label": "Oppenheimer",
      "type": "a"
    },
    {
      "href": "https://www.eia.gov/energyexplained/natural-gas/where-our-natural-gas-comes-from.php",
      "label": "US natural gas production",
      "type": "a"
    },
    {
      "href": "https://www.bloomberg.com/news/articles/2024-03-11/abu-dhabi-said-to-target-100-billion-aum-for-ai-investment-firm",
      "label": "free-flowing Middle Eastern money",
      "type": "a"
    },
    {
      "href": "https://news.microsoft.com/2024/04/15/microsoft-invests-1-5-billion-in-abu-dhabis-g42-to-accelerate-ai-development-and-global-expansion/",
      "label": "reports",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Weapon_of_mass_destruction",
      "label": "WMDs",
      "type": "a"
    },
    {
      "href": "https://globalprioritiesinstitute.org/leopold-aschenbrenner-existential-risk-and-growth/",
      "label": "peril of self-destruction",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Paul_Samuelson",
      "label": "Paul Samuelson",
      "type": "a"
    },
    {
      "href": "https://marginalrevolution.com/marginalrevolution/2010/01/soviet-growth-american-textbooks.html",
      "label": "predicting that the Soviet Union would outgrow",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Marcellus_Formation",
      "label": "Marcellus Shale",
      "type": "a"
    },
    {
      "href": "https://blogs.microsoft.com/blog/2020/01/16/microsoft-will-be-carbon-negative-by-2030/",
      "label": "Microsoft",
      "type": "a"
    },
    {
      "href": "https://sustainability.aboutamazon.com/carbon-methodology.pdf",
      "label": "Amazon",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Small_modular_reactor",
      "label": "SMR",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Federal_Energy_Regulatory_Commission",
      "label": "FERC",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/National_Environmental_Policy_Act",
      "label": "NEPA",
      "type": "a"
    },
    {
      "href": "https://amzn.to/4bA64Ih",
      "label": "Freedom's Forge",
      "type": "a"
    },
    {
      "href": "https://patrickcollison.com/fast",
      "label": "Fast",
      "type": "a"
    },
    {
      "href": "https://patrickcollison.com/progress",
      "label": "progress study stuff",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/William_S._Knudsen",
      "label": "William Knudsen",
      "type": "a"
    },
    {
      "href": "https://research.library.gsu.edu/c.php?g=115684&p=752252",
      "label": "labor organization and agitation",
      "type": "a"
    },
    {
      "href": "https://www.spiegel.de/international/germany/the-bad-news-bundeswehr-an-examination-of-the-truly-dire-state-of-germany-s-military-a-df92eaaf-e3f9-464d-99a3-ef0c27dcc797",
      "label": "the German military today",
      "type": "a"
    },
    {
      "href": "https://www.csis.org/analysis/contextualizing-national-security-concerns-over-chinas-domestically-produced-high-end-chip",
      "label": "able to make",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/7_nm_process",
      "label": "7-nanometer chips",
      "type": "a"
    },
    {
      "href": "https://www.wsj.com/tech/ai/sam-altman-seeks-trillions-of-dollars-to-reshape-business-of-chips-and-ai-89ab3db0",
      "label": "reported",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Sam_Altman",
      "label": "Sam Altman",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Atoms_for_Peace",
      "label": "Atoms for Peace",
      "type": "a"
    },
    {
      "href": "https://deepmind.google/",
      "label": "DeepMind",
      "type": "a"
    },
    {
      "href": "https://deepmind.google/discover/blog/introducing-the-frontier-safety-framework/",
      "label": "Frontier Safety Framework",
      "type": "a"
    },
    {
      "href": "https://www.nytimes.com/2024/03/06/us/politics/google-engineer-china-ai-theft.html",
      "label": "a guy who stole a bunch of really important AI code and went to China",
      "type": "a"
    },
    {
      "href": "https://scholar.google.com/citations?user=dOad5HoAAAAJ&hl=en",
      "label": "Alec Radford",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2203.15556",
      "label": "Chinchilla Scaling laws",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Mixture_of_experts",
      "label": "MoE",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.03762",
      "label": "transformers",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Heavy_water",
      "label": "heavy water",
      "type": "a"
    },
    {
      "href": "https://amzn.to/4dX8EJR",
      "label": "The Making of the Atomic Bomb",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Leo_Szilard",
      "label": "Leo Szilard",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/German_nuclear_program_during_World_War_II",
      "label": "moderator",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Enrico_Fermi",
      "label": "Enrico Fermi",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/George_B._Pegram",
      "label": "George Pegram",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Lavrentiy_Beria",
      "label": "Lavrentiy Beria",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/NKVD",
      "label": "NKVD",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Soviet_atomic_bomb_project",
      "label": "their version of the Manhattan Project",
      "type": "a"
    },
    {
      "href": "https://www.sciencefocus.com/science/who-really-invented-the-light-bulb",
      "label": "a bunch of people invented the light bulb around the same time",
      "type": "a"
    },
    {
      "href": "https://paperswithcode.com/sota/math-word-problem-solving-on-math",
      "label": "math benchmark",
      "type": "a"
    },
    {
      "href": "https://www.usaspending.gov/disaster/covid-19",
      "label": "Congress spent over 10% of GDP on COVID measures",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Pegasus_(spyware)",
      "label": "Israeli company had software that could just zero-click hack any iPhone",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Air_gap_(networking)",
      "label": "air-gapped",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Stuxnet",
      "label": "Stuxnet",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Zero-day_vulnerability",
      "label": "zero-days",
      "type": "a"
    },
    {
      "href": "https://amzn.to/451HTzO",
      "label": "Inside the Aquarium",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Viktor_Suvorov",
      "label": "a Soviet GRU defector",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/GRU_(Russian_Federation)",
      "label": "GRU",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/p/ilya-sutskever",
      "label": "Ilya",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Sensitive_compartmented_information_facility",
      "label": "SCIF",
      "type": "a"
    },
    {
      "href": "https://www.cnbc.com/2024/01/19/microsoft-executive-emails-hacked-by-russian-intelligence-group-company-says.html",
      "label": "executives' emails hacked by Russian hackers",
      "type": "a"
    },
    {
      "href": "https://www.reuters.com/technology/chinese-hackers-accessed-government-emails-microsoft-says-2023-07-12/",
      "label": "government emails they've hosted hacked by government actors",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Matrioshka_brain",
      "label": "Jupiter brain",
      "type": "a"
    },
    {
      "href": "https://www.cfr.org/timeline/us-russia-nuclear-arms-control",
      "label": "arms control on nuclear weapons",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Luftwaffe",
      "label": "Luftwaffe",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/p/richard-rhodes",
      "label": "Rhodes on the podcast",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Eliezer_Yudkowsky",
      "label": "Eliezer",
      "type": "a"
    },
    {
      "href": "https://nsarchive2.gwu.edu/NSAEBB/NSAEBB56/",
      "label": "encroach on West Berlin",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Spanish_conquest_of_the_Aztec_Empire",
      "label": "a few hundred Spaniards were able to topple the Aztec Empire",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Chatham_House_Rule",
      "label": "Chatham House rules",
      "type": "a"
    },
    {
      "href": "https://www.libertyjet.com/private_jets/IAI-1126",
      "label": "Galaxy",
      "type": "a"
    },
    {
      "href": "https://www.cnn.com/2023/01/09/politics/taiwan-invasion-war-game-intl-hnk-ml/index.html",
      "label": "maximum period of risk for Taiwan",
      "type": "a"
    },
    {
      "href": "https://www.yahoo.com/tech/plans-chinas-invasion-taiwan-could-235501383.html",
      "label": "deactivates",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Leslie_Groves",
      "label": "Leslie Groves",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Uranium_mining_in_Kazakhstan",
      "label": "huge deposits in the Soviet Union",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Boeing_B-29_Superfortress",
      "label": "B-29 bomber",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Boeing_B-47_Stratojet",
      "label": "B-47",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Boeing_B-52_Stratofortress",
      "label": "B-52",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Boeing_707",
      "label": "707",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Monopoly_on_violence",
      "label": "monopoly on violence",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Industrial_Revolution",
      "label": "Industrial Revolution",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Open-source_artificial_intelligence",
      "label": "open source",
      "type": "a"
    },
    {
      "href": "https://mistral.ai/",
      "label": "Mistral",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Moore%27s_law",
      "label": "Moore’s Law",
      "type": "a"
    },
    {
      "href": "https://conversationswithtyler.com/episodes/peter-thiel-political-theology/",
      "label": "muddling through",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Landfrieden",
      "label": "Landfrieden",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Holy_Roman_Empire",
      "label": "Holy Roman Empire",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/p/demis-hassabis",
      "label": "Demis",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Edmund_Burke",
      "label": "Burkean",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Quebec_Agreement",
      "label": "Quebec Agreement",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Treaty_on_the_Non-Proliferation_of_Nuclear_Weapons",
      "label": "NPT",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Superior_orders",
      "label": "not allowed to follow unlawful",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/United_States_Armed_Forces_oath_of_enlistment",
      "label": "unconstitutional",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Ethnic_bioweapon",
      "label": "bioweapons targeted to kill everyone but the Han Chinese",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Jake_Sullivan",
      "label": "Jake Sullivan",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Skunk_Works",
      "label": "Skunk Works",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Satya_Nadella",
      "label": "Satya",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Defense_Production_Act_of_1950",
      "label": "DPA",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/John_von_Neumann",
      "label": "John von Neumanns",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/YOLO_(aphorism)",
      "label": "YOLO",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Operation_Warp_Speed",
      "label": "Warp Speed",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Thermonuclear_weapon",
      "label": "H-bomb",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Intercontinental_ballistic_missile",
      "label": "ICBM",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/National_Institute_of_Standards_and_Technology",
      "label": "NIST",
      "type": "a"
    },
    {
      "href": "https://www.anthropic.com/news/anthropics-responsible-scaling-policy",
      "label": "responsible scaling policies",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Motte-and-bailey_castle",
      "label": "motte-and-bailey",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Tall_poppy_syndrome",
      "label": "tall poppy syndrome",
      "type": "a"
    },
    {
      "href": "https://undergrad.admissions.columbia.edu/academics/college/core",
      "label": "core curriculum",
      "type": "a"
    },
    {
      "href": "https://polisci.columbia.edu/content/richard-k-betts",
      "label": "Richard Betts",
      "type": "a"
    },
    {
      "href": "https://polisci.columbia.edu/content/war-peace-and-strategy",
      "label": "class on war, peace, and strategy",
      "type": "a"
    },
    {
      "href": "https://history.columbia.edu/person/adam-tooze/",
      "label": "Adam Tooze",
      "type": "a"
    },
    {
      "href": "https://amzn.to/4dW9BSR",
      "label": "very riveting books",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Log%E2%80%93log_plot",
      "label": "log-log plots",
      "type": "a"
    },
    {
      "href": "https://web.stanford.edu/~chadj/",
      "label": "Chad Jones",
      "type": "a"
    },
    {
      "href": "https://youtu.be/_Qq6dQwLh1s?t=47",
      "label": "Bruce Banner’s quote",
      "type": "a"
    },
    {
      "href": "https://letterboxd.com/film/the-avengers-2012/",
      "label": "The Avengers",
      "type": "a"
    },
    {
      "href": "https://www.forbes.com/sites/johnhyatt/2022/11/14/sam-bankman-fried-promised-millions-to-nonprofits-research-groups-thats-not-going-too-well-now/?sh=3a4af9f05ee8",
      "label": "Future Fund",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Sam_Bankman-Fried",
      "label": "Sam Bankman-Fried",
      "type": "a"
    },
    {
      "href": "https://apnews.com/article/sam-bankman-fried-ftx-fraud-timeline-be13e3fc0e074e2edd50ba59d1f8960e",
      "label": "revealed that Sam was a giant fraud",
      "type": "a"
    },
    {
      "href": "https://jan.leike.name/",
      "label": "Jan",
      "type": "a"
    },
    {
      "href": "https://fortune.com/2024/05/15/openai-sam-altman-ilya-sutskever-jan-leike/",
      "label": "That’s been the news of last week",
      "type": "a"
    },
    {
      "href": "https://fortune.com/2024/05/21/openai-superalignment-20-compute-commitment-never-fulfilled-sutskever-leike-altman-brockman-murati/",
      "label": "20% compute commitment",
      "type": "a"
    },
    {
      "href": "https://www.theinformation.com/articles/openai-researchers-including-ally-of-sutskever-fired-for-alleged-leaking",
      "label": "The Information",
      "type": "a"
    },
    {
      "href": "https://openai.com/index/introducing-superalignment/",
      "label": "superalignment announcement",
      "type": "a"
    },
    {
      "href": "https://www.nytimes.com/interactive/2023/11/20/technology/letter-to-the-open-ai-board.html",
      "label": "employee letter",
      "type": "a"
    },
    {
      "href": "https://karpathy.ai/",
      "label": "Andrej",
      "type": "a"
    },
    {
      "href": "https://www.vox.com/future-perfect/351132/openai-vested-equity-nda-sam-altman-documents-employees",
      "label": "an NDA",
      "type": "a"
    },
    {
      "href": "https://www.nytimes.com/2023/11/17/technology/openai-sam-altman-ousted.html",
      "label": "fired",
      "type": "a"
    },
    {
      "href": "https://www.nytimes.com/2023/11/22/technology/openai-sam-altman-returns.html",
      "label": "rehired",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Bell_Labs",
      "label": "Bell Labs",
      "type": "a"
    },
    {
      "href": "https://www.wsj.com/tech/ai/the-fight-for-ai-talent-pay-million-dollar-packages-and-buy-whole-teams-c370de2b",
      "label": "AI researcher salaries",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Modus_ponens",
      "label": "modus ponens",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Modus_tollens",
      "label": "modus tollens",
      "type": "a"
    },
    {
      "href": "https://www.factorio.com/",
      "label": "Factorio",
      "type": "a"
    },
    {
      "href": "https://www.jstor.org/stable/2120827",
      "label": "famous study",
      "type": "a"
    },
    {
      "href": "https://klu.ai/glossary/layer-normalization",
      "label": "normalization layer",
      "type": "a"
    },
    {
      "href": "https://www.dwarkeshpatel.com/p/will-scaling-work",
      "label": "scaling laws",
      "type": "a"
    },
    {
      "href": "https://commoncrawl.org/",
      "label": "Common Crawl",
      "type": "a"
    },
    {
      "href": "https://llama.meta.com/llama3/",
      "label": "Llama-3",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2305.16264",
      "label": "an academic paper by Boaz Barak that does scaling laws for this",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Monorepo",
      "label": "monorepo",
      "type": "a"
    },
    {
      "href": "https://chat.lmsys.org/?leaderboard",
      "label": "LMSys",
      "type": "a"
    },
    {
      "href": "https://www.anthropic.com/news/claude-3-family",
      "label": "Claude 3 Opus",
      "type": "a"
    },
    {
      "href": "https://www.anthropic.com/news/claude-3-haiku",
      "label": "Claude 3 Haiku",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Wright_brothers",
      "label": "Wright brothers",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Starlink",
      "label": "Starlink",
      "type": "a"
    },
    {
      "href": "https://www.nvidia.com/gtc/",
      "label": "Nvidia GTC",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Digital_twin",
      "label": "digital twins",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Production_function",
      "label": "production function",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Four_Asian_Tigers",
      "label": "Asian Tigers",
      "type": "a"
    },
    {
      "href": "https://platform.openai.com/docs/models",
      "label": "GPT-4o is like $15",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Apollo_program",
      "label": "Apollo program",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2001.08361",
      "label": "original Kaplan scaling laws paper",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Steven_Pinker",
      "label": "Steven Pinker",
      "type": "a"
    },
    {
      "href": "https://amzn.to/3VmtBGH",
      "label": "The Better Angels of Our Nature",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Cultural_Revolution",
      "label": "Cultural Revolution",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Red_team",
      "label": "red teaming",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Federal_Reserve",
      "label": "Federal Reserve",
      "type": "a"
    },
    {
      "href": "https://www.supremecourt.gov/oral_arguments/oral_arguments.aspx",
      "label": "SCOTUS oral arguments",
      "type": "a"
    },
    {
      "href": "https://www.neelnanda.io/mechanistic-interpretability/glossary",
      "label": "mechanistic interpretability",
      "type": "a"
    },
    {
      "href": "https://x.com/janleike/status/1791498174659715494",
      "label": "Tweet thread",
      "type": "a"
    },
    {
      "href": "https://www.forourposterity.com/europes-political-stupor/",
      "label": "Europe’s Political Stupor",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Treaty_of_Versailles",
      "label": "Peace of Versailles",
      "type": "a"
    },
    {
      "href": "https://amzn.to/4bFMWIQ",
      "label": "China's World View",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Marjorie_Taylor_Greene",
      "label": "Marjorie Taylor Greene",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Henry_Kissinger",
      "label": "Henry Kissinger",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Politburo_Standing_Committee_of_the_Chinese_Communist_Party",
      "label": "Politburo",
      "type": "a"
    },
    {
      "href": "https://www.realcleardefense.com/articles/2023/01/14/why_does_germany_keep_neglecting_its_defense_875723.html",
      "label": "defense spending",
      "type": "a"
    },
    {
      "href": "https://www.fpri.org/article/2023/04/the-beliefs-of-the-blob/",
      "label": "Blob",
      "type": "a"
    },
    {
      "href": "https://www.strategictranslation.org/",
      "label": "great work",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=qlJ2Zpyx31c",
      "label": "Alexey interviewed me",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Eastern_Bloc",
      "label": "Eastern Bloc",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Tucker_Carlson",
      "label": "Tucker Carlson",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=DfTU5LA_kw8",
      "label": "Joe Rogan episode",
      "type": "a"
    },
    {
      "href": "https://www.forbes.com/sites/stuartanderson/2024/04/14/more-than-1-million-indians-waiting-for-high-skilled-immigrant-visas/?sh=6bbd529d24ce",
      "label": "backlog for Indians",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/H-1B_visa",
      "label": "H-1B",
      "type": "a"
    },
    {
      "href": "https://www.paulgraham.com/articles.html",
      "label": "Paul Graham essays",
      "type": "a"
    },
    {
      "href": "https://x.com/JeffBezos/status/1514689258535145474",
      "label": "say something nice about me on Twitter.",
      "type": "a"
    },
    {
      "href": "https://www.theatlantic.com/magazine/archive/2021/01/the-most-american-religion/617263/",
      "label": "an article",
      "type": "a"
    },
    {
      "href": "https://plato.stanford.edu/entries/ethics-deontological/",
      "label": "deontologist",
      "type": "a"
    },
    {
      "href": "https://plato.stanford.edu/entries/consequentialism/",
      "label": "consequentialist",
      "type": "a"
    },
    {
      "href": "https://amzn.to/4aHqLk7",
      "label": "book",
      "type": "a"
    },
    {
      "href": "https://novelinvestor.com/stan-druckenmillers-worst-mistake-ever/",
      "label": "sell at the wrong time and buy at the wrong time",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Dot-com_bubble",
      "label": "dot-com boom",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/TSMC",
      "label": "TSMC",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Beta_(finance)",
      "label": "beta",
      "type": "a"
    },
    {
      "href": "https://web.stanford.edu/~chadj/Consumption2009-11-25.pdf",
      "label": "Euler equation",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Georgism",
      "label": "Georgist",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Thomas_Piketty",
      "label": "Piketty",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Efficient-market_hypothesis",
      "label": "EMH",
      "type": "a"
    },
    {
      "href": "https://amzn.to/3VmPrd9",
      "label": "Victor Davis Hanson’s summary of everything",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Blitzkrieg",
      "label": "blitzkrieg",
      "type": "a"
    },
    {
      "href": "https://www.wsj.com/world/china/chinas-shipyards-are-ready-for-a-protracted-war-americas-arent-d6f004dd",
      "label": "China builds like 200 times more ships than we do right now",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Samuel_Kier",
      "label": "geologist",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Effective_accelerationism",
      "label": "e/accs",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/August_Neidhardt_von_Gneisenau",
      "label": "Gneisenau",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Gerhard_von_Scharnhorst",
      "label": "Scharnhorst",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Sanssouci",
      "label": "Sanssouci",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Frederick_the_Great",
      "label": "Frederick",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Silesia",
      "label": "Silesia",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:03Z",
  "source_url": "https://www.dwarkeshpatel.com/p/leopold-aschenbrenner",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.03300",
      "bytes": 2074939,
      "reused": true,
      "local_path": "papers/repair/eabe88a3d618ccd5d1e4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2201.11903",
      "bytes": 891773,
      "reused": true,
      "local_path": "papers/repair/52cc4c4f6cf368123b97.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2104.03113",
      "bytes": 465709,
      "reused": false,
      "local_path": "papers/repair/deb2d1fbac664dbdbccd.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2403.05530",
      "bytes": 7184542,
      "reused": false,
      "local_path": "papers/repair/308e23cbb629727bb36a.pdf"
    },
    {
      "status": "ok",
      "url": "https://sustainability.aboutamazon.com/carbon-methodology.pdf",
      "bytes": 239362,
      "reused": false,
      "local_path": "papers/repair/569ba5a7ddf9b9aa020d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2203.15556",
      "bytes": 6004349,
      "reused": true,
      "local_path": "papers/repair/70766cef482201671d90.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.03762",
      "bytes": 2215244,
      "reused": true,
      "local_path": "papers/repair/c3ab01e47e614fe3cf33.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.16264",
      "bytes": 2160774,
      "reused": true,
      "local_path": "papers/repair/340921939a994c4d6f60.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2001.08361",
      "bytes": 2493004,
      "reused": true,
      "local_path": "papers/repair/9d7d7113c60c89afd004.pdf"
    },
    {
      "status": "ok",
      "url": "https://web.stanford.edu/~chadj/Consumption2009-11-25.pdf",
      "bytes": 225065,
      "reused": false,
      "local_path": "papers/repair/58dc6ca39de756ba2932.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 252660,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/cfae7be5aa63/pages/source.html",
  "local_html": "attachments/cfae7be5aa63/pages/article.html",
  "body_base_path": "attachments/cfae7be5aa63/pages/",
  "body_markdown_path": "attachments/cfae7be5aa63/pages/article.md"
}