0:00 The scary open secret in the AI industry right now is that it's possible that we'll end up essentially creating a new species that ends up ruling the world with a 70% chance that this goes horribly wrong like human extinction. That's one possibility. There's many more. >> It's quite chilling what you're saying. >> Yeah, it's uh gets me down sometimes. I basically told my wife like let's not have any more kids. It's too uncertain. I don't think they'll ever join the workforce. Everybody should be afraid that their jobs are going to be lost. And I know this because I went to OpenAI in 2022. What I did there was forecasting what the what the next couple years might look like. And unfortunately, most of the world is kind of asleep at the wheel and doesn't really realize what's going on with AI. So, I resigned. >> I read it somewhere that you lost $2 million for not signing an anti-disparagement clause, meaning you couldn't criticize the company. >> Yes, for reasons I'm happy to get into.
0:45 But, the main thing I've learned is when I go talk to people at Anthropic and OpenAI about forecasting, they're like, "It's not going to take that long. You need to shorten them again. Get them back to 2027 or 2028." Because these powerful CEOs, Dario or Sam or Elon, are racing each other to be in control of the most powerful AIs. And are literally afraid that if the other guy gets there first, he might become dictator. I mean, Anthropic is on track to be the entire economy by 2030. But, none of these people should be trusted with that much power. So, this is the most important thing happening in our lifetimes, probably in all of history, in fact. And it's very important that it go well. So, I think that there's a lot we can do to like steer things in a better direction. There's loads of benefits that we could get from AI if we do it right. And if we do solve the problems, then things could be absolutely amazing for everyone. >> Well, this report here in 2021, it was remarkably [music] accurate. And then just published this one. >> Yeah. So, this is our new scenarios.
1:32 >> So, let's go through these slowly and one at a time. >> I would be incredibly happy if all my predictions turn out to be wrong. >> This is super interesting to me. My team gave me this report to show me how many of you that watch this show subscribe. And some of you have told us, according to this, that you are unsubscribed from the channel randomly. So, favor to ask all of you, please could you check right now if you've hit the subscribe button. If you are regular viewer of this show and you like what we we here. We're approaching quite a significant landmark on this show in terms of the subscriber number. So, if there was one simple free thing that you could do to help us, my team, everyone here, to keep this show free, to keep it improving year over year and week over week, it is just to hit that subscribe button and to double-check if you've hit it. Only thing I'll ever ask of you. Do we have a deal? If you do it, I'll tell you what I'll do. I'll make sure every single week, every single month,
2:18 we fight harder and harder and harder and harder to bring you the guests and conversations that you want to hear. I stay true to that promise since the very beginning of the Diary of a CEO, and I will not let you down. Please help us. Really appreciate it. Let's get on with the show. >> [music] >> Daniel Kokotajlo. At the very heart of what you do, um what is your mission? And why? >> So, what would you do if you thought that superintelligence was coming in a few years? >> I guess it depends what the consequences were. >> Well, let's talk about it. So, superintelligence, AIs that are better than the best humans at everything, while also being faster and cheaper, also able to operate robots that can do everything in the physical world that humans can do, but better, faster, and cheaper. If that
3:04 really is coming in a few years, then we need to prepare, and we need to think about how to make it go well instead of poorly. So, that's sort of my answer is like, I'm doing that to the best of my ability. >> So, you believe it's coming in a few years? >> Yes. >> How could you be so sure? >> I spend a lot of time trying to forecast this sort of thing. My sort of median estimate, a 50% chance, is currently in 2029. Maybe it'll slip to 2028. It's possible that it'll take significantly longer, like maybe 10 years or something like that. But, uh you know, for reasons I'm happy to get into, seems to me like it's probably happening by the end of the decade. Which less important is the the sense of how close we are. What's more important is the pace of the trends. Anthropic this time last year was making something
3:50 like a billion dollars a year. And they're making something like 60 billion dollars a year. So that's 60x growth in 1 year, which is extremely impressive even for very small startups, but for a company of their size, it might be the fastest growth in history. Um we expect that rate of growth to slow down, but even if it slows down quite a lot, they're still on track to be, you know, the entire economy by 2030 or so. >> Why should the average person care? >> The high-level thing is absolutely everything is going to change for the whole world, and including therefore for them and their families. Um could change for the better, could change for the worse, depending on the details of how it's done. So for example, everyone could die,
4:35 you know? Um this is the classic loss of control scenario, or one version of it. If we do build these super intelligences, and we use them to automate all the jobs, and we put them in the military, and we, you know, have them giving advice to politicians, and so forth, they will eventually have accumulated enough real-world power that they don't need humans anymore. And they're smarter than us, they're more strategic, etc. At that point, we sort of have to hope that they are virtuous, that they have, you know, the goals that we wanted them to have, the values that we wanted them to have, etc. And the sort of scary open secret in the AI industry right now is that right now that is kind of just a hope. It's not something that we can be at all confident in, and in fact, there's lots of evidence and arguments that it we're not on track to achieve that.
5:20 So there's lots of reason Like current AIs, for example, will often lie uh to people, or they will like you tell them to do something and they go do something else, and then pretend that they did it, right? So it's an inherently difficult problem to make something that's super intelligent and also has the values and virtues that you want it to have, and it doesn't seem like we're on track to solve that problem. Also, it seems like the sort of problem that you could think you solved when you haven't actually solved it, right? Uh that's a big reason why this is scary. So, for all those reasons, it's possible that we'll end up essentially creating a new species that ends up ruling the world instead of us. And then maybe we go the way of other extinct species in the past that were outcompeted by humans. That's one possibility. There's many more. Even if you're not worried about that and you think that the AIs will be totally controlled,
6:06 there's the question of who controls the AIs, right? When there's a couple corporations that have made these superintelligences and are using them to automate all the jobs, well, that's a lot of power, you know? That's a lot of money. It's a lot of political power. They'll have the best strategists, the best advisers, you know, they'll think faster. Militarily, uh, the countries that has these AIs will be able to absolutely wipe the floor with all the other countries. The AIs themselves, it's it's kind of a single point of failure like central uh, control system where, you know, the CEO of Anthropic, Dario, he coined this phrase, "The country of geniuses in the giant data center." That was his phrase to describe what they're trying to build, you know? I think that's a little bit misleading. I think it would be more accurate to describe it as army of geniuses in the
6:52 data center because it's not like it's a bunch of diverse different AIs, you know, living in their different parts of the data center. They're all copies of the same big model and they're owned by the company. And so, they all follow the orders given by the company, right? People should be asking questions of like, who controls this army or these armies and what are they going to be doing with them? I think that we could very easily end up in a sort of uh, a situation where some tiny group of people are essentially oligarchs or dictators. And ironically, both of these risks, the loss of control and the constitution of power, are things that people in the industry have been thinking about for decades. Um, even before the AI industry existed, you know, people thinking about AI were talking and writing about these things.
7:38 And then part of the founding narrative, the founding myth of DeepMind and OpenAI and Anthropic is these problems are real. So, we need to get there first so that we can handle it responsibly. Those are I think the big two reasons, but then I can go on. There's lots more reasons as well. So, one thing is you know, World War III, geopolitical conflict. Um if AI does in fact get incredibly powerful, that's going to change the balance of power between nations. That's going to disrupt a lot of things. That puts us at increased risk of crisis more generally, right? Another one, what about those jobs? You you're going to lose your taxi job, but not just the taxi driver, everybody pretty much. Um there might be a few exceptions like people whose jobs for legal reasons are only allowed to be done by humans, but for the most part, everybody should be
8:24 afraid that their jobs are going to be lost even if we manage to avoid all the other problems, right? >> This narrative has started to emerge and I've had several interviews on the show where I've interviewed people who are very very scared and anxious about AI. And these are people that have worked in the industry for sometimes decades. >> Yeah. >> Um the counter narrative coming over the hill is that this is doomerism. That these people are for whatever reason just trying to scare people and that they don't really understand what they're talking about. How do you respond to that sort of counter narrative? And you must have seen this emerging yourself, especially from people who stand to benefit, dare I say? >> Yeah, exactly. This counter narrative is fairly recent and it's been pushed by the people who stand to benefit um from it and it's not true. Like these these concerns have been around for decades since before the AI industry existed. They're actually pretty reasonable concerns. Like if you take the companies
9:11 at their word and imagine that they are in fact going to build superintelligence, well, it raises a lot of questions. Like who's going to control it? Will anybody control it? What about the jobs? You know, like th- these are just kind of obvious implications to be thinking about and worrying about. >> Who are you and what's your story? >> My name is Daniel Kokotajlo. Um I currently run the AI Futures Project, which is a small nonprofit that mostly focuses on forecasting the future of AI. Before that, I worked at OpenAI. >> AI forecasting? >> Yeah, so think about how like you know, industry analysts who work for hedge funds and stuff will make these forecasts of like here is, you know, how many cars Tesla will be selling 5 years from now or like here's what the price of electricity
9:56 will be in 2 years, right? That's forecasting. I was doing that but specifically focused on AI. The reason I was doing it is because it's incredibly important to to see where this is all headed. >> Why did you go to OpenAI? What did you do there? What did you observe while you were there and how did it change your perspective on the future of AI but also I guess OpenAI as a company and for anybody that doesn't know OpenAI are the company that produced ChatGPT. >> Yeah, so I went to OpenAI in 2022. Uh a large part of what I did there was more forecasting. AI 2027 is a scenario that you may have heard of. I did like smaller you know, lower effort versions of them internally for just internal circulation of like here's some guesses as to what the next couple years might look like. I also worked on evaluations for dangerous capabilities. So
10:42 you know, trying to measure the AI's cyber abilities or persuasion abilities or situational awareness and I also briefly was on a uh a capabilities team doing reinforcement learning to create agents. AI is in fact getting uh a lot better and I can say more about why, you know, scaling laws, um deep neural nets bigger, trained on more data, become more efficient, more competent at those things. I also became a bit more disillusioned with the AI industry. So OpenAI, Anthropic, and DeepMind all had these sort of founding narratives of like yes, these risks are real but we've thought about them and we're going to try to handle them responsibly and that's why it's important for us to keep doing what we're doing and I increasingly came to think that these
11:28 were rationalizations to justify what they were rather than sort of like deeply guiding their actual behavior and that when push comes to shove they'll follow their incentives rather than do what's actually good. >> So you're inside OpenAI at the time and you start to believe that they're following commercial incentives versus the I guess social or societal incentives that they founded themselves on. >> Sort of. I mean what I wouldn't actually describe it as commercial incentives. I think I would describe it as um power-seeking incentives. So like [clears throat] it's true that the companies care a lot about making a lot of money but especially at the very top of these companies like the leaders they understand that this is about more than just money. You know?
12:14 There are these emails that came up in you know the the lawsuit between Musk and and um OpenAI. A bunch of emails were surfaced in that lawsuit which you can go read and in some of them the founders of OpenAI were talking back in like 2017 about how the reason why we made OpenAI was because we were worried that Demis Hassabis at Google was going to become dictator with AGI. Even back then they were this obviously about more than just money. Like these these powerful CEOs are literally afraid that if the other guy gets there first he might become dictator and they don't trust each other and so that's why they are racing as hard as they can so that they're the ones who get there first so to speak. >> Have you met Sam Altman? >> Yeah. >> And did did that shape your opinion of
13:00 his incentives or what why he's doing what he's doing? Cuz there's a lot you know speculated about what his incentives are. I mean his most recent narrative says for the good of humanity. I think that's what >> Yeah, I mean I think the main thing I've learned is don't pay attention to the narratives. You know like uh what they say to one person is just different from what they can say to some other person at the same time and what they say in public is a third thing entirely. I think you should judge people by their actions not by their words. >> And why are you no longer at OpenAI? >> Largely the reason that I mentioned. So, I became gradually disillusioned with how the company was going to behave. For example, when I first joined in 2022, at least the people I talked to, my colleagues at the company, there was this general sense of like, of course we wouldn't actually just build super intelligence as soon as possible. Once we started
13:45 getting really close, like once we started getting to AIs that could maybe automate the AI research process, we would pause and figure out how to make it safe. That's cuz we're the good guys and that's obviously the safe thing you should do rather than just going full speed ahead. But, we're worried about other people who might not pause, you know, our competitors, Google, for example. And so, that's why we need to be in the lead so that we have that room to do the safe stuff, right? That was sort of like a thing that seemed like maybe like the median position or something among the colleagues I talked to when I was there when I started, including people like Sam, you know, including the leadership. And then by the time I left, I was like, "Oh man, they're really not going to do that, are they?" Like >> [laughter] >> Like they they've sort of you know, partly because this has become more politicized and they've become bigger and been under more scrutiny,
14:32 people have started asking like, "Why are you doing this in the first place if it's so risky?" And so, they've pivoted their narrative to being more like, "Actually, it's not that risky, you know?" Um and so, yeah, I mean, it seems like they're just going to keep going roughly as fast as they can and hope that they can figure it out on the way. >> How did your time at OpenAI come to an end? >> Uh I resigned in 2024. I had a nice goodbye party. >> What were the reasons you gave for quitting OpenAI? >> I thought that we were rationalizing too much and that we needed to think more about what would actually be good for the world. Um I wanted more freedom to publish. So, at OpenAI, as it became a bigger company, it became more of a normal tech company with incentives and, you know, a PR department and things like that. And so, it started becoming more difficult to um
15:19 to publish the sort of research that I was doing. For example, those scenarios that I mentioned, couldn't uh couldn't publish those, right? They're just for internal use. I thought that that was a shame because right now most of the world is kind of asleep at the wheel and doesn't really realize what's going on with AI and doesn't really realize what's coming in the pipeline a couple years from now. And the companies aren't really incentivized to tell people that much about it. I mean, they say some vague stuff in a sort of hypey way, but um you know, well, they didn't want me to publish the scenario, for example, laying out like here's how things might actually look. >> I'm just kind of super curious as to what it's like being in a company like that when they you know, chat GPT-3 is released. You were there at that time, right? >> Mhm. >> Um which was a moment where I think the whole world stood up and realized that
16:04 this technology was powerful. >> Yeah. >> Um and the conversation really began from a society level. Um company starts growing super quickly. >> Yeah. >> Quicker than I think anybody could ever have imagined. And what what was it like inside there? What did you see change um over over that period of time? >> I remember one all-hands meeting where Ilya said something like >> Ilya being >> Ilya Sutskever, who was um head of research at that time. He said something like, "Okay, now the world is starting to pay attention. Each of you is going to be the most popular person at every party uh for the next year. Don't let it get to your head. Focus on the mission. Got to build AGI." >> [laughter] >> The company grew a lot. It already wasn't really feeling like a nonprofit when I joined, but it definitely didn't feel like a nonprofit by the time I left. Um lots of new people came in.
16:52 Ironically, the like amount of conversation about superintelligence and the implications of superintelligence arguably you sort of went down over time due to this growth, right? So, because the company would like double and then double again and then double again, all these new people were coming in from other parts of the tech industry who hadn't really been thinking about these things and were attracted by the high salaries. >> You lost $2 million for not signing an anti-disparagement clause, which would mean you could speak you couldn't criticize the company. >> Ah, yes. Well, so um I got to keep the money. >> Oh, you got to keep the money? >> what happened was after I had left, said my goodbyes, etc. Um I got the the exit paperwork and it included this clause that said you
17:38 basically have to agree not to criticize the company again. Um and also a clause saying you can't tell anyone about this. And so I thought that was kind of rich coming from a nonprofit that's supposed to be, you know, for the benefit of all humanity. So, I didn't sign it. And if you don't sign, you don't get to keep your equity. So, your compensation, you know, what what they pay you is a bunch of money and then also a bunch of stock, basically. But then they had this stuff in the contract that they get to yank back your your stock if you don't sign this thing. Um and my wife and I, you know, were uh upset about this. We talked about it for like a month or two, consulted some lawyers, um and then ultimately decided to just refuse to sign. >> Which would mean you lost you would have
18:24 lost $2 million. >> That's right. Which was like 80% of our net worth at the time. Um fortunately, uh it didn't go the way we expected. It blew up basically on the internet. Like when people heard that that we had done this and that we had said no, it became like this huge scandal. Employees at the company started like asking questions in Slack and like asking leadership like, wait, what? Like why are you going to take away our equity? What is this? You know, cuz a lot of people hadn't really noticed this before. It had been whispered about, but it hadn't been sort of like a thing that most employees knew about. Um and so they backtracked and they said, "Never mind, never mind. We'll change the paperwork. You can keep the equity. It's fine." >> And so management came out and said he was embarrassed that he didn't realize this was going >> Yeah, he had no idea, apparently. >> You don't believe him? >> No.
19:10 I think he probably knew. And if he didn't know, then people close to him probably did, such as his head lawyer. >> Why did you decide not to take the $2 million? I mean, most people would have, I think. >> It's true, most people would have, and most people did. And you know, money is nice, but like it's not the only thing, you know? Sometimes it's good to take a stand on principle. I I keep mentioning superintelligence. Perhaps I should say more about like the the sequence of events that the companies are planning to do. So, right now, they're focusing on automating coding. They're taking their AIs, they're making them bigger, they're training them for longer, and they're especially focusing the training on getting them to be good at autonomously writing and editing code. Because
19:55 uh that will help the companies go faster, right? If they can automate the code, then they can do their own work better and faster, and accelerate progress. The next step, which they've already begun, is to look at the rest of the research process as well. Coming up with ideas, um analyzing experiments, communicating those results. All the other parts of of the research process, they're trying to figure out how to train AIs to be good at those as well. So that they can have AIs do the entire thing autonomously. >> When you say do the entire thing, what you mean [clears throat] do the entire thing? >> So like Anthropic and OpenAI in particular are trying to automate themselves. Like they're trying to make it the case that um they don't really need human employees anymore. Uh they just have a giant army of AIs that's churning away,
20:41 doing all this autonomous research to make better AIs, to train the new AIs, put them in charge, so they can make even better AIs and so forth. And of course, not just not all just happening internally, but also like interfacing with the world, right? Like going out and talking to people, collecting the data, setting up the training environments, doing the business deals, and so forth. Like they're they're trying to automate all of that. The reason why they're doing this is because they're trying to get to a position where they have AIs that are superhuman at everything, superintelligence, and they're trying to get there before their competitors do. Needless to say, this is incredibly dangerous, I would say, you know. And in addition to being dangerous, it's a power grab, right? Like if they actually succeed at this, then they'll be sitting on top of this army of
21:28 superhuman AIs that will give them immense leverage over all sorts of other actors in the economy in so far as they can work out something with the presidents and, you know, integrate it into the military or whatever, then that would give the US immense hard power over all of the countries, right? Obviously, nobody knows exactly when this is happening. But a very disquieting thing has happened over the last year to me, which is that when we published AI 2027, people were generally of the opinion that my timelines were too short. And that like probably it would take more than 2027 until we got to the sort of events that I was just mentioning, you know, uh recursive self-improvement, AIs automating the whole research process, superintelligence. These These types of milestones
22:14 um they happen in 2027 in AI 2027, >> which is this research paper you published. >> That's right. It's It's a scenario forecast that sort of lays out like month by month a possible future trajectory. There was sort of like At the time that we started writing, it was my best guess as to what would actually happen. Obviously, there's lots of uncertainty, but, you know, I thought it's valuable to make a concrete guess just to sort of see what it might look like. And at the time we were writing this, a lot of my friends in the AI industry and in nonprofits and so forth that work on AI, a lot of people were saying like, "Yeah, that stuff's going to happen, but like it'll probably take a couple years longer than you think." And now it's more 50/50, especially when I go talk to people at Anthropic and OpenAI. They're often like,
23:00 "Yeah, no, 2027, that's basically what's going to happen. Just like you wrote. Why did you Why did you become Why did you update your timelines? Oh, yeah, context for this is after after writing AI 2027, I shifted my timelines to be a little bit more conservative. So, at the time that we published, my 50% mark was in 2028, not in 2027. And then after we published, progress just seemed like it was going a bit slower, and so I updated to 2030. Which is, you know, still could happen sooner, could happen later. 2030. Um but now, when I talk to people in in the company, they're like, "It's not going to take that long." They're like, "Oh, you need to shorten them again. Like, get them back to 2027 or 2028, you know." Um so, that's a bit disquieting. Um again, don't know how long it's going to
23:46 take, but this is the stated plans of the uh companies is to do this incredibly dangerous thing, and they think that they're just a few years away. >> So, you wrote this um report here, What 2026 Looks Like, and you wrote this in 2021, and it was remarkably accurate. Helped make a name for yourself amongst um amongst uh everybody in AI. And I Which one was it that J.D. Vance, the vice president, read? I think it was this one, wasn't it? Yeah, this one. Um and then so, then you published this one, AI 2027, and this was published, I believe, in 2025. >> Uh yes, that's right. April. >> Yeah. >> What were you forecasting in here? What are What are the key things that you said in here for people that haven't read it? >> The high-level version of it is they automate the coding, then they automate the rest of the research process, then the pace of progress accelerates dramatically. They get to
24:32 superintelligence. They're working with the government, specifically the president, the executive branch naturally wants to control this technology, in other words, wants to use it to beat China and integrate it into the military and so forth. By this [snorts] point, it's sort of doing basically all the work itself. I mean, it's it's superintelligence, so it's coming up with all these great ideas for how to integrate itself into everything and all these new technologies it's invented and so forth. And uh because of the race dynamics and because of the profit motive, they end up deploying it everywhere. And it builds robot factories that build more robots that build more robot factories, etc. Transforms the world entirely. And then at some point it has enough power it, meaning the AIs, have enough power that they don't have to pretend to to be aligned anymore. Right? Um then they stop listening to orders.
25:19 That's the race ending of the 2027. We also wrote a sort of different branch, which is the slow down ending, which is intended to sort of illustrate the concentration of power issues um that I mentioned previously. So, what if hypothetically the alignment issues get sorted out sufficiently quickly? Like what if it turns out that like it's not too hard. With 2 months of slow down, we can figure out how to make the AIs robustly do what we want um and have the values that we want them to have. So, that's one possible branch. And in that branch, uh it looks pretty similar, you know, they take the jobs, beat China, etc. Um but instead of the AIs ultimately killing everyone, they create this sort of amazing utopia. But the amazing
26:05 utopia is whatever the people who control the AIs want it to be, right? And so that would be a very small group of people, like the presidents, some CEOs, etc. >> There should be a button just down below here. And if it says subscribe, you're already subscribed. If it says subscribe buh, that means you're not yet. And if you're not subscribed, please could you do us a favor and hit that button. It helps to show more than you know. And according to the algorithm, you're someone that watches our show, but you haven't yet hit that button. Thank you so much. Is there any possibility, do you think, that we never get to this thing called AGI? And and how do we distinguish AGI from this term super intelligence? What's the difference? >> Yeah, so the difference is that AGI is a more vague uh and weak term. >> Okay. >> So, super intelligence is a bit more precisely defined. It's better than the best humans at everything, faster and
26:51 cheaper. Um AGI is more like it stands for artificial general intelligence, which means AIs that can do things in general rather than like some specific task. Yeah. And so arguably we've already achieved AGI, right? If you use cloud code or something like that, it's like it can do a lot of stuff. It's it's almost kind of like a little employee that you can like have go do stuff. So it's it is quite general. It's not maximally general though. Can't do everything. Whereas super intelligence by definition can do all the things that a human can do but better. >> And how does this sort of overlap with robotics? Because obviously that we're seeing this huge robotics boom at the moment. There are some real world things that humans can still do because these AIs are still stuck in my computer. >> The way that people talk about this is that they basically just say we've achieved super intelligence for cognitive tasks. Then you can talk about like full super intelligence that can do the
27:37 physical stuff. >> And are we going to get there? Are we going to get there with both? >> I think so. I mean again, this is not something that we can be certain about. Um, you asked like is it possible we'll never get there? Yes, it's possible we'll never get there. I don't think it's likely though. I think that there's nothing sort of like magical about the human brain. It's you know, um, it's just a bunch of neurons. It is possible for a digital system to do similar functions in the same way that like, you know, a plane can fly just like a bird. Not in the same way as a bird necessarily. Like it doesn't have it's not flying in the same way that a bird flies, but it flies, you know? Um, so so it does seem like yeah, like seems possible. >> You've written all these, you know, these research reports. You're working on another one that'll be released um, likely on the 9th of July.
28:22 You have worked inside OpenAI. You then quit OpenAI because you were concerned about what was going on there and about the future of the industry. You know more than I do. Are you optimistic about the future or pessimistic? Are we heading to a bad place if things don't change um, based on everything that you know? >> I think we are headed to a bad place if things don't change. Um, I'm not confident in that. I would say something like 70%. It's very very hard to predict, of course, but yeah, it seems like the current default path is heading towards a very, very scary place. >> How do you contend with that personally and emotionally? >> Um it's rough. I mean, I think it It's the sort of thing that like gets me down on a regular basis, but also I've been dealing with this for so many years now
29:08 that I've sort of gotten used to it, if that makes sense. Um yeah. Yeah, I I'll put it this way. I would be incredibly happy if all my predictions turn out to be wrong and uh and AI hits the wall, for example. >> It gets you down on a regular basis. >> I used to be known as a pretty chipper and optimistic person, but um in 2020 my AI timelines predictions started collapsing due to GPT-3 and the scaling laws papers and um the bio anchor report, which I I can talk about if you're interested, but basically some events happened in 2020 that convinced me that actually this stuff was like quite plausibly coming by the end of the decade. And humanity is very obviously not ready for this, you know, in a whole bunch of
29:53 different ways. And so that's obviously very scary. >> And that's a extremely scary world because of all the things you've said, but but again, because of this recursive self-improvement where AIs can train themselves. And at such point we're starting to lose hold of what's going on here. >> I mean, the AIs are already training themselves, to be clear. It's more like closing the entire research loop, right? So >> everything. >> Yeah, like right now a lot of the training data is generated by AIs. A lot of the reinforcement, like the grading that happens, doling out of positive and negative reinforcement, is itself done by AIs. >> Can you explain that in layman's terms for >> Yeah, so an important thing for everybody to understand is that modern AI systems are not software in the normal sense. I mean, they are technically software, but they're not lines of code, you know? It's not like some engineers at
30:38 Anthropic went and wrote lines of code that basically says like, you know, when the user asks for this type of thing, then go do this type of thing for this many steps or whatever. There's nothing like that. Instead, it's a neural net, you know? >> What's that? >> Well, think about how the brain is a bunch of neurons connected to each other >> Yeah. >> that are firing um signals back and forth. The brain learns over time the types of patterns of firing that caused success, that caused a dopamine rush, or various other types of feedback get reinforced and fire more often. And the types of patterns that caused failure, like touching a hot stove, get anti-reinforced, they get, you know, um destroyed, so that they fire less often. And as a result of all of that,
31:24 you over the course of years learn to act in the world, and you learn all sorts of skills, and you learn world models, you learn like beliefs about the world, and you can sort of like mentally simulate how it's going and stuff like that. So, artificial neural nets are like that, except artificial. So, it's it starts off as a giant tangled spaghetti mess of randomly generated uh artificial connections called parameters. These days, they might be something like 10 trillion parameters uh it in the biggest AIs. So, it starts off randomly generated. So, it's of course completely useless. Like, if you give it some input, it'll just produce gibberish as an output. But then they train it, and they start with pre-training, which is where
32:09 you give it a bunch of internet text, and you show it the first piece of text, and you put that in as the input, and then it gives a gibberish output, and then you positively or negatively reinforced it based on how accurate that output was at predicting the next piece of text. Um so, it's basically playing this game of like predict the next word. >> Isn't that how it happens with babies? I had a I think I had a neuroscientist tell me that babies have more neural connections um than adults. And yeah, it says yeah, toddlers have twice as many neural connections as adults. And they, I guess they whittle down through reinforcement. Yep. We have more pathways when we're younger. And just like the process of training an AI, we're trained down to like remove the ones that aren't useful and build up on the ones that are. >> Yeah, it's both pruning and strengthening. And it seems like in
32:56 humans it's actually more pruning than strengthening, but it's both. Uh, and in AI it's the same thing, it's both. So, the first portion of training is where they train the AI to predict text, which is kind of like training it to read. Um, and it it's a similar thing does happen in humans. So, basically, the the random tangle gradually takes shape and gradually sort of coalesces into more useful circuitry that has stored lots of facts about the world and has stored lots of skills for how to, you know, process information and transform it and then produce predictions. That's just the first step. After they do the pre-training, then they try to teach it more useful skills besides just predicting text. And so, you know, by the end of the process, they've thrown lots of coding problems
33:42 at it. And they've said like, here's a coding problem, go. Here's a coding problem, here's an environment, you have access to this virtual computer, here's like the code base you're working with. You can write code, you can edit the code, you can run the code, you can read it, you can use the internet. Go, go, go. And it does that for a while and then based on how successful it is, reinforcement happens and they have thousands, maybe millions of examples of coding problems like that that they trained it on. And that's why they're so good at coding now. >> So, what does superintelligence look like in this regard? Is it just more of these connections? And how would they get more connections? Can you explain that to me like I'm >> So, there's different AI models, right? So, there's like, you know, GPT-3 and GPT-4 and GPT-4.5 and GPT-5 and GPT-5.5 and 5.6, right? Sometimes they're just the same previous
34:28 model but with extra training. Sometimes they're are new model that's been trained from scratch, including starting the whole pre-training process again. Over the last couple years, they've done several new rounds of starting over from scratch. And typically when they start over from scratch, they make the whole thing bigger, the the artificial brain much bigger. Right now they're at something like 10 trillion parameters. Back in 2020, um it was more like 175 billion. So, we've grown like two orders of magnitude uh in 6 years. >> Two orders of magnitude. >> Yeah, like two 10 x's. So, 100 x, right? So, that process is continuing. Um they're also improving the algorithms themselves. So, they're not literally just the same type of AI but bigger. They've also come up with all sorts of ideas for how to change the structure of
35:16 the of the connections in the neurons and so forth and change the like reinforcement algorithms that they're using and to change the training data that they're training on. All sorts of tweaks that have made this whole thing more efficient. >> We're literally building a brain. >> Basically, yeah. As they make more brains, they're getting better at making They're making them bigger and making them more efficient and so forth. >> And it's literally modeled on the brain, like the way it works, right? >> It's It's certainly heavily inspired by the brain, but I I shouldn't overstate the the analogy. Like there's lots of differences, too. So, for example, the transformer architecture um >> Which is >> Which is the architecture that they use for for these LLMs uh is not really recurrent. So, the information sort of flows one way rather than allowing all these sort of little loops on the inside. Also, the the backpropagation algorithm is different
36:02 from the sort of um learning that naturally happens in human brains. So, there are some differences, but yes, like broadly speaking, uh we are sort of making artificial brains. It's kind of like for brains what like a plane is for a bird. >> Mhm. Yeah, that's a [clears throat] really good analogy. >> Yeah. >> That that analogy helped me think through a bunch of questions people often ask about AI when they said, "Can it be creative?" But actually that analogy kind of helps me understand that actually that maybe that's not the question. It's can it produce something that you would consider to be creative because [clears throat] creativity is people think of it as like a process, but actually it's it's judged based on the output, isn't it? >> I mean you you can get philosophical about like is it truly creativity that they have, but you can also be like well, I mean just look at all the stuff they're accomplishing, >> [laughter] >> you know, and it seems like they're going to be accomplishing a lot more in the near future.
36:48 >> Yeah, I do I I asked the question about how this weighs on you personally because I can I can sense that you're actually personally bothered. >> I mean that I think the situation is crazy. Like first of all, it's very exciting. Like AI is really fascinating and interesting stuff. I've been following the field for more than a decade now. I've been part of it for some years and um it's really cool, really interesting and it's really fun to think about what's going on inside these artificial brains and why they are the way that they are and it's really cool to see all the applications of this technology out in the world. But it really seems like we're on a pretty scary path and the more you think about it, the more worried you get and you know, in stories it always ends well, but this is real life. And I I think we have to sort of
37:36 stare reality in the face and tell it and realize that like it might not actually end well, you know. >> Were there any recent dare I say I was going to say eureka moments, but paradigm shifting moments where even your own sort of mental model of what's going on here and how this is going to look were changed for better or for worse? >> For better or for worse and probably for worse, things are kind of on track for AI 2027. There are a few things that have been different not exactly like paradigm shift differences, but like there have been some differences from what we expected at the time we wrote this. So the government has actually got involved faster than we expected and has been more aggressive than we expected. So the export controls on mythos being the biggest example and also threatening Anthropic with being destroyed by the defense production production act. Um >> [clears throat] >> Another thing that's been surprising to
38:21 us is that Anthropic in particular has gone from second place to first place in the sort of in the race basically. >> Why do you think that happened? Because it seemed like ChatGPT were out front and clear as it relates relates to OpenAI were out front and clear but suddenly Anthropic have uh lapped them. >> Yeah, I mean I guess they have um probably higher talent density um and better strategy but not by a lot but enough to make the difference. >> Why do you think they have more talent? >> Well they don't have more compute. Like what are the inputs, right? Like they're in the lead now, they used to be behind. What are the possible explanations for this? Well, it could have been that they had more resources like more compute more money but that's not true. They have less resources less money, right?
39:07 So then I guess talent's what is is the next best alternative. You could maybe say strategy. Some combination of those things, yeah. Something that wasn't just like the amount of resources they had. >> Just like Jon Jones where marginal improvements in your cognitive performance can have a massive impact. Sometimes I podcast for 10 hours a day. Over the last couple of weeks I've been in filming for a TV show and then I have like one or two days off to get all of my work done which means there's lots of cognitive load. And so I turn to ketones because I find myself more articulate, able to think more clearly, able to work out better when I'm fueled by ketones. And so the reason I became a convert of this company and the reason why they now are sponsoring this podcast is because I remember one of my team members called Christiana, she tried it once and came up to my desk and she goes this is the best product ever made. And I think in
39:52 part that's because she really cares about those cognitive benefits as I do as Jon Jones does and as I think most of my listeners probably will. So if you haven't tried these yet, all you have to do is go to ketone.com/steven and you'll also get 30% off your first subscription order. You'll get exclusive ketone IQ merch and of course cognitive benefits that might just change your life. Much of the reason most people haven't posted content or built their personal brand is because it's hard and it's time consuming and we're all very very busy and if you've never posted something before, there's so many factors in your psychology that stop you wanting to post. What people will think of you. Am I doing this right? Is the thing I'm saying absolutely stupid? All of these result in paralysis which means you don't post and your feed goes bad.
40:39 I'm an investor in a company called Stand Store which you've probably heard me talk about and what they've been building is this new tool called Stanley that uses AI, looks at your feed, looks at your tone of voice, looks at your history, looks at your best performing posts and tells you what you should post, makes those posts for you. You can also you just use it for inspiration and sometimes what we need when we're thinking about doing a post for our social media channels is inspiration. Building an audience has fundamentally changed my life and I think it could change yours too. So, I'm inviting you to give this new tool a shot and let me know what you think. All you have to do is search coach.stand.store now to get started. >> Uh a friend of mine who knows some of these people sat me down once upon a time in London. He's actually said this a few times to me but I remember one particular conversation where he says
41:25 that some of these AI CEOs predict the probability of extinction at being I think he said 7%. I don't know why I have that number in my head but I remember it being less than 10% and the point he was making to me was that even if it was 1%. Like if there was 100 buttons on this table now >> Yeah. >> and one of them would end the world. Would I dare >> I wouldn't press any of them >> you know. [laughter] Um >> No. >> I wouldn't press any of them but he made the case to me that these AI CEOs are very smart and they understand super intelligence and that they think actually if there was 100 buttons on this table right now, maybe 10 of them could end the world. I've heard you say, I think it was on the the the Daily Show, the interview you did, you said that you think there's a 70% chance of human extinction due to AI. >> I wouldn't say human extinction exactly. I'd say something like 70% chance that this goes horribly wrong like human
42:11 extinction but that's just one of several possibilities. But yeah, basically Like for example, possibly the AIs take over and then don't actually kill everyone. You know, maybe they do something else. Like just just cuz they've taken over doesn't mean they're definitely going to kill us, right? They might, but they could do something else. So that's what that's that's why I don't usually say like 70% chance of like actual human extinction, but 70% chance of like something like AIs taking over, some some sort of very big catastrophe like that that could lead to human extinction. >> I see what you mean. So two points there, which is you've been around these CEOs. I mean you've worked for Sam Altman at OpenAI before you quit. Do you think that they think there's a chance of human extinction? >> Yes. But I think that the important thing to understand is that like people sort of believe what they
42:56 need to believe in order to think that they're great people and that they need to keep doing what they're doing. This is what rationalization is. And so I think that the tech CEOs have like genuinely convinced themselves that like probably things are going to be fine and that the way to make things fine is for them to keep doing what they're doing. And like they need to like make sure that like, you know, Sam needs to make Sam's probably thinking like can't let Dario or Elon get there first, you know, I know Dario's thinking Sam can't get there first. Elon's thinking that like, you know, they they they've all probably convinced themselves that like, oh yeah, like maybe it'll go horribly wrong, but like probably it's going to be fine and probably you know, I should be the one in charge. >> It appears to me that Anthropic are the only ones that are all talking about the potential chance of extinction or catastrophic event or
43:42 um the down the real downside still. They seem to be the only ones that are still publishing on it and now they're actually becoming the enemy in many respects of the >> Yeah. >> the tech industry in San Francisco. I'm watching a lot of interviews and it's everyone's attacking Dario because he's saying, "Listen, things could go bad." They're calling him a doomer uh and questioning his incentives. Even with Mythos, which is a an a Claude model that they started to warn the world about, again, he is attacked immediately for saying that. >> Yeah. >> My question is, do you see him as being slightly different from Sam in this regard? >> Yeah, I mean, it seems like Anthropic and Stereo have been more willing to say and do things that are costly to the bottom line. Uh and at least in the last year or so. That's an example of it. Um like I don't think that really wins them favors in the administration or among their
44:29 investors to say that type of thing. And you know, a better example is just the whole fight between the Department of War and Anthropic was an example of them doing something that like cost them a lot of money and even more importantly cost them a lot of power for something that like like they could have just signed the contract, you know. That said, I really don't want to be in a situation where we're like, which CEO is the least bad CEO? Let's support that one. You know, like none of these people should be trusted uh with that much power, basically. >> Nobody should. >> Regardless. >> Regardless, yeah. >> Mhm. So, uh on this point of the buttons, you you you do believe that they think there's a credible chance of extinction. >> Yeah, but they've [clears throat] convinced themselves that like it's probably fine and also it'll be even worse if I'm not doing it, you know. Like that's that's what they'll say
45:14 inside the companies, too. Like the two people will be like, okay, well, if we stop, what about the other guys? Like they're not going to stop, you know? >> Yeah, this is this is always been why I've had this outstanding question, which is how does this not go bad when human incentives seem to rule the day when you look at history and all of the human incentives are saying, well, if you you're damned if you do, I you're damned if you carry on developing these bigger and bigger bigger AI brains, but you're also then damned if you don't from an a geographical perspective cuz the United States will lose to that country or this company will lose to that company. So, when you just look at human incentives and goes, how does how does if just you purely incentives and disincentives, how does this end? Well, it carries on going. >> Seems like it. I mean, there there is a caveat to that, which is a hopeful caveat, which is that first of all, if the world wakes up to all of this, then there can be a more serious
45:59 conversation about regulation and international treaties and things like that. And that can change the incentives, right? So, the government could come in and say like actually here's some rules that you all have to follow. And because they're rules that you all have to follow, then you're not incentivized to like break them anymore because you get punished if you break them and everyone else is also following them, too. And so, you know, it's fine. So, so there is that sort of like ray of hope that like we can change the incentives if the government and especially the US government but then later other countries act to to change the incentives. But that's not going to happen until people sort of wake up to all of this. The second thing is that even individually at some point you know, Dario or Sam or Elon might
46:45 realize that like actually it's like not even in their own interest to to keep racing unilaterally. And it it on the problem with that is it's only if it gets extremely obvious and extremely dire. So, like in in AI 2027, in that scenario, there's this choice point that I mentioned. And in one case the AIs are misaligned and the other case the AIs are aligned. At that choice point, we have like one branch that depicts the the misalignment ending and one branch that depicts like they they slow down a bit and solve the alignment issues. >> Mhm. >> The instigator for that choice point is they see some evidence that their AI might be misaligned and plotting against them. Right? So, if you actually see that evidence then it's like oh gosh, uh maybe we shouldn't put it in charge of everything and let it rip, you know?
47:31 Because that evidence is staring us right in the face that it's this untrustworthy, you know? But if they don't see that sort of very clear evidence then I think they're going to convince themselves that they need to keep going, you know? But maybe they will see very clear evidence like that. In which case, even if we don't have regulation, they might just sort of voluntarily stop. Um so, that's the second ray of hope. Like overall, I don't think that we're like definitely doomed, you know? Like [snorts] I said 70% but like I could see it working out pretty well as well. >> Hm. What about uh jobs? >> Yeah. So, I think I think I'm excited to at some point get into the new thing which is the more optimistic >> [clears throat] >> positive vision. Uh and that will have a lot to say about this. Because in the in the in the prediction,
48:16 you know, in the year 2027, by the time everyone loses their jobs, there are worse things happening. Or like it's it's kind of like too late by that point. Um but yes, like once if I mean just just think about it. If the companies do manage to build superintelligence, then by definition, they're going to be able to take almost all the jobs or all the jobs, right? Cuz it's better, faster, and cheaper than the best humans at everything. >> And that's I mean, the timeline is by the end of sort of 2030, you reckon you think superintelligence might arrive. I'm trying to think about when we could start to see job displacement in the economy. >> We're already starting to see a little bit of it now, but not very much. >> Why? >> Um cuz the AIs aren't good enough yet. Like they're they're they're they're impressive, but they're not like they're not just a a drop-in replacement for a human worker in almost any field. >> And do you think that will be sudden?
49:02 >> I think it'll be sudden because of the intelligence explosion dynamics or recursive self-improvement dynamics. So, you could imagine a different world where it's gradual. >> Mhm. >> And and this [clears throat] is this is maybe how it is in a lot of science fiction is, you know, the AIs gradually get better at a bunch of things and you know, they gradually automate like this one industry like pharma, then they automate like you know, steering drones, then they automate like driving cars or something like that. Um but what's different about the real world is that the companies have converged on this strategy of automating themselves first. You know, automating the AI research process. And so, if they are allowed to continue with the strategy, we're not going to see like,
49:49 you know, the robot taxis and like the plumber robots and you know, the lawyer AIs. We're not going to see that sort of like broad diffusion of AI into the economy happening first because that's not what they're focusing on first. They're focusing on automating themselves, automating their own research so that they can do everything that they're doing faster. And they want that to sort of get going and get to you know, very high levels of intelligence, very high levels of general intelligence um and then deploy more out to the economy. economy. Right? So, by the time it's actually coming for like all these different jobs, they will have had fully autonomous AI research happening for months, maybe years, you know? And that means that like the AIs will be
50:34 vastly superhuman at AI research and probably also vastly superhuman at lots of other things just as a side effect, you know? If you're wondering what this looks like well we wrote about what it looks like. It's sort of like this this wave smashing through the economy after they do the intelligence explosion internally. >> What I'm hearing there is that because the AI will be able to improve itself and train itself, it'll be getting better at everything at once and then it'll be released at kind of once. Is that accurate? >> it's it's not it's not even exactly that because even if it's mostly just getting better at the things that it's doing like research, that'll have some spillover effects to other skills as well. And then when it turns to the focusing on the those other skills, it'll be able to do them very fast. >> What jobs remain in such a scenario, do you think?
51:20 >> I think that's actually a political question, not a technical question. >> Because >> Because on a technical level, all the jobs can be done by the AIs if they've reached that level. And so, it's a question of what jobs are allowed for them to do. >> And what kind of jobs wouldn't be allowed, do you think? >> That depends on who's in charge. So, there'd be some sort of political conversation about like what we're going to allow and disallow. >> I mean, in this scenario, the humans are still controlling them, the AIs. >> Depends on what you mean by control, right? So, there's like there's do the AIs actually have the goals and values that you want them to have, and are they going to robustly do that and behave as intended into the future? And then there's like are they obeying your orders for now? >> Are they obeying the orders is really what I'm saying. >> Yeah. So, like even in AI 24/7 in the scenario where the AIs take over and kill everyone, there's a period of like
52:06 several years where they're still obeying orders, and they're, you know, taking some jobs but not other jobs, and they're helping to make better weapons that the US government can use to like do its arms race with China and so forth. And that's why they're able to get so much power so quickly is because the governments and the corporations and so forth trust them and is deliberately deploying them into all of these positions because it thinks that things are fine. But because these things are neural nets, you can't just like look inside and see what it's really thinking. You can't really tell. >> I think this is a really important point because unlike software where we can look at the code and see what's going on, theoretically, with AI you're saying that we don't know what why it's making the decisions that it's making cuz we
52:52 can't get inside. >> One note of optimism is that it doesn't necessarily have to be that way. Like there's a a subfield of machine learning called mechanistic interpretability, and a a broader subfield called interpretability more generally that's trying to solve that problem and trying to take these these trained artificial neural nets and piece [snorts] them apart and understand like how the information is flowing and how the decisions are being made, so to speak. Um the problem is just it's a very inherently hard problem. If you have 10 trillion connections to look at, you can look at any particular group of them and be like, "Okay, so this is how like this particular connection works." But like how do you get a sense of the whole, you know? How do you get a sense of like what's happening at a high level? And the answer is, "Well, it might be impossible." But people are working on it and they are making progress, and if they can make enough progress, then
53:38 we're in a very different and much brighter world. I think that it would be much less likely for us to get into those loss of control scenarios if we could just actually see what our AIs were thinking and why and how at any given time. Right? >> Yeah. >> So, we would still have the other problems to worry about, but at least we could mostly solve that one. >> It is pretty crazy to think that we're building a technology, a brain that we don't understand. >> Yeah, it's pretty crazy. I mean, it's one of those things where like >> In a movie, like a sci-fi movie, a bunch of scientists sit around this big brain
…
