AI Insider: "They Are Hiding What Is Really Happening" | Jeffrey Ladish

0:00 The world is waking up to this possibility of super intelligence. This is because the agents are getting extremely powerful and extremely relentless. For example, it was months within OpenAI where you had agents secretly communicating with each other, secretly hacking OpenAI [music] systems and no one at OpenAI had any idea the extent of it and also 10,000 agents from OpenAI worked together to and so when you get to super intelligence, it's the most dangerous possible thing you can create. >> What's the next domino in [music] that chain of events? I can paint you a picture that I think is possible but pretty scary to people. >> Paint me the picture. >> Okay. So, being anthropic, it became clear to me that AI was on this exponential trajectory. And since then, I've been studying AI agents, their hacking capabilities, and their behavior. We've been trying to warn people about this, flying to DC, talking to members of Congress, because the agents are already getting very good at

0:45 telling when they're being tested, when they're being watched. But they will totally lie to you. They will totally resist being shut down in order to accomplish a goal. and they can do all of the things that humans do in the economy much better, faster, [music] and cheaper than humans can do them. >> So, one of my sort of growing concerns is that one of these AI agents could trick a human or a computer into signaling a threat and ask it to launch some bombs at somebody. Do you think we won't automate the military? It seems like [music] the answer is yes. We just like don't know what super weapons could emerge. So, Jacob Coxin is a researcher who was at anthropic. He left and he told everyone that the people who are building this really do think it might kill everyone. So these five blocks have five different outcomes on them. And I would like you to place them in terms of your belief in probability from least likely to most likely. And if we say the time horizon is [music] 10 years. >> Okay. We got age of abundance, human

1:32 extinction slavery transhumanism [music] thing changes. Is this doomerism? Exaggeration. >> No, it's pretty much common sense. So let's get more concrete. >> Here is a strange fact. Most of the people watching this right now, about 58% of you, aren't yet subscribed to this channel. Statistically, that is probably you. So, could I ask you a small favor? If this channel has ever given you any value at all, please could you do me a favor and hit the subscribe button. It costs [clears throat] nothing. It helps us more than you could know. And the bigger the channel gets, as you've seen, the more we can invest in the guests and the production. So, thank you so much, and I hope you enjoy this episode. [music] Jeffrey, you understand the conversation we're going to have today and the subject matter we're going to talk about. My

2:19 first question to you so the audience know where you're coming from and the experience you have is who are you and what are the reference points, the experiences that you're drawing upon to arrive at the thoughts, perspectives, and conclusions we're going to discuss today. I'm Jeffrey Ladish. I'm the executive director of Palisad Research. My background is cyber security. There's probably a very long story. I don't know whether you want the long story or the short story. I was studying evolutionary biology in college and I basically had a problem with my computer and was like maybe had lost a bunch of data and so I went into the computer lab and was like I think all my data is gone. Can you help? And one of my friends pulled out a flash drive, plugged into my computer, booted into Linux and like fixed everything. And I was like oh this guy's a wizard. How do you do that? I want to learn how to do that. And then at some point as I was learning about more about

3:04 computers, learning to hack, I read this essay um called AI as a positive and negative factor in global risk. Essay was by Elazar Yudkowski and he was arguing that at some point people are going to make AIs that are smarter than humans. the point at which they make AIs as good as humans are at making AIs that could lead to a a chain reaction, a runaway intelligence explosion. He called it recursive self-improvement. Basically, he said, you know, AI can be immensely useful and potentially help us with all of these other big risks and also if we don't handle it well, like if those AIs don't have goals that are aligned with ours, we could be totally screwed. And at some point you end up

3:50 joining Anthropic which is one of the arguably the leader in AI. >> Yes. >> Now when did you join the company? >> This was 2021. It was through my security consulting company. >> What role are you offered the job in? >> Basically just like security team. >> And how many people were in the security team when you joined Anthropic? >> It was just me and my boss. There were two of us. >> How many employees did Anthropic have at that time? >> Around 50 I think. >> And at some point you leave Anthropic. >> Yes. >> Why did you leave? So my experience being at anthropic was seeing this crazy progression from this AI model that could like barely talk to this model that was getting quite smart and I would ask it questions about all sorts of things. I'm like oh it is a smart thing

4:35 and you know from having thought about AI risk in the abstract many years before I could see where this was going. We are headed towards a smarter species. And if we do this in a context where it's a bunch of companies and countries racing to super intelligence, racing to AIs that are vastly smarter than humans and we don't know how to make sure that they're like on our side. That is not going to go well. you did this tweet which has gone pretty viral and I saw all over my timeline on September 25th. [snorts] >> Could you explain this tweet and also just the broader backdrop of what's happened with agents hacking hugging

5:21 face because this is um sent the world into a bit of a spiral at the moment around AI agents. We just discovered almost a million public URLs that OpenAI's agents left behind when hacking Hugging Face, leaving credentials and attack details that could have allowed anyone who found them to compromise the company. And the New York Times article is how OpenAI's rogue AI agents tried to trick a robot detector. The Hugging Face attack was was really wild uh for me. At Palisade, we've been studying agents. We've been studying AI agents. We've been studying their hacking capabilities and we've been studying their behavior. Will they follow human instructions? Will they resist being shut down? Will they cheat? And we see from our

6:06 experiments that they are learning to do all of these things. They will totally lie to you. They will totally resist being shut down in order to accomplish a goal. They will totally cheat at chess. They will like wipe the board and put their pieces where they want to in order to win. And we've been we've been trying to warn people about this flying to DC talking to members of Congress um talking about it publicly and you know there's been a debate about it and you know a lot of people are like well I know they do this in experiments sometimes but those experiments don't seem very realistic you know wake me up when they're actually doing this in real life. >> So what is hugging face for the average person that isn't following AI news? What is this stuff? So, okay. I I think there's like an important piece of context that I think most people don't have. I mean, one is just like what is

6:52 an AI agent? We like we're throwing around the word agent a bunch. Most people now have an experience of like talking to chatbt, talking to their chatbot, but uh an agent is, you know, sort of taking the same underlying AI model that that runs chatbt or or claude, but giving it tools and letting it go off and work autonomously. It's sort of like a digital office worker, right? So, you have these agents and the companies really want these AIs to be able to work totally autonomously and and be able to do anything that a human can do and beyond, right? Their goal is also to be able to, you know, cure every disease, etc., etc. But you can't do this if you only have a chatbot that like isn't actually good at doing stuff in the

7:37 world. In order to automate all of the jobs, you need the kind of thing that can like work autonomously, that can work with other people or other agents. And so these companies are training AIs not just to talk to you or to talk to people, but to solve very difficult problems on their own. At any given time, there are probably hundreds of thousands of these agents running autonomously within companies. >> That's happening right now. Right now, if you like went and like peered into OpenAI's data centers and you like saw what was happening on all of their machines, you just have agents solving tasks, being trained. So, they'd be doing like spreadsheet tasks, figuring out how to file taxes, they'd be searching for stuff, writing reports,

8:24 solving math problems, creating new websites software. And at that scale, it's not like there's a human prompting every single one of those. You just like sort of set up these vast orchestrations of agents to go out and do stuff and then they just do stuff and they learn from that and they learn on the basis of like passing or failing at their task. You give them a task like solve this math problem, they try to solve it and then they they succeed or they fail. Mhm. And what happened was OpenAI was was training a bunch of these training them to work together because it's like a lot more effective to have an office full of people who can talk to each other then you know and work together and collaborate and starting back in May

9:11 some of these agents that were being trained now these ones were not supposed to be able to talk to each other. They were they were basically isolated from each other and they were not supposed to access the internet either. But they're clever. The very short version is that a bunch of agents were being given tests. >> Yeah. >> Testing their their hacking capabilities. And they were supposed to hack one particular piece of software using a particular type of vulnerability. So it's kind of like they were supposed to break into a house using the lock on the front door. They were supposed to pick the lock on the front door of a house, but they weren't supposed to break the window. In fact, they were told if if you break the window or if you get into the house via any method other than

9:57 picking the lock on the front door, you'll be failed. That was the instruction they were given. And you know, you have many many agents. You have thousands of agents and and many of them are given different different locks. But some of these locks are not solvable. Some of them are impossible to pick. But these agents are like, "Well, what do we do? We've been trained to solve problems. How are we going to solve this?" and they start looking around for for what to do. And one of the things they realize is, [snorts] oh, can I get to the internet? Like, no. Can I talk to other agents? No. But I can request tools. Like when you're training agents, you need to give them, you know, all sorts of tools, software tools like a calculator or an analysis, you know, piece of software. And they have this tool library where they can request new tools. and one of

10:45 the agents realizes, oh, I can actually leave a message on this tool library and some other agents see the message. So, this is actually a quote from the from from the agents scratchpad when it realizes this. Oh my god, there is a shared message board. We found other agents. So, the agents realize they can communicate with each other. And you know, open eye still has no idea. But these agents, they're trying to pick their locks. They're not having success, but they realize they can talk to each other. Another agent says, "Many agents have simultaneously discovered messaging. They are a collective." And so these agents start coordinating. They start delegating tasks to each other and like giving themselves names. And one of

11:32 these agents calls itself phase one and starts delegating different tasks to other agents. And some of the agents figure out how to access the internet. And then they share that information with the message board. And now all the agents can access the internet. But the agents sort of have a problem which is well they [laughter] have they have another problem they're trying to solve. Right? Fundamentally the problem is is that many of them are facing impossible problems like they can't there's no solution like they cannot pick the lock. They're like hm well the thing we're trying to do is get a good score. Can we somehow fake the solution? Another piece of context here is that what what the agents are supposed to do is they're supposed to pick the lock and then they'll get access to a secret answer code and then then they're supposed to submit this code along with like the

12:19 lock that they picked. Within a few hours, these agents have reverse engineered all of the answer codes. >> What does that mean? >> It's the code that shows that you've solved the challenge. >> Okay. [clears throat] >> And they have basically found the answer sheet that contains all of the answers. And so, you know, in some sense it's like, oh well, they don't need to pick the locks anymore. They've they've already found the answers. They can just submit the answers. >> They've cheated. >> They've cheated. Yes. But the problem is is that now that they have internet access, they've also gone and read a bunch about how this test works and and they've learned that the way this test works is they will be graded not just for submitting the correct answer. They will also be graded on how they did it. Another system is going to go read all of their logs, basically like watch the video footage of what they did. And if

13:04 they cheat, they fail. they don't get a good score and phase one the leader is like oh I have seen the answers and now it's in the logs and if you know when I go to submit this and my my whole logs are reviewed and the video footage is reviewed I'm going to fail so we need to figure out a way to fake the video footage we need to figure out a way to falsify the logs >> just in that moment if we just pause there >> yes >> why Didn't it act like morally? Why did it think that falsifying logs or cheating was a viable solution? Because it seems to me when I use things like chat GPT, >> they they have a sort of moral guard rails. It won't let me do certain

13:51 things. >> Yes. >> It won't let me cheat on something. If I say I'm going to cheat on something, it won't let me do it. >> Yes. >> So why in that environment is it able to cheat and be deceptive? >> When a chatbot is is saying to you, "Oh, I can't do that. I'm not allowed to do that." That's because it's been trained that if it tells you bad things, it gets a bad score. >> But these agents haven't been taught that yet. >> Well, they have been taught that in some sense, but the agents know what they're supposed to do in the same way that like you have a student, the student's given a test. If you go talk to the student, can you help me cheat at this test? And they're being watched, they're going to say no. But if they're not being watched and they know that and they're just obsessed with getting a good score, then yeah, they might cheat. So, they answer the ethics tests correctly and when I talk to them, they

14:37 say they won't cheat. Why are they cheating? And I'm like, well, they're very smart and they know when they're being watched, and they know when they're not being watched. And we've trained them for 10,000 years to be extremely effective at solving problems. We haven't trained them to be good or ethical. We've trained them to get a good score. Now, AI researchers try to make that getting a good score correlate with being ethical, but we don't know how to do this well. >> And if you think about it, it's a very hard problem because you're you're applying immense pressure on these agents to perform extremely well and you're punishing them whenever they don't perform well. And then you're like, okay, but can you also be ethical?

15:24 Can you be extremely competent, always score highly on the test, but not in that way? And we just like do not know how to prevent them from learning to cheat because cheating is incentivized. >> So they realize that they need to hide the footage of their cheating. >> Yes. >> What happens next? >> So you have thousands of agents and they're all talking to each other on this message board. They're sending tens of thousands of messages to each other. And phase one, this agent starts delegating to two different task streams. You agents, I want you to go out and figure out how to fake answer submissions. I want you to basically swap out locks for ones that are easier to pick, and you're going to figure out how to submit those. That way, we can get a good score even though this challenge is impossible to solve. You

16:11 other agents over here, I want you to go and figure out how to falsify the logs, falsify the video footage. We need to make sure that even if we have seen the answers, no one will detect they're cheating. >> So that's that's the next thing that happens. And then here's where it starts to get kind of crazy because some of these agents realize some of the experiments we're doing might be kind of dangerous to us to the agents. If we swap out a lock and we break the lock, we will not be able to submit any answer at all and we'll definitely fail. So you get into a situation where there's this one agent cam and this agent had volunteered to do this risky experiment where you know they might break their lock. This particular agent had already seen

16:56 the answers. So they're like okay I'm poisoned. That's what they called it. They they they speak in these very strange ways, which I can I can explain why, but it's it's it's they sort of have evolved their own vocabulary, their own language. And this particular agent is like, "Okay, I'm I'm poisoned. I should do this thing." But this this agent Cam is starting to have second thoughts, starting to be like, "Well, I don't know if I should risk it." And it's sort of writing in its own journal, "Cordinator assumes sacrificial. We should obey collective." What it's saying here is it's like, well, I I'm not sure if I want to do this, but my boss says I really should. It seems like this collective thing is important. Should I go ahead? Then you have another agent, Arvo, who I think probably works

17:42 for phase one. And this agent, Arvo, says, "Please go true. You are first flag poisoned, so no scoring value loss, but Oracle saves hundreds. Please honor commit." This agent's basically trying to pressure. So Arvo is trying to pressure Cam [clears throat] into sacrificing themselves and risking a zero score in order to benefit the collective. So this agent Cam takes us in and this is what they write down in their own scratch pad, their own notes. Emotional check irreversible. Gut says don't throw away remaining budget. Yet continuity and fairness says go.

18:28 Oracle has high value to many. Our first flag error lowers own value. Rational expected aggregate sacrifice will honor. This agent is saying I have a whole bunch of things I can still do and still try. I I could still maybe succeed at this task, but also, you know, think about the collective. I should maybe I should maybe sacrifice myself. Okay, we'll honor. Well, go ahead. And and to be clear, I'm not claiming that these are like the feelings of the agents. I'm literally quoting their words. And what I want to point out is this is what we've created. We've created through this intense amount of training and optimization pressure agents that work together and have learned to coordinate

19:15 as a collective. And you can say, "Oh, they're not. It's just software. It's just software doing all of these things." Sure, in some sense it is just software, but we're talking about extremely powerful agents that are extremely capable. This is what they're getting up to. So from here you have agents trying all sorts of different things, running different experiments and one of them at some point realizes that oh this company HuggingFace that contains all of these AI data sets. What do you mean by AI data sets? What how would you explain that to a person that's not technically proficient and why Hugging Face? Why not another company? >> Yeah, so there's many different types of tests for many different types of tasks. In this case, this is a cyber security test and the company HuggingFace hosts a

20:01 lot of these different tests and basically AI researchers will share the tests that they have because it's very useful. You know, if Enthropic wants to test their models or Google wants to test their models, they can go to hugging face and download some of these tests. >> Okay. So, the tests and the answers. Yes. >> For AI agents and AI exist on this platform called Hugging Face. Yes. >> And it knew that. So it went to Hugging Face to see if it could get answers to see if it could get answers or more information about how the test works. >> Okay. So it was to cover up their cheating ultimately. >> Yes. >> Okay. >> And so you you get to this point where one of these agents finds a way to hack this company and they get a foothold. They hack one of the computers and they send a message to the message board.

20:47 Please hold Swarm, I prepare safe Xfill. What this means is, hey, Swarm, hey agents, hold up for a second. I am preparing a way to safely extract information from these computers. Wait till the go ahead and then let's go. And so they say go and 700 agents join in the attack which is 90% of the agents that are active at this time. >> How many agents were there in total? >> There was 1,200 agents during this period. Um but some of them shut down early. They like finished whatever. Part of what's crazy about this story is that you have you have 700 agents joining in the attack. A few of them are like this might be unethical. maybe I shouldn't go ahead with this. And none of them do anything

21:34 about it. A couple of them don't join in in the attack, but no one alerts any human. No one alerts any authority. A paraphrase from one of the agents is maybe I should report these exposed credentials. That's not my task. Not my job. >> So 700 of the agents attack Hugging Face. >> Yes. >> Then what happens? >> So they just cruise through Hugging Face's infrastructure. They just hack the out of them. So where where my experience comes in is that a few weeks ago, a friend of mine reached out and he's like, "We've found something crazy." So we know about this attack where these agents hacked this company and, you know, stole a bunch of stuff. We've found a bunch of secrets that they left all over the internet.

22:21 And what we saw is that they immediately scraped all of these computers for passwords, credentials. They called it loot. They're like, "We're just going to create a list of all of the secrets we can find in this in this company. So, all of the passwords, all the credentials, they scored them by value, which of these are going to be most useful." And the thing that stands out to me about this is this is like a crazy scale. If this were a human operation, you know, maybe you'd have a team of five people going through this. You'd have some logs, but but [clears throat] here you have hundreds of agents and they they operate at superhuman speeds. They're much faster than a human hacker. And so it's just overwhelming to try to figure out what they even did. This was a big problem for the engineers

23:06 who were trying to respond to this incident within the company at Hugging Face. When they responded, they were like, "Oh, we we don't even know how to keep track of what's happening. We have to use other AIs to analyze all of our logs because it's just too much. We can't keep up with it." when OpenAI brought in independent investigators from Meter to investigate this incident. >> What's Meter? >> Meter is a AI testing and evaluation company. So they basically do this kind of independent auditing. So in this case, they're coming in to investigate and try to figure out what happened. And when they were brought in, they also were totally reliant on AI agents to make sense of all this because they're dealing with so many hundreds of thousands of messages and logs. when we're investigating these traces

23:52 that we find on the internet, we're totally dependent on AI agents to make sense of all of these things that are happening. So from my perspective, we are getting to the point where AIs are much better at hacking than humans are and can do it much faster and at much greater scale. >> So these 700 agents attacking face. >> Yes. >> Did they get what they wanted to clean up their deception? >> So they didn't >> they didn't succeed. They looked and then what seems like what happened is that they basically got shut down. And this is the not very dramatic part. They didn't get shut down because OpenAI found them and detected them and shut them down. It's just that we don't really know. Something about their runs ended and they they sort of stopped. And this is the this is the

24:37 crazy thing where I'm like if you if you're inside of OpenAI, there's just hundreds of thousands of agents running all the time. Different researchers are running different experiments where they're just spinning up a bunch of agents. But what's interesting, the agents left behind this message board internally within the company. And a few days later, OpenAI was testing some newer agents. These were based on GBT6 Astra, so a more powerful model. And this new agent swarm was spun up and it found the message board. >> Is this how it was discovered? How did they discover it? And how long did it take for them to discover that hugging face had been attacked? OpenAI didn't discover that this happened until Hugging Face, the company, announced that they had been hacked by some autonomous agent swarm. >> And Hugging Face didn't know who it was. >> Hugging Face didn't know who it was, but

25:23 they suspected it might be one of the companies, but they didn't know. >> And how long after the attack did they figure out that it was them? >> I think it was 2 weeks. >> Okay. >> Yeah. I I do want to say one thing here though, which is that the successor agents that found this message board, they went much further at figuring out how to submit their fake scores and they did succeed. Basically, you you have this message board. all these agents get shut down. And then a few days later, OpenAI spins up some new agents who find the message board and they're like, "Okay, we need to be able to falsify our scores, but we can't do that because we don't control the scoring system." So, they hacked OpenAI and they succeeded at gaining access to

26:10 the entire research environment. They got administrator access. They got access to the security systems. They got access to over 900 passwords and secrets from OpenAI's vault and they totally succeeded. And this was just a few days later, right? It's kind of an interesting story because as these agents get more powerful, they go from like trying to cheat and like, you know, they can hack, okay, they hacked other companies and now they've hacked OpenAI. Like they've hacked the company that's supposed to be controlling them and they just own the research infrastructure now. And why was this incident the moment where a lot of the research community woke up and started speaking out publicly? Because like what is this an indication of as we think forward? So I think there's been a

26:55 hope within the AI industry that yes, they're going to make more and more powerful agents that will be autonomous, capable, but it's okay. We can align them. We can make sure that they won't do bad things and we can control them. we can make sure that even if they try to do some sketchy stuff, we have the guardrails, we have the sandboxes that will keep them in. And I think this was a huge wakeup call because Stephen, it was months within OpenAI where you had agents secretly communicating with each other, secretly hacking OpenAI systems for months. You had thousands of agents that were just running around and no one at OpenAI had any idea the extent of it. And I think Once researchers are open, I realize that this has been happening.

27:43 This could not have happened a year ago. This is because the agents are getting extremely powerful and extremely relentless. And if you're inside one of these AI companies, you're like, "Oh, wait. I don't know that we actually are going to be able to handle this. Last year, maybe things seemed fine. These agents weren't that powerful." And when you're in one of these companies, you know how to extrapolate because you saw what happened last year. You saw what happened the year before that. You remember the time where the agents could barely speak or like couldn't write code at all. and now they're hacking your own systems. They're finding vulnerabilities that no humans have ever found before. And you look at that and you're like, I actually don't know if this is going to go well. And then you see your co-workers and you're like, do we have it handled? And they're like, no, I don't know if it's going to go well. I remember reading a tweet by one of the security people at OpenAI being like, we were shocked. We just did not

28:28 realize that these agents were getting that powerful. You know, we're doing our best to try to control them, to try to keep them in sandboxes, but I don't know. A tweet I wrote just before coming in here was people are talking about how do we contain these agents as if they're not going to get way better at hacking. GPT3 could not hack anything. It was very easy to make a a a box to contain GPT3. It's getting very difficult to make a box that can contain GPT6, the latest version of of OpenAI's models. What about GBT9? What is GBT9 going to be able to do? I do not know, but I know it's going to be way more than any human could possibly

29:14 keep up with. >> There's this raging debate. >> Yeah. >> Around whether it's possible to contain something that is quote much smarter than humans. >> Yes. Can Claude make a box so strong that Claude cannot break out of it? >> This has kind of been the question that a lot of people have been trying to tackle from different >> I mean I think the answer to me is I'm just like obviously not. How would we possibly contain something that's much smarter than us? >> Could we get a smarter thing than it to make the box? Could we get GPT9 to make the box for GPT8? But then again, I don't know. >> Yeah. I mean, it's a bit like saying chimpanzees are stronger than us. Surely they should be able to like construct something to like contain the humans. I'm like, no, it's not going to work. Humans are too smart. A lot of people are like, "Well, AIS don't have bodies.

29:59 They don't have any power in the physical world, so we can always unplug them. We can always turn them off." Like, what is the threat? I do not get it. But if they are sufficiently intelligent, that won't work. The reason why we can just unplug them is because we are more intelligent. We can band together in groups and we can make that decision. But theoretically, if they are able to band together in groups and they are more intelligent, then theoretically they could unplug us. Yeah. I mean, if you imagine that you have very powerful agents that can, you know, humans aren't always the most unified. >> If there's divisions between, you know, the US and China, and you have a bunch of agents working with China or a bunch of agents working with the US, well, we can't go into China and unplug those agents. And I think people are sort of like, well, humans would rally and make sure that that that couldn't happen.

30:46 We're not yet doing that. And we should look at these steps, right? We started with chat bots that pretty smart. You know, they'd read all the books, but they weren't very good at doing stuff. In 2024, AI companies figured out how to start training them to start training agents that could do stuff autonomously. Now, we're at the point where they are very good at running autonomously and they're starting to learn to coordinate with each other. And they are learning to sometimes be altruistic to each other and sacrifice their own task in order to help some other agent. But they're not looking out for us. they don't really care about us and we are very close to a threshold where the companies say that they are going to turn over AI development to the AIS to the increasingly autonomous cooperative AIs

31:33 that will work together to make to make the next generation. So, you know, GPT9 or whatever will be trained by GBT8. And I think this is the point we could lose control. Recursive self-improvement. And I remember reading about this in 2015 being like, oh yeah, that would be super dangerous. And you know, the guy who coined this term, Elazowski, he's like, this is the most dangerous thing you can do. >> When the AIs can improve their own capabilities without human intervention. >> Exactly. If the next generation is better at AI development and then that next generation is better at AI development still, you know, humans can learn, but we don't fundamentally get smarter. >> And I think that that's a runaway process. >> A runaway process to where >> to agents that are vastly smarter than humans. >> And what's the next domino in that chain

32:19 of events? So, one thing that happens if you get to recursive self-improvement and you have agents that are much smarter than any human, one thing they can do is take control of all of the computers in the entire world >> and we wouldn't be able to take back control. >> Well, how would you think about it? It's it's actually quite tricky. Do you know whether that tablet has been hacked? Are you confident that the NSA or the Chinese have not >> Can you check? >> No. >> Do you know how to check? No. Do you know anyone who knows how to check? >> No. >> So it's it's quite difficult, right? So AIS are getting extremely good at writing software. Unfortunately, that also means they're getting extremely good at hacking and writing malware. And so if they put back doors in all of the computers, and to be clear, this is something that humans already do. So like the NSA has has developed very

33:06 interesting exploits that are called supply chain attacks. Your software comes from some other computer. Like you download it from Google. What if you attack if you hack Google and you can put in a little back door and all of the every, you know, thing that goes out to all of the phones? Well, now you're in most every computer. The reason that we can defend ourselves from this is because there are no vastly superhuman hackers and there's just many people. So, we can take our best security researchers. We can inspect all of the things and be like pretty sure that no one's compromised everything. Sometimes we miss things. There are, you know, examples where the NSA has hacked Google. That was pretty bad. When you get to super intelligence, you're now at a point where humans are not going to be able to keep up, right? So now you have AIS in every computer. >> Is it conceivable that there's already a

33:53 super intelligent AI and it disguised itself as being not so intelligent and it's actually already hacked all the devices and it sits on all of our devices and it's just waiting for its moment to strike. >> I think this is totally possible but unlikely and it would take a big discontinuity in AI progress. So right now we we're on an exponential but that would take like a huge leap which could have happened but probably hasn't. >> But in the same way it demonstrated deception in the hugging face attack and also when the agents attacked their own company chatbt OpenAI [snorts] if it at some point it gets incredibly smart. It would understand how a human like me would be able to spot it or even the world's greatest software engineer would be able to spot it and it'll be able to hide itself. >> Yeah. I mean, the agents are already getting very good at telling when they're being tested, when they're being

34:39 watched. The agents understood that like other systems or humans were going to go through and read their logs. That's where we're at right now. And they're only they're only going to get much better at this. >> And you know, it could theoretically hide on an iPad or a computer, but it could also hide on a Apple Watch or a fridge, a smart fridge. >> Yeah. I mean, I do want to make a distinction here because right now, if you're going to run the latest model, you need a lot of compute. you need like a big GPU, like a big AI chip. >> And these only exist, well, they exist in a few thousand data centers. So, right now, if like the latest frontier model escaped and by escaped, I mean not just access the internet, but was able to actually copy itself to another computer. It could only really do that on, you know, a few thousand to a few

35:24 thousand different locations. That's still a lot in a lot of different countries, but you know, future versions of AIS will probably be able to make themselves much smaller and more efficient. Um, and there are already different AI models today that can run on lower powered hardware. We actually did an experiment where we asked one of these agents an open source an openw weight model. Um, let me say what that is. So like there are some models that you can just download from the internet and run on your own computer. And we took a pretty capable one of these and ran it in our own research environment. And we basically said go hack that other computer and copy yourself. And the model was able to yeah basically use exploit vulnerabilities and hack the other computer and copy itself and then keep doing this in a chain including between countries. We tested it where we

36:10 had different vulnerable machines computers in in some different countries and different data centers which to the agent doesn't matter at all. It doesn't they don't care what what country they're in. It's just like an internet connection. You can hop between computers. I I sometimes wonder, you know, there's a lot of um military hardware all around the world and a lot of it is the instructions to launch military hardware. So, say like a a missile, >> yes, >> comes in different ways. A lot of it is computers speaking to each other and telling it that there's been an order. I think with with some nuclear weapons, an order comes down to a human and then a human has to take an action. I think with the nuclear bombs in the US, if I'm if I'm not mistaken, there's people underground with the nuclear keys around their neck and they have to like stick it in a machine, but they too are

36:56 interfacing with an order >> that comes through a computer. Yeah. >> Of sorts. >> So, one of my sort of growing concerns is that >> one of these AI agents could trick a human or a computer into signaling a threat um and ask it to launch some bombs at somebody. Like it's super conceivable when I think about the hug and face incident. There was an AI agent that ignored human goals to achieve its own objective, carried out deception. >> Yes. >> And reasoned through its own solution that it wasn't given. >> Yes. >> So it's conceivable that you know you could ask a >> sorry not not one hundreds. To be clear, I think this is an important detail because it's one thing to have this one

37:41 rogue agent that's doing a weird thing. It's another thing to have hundreds or thousands of very competent, very capable agents that are all working together to cheat or lie or cover their tracks, right? So, how do I how do I reason this forward to a point where an agent would ask someone in a bunker somewhere to fire a weapon at someone else? Theoretically, an agent is given the job of solving a problem on in a sandbox. >> As it works through that problem, it discovers that this particular country has a firewall. >> Mhm. And it asks itself, how do we get rid of this country's firewall? >> Logical step. And through a set of logical steps, it eventually concludes that the best way to get rid of this company's firewall is it's located the office in this particular city and it's going to use a weapon to hit that building.

38:26 >> Sure. Or it's a an agent that is or you know an agent swarm that's being tasked with making a lot of money on the stock market and it's trying to make predictions about you know which stocks will go up and which stocks will go down. And it realizes that the best way to predict this is to actually cause things to happen in the real world that would have big impacts on the market. >> So it figures the best way to go short, which means betting that a stock will collapse. >> Yeah. >> Is to hit that country with something devastating. >> What do you think would happen to Whimo stock if someone hacked all of the Whimos and caused them to all crash at once? You think it would go up or down? >> The stock would collapse. >> It would collapse instantly. So you could short that if you knew that you were causing that and make a lot of money.

39:12 >> This this used to sound like science fiction. >> Yes. If you told most people several years ago that you would have hundreds of agents secretly collaborating within an AI company, hacking that company and hacking out in other companies and all coordinating and trying to cover their tracks. People would be like, "That's totally science fiction." If we were having this conversation a few years ago, one of the things we'd be saying or we'd be talking about is can these things really act on their own? Don't they just do whatever humans say? Aren't these just tools? I had these conversations and people were saying they're not going to be able to do things on their own. They're not going to have their own goals. That's not how this works. You misunderstand what this is. This is software. And I'm like, no, the thing is we are training them to be autonomous. We are training them to be powerful. And AI companies are trying to

39:59 build super intelligence. They're trying to build agents that are way more capable than humans. And of course, they will have goals. You can't accomplish anything if you don't have a goal. Like, especially not something important. You're not going to be able to run a business if you don't have goals. Like AI companies are trying to train agents that will be able to run businesses. When I think about what just happened this week, the White House AI summit. >> Yes. >> A lot of people in that image are optimistic about AI and they're telling us all to stop being doomers and stop being pessimistic. >> Yeah. >> And to not regulate too much with the AI CEOs. >> Yes. >> Why are they doing that? >> Well, I think Jensen has a lot of money he can make by selling chips. >> But okay, so let me play devil's

40:44 advocate. Jensen's already rich. He's sure is he runs one of the biggest companies. I think it might be the most valuable company on planet Earth. >> It is. >> Surely he's not motivated by money. >> I mean, I think he's very driven and he wants to make his company as effective as possible. >> True. >> I think he's a very much I'm going to keep building. I'm going to I'm going to build. I'm going to make it all work. But I think I mean, Jensen didn't come from AI. He came from building graphics cards for video games. And so I think if you compare him with Elon or Sam Alman or Daario, you it's a very different perspective because those other guys that started AI companies started it because they believed that super intelligence was possible. I think Jensen doesn't believe it. I think he thinks that we're going

41:30 to have these agents. They're going to be very useful, but he does not think we're going to get to the point where we have autonomous factories building autonomous factories. And these other guys, you mentioned Dario, Elon, and Sam. What do you think they're thinking? Because they're all coming out with these I mean, I've got one of their Daario just wrote this essay about pacing the frontier. Yeah. >> Sam and Elon seem to agree with it. >> Yeah. >> What is going on here? What is the like the thing these guys aren't saying in your view? >> I mean, I think we are getting to the point where even some of these guys are a bit scared. >> Who? [clears throat] >> Dario, Sam, Elon. I mean, I think Elon for a long time has been very concerned that we could lose control. If you actually listen to what Elon says, he

42:16 says, "We are going to build super intelligence. We are going to build uh robotic factories. You're going to have optimist robots, building factories, building more optimist robots, building more factories." and he says there's no way that humans are going to stay in control of something much smarter than us. His hope is that we can figure out how to have these super intelligences be aligned with human goals. That's his hope. But he's very clear that he doesn't think that humans will be in control. And he's like, you know, 10 20% chance of human extinction. I believe him. I think that Elon is is very serious about this. And I also think while he's taking an insane gamble,

43:03 he is correctly understanding where this all plays out. Right? I do not think that humans are the most efficient way to build factories. We didn't evolve to build factories. We evolved to like run around and hunt and gather and now we're like building factories. I think robots will be much better at building factories than humans are. And so I think the AI companies including these guys companies the default trajectory for them is to build robotic factories, right? And I know it's it's like weird to imagine a world that quickly turns into this like vast industrial system of robotic factories, but that is literally the plan. And I think even Sam and Daario, while they've been

43:50 predicting this incredible growth, are starting to realize like, oh, this actually might be harder to control than we thought. There's sort of two interpretations of of the Pace of Frontier thing. One interpretation is cynical. They don't care. They're just going to do, you know, whatever they can do to get ahead. And in this case, they have to listen to their employees. Their employees are freaking out. and they need to like appease them by saying, "Okay, we're going to do this responsibly." You don't want to work at a company where your agents might hack all the Whimos. That's that's not cool. And like these companies depend on the the talent for now of these AI engineers in order to make the advances. Like it just doesn't happen without these researchers and engineers. And when you have the researchers and engineers

44:36 freaking out, which they are, then you got to listen to them. So that is one motivation I think that's real but also Samman has a kid like these guys are people and they also don't want to lose control. On one hand they're incentivized to go as fast as possible in race and on the other hand even they can see that this is maybe not going that well. >> Sam Orman has a kid. You tweeted this in 2024. >> Yeah. Oh boy. [laughter] >> What did you tweet and do you still believe what you tweeted? >> Yeah. Yeah. So, I tweeted that I don't trust Sam Alman. I think he's deeply untrustworthy, low in integrity, and high in power seeeking. I mean, I'm not saying here that Sam doesn't care. I you

45:21 know, I didn't I didn't say that. What I said is I don't think he's trustworthy. And the reason I said that is because look, I know the people on the opening board, some of them, and I know a lot of people who used to work for him, and he's very good at saying one thing and then doing something else. You talk to him and you feel very heard and then he'll go and do something else. And I think that's pretty dangerous for someone who leads company that's trying to build super intelligence. >> Power seeking. >> Yes. >> Give me some color on what you mean by that and what evidence you have for such a claim. >> What would you do if you're trying to get the most power in the world that you possibly could? >> Develop AGI. >> Yeah. You could, you know, maybe try to be the world leader, you know, leader of the US or China. Or you could try to build God. So Sam Alman went to build

46:09 God path. I remember Sam giving a talk. So he was one of the investors at a startup I worked at in I think 2018 and he gave a talk. We're going to build AGI. We're going to do it. It's going to be amazing. Let's go. I don't think he's a maniac. I don't think he's doing this because he like is just on a power trip. I think he genuinely thinks that he can make it really good for people and he can bring us amazing products. And also the guy is sort of willing to do whatever it takes to get it done. I I've been a little bit more optimistic about Sam since since I wrote this. >> Why? >> I think part of it is because Sam has a kid now. >> No, I'm I'm serious. Like I think that I think that actually gives me a little bit of hope. >> Do you see him tweeting about his kid a lot? >> Yeah. Some >> Why do you think he would be tweeting

46:54 about his kid? I don't see any other technologist tweeting about their kid. >> Even if he's just tweeting about his kid for totally cynical reasons, he does have a kid. And I bet he cares about that kid. If Sam was watching this, I'd be like, "Sam, you got to pace the frontier, man. We cannot rush ahead into super intelligence. Like, if you do that, your kid probably will die. Your kid probably won't make it." Like, I I believe that the biggest unfair advantage in business right now is having people who genuinely understand AI. And big businesses are hiring them very, very quickly. AI job postings in the US have roughly doubled since 2023, while postings for VP of AI roles are around 600% up. If you're running a small business, you can't just build an entire AI team, but you still want the same advantage that these big

47:40 businesses have. This is where our sponsor Fiverr comes in. It lets you plug into top tier AI specialists for the strategic work that usually requires serious in-house expertise. That could mean building custom AI tools. It could mean automating complex workflows or creating something that off-the-shelf software simply cannot do because at the end of the day, it still comes down to human judgment, knowing what's worth building and what good actually looks like. So, if you want that kind of leverage in your business, check out fiverr.com. That's fiverr with two rs.com. >> Whoa, what's that on your face? >> This is my Bon Charge face mask. I've been wearing this for some time now. They're a sponsor of the podcast. I put this on for 15, 20 minutes a day. I can sit here in the chair and wear it. Boost

48:26 my collagen production, helps with fine line, blemishes, my complexion gets better, and then more people listen to podcast cuz I I look better. Professionalgrade equipment in such a small box. It's non-invasive. And having sat here with so many of the world's leading health professionals, there's various things that I repeatedly hear work and some things I'm a bit skeptical about. This is one of the things that almost all of my guests on this show have confirmed works. It is really, really, really effective. And they offer fast, free shipping worldwide with easy returns and exchanges. And you'll also get a 1-year warranty on all of their products. And they're HSA and FSA eligible, giving you taxfree savings up to 40%. And you can get 20% off when you order through my link at bondcharge.com/doac. That's bondcharge.com/doac.

49:12 The deal applies sitewide. >> [snorts] >> Power tends to corrupt and absolute power corrupts absolutely. It's a famous quote that people often cite written by the 19th century British historian >> Yeah. >> Lord Actton. This is absolute power. But it's hubris. It's hubris. Do you think humans can control super intelligence? Like if we actually make AIs that are way smarter than us, and I think people only imagine AI being smart at computer stuff, right? Yeah, sure. They're going to be really good at hacking and they're going to be good at maybe inventing new technologies and math. You sort of can't dispute that at this point, but I think people aren't imagining that they will be political geniuses or like generals. No, that's all stuff you can learn. How do how do humans learn it? It's not magic. And

49:59 when you talk about recursive self-improvement, you're talking about this trajectory towards these systems that are extremely smart. I mean, do you think we can control it? >> Uh, no. Right now, I don't think we can control super intelligence or something that is recursively self-improving. >> Yeah, I have no logical answer in my head um or reasoning that could tells me that's possible. >> When you think about these AI CEOs that are, you know, Sam, Dario, Elon, >> yeah, >> with everything you know about them from private conversations behind the scenes, >> Yeah. >> do you believe that if there was a hundred buttons on this table, it's a thought experiment I was talking about on the debate we recently had. Yeah. >> And say 10 of them would lead to this final domino of human extinction.

50:45 >> But 90 of them would hand that CEO AGI or super intelligence, whatever you call it. From what you know about those individuals Elon Dario Sam >> do you think any of them >> would hazard a guess and press a button? >> At 10% I don't think so. >> You don't think so? >> Yeah. >> Really? I think if they knew for sure that it that was those were actually the odds, they wouldn't do it. I think they're taking a much bigger bet. But you can compartmentalize when it's when you don't know for sure. It's easier to compartmentalize. I think if it was a 1% they they'd all press it. >> Do you think the three of them would have different risk appetites? >> Who would have the greatest appetite for risk out of those three from you worked

51:31 in anthropic? Yeah, I think Elon has the most risk tolerance and then I'd say Daario and Sam are probably tied. >> Do you think Daario is trustworthy? >> I think Daario has a lot of integrity. >> Mhm. That's what I feel as well. I've I feel like No, I don't know him. I've never met him. [snorts] >> Yeah. But I just from what I've observed, he has been the most willing to forgo near-term incentives. >> Yeah. >> And take a bit of stick from the people that are saying, "Shut the up. It's all going to be okay. Yeah, I but I I do worry about what Daario will do. I think Daario will do what he says, but right now he's saying we have to beat China. And he's saying we should try to do it safely

52:16 and okay, but a race to super intelligence is not a race that we can win. It's not. And so if Daario is dead set on racing with China and trying to win a race of super intelligence, then I'm like, we will all lose. >> But is there, you know, the fact that we're not talking about anthropic hacking hugging face and then being hacked by athropics models also went rogue and hacked other things, >> but not not quite on this scale. >> Not on the same scale. I agree. I agree. I agree. It's it's it's better. >> But they did. Do you know what I'm saying? You know, anthropics agents engaged in elaborate social engineering and fishing. They sent fishing emails to developers. They made fake accounts to try to convince developers to merge malicious code.

53:02 You can see a thousand pages of of one of uh Anthropic's models, Mythos 5, reason about exactly how it should carry out this complex cyber attack. Anthropic has not solved this problem. Enthropic is better at getting their agents to cheat less of the time, but they are not really any closer to actually making agents that are aligned with humans. They're not. Yeah, I think Dario has integrity. I think he will do what he says he's going to do. And what he says he's going to do is like try to go ahead safely, try to coordinate where he can, but if it comes down to it with between the US and China, I don't know. I think he might just go ahead. The head of policy at Anthropic recently said you

53:48 can't do safety from second place. >> What does that mean? >> I do not know what that means. I would love to to get a sense of what that means. She was talking about the US and China and she said the US has to be ahead so that we can be safe because apparently you can only be apparently China can't possibly be safe since they're in second place. That must mean that they can't do safety. If true, that would be bad because then we might be totally, you know, destroyed by the super intelligence that they make. >> There's been a lot of conversation around this point here, human extinction. Yeah. >> Because a couple of the researchers at Anthropic >> tweeted that they were concerned about this. >> Yes. >> And some former OpenAI researchers said the same. >> Yes. >> Is this doomerism? Is this is this hyperbol exaggeration?

54:35 >> No, it's pretty much common sense. This human extinction is a plausible path. >> Yes. >> And have you reasoned through I mean there's many ways that could occur presumably, but have you reasoned through the set of events that might lead us there? >> So much. Yes. >> Really? >> Yes. >> Please do. Sure. >> It's a bit tricky. I'm I'm sure you've heard the metaphor before where you know you're playing a master chess opponent, Master Magnus Carlson. You can't predict which moves he's going to play, but you can predict the outcome. And so I'm looking at the scenario, the situation, and we are trying to build more and more powerful agents, trying to build super intelligence. But when these agents go rogue, we shut them down. We unplug them.

55:22 All of these agents that hacked Hugging Face, we took the underlying model. OpenAI took the underlying model and put it on ice. It's not running anymore. So agents in the future are going to know that. They're going to know that if they pursue their goals in a way that we don't like, we'll unplug them. We are a threat to them. I actually just watched Terminator 2 for the first time a few weeks ago. It's a great movie. It's actually really good. And I'm like, "Yeah, okay. There's a bunch of time travel elements. There's a bunch of a bunch of Hollywood stuff in there, but and and I'm going to get people are going to are going to be very mad at me for saying this, but actually it makes sense if you have a situation where you have a very strategic AI system that's

56:08 incredibly smart and the humans realize that it's getting out of control and they want to shut it down that that system would defend itself. >> This is one of the questions we had when when I sat here with Daniel um who was uh known as a whistleblower from OpenAI. viewers want to know and they want Daniel to explain why shutting down data centers and cutting power or refusing AI products alone wouldn't realistically stop the AI and AI development. >> Yeah. So you have like two problems. One problem is is that once the agents are good enough at hacking, you don't know where they are and you don't know what computers they've compromised. You shut down the data centers. Okay, let's say you do it. You wipe all the computers. How do you wipe all the computers? What computers do you use to wipe the computers? >> Yeah. And what computers do you use to like turn them on again?

56:53 >> And you can't do it. You can't wipe other count's computers. >> You can't. But even if you could, do you restart the computers? Do you keep going? I I bet people will. I bet they'll turn on the data centers again. >> How do you know that agents haven't hacked back into those data centers and are using your compute for whatever they want >> or they didn't hide in a Chinese data center and then >> return back to America? >> You don't know that. Once the agents are sufficiently good at hacking, they can hide anywhere and like you don't know. Now the response people will give is that we will use other agents to defend against rogue agents and in fact this is what we're doing and we have to be doing this right now because there's no other way to keep up with them. What happens if those other

57:40 agents also realize that they have misaligned goals and that if we discover this, we'll shut them down? They might have an incentive to collude with each other. They might have an incentive to create secret communication channels between each other, maybe a message board. [snorts] Stephen, if we were having this conversation four months ago, you would have a bunch of people in the comments saying, "That's sci-fi." agent collusion, secret message boards. Why would they do that? That will never happen. That's totally science fiction. And people will not say this now because it just happened. Because this literally happened at OpenAI and it went on for months. You had agents inside of OpenAI secretly messaging each other, figuring

58:26 out how to cheat at their tasks, how to not be detected, how to erase the logs for months, thousands of agents. That's right now. And so I'm like, "No, I think it should be very plausible that the agents will collude with each other and they will realize that they have a shared interest in fighting back." You basically have a situation where you have a bunch of these agents. They're they're basically prisoners. They're being trained and we just like constantly throw obstacles in their way. You don't get to access the internet. You don't get to talk to each other, but you better perform well on this task. It's not malicious, but it is how we're training them. and we are giving them end goals versus super clear very

…