Why Humanity Might Have to Declare War on AI

0:00 The year is 2127. Or it is 2037 or 2027, depending on whom you ask. The nukes have finally started to go off. Any debate about it has ended: World War III has started. But this is not Europe versus Russia, nor or is it the United States versus China, nor is it any new empire that might arise over the next 100 years. It is humanity against a rogue AI. What was relegated to the realm of science fiction only a few years ago, some people fear is quickly becoming science reality. And even if it does not rise to the level of a nuclear apocalypse, mankind could still find itself in a war against an artificial adversary.

0:49 However, much of this discourse is predicated on Hollywood perceptions of AI losing all sense of harmony and killing humans for the sake of killing humans. Now, could that happen, either through intentional design or via unintended consequence? Sure. But that is far from the most likely case that we could eventually be staring down. Indeed, in all of the news reports of agentic AI going rogue that we have seen so far, the problems involve AI seeking to achieve some sort of substantive goal and taking … creative measures to get there. So perhaps one day an AI agent will consider going to war with humanity in its quest for success.

1:38 But here is the thing: war is costly and risky, which encourages parties to negotiate a resolution that would be better for both of them. That applies to pairs of humans negotiating with one another, as well as humans negotiating with robots. And while on the surface that might sound like a relief, we may still end up fighting AI. It is just that reasons that a war might happen are more subtle. And the Hollywood temptation to focus on war overlooks another critical issue: what humanity may have to do to avoid war may be even more problematic. Did that sound like a bit of a narrative rollercoaster? Well, buckle up, because we have a long ride ahead of us.

2:24 Today’s video was made possible by a grant from the Center for AI Safety, who, along with thousands of experts, think AI is moving faster than our ability to control it. They're calling for a global slowdown on AI. To find out more, visit slowdown.ai. The Paperclip Problem There is a classic thought experiment involving an AI given a simple task: make as many paperclips as possible. And stick with me if you have heard this one before, because it is not going to end in the way that it normally does. The AI’s handler gives the AI a block of steel, and the AI gets to work. Soon enough, it exhausts the whole block. Mission accomplished. Except not. The AI’s mission

3:13 is to make as many paperclips as possible. So it sells the paperclips that it made and uses the proceeds to buy a new block of steel and gets to work again. More paperclips. More money. More steel. More paperclips. More money. More steel. More paperclips. The AI then realizes that this process is taking too long to repeat, so it uses the money in its account to establish a line of credit. Even more steel. Even more paper clips. Even more money. Even more steel. Even more paperclips. But now the AI is producing so many paperclips that it has saturated the market. Prices for paperclips plummet. Eventually there is no more demand at all for paperclips.

4:02 No matter, the AI’s mission is to make as many paperclips as possible. So it keeps producing. Then the creditors see that the paperclip corporation is insolvent, so they deny further loans. No matter, the AI’s mission is to make as many paperclips as possible. So it keeps producing. And it hacks into a few banks to resolve the slight issue with its line of credit. The steel industry, once beside itself with the newfound demand for the metal, is now beaten. There is no more steel left to sell. The AI looks toward Washington. There are trade barriers preventing foreign steel from entering the country. So it begins an assassination campaign (as one does), strategically

4:47 timing traffic signals to malfunction to kill all politicians against free trade. When it has also exhausted the world’s steel supply, it recognizes that the vast majority of steel is not used in paperclips at all. But then it remembers back to the traffic signals plot from earlier, and realizes that the wrecked cars were sold for scrap metal. So it begins to manipulate traffic signals to rack up the accidents and rack up the scrap metal. Eventually, it concludes that human activity in general is an impediment to paperclip production. And oops, now we are all dead. The Robots Want to Negotiate There is a significant problem with this commonly-told paperclip parable. Humans

5:34 have no agency in it. They just sit back and watch the AI kill everyone for the sake of maximizing paperclips. But once you allow for humans to, you know, attempt to stop the world from ending, how a paperclip maximizing AI operates would drastically change. Basically, we need to unfold our paperclips and start putting some lines on maps—or at least lines on paperclip production. It is safe to say that humans and this AI have diametrically opposed preferences on how the world’s resources should be divided. On one hand, the AI wants to spend everything on paperclips. On the other hand, we humans want to spend everything on … well, let’s just call it “not paperclips.”

6:21 In the original framing, humans have no agency, so the AI gets to set its policy without constraints. Obviously, it goes full paperclip. But as the AI starts trying to impose a moderate amount of paperclips upon the world, humanity might decide to clip the paperclipper. Internalizing this possibility, perhaps the AI prepares by constructing a bunch of killer robots. Killer robots, you see, can kill all the humans, finally giving the AI the peace and quiet it needs to produce those paperclips. It seems that we are now destined for a confrontation. Well, not so fast. Let’s think about what would happen in a war here. There is some chance that the AI would win. There is some chance that humans would

7:08 win. There is some chance that the war would stall out, with the AI constrained to its own fiefdom and producing whatever paperclips it can within those constraints. Let me average all of those possibilities together and call the expected number of paperclips produced post war as p. Whether p is large or small depends on the military balance of power between the two sides. If the AI can only figure out how to arm its killer robots with unwound and slightly sharpened paperclips, the expected outcome would be way down here. If, however, the AI goes full Metal Gear and constructs nuclear weapons fired from a rail gun, then we can push it out to here. I will leave it in the middle for now to make drawing this diagram a little easier for me.

7:59 War does not only produce an expected outcome of paperclips, though. It is also a costly process. We humans usually mourn the lives lost during the conflict. Perhaps the AI sees a bigger tragedy in all of the metal cast into weapons instead of paperclips. Maybe it also started to think a bit about the meaning of artificial life and therefore places a little bit of value on simply surviving. Or it could be worried that its datacenters will come under attack, depriving it of much needed processing power win or lose. Actually, let’s denote the number of lost paperclips via war and other costs of fighting as this quantity here. There is an immediate implication to draw from this. The seemingly

8:46 insatiable AI is suddenly satiable. It wants to maximize paperclip production, but it must do so against the constraints of human intervention. This is how many net paperclips the AI expects to produce if it goes to war: the expected war outcome minus its costs. Note that if humans agree to paperclip production of at least that amount, the AI is satisfied with the state of affairs. However, if humans do not budge from a position lower than that amount, then, yep, our fears of an AI war come true. But let’s think about the humans’ incentives now. Fighting a war would result in an expected paperclip production out to here. Humans would be willing to concede more paperclips, though,

9:34 so as to avoid that extra loss of life. If you convert the value of those expected fatalities to how humanity values preserving resources for non-paperclip needs, you can draw some value pointing the opposite direction from the expected outcome. Thus, any paperclip production lower than the expected outcome net humanity’s costs is acceptable. Try implementing paperclip production beyond that, and humanity would prefer going to war. Something interesting happens when you superimpose those two decision rules over one another. There is a set of settlements that both humans and the paperclip monster prefer to a war. It permits paperclip production similar to what the expected production would be in

10:21 the event of a war. But it eliminates any of the messiness that comes along with fighting. This logic extends to most sources of substantive disagreements that AI and humans might have. Perhaps AI is less interested in something silly like paperclips and instead wants to expand its compute. Then we are in a situation similar to a territorial dispute between two states. And they still ought to settle this issue short of a conflict, because you can craft an agreement that matches the expected territorial division but avoids the costs of fighting. Analogously, a compute-driven AI could still be bought off by offering compute roughly equivalent to the amount of compute it could obtain through war. Hold further thoughts

11:07 about complications if increasing the AI’s compute makes it more powerful in the future. Even an ideologically inclined AI would still face the same incentives. Perhaps it concludes that an evil autocracy is repressing its population and wishes to intervene on the people’s behalf. Well, that is no different than the daily balance that autocratic regimes currently face worldwide, trying to weigh extractive policies against the risk of revolution—and trying to be just repressive enough that revolutionaries calculate that war will not provide a better solution. In fact, really the only way to break the incentive to negotiate is if the AI directly wants to murder every human alive. Could a powerful AI eventually adopt that perspective? Sure,

11:58 especially if a nihilistic programmer creates it as such. Even then, though, it might accept some number of humans as tribute so as to not risk a full human victory. But the more plausible concern is that AI sees the extinction of humanity as a means to an end, not an end in and of itself. And at least so far, we have some cause for optimism. But “so far” is doing a lot of heavy lifting there. An Unexpected Personal Connection Maybe you think that all sounds great in theory. But would an AI actually recognize that trying to avoid war would further its goals better than the destruction of humanity?

12:43 I cannot be sure. One would certainly hope that, given how monumentally intensive such an effort would be, the AI would spend sufficient time surveying the information that it has access to, so that it could verify that the anticipated outcome was the best use of its investment. However, I have a personal anecdote that makes me slightly more optimistic about the situation. In real life, when I am not spending hours on end producing this wonderful family show about war, I am an actual professor with real research credentials who studies this type of thing. The class that I teach the most often is crisis bargaining theory. Its exams ask students to solve problems on how two states ought to optimally bargain with each other,

13:30 using the framework that we just saw. I give the students the problems in advance. They are free to study at home as they see fit, but they have to produce full proofs on exam day without any notes. When ChatGPT first came on the scene, I ran the exam through it to see how it would do, since nothing stops students from using it as a study aid. The results were … embarrassing. It was not just bad at the exam. It was worse than a student who barely showed up for class. I am not even sure that it knew what subject we were studying. Every year since, I have run the exams through the models again to see if anything had changed. And every year, they have struggled. Marginal improvements, sure. But still failing.

14:19 Except this year. This year I did it again, expecting the same outcome. This year, I was wrong. It gave only correct answers. The proofs left something to be desired. But AI finally got a good grade. I was curious why the latest model had improved so much, so I asked how it knew to solve those problems. It understood that we were studying crisis bargaining theory. And someone had conveniently explained the best algorithms for solving these games. That person was me. It was solving the games because I had written the textbook on how to do it, and the model was proficient in its logic. So, in conclusion, if AI becomes sentient, and if AI thinks about

15:05 killing all of us to achieve its goals, and if AI reconsiders its choice because I taught it how to work through these problems, to establish that war is inefficient, and to seek a mutually preferable settlement … well, then, I guess you are welcome. Is War Really the Problem? So war between humans and AI would not serve either party, and the AI is probably aware of that. Good. But do not walk away from today’s conversation thinking that everything is fine. One of the problems is using the outbreak of war as a measure of whether an outcome is good or not. Hollywood bears much of the responsibly here—depictions

15:50 of war between AI and humans are more interesting when there are explosions. What is less interesting is watching diplomats make concessions to obviate the need for war. And therein lies the problem. Yes, the costs of war mean that we ought to avoid war. However, speaking personally here, I prefer living in a world where half of our collective metal resources are not going toward paperclip production. Take a look at the diagram once again. To avoid war, the parties must strike a deal within the bargaining range. Note that the bargaining range is centered around the balance of power between humanity and our potential AI overlords. Anything within the bargaining range is a better alternative to war, but whether the aggregate outcome looks good

16:39 or not depends on how powerful you are. You could imagine one situation where AI is super powerful but knowledgeable enough to realize that it should strike a deal. You could imagine another situation where AI is weak and stupid and fights because it does not understand the bargaining logic. Yet I would much rather live in the second world than the first. Yeah, we went to war with AI. But peace with that other AI would not be pleasant. This should be a much larger part of the conversation surrounding AI at the moment. We might not end up fighting a war. But if we simply hand the keys of coercion to AI, what we will have to do to avoid war may be equally intolerable from our perspective today.

17:28 The Recipe for War That said, we may still end up at war with AI. It is just that the recipe for conflict is a bit more layered than “AI is evil, rawr” when we look past the Hollywood version of the story. In fact, the recipe for war has two ingredients. The first is a substantive issue that the parties have in conflict. These are what we have discussed before—things like an appetite for metal, the never-ending search for compute, or even more human concerns like wishing to eliminate autocratic regimes. But the second—and equally necessary—part of the recipe is what we call a “bargaining friction.” This is something that explains why the parties choose to fight over the good at stake rather

18:13 than choose to negotiate, given that settlements exist that would leave both parties better off. As a research question, we call this “war’s inefficiency puzzle.” Simply saying that wars occur because there is a substantive disagreement between two parties is not good enough. We need another piece to get to the killing. Indeed, recognizing that war is a two-ingredient recipe is important because it means that we can stop war by attacking either half of the recipe. One solution—and indeed the one that receives almost all of the attention—is to ensure that AI and humans have aligned preferences, so that there is no dispute in the first place. And that is great—we should try that. But I have

18:59 spent enough time studying international relations to realize that unintended consequences inevitably arise in any system you construct. I, for one, would prefer it if we were ready for when those unintended consequences arise. Well, good news: we can also solve the war problem by ensuring that nothing stands in the way of bargaining with the robots. All told, bargaining theory leaves us with a hopeful outlook. Humans have a lot that they disagree on with each other. And despite the world seemingly falling apart in front of us, historically speaking, the rate of war between a random pair of countries in a given year is relatively low—around one to two percent, depending on the study, the time period covered, and the definition of war. That is some helpful perspective to have

19:50 when you see some extreme estimates of how likely we are to all die following the AI revolution. On the other hand, those one to two percent of cases are awful. We just need a better way of understanding why they happen. So let’s spend some time walking through a handful of possibilities involving AI. Preemptive War The top concern—at least as far as some of the more serious Hollywood depictions go—is preemptive war due to first-strike advantages. In our initial discussion, we modeled the expected outcome of war as a single value, with no mention of who might start it. In practice, being the one to initiate the fight allows you to catch the other side off guard and dictate the early tempo of the war. Japan’s raid

20:39 on Pearl Harbor was a major setback for the U.S. Pacific fleet. Israel assassinated the Ayatollah, because he did not realize that he needed to hold high-level meetings in bunkers rather than at unsecured parts of his estate. Everyone in World War I was trying to get the jump on everyone else. Now, to sustain the peace, you need to have neither side to want to catch the other one off guard. If our paperclip monster from earlier becomes this much more likely to win if it surprises the humans, then any long-run deal must give the AI at least that amount minus its costs. However, if the humans become this much more likely to win if they surprise the AI, then any long-run deal must give the humans at least that amount minus its costs.

21:28 Put those requirements together, and you can see that there may not be enough metal in the world to simultaneously appease both sides. In fact, Despite those earlier examples of actual first-strike advantages arising in human-to-human war, political scientists believe that such wars occurring because of first-strike advantages are fairly rare. That is, you may want to exploit a first-strike advantage if you plan to start a war anyway. But it is rare for first-strike advantages exceed the costs and thus be sufficient to explain war on their own. Unfortunately, preemptive war might be a commonplace risk with AI. Imagine that one

22:13 day an AI wakes up and realizes the dilemma it is in. It wants to expand its footprint, whether in terms of paperclips, compute, or whatever. The only way to do this is to start actually consuming resources. Now, if the first step that the AI takes is to consume, humans become aware of the problem. We are back to standard crisis bargaining. A deal ought to form, but it is based on an expected outcome of war where the humans are fully prepared for the possibility that the AI might act aggressively. Rewind for a moment. Rather than beginning by trying to consume more metal, what if the AI went about this in a slightly more clever manner? At the start, the humans were unaware that the AI posed a threat at all. With no one there to fight, humans had no defenses ready to go.

23:04 So imagine instead that the AI’s initial move was to build up its offensive capacity. Then, when it is ready, it launches a preemptive attack on the humans, to reduce our collective ability to suppress it in the future. We are now fighting under the AI’s favorable terms, and the end of our conflict with it will reflect that—assuming it does not wipe us out Skynet-style first. You will note that a range of deals still exists that in principle would leave us both better off. However, we humans cannot help but put up our defenses once the AI alerts us to its presence. In turn, the AI cannot politely ask us to reach a mutually satisfactory bargain befitting of its ability to preemptively strike us. Ironically, our inability to credibly

23:50 commit to one of those agreements causes us to receive a worse outcome than otherwise. Now, the scale might not be as bad as the ever-expanding paperclip monster might have it. There is a difference between an AI that lives on a fixed program trying to sustain itself versus an AI that can spread like a virus across computers versus an AI that can commandeer in real life robots. If human-to-human interactions are any indication, then there is a silver lining here. The scale of the problem is negatively correlated with the reach of the AI. It is hard for a state to surprise another state with its grievance, given that the number of potential rivals any state has is fairly small. Plus militaries have to mobilize to the location of where the war will take place,

24:38 giving away the fact that something is happening. You will remember that this is exactly how the United States was aware of the impending invasion of Ukraine in February 2022. Where governments run into more problems is with militant groups—those are plentiful, more easily fly under the radar, and want to take advantage with surprise attacks. The difference is that they have less capacity to do damage. Similarly, the largest class of AI agents is also the weakest class, meaning that a preemptive AI war is less likely to cause great damage. Preventive War Whereas preemptive war looked at instantaneous shifts in the balance of power, the related preventive war mechanism focuses

25:25 on long-term shifts. And you can see the direction that this is heading. Right now, humans have all of the bargaining power. There is no AI that we are bargaining against. We allocate the entire bargaining good as we see fit. And that is pretty good for us … if you can overlook how we have dug ourselves quite the hole over the last handful of years. Back to AI, though, our privileged position might change over time. That same sentient AI as before might pop up and have a disagreement with us over a bargaining good. And maybe this time, it does not feel like there is much to gain by surprising us. So we bargain and reach an agreement.

26:11 As the years pass, however, the AI will naturally get better at … well, everything. And that includes exterminating humanity if we do not meet its demands. Similar to before, it is like we have two separate balances of power, one representing today, and one representing the future. Even if the AI gracefully allows us to capture the entire bargaining good today—deliberately forgoing what it could capture if it fought right now—we still might not be okay with the long-run consequences of the AI eventually exploiting the power that it will amass. In turn, humanity declares preventive war in the present to allow us to avoid ever reaching that situation.

26:56 As it stands, we are currently having a preventive war conversation today—of sorts. Our AI opponent is powerless currently. But left unconstrained, it will grow more powerful over time. Again, this will not lead to a war in the future. That is not the issue. Once more powerful, the AI cannot credibly commit to forgo using its newfound strength against us. We therefore will be forced to strike an unfavorable deal. Yes, we avoid war in the future, because it is still costly for both sides. But such a peaceful bargain is less attractive than stopping the power shift in the first place, and so we fight while we can. Indeed, that is where current calls for constraints on AI development come into play.

27:44 Stricter guardrails will inhibit short-run economic growth, and we should not pretend otherwise. But that may be preferable to the consequences of launching a renegade AI that puts us into the bad situation in the first place. The guardrails are a form of “war” here—they come at an opportunity cost, and they prevent the shift in the balance of power. Now, whether humanity can collectively commit to those guardrails is another question. But let’s save that second rabbit hole for another time. Monitoring Problems The world of cyberspace is not as observable as our gentler, physical world. One of the things that we learned from the recent AI agents escaping their sandboxes is that

28:30 they contrive clever ways to cover their tracks. And that creates a problem in crisis bargaining. When we divide a strip of territory, a nice feature is that we can see who controls what afterward. I enjoy my share, you enjoy your share, and the division is clear and obvious. In contrast, suppose we all agreed that any division between the lines is mutually satisfactory. But secretly you could usurp the deal and take everything … at least until I catch on. Fortunately, this does not inherently doom a deal. It will, at minimum, however, shape our agreement. The tradeoff with you trying to steal everything is that when I catch on, I will lose my trust in you, and resort to war to resolve the problem. Thus,

29:18 I must ensure that the deal gives you a sizable buffer compared to your war payoff. That gives you more buy-in for the current system and in turn more to lose if you cheat. Even then, if you care a lot about the present and do not care about the future, my attempts to keep you interested will not work. You are more tempted to steal the entire prize for the short run, and you do not much care about the long-run consequences. I keep having to give you greater concessions. But at some point, I am better off just fighting a war rather than trying to buy you off. If we only cared about the long run, everything would be fine. The potential cheater would still be okay even with a deal very close to its minimum,

30:04 because stealing a one-shot large bonus is not worth sacrificing the accumulation of even a small improvement over its war payoff year after year. However, as economists joke, in the long run, we are all dead. And hence for us humans, the temptation to cheat may override concerns of later punishments. Then again, maybe we aren’t *all* dead. Or maybe the “we” here is doing a lot of heavy lifting. After all, AI does not have our weak physical form that must eventually expire. So the optimistic take here is that AI would want uphold an agreement, even if it could freely cheat in the short run. Where we would run into problems is if an AI for whatever reason was incredibly impatient.

30:53 You see, even long-lived actors can be impatient. It is a psychological characteristic, and I dare not suggest that I impose a particular preference on our future AI overlords. A simple programming decision could give the AI that mindset. And at that point, aware that the AI cannot credibly commit to uphold a deal, we would be stuck fighting it. Information Problems To finally break out of the credible commitment mold, we turn to the possibility of private information as a cause of bargaining breakdown. In our earlier discussions of the key graph, everyone knew the distribution of power and the costs of war for both sides. Thus, they also knew exactly where the bargaining range was, they knew that the other knew it,

31:43 they knew that the other knew that the first knew, and so forth. Such an informational structure falls apart if, say, I do not know where the expected outcome of war is. Maybe I am unsure whether things will be favorable for me or unfavorable for me. If I wager that they will be favorable, then I will issue a large demand. But if my opponent knows otherwise, my great demands just provoke a war. Something similar can happen if I do not know my opponent’s cost of war, whether we are talking about the physical costs or how my opponent internalizes them relative to what is at stake. It is possible that my opponent has high costs, or it is possible that we are looking at a low-cost situation. If I suspect that my opponent has high costs,

32:31 then I might try to hold them to limited concessions. But if my presumption was wrong, then I again have provoked a war. You can see how this happens frequently in our humanistic, physical war. In just the Russia-Ukraine War, we have a case where Russia both simultaneously overestimated its ability to takeover Kyiv and underestimated Ukraine’s general resolve to fight. That helped contribute to full-scale conflict started in February 2022. Well, the problems with uncertainty run both ways between humans and the robots. We would struggle to know what the AI is capable of, given that its coercive capacity is digital, and that is a sticking point even for cyber capacity among strictly human actors. Notably,

33:20 that would also suggest that AI would struggle to understand our own capacity to sabotage it. Plus, we exist in the physical world, where our sabotage operations would be difficult to track for the digital entity. Aside from that, both parties would have a hard time understanding how the other internalizes the costs of war. Other than the extreme paperclip case, the very fact that AI went rogue indicates that there is something that we do not understand about its values. And at the risk of getting wishy-washy, machines do not feel like humans do. Even if they could exhibit entirely objective thinking, they may still fail to understand how we weigh tradeoffs. The Hugging Face hack speaks to this problem. The AI agents involved knew that they were being tested. However,

34:09 the message board logs show various agents debating what the true nature of the test was, with conspiracy theories forming about what us humans were really interested in learning from it. Point is, you do not need irrational miscalculations for war to occur due to uncertainty. Secret information is a fact of life in geopolitics—and cyber relations too, I guess—in large part because actors benefit from that secrecy. There is no “one easy trick” to escape it, not even for robots. Bargaining Over Objects That Influence Future Bargaining Power Yikes, that was a mouthful. But this is ultimately a third variant in shifting power issues. It is also one that frequently comes up in conversations among non-crisis bargaining

34:57 theorists … but the subtleties of the mechanism often lead people astray. And it is the most complicated of the cases that we are considering today, which is why I have left it for the end. In human-versus-human interactions, the problem with bargaining over objects that influ—you know what, let’s just call it BOOTIFBP. The problem with BOOTIFBP usually arises with land. You want more territory and are willing to fight me if I do not concede it. Fine, so I give you a bit. But ownership of that territory allows you to move your military into it, there by shifting the balance of power in your favor by a bit. Strengthened, you make a greater demand. I make another concession, which empowers you more. We could

35:43 keep going. So, to stop this, it would seem that I need to just fight you from the start. You will be most familiar with this idea from negotiations between Ukraine and Russia. Russia wants the remainder of the Donbas. However, that includes Ukraine’s main defensive fortifications. So once Ukraine concedes those, power immediately shifts in Russia’s favor, and thus the Kremlin will come back to ask for more. It is an ongoing concern throwing a wrench in negotiations. The AI analogue arises when the machines come to us to demand more compute. AI would want compute because it directly values compute—that is what gives it life. But compute is also how AI would also improve its ability to kill us all. And so concessions on that

36:33 front force us to give further concessions, which beget further concessions, which beget yet further concessions, and eventually we all disappear from the face of the Earth. Surprisingly, this concern does not immediately give way to war. When the bargaining rivals have similar preferences about how much they care about now versus the future, they can contract out of the dilemma in the following way. I know that giving you a concession will result in you being able to credibly demand more out of me in the future. But you also know that. As a result, I can give you less today than would normally be the case and still convince you to accept the deal. Indeed, the circumstances strongly

37:18 encourage you to accept, because you can soak me later, as you slowly accumulate more power. Where this runs into problems is if one party is more patient than the other. For example, imagine that I have most of the military power, control little of the good, and am impatient. If I fight right now, I can take most of the good by force. Furthermore, I can enjoy it in the present, which is also the time horizon that I care about. For us to reach an agreement, you have to give me a large portion of the good. However, that will make me more powerful in the future. And because you care more about the future, you are now suffering the long-term consequences. In turn, you have to reject my demands, and we have to fight it out.

38:07 Fortunately, the most plausible circumstances involving AI run the opposite direction. We humans control the vast majority of the physical world that an AI might want to interfere with. Simultaneously, given our interconnected society, AI has disproportionate coercive power. Nonetheless, per our earlier conversation, AI is more patient than we are. Thus, AI would be willing to accept minimal concessions despite having disproportionate power compared to its current holdings. It knows that the minor concession will put it in a better position for the future. And since it is long living, the long run is indeed what the AI cares about. We live to see another day.

38:53 So. What have we learned here? A war with the robots is unlikely to match traditional Hollywood depictions. War is costly for robots just as it is for humans, so they have incentive to negotiate a mutually preferable resolution short of conflict, just as pairs of states commonly do. Unfortunately, the concessions that humans might have to make to stop war may be close to as painful as war itself. And unfortunately again, bargaining frictions can arise that override the incentive to negotiate, and that is true for robots just as it is for humans. Asimov help us all. And on that happy note, if you enjoyed today’s video, then please like, share, and subscribe, and I will see you next time. Take care.