0:03 Okay. Um My name is Alex Morozov and I am professor at the Department of Physics and Astronomy and Center for Quantitative Biology at Rutgers University. And today I'm going to talk um about a new mechanism we have discovered for evolution of cooperativity in the game of prisoner's dilemma. So let me start with um this amazing process of Darwinian evolution uh in which 3 billion years ago um first organic structures appeared on Earth and then about 2 billion years ago uh first single-cellular organisms appeared
0:49 and ultimately through this process of um cooperation and um producing more and more complex forms this amazing variety of life forms appeared that you can see in this picture. So of course uh Charles Darwin was um one of the first people who uh talked about the theory of these evolutionary processes and uh what he pointed out is that um there are mutations that uh new mutant variants appear and that drives this this gives the raw material for evolution and drives evolution forward. But then new mutants and the old members
1:35 of the population are acted upon by selection uh which kind of chooses uh you know, which individuals propagate and which die off. Then uh Darwin didn't focus on it so much, but there's stochasticity in finite populations uh known as genetic drift and this stochasticity may lead to dramatic effects where um you have a beneficial mutation, but you might disappear from the population anyway. And finally, we have spatial effects such as migration, isolated populations and and sort of effects that have into do with uh spatial extent rather than evolution with time.
2:22 So, this um is modeled quantitatively these days using the tools of population genetics. So, that's one aspect of my talk, and the other one is of course, prisoner's dilemma, which was formulated in the 1950s by two people from RAND Corporation and it was applied in the context of the Cold War, and they talked about um they talked originally about two people that are getting prison sentences, but it's easier perhaps to think about payoffs. So, let's think positively rather than negatively. And a typical two by two payoff matrix in uh in a prisoner's dilemma in prisoner's dilemma
3:07 uh is like this. If So, there are two players, X and Y, and if X chooses to cooperate, and Y cooperates as well, they both get a plus 10 points, as you can see in this table. So, that's the good state to be in. However, there is this temptation that if X already chose to cooperate, and Y now decides to defect if you if you look at sort of the off-diagonal cell of this table, then X gets minus 10 points, minus 10 dollars, but Y gets plus 20, which is bigger than the payoff than uh the uh you know, in the case where they both cooperate. And of course, it's the same for Y on X. So, if Y first, you
3:54 know, steps forward and cooperates and X at that point chooses to defect, then Y is screwed. It's minus 10 and X is plus 20. So, the only logical conclusion with with this kind of payoff matrix in this kind of game is to say they both should be defecting. But, that only brings them zero each. So, it would have been much better to be both cooperating, but just the temptation to defect brings them into this defect defect state, which is worse. So, so this is kind of the the dilemma in prisoners' dilemma. And even though it was formulated in terms of people playing games, um
4:39 later on researchers implemented um implemented game-playing microorganisms. So, in particular, Jeff Gore published this well-known paper in nature a while ago in 2009, where they made yeast play games through sharing of a common resource. So, they took sucrose in the medium and you need to digest it, break it down into glucose and fructose. And you need a special gene to do that called suck2. And cooperators have it. So, they they do that. They make glucose and fructose, but they cannot retain it inside the cells. What happens is that
5:27 these molecules of glucose and fructose, they diffuse away and become available to other cells, in particular to cheater cells that don't have the suck2. They don't have to incur the energy cost, incur the effort of making glucose and fructose. They can just use what comes their way from um you know, glucose-fructose producing cells. >> [gasps] >> So, uh that potential it sets up a scenario where um um you know, the [clears throat] cheaters will take over because uh it's because they don't have the succus associated costs. And uh but then by themselves they cannot produce the nutrients and the population will collapse. So, that
6:13 particular paper found that there is coexistence and the mechanism that put forward is the nonlinearities in the fitness function, nonlinear response. So, here I'll talk to you about um a different response, perhaps a complementary sorry, not response mechanism uh that we have developed that does not rely on nonlinearity of fitness functions. So, evolution of biological complexity is a big question. Um you know, complexity in biology appears at all scales from molecules to populations of organisms. For example, cell make multi-multicellular organisms, right? Organisms uh take part in communities and ecosystems.
7:00 And the evolutionary mechanism for um for the appearance of this complexity is is very far from clear at the moment. Of course, over the years it has attracted a lot of attention from uh evolutionary biologists. Uh in particular from evolutionary biologists and was the subject of of books, evolution of complexity. So, however, it is an open question. It's still a major puzzle. Uh in particular, Karl Sigmund, who uh for decades worked on sort of mathematical underpinnings of evolutionary game theory, um um and in particular published this book called The Calculus of Selfishness, says in the first few pages that first of all the evolution of
7:46 cooperation is one of the 10 most challenging problems of the century. That that would be 20th century or maybe the the upcoming 21st century. Um and his book certainly did not claim to solve that problem. So the basic conflict is as follows, you know, it's an evolutionary conflict between selfishness brought about by selection. So rational behavior in prisoners dilemma is to defect. But that abandons common interests. It's in contrast to common interests and that gets everybody in in a bad place in the course of evolution. So that's the problem. You act rationally but at the same time you end up in a bad place. So
8:31 there's this this fundamental puzzle. Of course people have spent considerable amount of time trying to understand how evolution can arise under different um circumstances and in different models. Um and this paper this review paper by Martin Nowak published in 2006 in science shows five mechanisms for the evolution of cooperation and one of them is kin selection. You help you know, your genetic relatives. Another one is direct reciprocity which means that you encounter the same individual multiple times and you decide in the process of these encounters whether uh you can you know, be nice to one
9:16 another. So this does not work in modern human societies because we rely on sort of faceless transactions and therefore people have worked out the notion of indirect reciprocity which means that I guess institutions and individuals carry around a reputation, some sort of reputation score. Like you can trust this bank but not that bank. And you know, individuals with with high scores likely or more likely to be cooperated with. So, another aspect is is really a spatial aspect that's usually discussed in terms of networks. And if you evolve cooperators and
10:02 defectors on networks as opposed to just sort of in well-mixed systems, then um on networks, clusters of cooperators can outcompete defectors. And and finally, there's this idea of group selection which adds fitness terms for being within a group. So, you cooperate with the members of your group more than you know, with individuals that belong to a different group. So, all of these um All of these are are valid approaches, but um my feeling at least is that they're not basic enough. For example, a reputation score is hard to imagine in microorganisms and
10:49 um there maybe maybe perhaps more fundamental mechanism for developing evolution for developing cooperation sort of as an emergent property as opposed to imposed by as a property imposed by um you know, again terms to the model. So, that is the idea that I'm now that I will now try to explain. So, we use this two-by-two matrix that's listed here called EPG and it depends on only two parameters. It's parameterized in this way which um which is also called the nation game parameterization just to make things technically simpler.
11:36 So it's not fundamental that we use this A and R, but basically A is an overall scale of the payoffs that you know, comes in your game and R and both A and R are positive numbers and R the larger it is, the more tempting it is to cheat and the harder it is to cooperate. So you expect more cheaters to appear when R is large and positive. So now the standard sort of evolutionary approach to prisoners dilemma is illustrated in this scheme here with just three individuals. And um uh these individuals are characterized by the probability to cooperate. I guess the simplest scenario would be that
12:22 somebody is a perfect cheater or a perfect cooperator, but um sort of a next step in the complexity of the model is to assign these probabilities. So if you're individual I, you have a probability to cooperate and in this model, which is the the standard model of um you know, evolutionary game of prisoners dilemma, um in this model, it's a single number that characterizes you. It's this probability to cooperate assigned to individual I and you apply this probability to all encounters with other players, with other agents in the population. And this is what's shown here. For example, PC2 is the same
13:07 if two interacts with one or two interacts with three. So because we're in this talk, we're going to look at large populations, but we'll discard stochastic effects even though they're very interesting. And we will look at a at a very standard way of describing evolutionary dynamics, which is based on replicator equations. And And replicator equations just have you know selection and mutation in them. This uh selection and no mutation is is [snorts] is shown on the screen right now. Uh F is the average fitness. You have to subtract the average fitness to uh uh to ensure that the sum of X I's and X I's just the frequency of a given genotype and it should always sum to
13:53 one. So, in equation this equation has this form to ensure that. And F I is the fitness of genotype I. And of course, it's also possible to uh put in mutations. For example, here I put in a very simple mutation matrix that allows uh for NB uh mutational neighbors. For example, in DNA you would have three mutational neighbors because A at some position can mutate into C, G, and T. So, you can put that into your replicator equations as well. And we will use that uh in the future slides. So, let me talk for a minute about a standard uh fitness landscape model. So, fitness landscape is something that
14:39 um you know, do you have a representation of a genotype? For example, here it's represented as a binary string, string of zeros and ones. And then to each genotype state you assign a fitness and that uh creates a multi-dimensional fitness landscape. So, in this particular model uh which I'm just using here as an illustration, it's it's not uh central to to what's to come. But in this particular model uh fitness over genotype is a sum of randomly chosen contributions that depend on uh K neighbors of site uh of site I. And uh this model is well known for sort of its ability to produce smooth as
15:25 well as rough fitness landscapes. And this is controlled by the number of neighbors, by this K. So, if K is zero, it's a very simple model and all fitness trajectories go up and there is a single global maximum of fitness. Whereas, if uh K is large, you know, you get uh something that's uh more akin to spin glass with many uh metastable maximum local maxima um you know, and then global maximum is uh not that easy to reach. Uh depending, of course, on the details and the size of the system and so on, but the idea is that uh in this model you can control roughness of the landscape. However, you know, on the ensemble of
16:10 this paths in the absence of mutations, um these trajectories are not not really uh arbitrary. They follow what's called Fisher's fundamental theorem of natural selection. Um in the absence of mutations, it just means that average fitness uh of uh you know, of a population of individuals goes up because its derivative with respect to time is just the variance and variance can be either positive, you still evolve, or if everybody has the same genotype, variance is zero and then evolution stops. So, you go, for example, from points around this global maximum to the global maximum. And uh and then once everybody is there,
16:56 uh which is one possible outcome, but maybe not the most likely on such a rough landscape, then d f d t d average f d t is zero and the evolution stops. So, so this is the idea of Fisher's fundamental theorem. So, now uh we have um So, these landscapes, once you assign a fitness to the state, it always stays the same. But we don't have this situation because our individuals play games with one another and the outcome of these games determines fitness. So it turns out that playing the game and what what you can imagine is for example a population of 100 individuals
17:43 and individual one plays plays a game with the other 99, so one at a time, and plays a series of games and equals the expected gain or loss vis-a-vis individual J. And then once you average over all the other individuals, that's your fitness. So your fitness actually depends uh uh very much depends on who else is in the population and who is playing with you. And it turns out that with the 2 by 2 AR parameterized metrics of prisoner's dilemma, it's actually uh frequency-dependent selection. It's It's a linear function of frequencies with some coefficients gamma IJ. So if you work out the math
18:28 of what I just said, you will see that fitness is given by this formula here. It has to do with the average cooperativity and how different it is from the cooperativity probability to cooperate, probability to cooperate of the individual in in question. So and it's by by averaging fitness, you can see even from this formula that average F is average probability to cooperate. And it turns out that you can work out an equivalent of Fisher's formula, you know, Fisher's theorem, and it has a minus in front. So that's a bit surprising. It says that it's anti-Fisher behavior, right? Because variance of PC is always non-negative, but it's times minus R. R
19:16 is always positive. And um you know, therefore D of DT in a standard prisoner's dilemma setting is is destined to go down, not up, as you would have in sort of simple landscapes. So, if you don't have new mutants come in, then actually the least cooperative player type takes over. Uh if you have new mutants, you will actually keep driving towards less and less cooperativity. This is the prisoner's dilemma. So, but frequency and and the reason for it is that frequency dependent selection is like trying to hit a moving target a little bit. Because you depend on your neighbors, and the neighbors change all the time. By the time you try to improve, the landscape changed. And
20:01 people use this metaphor that static fitness landscapes become population state dependent seascapes. Uh it's it's like, you know, changing the height of waves and the spacing of waves in the sea. That's that's sort of the metaphor. So, do our cheaters destined to win in the standard uh setup? Yes. And this is what, you know, has puzzled people for uh for, I don't know, half a century or so. More than half a century. For example, if you run uh a system of replicator equations starting with just one defect defector. Defector will quick fairly quickly take over. There will be fewer and fewer descendants of the cooperator types and
20:49 more and more uh defector types. And if you just, as a function of time, if you plot average fitness plus minus standard deviation, you see that it sort of reliably just goes towards the uh the the type that's least cooperative. So, that's that's the problem. But now we made one change to the model that uh uh that changed um that changed this dynamics uh in a big way. So, what we said is that these are pairwise encounters, and therefore when individual I encounters individual J, they don't have to react in the same way. That's not what happens with animals, certainly not what happens with humans. For example, if individual two sees
21:36 individual one, he might be compelled to cooperate because one um maybe in animal societies is a big healthy individual, and uh it just makes sense to cooperate, but vis-a-vis individual three, you know, two to three, the decision might be different. So, probability to cooperate of uh of individual two given three or when faced with individual three might be different. Um you know, it might depend on the opponent. So, this is what we call opponent-specific recognition, and here it's illustrated with three individuals. And um you can turn this idea into a mathematical formula for fitness. And the new component here is
22:21 uh is these um average sort of partial averages. There is an fitness depends on the difference between how other individuals react to you minus how you react to the other individuals. So, if uh in a population other individuals like you, but you don't like them quite so much, that actually gives a boost to your fitness. And this is a new element in this model. So, let's see now how um you know, how fitness or average fitness evolves with time. You can work out mathematically uh something that we call the variance covariance theorem and uh so this it places or it's it's more
23:09 general than Fisher's theorem and the minus R variance PC version on the previous slide is a a a um is a special case of it. So so this is more more general. So to derive it I had to divide um fitness into uh um actually symmetric and anti-symmetric contribution because fitness is linear in uh in frequencies. It's if you recall gamma IJ times XJ sum over J, you can always divide gamma IJ into symmetric and anti-symmetric contributions. And this turns out to be important for this particular formula. Um So if you do that um you know and see
23:57 just on a computer how average fitness uh how average fitness evolves uh by just uh solving you know a system of replicator equations under this new model, you see a completely different behavior. So the old one uh the red curve was always down but the new one is actually starting even from the same point uh goes through plateaus of um you know partial cooperativity and then a period of upheaval then another plateau of relative calm where uh one genotype is dominant and then another upheaval and and so on and so forth. So it's a completely different dynamic from the red curve which just always goes down. So here's another one where you can see that there may be a long period
24:43 of upheavals and going up and down and then relative calm and then someone new comes in and takes over uh and and changes the state. So this is in the absence of mutations uh which means that um at times, you know, some individuals are driven to very low frequencies, but they can get reselected uh because in the new circumstances with the new population, they now become >> [clears throat] >> you know, they now possess high fitness. So, this is what's going on here. There are no new mutants. And of course, the first thing you want to ask is whether all of this is stable with respect to mutations. So, we don't really have genotypes in the model. What we have is phenotypes that are characterized by the
25:30 probabilities to cooperate. So, we worked out um a system, you know, a a system for mutations where you basically mutate one of the probabilities to cooperate uh uh you know, if an individual is chosen to mutate. And of course, you have to stay between zero and one. You have to stay between zero and one. And uh the old model, which you know, always shows collapse of cooperativity, uh shows it here as well. Uh this is basically final uh average fitness minus initial as a function of R. Uh and in the presence of mutations at high mutation rate, 0.2, and you can see that actually this
26:16 change is um persistent, you know, it's not negative when you have a full model, the opponent recognition model. And uh and this is done over just multiple trajectories uh and the the trace um like this dashed line is traced through the center um of each histogram. But basically, what you see is that you actually gain, you know, with the new model, uh it cooperativity is persistent, and with the old model, it always collapses. So, that was uh um know, that that was interesting. That shows that uh mutations don't destroy um our result, the ability to to our of our
27:01 model to develop cooperate activity. So, now next we asked if we start with the defector state, but allow for mutations, including cooperate mutations, you know, can we recover from the from this sort of low fitness all defector state? And sure enough, the old model collapses even more. This is the red curve, but the uh the blue curve, you know, when you start uh with defectors, meaning everybody has the probability between uh basically zero and 0.2, as as shown here, but then you allow cooperate mutant to arise purely through mutations, you actually get a curve that uh climbs up in fitness on average in the population.
27:47 So, so, you know, defectors can be overcome by cooperate mutants, which is kind of the opposite of uh of what the prisoner's dilemma wisdom is. So, now we asked also if we start with uh opponent with an opponent blind society, but allow opponent specific recognition to arise uh through um uh you know, through mutations. So, you start your collapse here. You start your collapse. But then mutations allow for start to allow for opponent specific recognition, and uh you actually recover. You don't go to a full collapse, which would be something like this, but you you start a recovery.
28:35 So, this this was interesting as well, I thought. So, now, because we know how our models work, we can imagine um a system in which we design what I term a rise of a populist, the rise of a popular person. And that means in terms of this model, strictly in terms of this model, not in terms of politics or something like this, that we have 500 individuals in a population and one of them is liked by the other members, the other all other members of the society, but this is not a turn, you know, he doesn't like them, he or she doesn't like them in return. So, there's a big difference between
29:21 0.98 and 0.1, but the descendants, you know, you know, all like-minded people, which is the probability of basically genotype one to genotype one is very high. So, in this case, fitness goes up and in the end, very quickly, the populist or the descendants of a populist, they take over. The this NS curve is just the number of species and this goes to one and they're all dependents of the populist. Um 100% they're not dependents but descendants of the populist. And the the fact that fitness goes up is driven by um you know, you can trace it through the fact that variance and covariance have
30:07 the same sign and therefore dF dF average dT is positive. But now imagine this scenario where it's all the same, all the same system except the descendants of of of the single populist that invaded the population are >> [clears throat] >> all they all hate one another. So, their probability to cooperate with one another is only 0.1 and before it was, I believe 0.99. So, now in this scenario, you actually have a collapse. So, you you go from 0.5 to 0.1. Everybody is a descendant, you know, all the people who supported the single populist in the beginning have died out.
30:53 So, the the population is again is just 100% descendants of the populist who, however, hate one another. So, so this is a very bad state to be in and this, as you can see in this picture, is driven by the fact that the covariance term over powers the variance term and drives the average fitness down. So, now the last thing I want to talk about briefly is that we actually evolve populations of small neural networks that play prisoner's dilemma. So, they play it exactly as I've told you about, you know, what So, these are so-called LSTM networks. Um, and they exchange So, they have fitting weights, just a few hundred fitting
31:39 weights per network. And what they do, they actually let one another in a pairwise encounter read each other's weights. And these weights are a proxy for phenotypical characteristics. Of course, they're much more obscure than things like uh I don't know, the the height of the animal or the weight or or the overall appearance, which is what is read in presumably others. But this is some sort of a simplified proxy for that. Right? So, they in an encounter between agent I and agent J, they exchange their weights. Those weights become the input to their own thinking machinery, to their own network, and then they spit out a single number, which is the
32:25 probability to cooperate of I to J and then J to I. The 2 by 2 matrix is still the same. The fitness is computed still in in same way, meaning that individual one plays the game with individuals two through n, and then the average of the expected payoffs, so it's a series of games because we're we're taking like an expected payoff um instead of sort of an instantaneous payoff, but then these expected payoffs of all these pairwise encounters are averaged, and this is your fitness. So, it's actually as close as possible to the previous model. So, uh we use Wright-Fisher population
33:10 dynamics, which means you have a box with n individuals, and you apply mutation and selection, and you populate the box. This is discrete, non-overlapping generations. So, it's illustrated here with just four individuals, and on the left, it's step L or generation index L. And you have four different individuals, and then after one uh step, you know, it's still four, that's enforced, but W1, you know, agent one mutated, and and now is a A1 new characterized by new weights, W1 new. Uh W2 created a descendant, so now there are two copies, and they actually replace W3 or agent three,
33:57 which died off because there's no place for it anymore. And W4 did not mutate and did not reproduce, and is still surviving in this generation L + 1. So, this is apart from the neural network angle, this is a very very standard um type of population dynamics that people use in a mathematical population genetics. So, when you run it and you look at the evolution of average fitness, let's say with a hundred or a couple of hundred of these neural networks that evolve, uh, by right Fisher dynamics with selection and mutation, you actually see that, um, it's not that unlikely for them to be, uh, as you can see from these two
34:43 trajectories, in a high cooperative state. And a little bit like our simple model, uh, you know, sometimes it collapses down to something like 0.4, 0.2, but then it comes back up. So, there's this plateaus of stability and then intermediate regions. So, it's not exactly the same because these are completely different, uh, or fairly different mathematical frameworks, but there's some similarities, at least qualitatively. For example, if I focus on this region here, which is, uh, um, you know, outlined by a box. So, here, uh, average fitness on trajectory one, the the green one, goes down and then it goes up, kind of a non-uniform way. If we make four snapshots
35:29 of of, you know, four intermediate states on on this way, you know, four intermediate number of generations, and plot kind of a heat map of PC IJs, you see that they, initially, it's in a bad place. Everybody is with probability to cooperate about 0.1, even below. Then there's a transition. Um, so, this is the second point, the third point, it's better still, and then most of the individuals end the high cooperative state, you know, are at high cooperativity. So, this most of the points are close to one. So, this is this transition from basically 0.1 to one that's, um, to to see this transition, we needed to zoom in on them.
36:14 Uh, into this long trajectory and just, uh, look at four, um, closest space generations. So, I would say this is an implementation of the same mechanism because these networks um again if you look at how they exchange their information, you know, these networks they're actually computing PC of I given J. This is opponent-specific mechanism or opponent-specific recognition exactly of the type that we talked about before. So, that's it. Um so, this uh concludes my talk. So, the summary um of the main points is that cooperativity can emerge in the game of prisoner's dilemma if individuals uh rely on
37:02 opponent-specific responses instead of using a single probability to cooperate with everybody. Um then next opponent-specific responses I would claim can arise naturally in response to uh phenotypic characteristics that individuals see in pairwise encounters. Uh for example, in humans, right? When you see someone, sometimes you like some people more than others and this is uh this is a very innate thing to to do and this is the type of response I'm talking about here. So, we're not claiming that this is the only mechanism or it's uh incompatible with other mechanisms. It's more like a fundamental, perhaps the simplest
37:48 emergent mechanism because we didn't put that into the fitness function. The cooperativity emerges by itself. And it can be reinforced and refined by other mechanisms such as kin group selection direct and indirect reciprocity, especially in human societies that um you know, where individuals have um memory and have um you know, sophisticated recognition mechanisms. And the moreover, as I showed you with uh this game of a populist, opponent-specific responses sponsors to sort of achieve specific such as a complete takeover of the descendants of a of a populace or by them you know takeover of the entire population and complete die out
38:36 of the initial supporters of such a populace. So this work was done in a close collaboration with a collaborate you know with an old friend of mine Alexander Sasha Figel. He is in the physics department of the Hebrew University of Jerusalem and here he is enjoying uh you know a Volvo car museum and of course I have to thank my lab um my former and current students. These three students highlighted here are currently my PhD students and all the sources of funding and thank you very much for listening.
