Canva and the Thundering Herd

0:00 Greetings, fellow Incident nerds, and welcome to season 2 of the Void podcast. The main new thing for this new season is we're now available in video. So, if you're listening to this and prefer watching me make odd faces and not a lot, you can find us on YouTube. The link is in the show notes. The other new thing is we now have sponsors. Don't worry, these folks help make the podcast possible, but they don't have any say over who joins us or what we talk about. So, fear not. This episode's sponsor is Uptime Labs. Uptime Labs is a pioneering platform specializing in immersive incident response training. Their solution helps technical teams build confidence and expertise through realistic simulations that mirror realworld outages and security incidents. When most of

0:47 investment these days in the incident space goes to technology and process, Uptime Labs focuses on sharpening the human element of incident response. All right. In this episode, we talked to Simon Newton, head of platforms at Canva. They just published their first public incident report. It's not their first incident, but it's the first time we get to delve into the details of what happened, and we really appreciate them sharing that with us and the world. So, let's get into it and talk to Simon. Simon, thank you so much for joining me on the Void podcast. Will you do our listeners a favor and introduce yourself and tell me where you work and what you do? Yeah. Hi, I'm very happy to be here. Um, I'm Simon Newton. I'm the head of platforms at Canva. Uh, I've been there

1:33 about 3 years or so. And, uh, platforms at Canva consists of a bunch of different areas. Uh, I guess most interesting today, uh, that includes the teams that run our edge and gateway, um, sort of the the front door into Canva. um as well as the teams that u manage all of our cloud resources. So yeah, Canva, Canva, in case you haven't heard of it, um is a visual communication company. Uh what we do is make it really easy for people to express their creative ideas uh to others uh in a way using visual content. Um we have a bunch of different products uh built into what we call the the editor. Uh and so we have whiteboards and presentations and responsive docs and websites uh and drawing tools for making social media posts. Um, so yeah, there's a very there's a huge amount you can do.

2:20 Yep. That's how I'm going to make the social media posts for this podcast. So, uh, thank you all very much at Canva. That's why I was very excited, not for you really, but when I saw this incident report, uh, go by. Uh, I'm in a Slack group, this resilience and software group, and we have a channel that's just called cases. Um, and that's honestly I just like I could sit watch that all day. like so whenever you know sometimes we know because things happen to us but I found out about this this incident report and the outage I did not experience the outage personally uh but I did find out about the incident report from the Slack group that I'm in and I think also I noticed that it's the first one you've ever done right I don't recall ever reading a Canva incident

3:06 report before so you've had and in the report this report and you say you've had an internal process since we're going to get into the incident, but you've had an incident process since 2017 and I'm always so curious about what it is that pushes an organization over the edge or I mean I don't want it to sound so negative but like what brought you to the point where you said okay we're going to go to the effort of publishing our first public incident report. Yeah and it is it is a bit more effort right I mean we've been publishing uh incident reports internally um like you said for many many years um but there is a difference between you know a report for internal consumption and then a polished report for external consumption um so in terms of the why behind it uh it's really a reflection of how we're

3:51 evolving as a company um so Canva you know st got it start uh with a bunch of um individuals and small businesses um that was sort of typically the the the user groups um and they'd you know be creating social media content and marketing um for their small businesses uh and that that sort of material. uh but in the last couple of years uh we've really seen increasing adoption within enterprises uh and enterprises bring their own like new and different set of customer requirements right and customer expectations um and so as part of us wanting to show our commitment to transparency um to those enterprise customers um and also because we believe uh and I very much believe uh that sort of being open about incident reports does benefit the broader industry um and I think the last point there is sort of

4:36 really important Yeah, and I I I love I I love what what Void's doing here, right? Um you know, my my take is that I think software engineering generally as a field does a fairly poor job at learning from previous failures. Um if you look at other engineering fields, uh you know, often they're like more highly regulated and there's like processes around this where uh where you know, failures are like well understood and investigated. Uh but in software engineering uh we we seem to make the same mistakes uh many many times, right? Right. And there's not that that not there's not that sort of like learning uh as a broader field. Um and so yeah, very very keen on on sort of like void's efforts to see uh sort of improved education around this, right? So that that we can all uh uplift the industry as a whole. Thank you for the shameless plug. I

5:23 appreciate it. I want to ask you the same question I ask everyone at the start of this podcast, which I think is cruel and yet I do it every time. Can you give a brief summary of what happened? Sure. Uh I will I will try and keep it brief. um the a as I'm sure you're aware right uh the the large and interesting uh failures are are because of a confluence of factors right so in this particular case they were there were multiple contributing factors that sort of lined up in a way that caused this much much much larger incident but maybe a little bit of background um so the canva editor is a single page app um we deploy that multiple times a day um and amongst other things as part of that deploy uh we uh build the javascript script assets um and then we publish those into an S3 bucket. Um and so then

6:08 when clients uh reload the page um or the the editor self updates um those clients then go download the new assets uh via our edge provider um and once those assets have loaded um and the editor is sort of like functional and running um then it starts making API calls to to load in all the content. Uh and those API calls also go via our edge provider um to what we call our Canva API gateways. Um and then those gateways route those requests to the various services um sort of within within our systems um which then will will handle. Um and I guess the the other key bit of information is those gateways run as autoscale groups um within our cloud provider AWS. Uh so in this incident the first uh the first like factor that that contributed towards it was we did a

6:55 deployment of the editor um at the same time that there were network issues that were occurring within our edge provider. Um and now normally uh these these network happen issues happen all the time right like the internet is a messy place. Um this is this is happening day in and day out. Um and so normally what happens is our edge provider has automated software uh that can detect these issues and can mitigate them and route around them. Um unfortunately in this particular case uh and and again we only learned this later uh they had a bit of manual stale configuration in place that have prevented that automation from running. Um so first contributing factor is there's a network issue uh and it hasn't been mitigated uh automatically as as often happens with those things unfortunately. Yeah. Yeah. Yep. And so

7:42 then uh so then maybe the next bit to to discuss is if you consider how the edge proxies work, right? You've got a lot of clients making requests for resources. Uh and if that resource isn't in the local c of the proxy, it'll have to go and fetch it from the origin server. Um and it's very likely that there's multiple clients requesting the same bit of content, right? Uh and so it's pretty inefficient to have the or the proxy server then go and turn around and request multiple times from the origin. So what you want to do is you want to coales all those requests uh do a single origin fetch uh and then send that content back out to all clients. Um and you can probably see where this is going. Uh but you know with a bunch of clients requesting the same JavaScript asset uh and with networking issues. Is that the sound is that like the sound of a thundering herd I hear in the

8:28 distance. Exactly. That's um but but the interesting bit here was you know like if if everything sort of behaved as it was as it would normally uh you know your latency would go up because the origin fetches are taking a bit longer um and and you know that that's fine like you know it's just a bit of increased latency for the clients um but where we got really unlucky was one of those origin fetches didn't just go out from like you know measured in milliseconds to seconds the origin fetch actually took 20 minutes to occur right and and as I'm sure you know 20 minutes in computing time is an absolute eternity. It's like geological time. Yeah, it's yeah it's geological time. So yeah, it's rough. And then when we got particularly unlucky again was that this one one particular asset uh was the

9:14 JavaScript that loads uh what we call the object panel in Canva. Um and so this is the the sort of main place where you interact with content uh in your designs, right? And so without the object panel loading, the editor is is essentially you know frozen and dead. Um, and what it also means is that once that one bit of JavaScript loads, it then triggers a bunch of API calls because it can start loading the content. So, yes, you know, if you've got a fetch taking 20 minutes, uh, you've got all these clients like coalesing and waiting for it, uh, and then suddenly it becomes available. Uh, that turns into a very very large number of API requests uh, hitting the gateways. Yeah. Uh, it's pretty ugly. Um, at that point things fell over in terms of like the autoscaling and whatnot, right? At some point it all just like collapsed on

10:01 top of itself there, right? Yeah. And so that turned into uh about I think it was about one and a half million uh API requests in a very very short amount of time. Um and that was enough to yeah overwhelm the gateways. Autoscalers can't react fast enough that um the gateways went out of memory and then that started sort of the cascading failure due to overload. Okay, that's the shortest version of this incident, which is our um and so I think we'll we'll talk a little bit more about some of these details. Uh but how did you all find out about this? Yes, I can run you through that. Um so the the first signs that something was wrong uh was about 10 minutes after the deploy uh because we started noting noticing a drop in search traffic. Um and so our

10:48 search on call team was paged uh for a drop in traffic. I think they went down around 20% or so. Um and so they started investigating uh and at first they thought like oh you know maybe this is something like very sort of you know isolated to search. Uh but as they started uh communicating in an incident room it was a SE one which is sort of our our second highest tier of uh incident at that point. Um as they started communicating uh other service owners started getting involved and saying like oh actually like we've also noticed a drop in traffic. Um and as they were doing that uh that was about the time that the fetch itself completed uh and then our gateway and edge teams got paged for you know massive massive gateway failures. Um at that point the incident was upgraded to a sez zero. The gateway teams were pulled in uh I was

11:34 pulled in and we you know the incident uh incident coordinator was was activated. Um and then and then we sort of went from there. Yeah. What does your incident response look like? How you know do you all have sort of some you know protocols or what does that look like and how how many people ended up being involved if you could tell me because it sounds like it was a doozy. Um and a lot of people got called in. I'm always really curious like how many hands are on deck. Yeah, I would I would guess probably at least 20 or so. Um so let me let me describe the process. uh you know typically uh alerts will page uh service on call teams um I think it's a it's a very typical process right those service teams will triage um and open the incident uh if it's a sev0 zero

12:19 or sev one um that will activate the incident coordinator rotation which is sort of a it's a small group of people um that have a lot of training and skills around managing and coordinating these large scale incidents u zz zeros will also also page myself uh and Then it it really depends on the incident as to the the structure of it, right? If it's a smaller incident, um you might have only a handful of people uh and they'll they'll be sort of fulfilling all the roles, right? Um if it's a large incident like in this one, uh you know, we'll have we'll set it up so that there's dedicated people running running each particular role. Um I think maybe one one thing that's important as well is that uh you know the larger or the the highest severity incidents uh will automatically assign a representative

13:05 from our customer support organization. Um and they'll be the ones that will be coalescing all of the user reports um and and sort of looking for signal there to not overwhelm the the sort of the responding team. Um and then they're also the channel back into the user support. Um so they can feed back like look you know this is the state of things if people are you know contacting or reaching out to Canva this is what we should be telling them if there's anything they can do um you know to to mitigate it from a user perspective um that as well. Uh yeah and then you know post incident uh like I mentioned we'll write those incident reports and then we also uh we've just uh you know fairly recently started using AI to try and extract common themes uh across all of those reports as well. um to to sort of look for areas where we can uh where we can improve. I have so many questions.

13:50 Um so the IC's getting pulled in sort of automatically if it's like sub one subzero. How is that team are they are they uh also engineers on on different teams? Are they kind of a a dedicated sesque function? And when they're not ICing, what are what are they doing? I'm kind of curious what the shape of that looks like. Yeah. So early on uh I would say our our IC's were just sort of a bunch of very experienced uh sort of battleh hardened engineers at Canber right um and then over time of course as we get bigger uh you know we need to sort of set up a dedicated function um and those you know because those those folks are also they've got their own projects right their own their own deliverables um that they're working on

14:35 so yeah IC is a dedicated function at Canva um when they're not on call uh they're doing a bunch of other different Um some of those is improving improving the incident process itself. Uh looking for those patterns in incidents. Uh plus also there's a bunch of work that goes into our large uh launch events. So we do two two main launches a year. We've got one coming up now in April. Um and so they're they're sort of like doing a bunch of planning for that. Uh there's a bunch of capacity planning involved. Um you know like risk risk uh understanding the risk and mitigating um the the risks that are sort of unique to each one of those launches. Um, and so yeah, that's a that's like a dedicated reliability function. When you have like a pretty big incident like this one, do you have execs in the in the channels in the uh

15:21 active incident channels? You like or do you have a dedicated role who's talking to folks and kind of coordinating back or forth? What does that look like? Yeah. So, in this particular case, the founders, our founders did join the instant channel. Um, typically what will happen, you know, sometimes they're there, sometimes they're, you know, they're busy with other things. um typically like myself or sort of one of the other senior leaders will act as that channel um to the to the founders um to give them updates and and help them understand what's going on. So you mentioned you found out later. So was was the Cloudflare wrinkle of their traffic and you didn't mention this explicitly but I think you're getting the network configuration. So their traffic was going over the public internet not over their private backbone. Is that a piece of that you found out later? And so that was sort of like we didn't know why that was

16:07 contributing to the problem. Yes. Yes. Yes. I mean we don't we don't have visibility into that traffic between you know cloud cloudflare and our our origin running in a AWS. Um so yes that was a detail that we we didn't find out until I think maybe like one or two days after the incident as we were working with Cloudflare to understand what happened at at some point. Did anyone have the intuition that this was going to get worse in the way that it did? Like once the files were downloaded from the origin server and and there was did you see that hear the thundering herd coming? Like I'm always a little curious what the what the existential dread might be like like in an incident or was that did that catch you by surprise as well? Uh yeah, it it all happened uh like fairly quickly in terms

16:53 of the um okay you know the the unfolding incident. So yeah, by the time that folks were in the room saying like, oh, this is this is broader than search, um, that was about the time that that origin fetch completed. Uh, and then the gateway started starting falling falling over and so that so even so the like the all the clients hitting the API gateway, you've got a known performance issue. So that all happened really quickly and but or were piece of those only aware after the fact as well? Yes, I didn't actually touch on the the performance side. So um you know as this is all unfolding then yeah I jumped ahead you know as the as the as the gateways are now you know now crashing uh we have a bunch of people on the call and and that's sort of where we started breaking up into different work

17:40 streams right so you know there were a set of people that were responsible for uh contacting our vendors um and getting them involved uh and that process differs across vendors right like some of them some of them will be like we'll go to our portal and submit like a you know a P 0 request others will be like email this address and it will page sort of your your on call account managers. Uh and so that process there um you know needs to be well practiced, right? And I think this is actually something that um something we improved afterwards as well. Uh we have internal docs on how to escalate to each of our vendors. Uh but we tidied up that documentation and just made it a bit a bit clearer because when you're when you're doing that, you're already in a high stress environment, right? And so you don't want to load a documentation page and have four pages of workflow to go through. It needs to be a very clear do these four things

18:26 right. Uh so yeah we had a set of people set of people sort of contacting vendors. Uh we had one event the engineers who was not on call at the time um but was you know saw the incident and jumped on uh as as as many people sort of can do right where there's a there's a very good um culture of of sort of like banding together especially for these for these large incidents. Um so we had one one engineer who went off and started profiling the gateway. Uh and so as this is all emerging um you know he was reporting back in and saying like oh look actually you know the profile looks different. Um I think I think uh if I remember correctly uh he actually had he he'd done this before other times and so he had he had past profiles to compare it to. Um, and he was like, "Look, you

19:12 know, when I did this like two weeks ago, uh, you know, like this this one looks different, right? And I'm seeing lot contention here." Uh, and so that was that was sort of like one one avenue that we thought uh might have been the contributing factor. Um, and so we had a bunch of people exploring that. Uh, that turned out to be uh a change in our metrics library. Um, what we were doing was changing making some changes to the way that metrics were collected. uh and so we were integrating a like a third party library there and what that had done is uh inadvertently put metric registration behind a lot uh and so the capacity of our gateways were reduced um which would would then also contribute to the to the sort of like you know the the overload situation um and what we saw with the cascading failure um but that was all sort of emerging in

19:58 parallel to the you know to the to the rest of the activity that was happening on the call. So, you got the performance issue. Um, which must have been and and if I'm not wrong, there was this really juicy detail in there. Um, these are the kinds of things that I love it when companies share where it was a known issue and there was a fix for it in like the deployment pipeline for that day that had not gone out yet. And in many organizations, there could be a lot of blame and like post hawk like I can't believe we didn't do that. we should have like what was the general sense when you all realized that that also was contributing to this incident? Yeah, I mean it definitely there was never any

20:44 blame, right? Um and it it was just it was just very unfortunate that yes, you know, the metrics team had already identified this, they'd already fixed it, it was already merged. Uh and and you know, I guess what would that have been in in like 12 hours later? uh the new gateway would have been deployed and and this this may have looked very very different, right? Um so that was that was just just unlucky. Um you know we do we do have a project underway um to uh move to incremental deploys. Um right now I'd say most backend components uh are deployed on a daily cycle and then uh and then sort of the components individually can opt into incremental deploys um and and sort of deploy on merge. uh but we do want to sort of move more and more of our components to like over to that incremental model um which

21:31 would have which would have you know ideally caught this uh in this case and so then the last but not least certainly I believe again if I've got the timeline right here so you've got that going on and then basically you had like the load balancers on your ECS containers just they couldn't like they were just getting hammered at this point as well right and was that also uh a matter of sort of this was reaching out and trying to figure out what was happening with AWS or like what what was that last piece of the puzzle like? Yeah. So we did you know we had a a set of people uh escalating cloud flare um we had a set of people reaching out to AWS um because uh you know we were at that point this was sort of just after that origin fetch had completed um you

22:17 know we were concerned that there might be something going on the AWS side. Uh but thankfully uh we have I would say like reasonably good observability into all of this right um and a cascading failure due to overload is a very distinct pattern in a graph right where you get this this sawtooth pattern like if you're looking at instance sort of instance uptime you get this sawtooth that's sort of offset with every instance um and so you know I I've certainly seen that before uh a number of other people on the call have seen that before um and so it was very clear at that point it's like oh no this is, you know, this is an overload situation. Um, and then, uh, again, we've we've sort of, uh, faced this before, right? I've I've seen I've seen quite a few of these. Uh, and there's there's really only one option, um, which is like

23:03 you've got to, you know, bring the load down and get the system back into a stable state, right? You can you can try and add capacity. The problem is unless you can do that in sort of a an atomic flip fashion, uh, any new additional capacity that you bring up is just going to get hammered into the ground. Um, and so you've got to cut that load back. Uh, in this particular case, you know, what what would be really nice if is if we had a sort of a lever where we could just say like, oh, okay, great. Like dial the load to 20% of of sort of the the incoming demand. Um, we didn't have that sort of control within the cloud provider systems. Um, so instead what we did was reach for uh the the sort of country level controls, right? Right? And so we put in a block that said uh you know for any country in the world uh

23:49 we want to display this status message instead of forwarding the requests onto the onto the back end. So that was a that that was the the lever that we had at the time right um and then what we started doing was uh once you know once we got that block in place we saw the traffic drop to zero we saw all the gateways come up and stabilize uh and then we started admitting more load. Um I think the maybe the interesting bit here is that uh we added Europe back in first because that was where the peak load of the time was. Right? This happened in sort of Australian like early to mid-eing time. Um which would have been European daytime. The US US was mostly asleep at that point. Um and so we targeted Europe to be admitted back first. Um and then and then rolled out uh rolled out to the rest of the world. Yeah. Yeah, I mean that was an

24:34 interesting piece of what I like to think of as hardearned expertise where instead of just turning everything back on again and literally recreating potentially the same problem, which I've definitely seen incidents where they're like, "Oh no," and it like literally happens again. Uh the only other piece I was curious about is in terms of what that instant response looks like is like are people remote? Is everything in a Slack channel? How does that work? Is everybody in the same time zone? How complicated does that look for you all? Yeah, so uh Canva is a is a hybrid work setup. Um so I'd say the bulk of our engineering is sort of in the New Zealand to west coast of Australia time time zone. Um uh and so yeah, this was this was sort of like early to mid

25:20 evening for them. Uh but yes, as part of uh the automation when an incident is triggered, uh it automatically creates a slack room. um it ties that Slack room to the main uh incident Slack room so people people can go in there and find it uh and it sets up the Zoom call as well. Uh and so then I'd say the majority of the discussion occurs on Zoom. Uh maybe another interesting point is that we we have deliberately set up that Zoom in a way that uh the Zoom chat is disabled. Uh so it funnels all of the chat through the Slack room um so we we can capture it for later and then B you don't have two two different uh sort of you know text. Yeah. Yeah. Um and so typically people will share you know we'll be sharing links to dashboards or logs or etc in the in the slack channel. Uh and then we

26:07 have a way of annotating within the slack channel sort of key moments right uh in a way that it's very easy to when you're going back later trying to write that incident report you can sort of pull out the key events and say like okay this is what we knew at this particular time. Is that a homegrown tool that you all use or is that a that's a third party tool for the for the incident sort of Yeah, I think it's it's mostly a bunch of bunch of tools developed by that same set of uh incident coordinators. So, some companies say, you know, for for sev whatever number, we're going to always have an incident review and we're going to try to do it within this amount of time. And do you have like a dedicated incident analysis team and with analysts and do they have processes

26:53 around that and and time frames for when you want things to happen or is that more ad hoc? Yeah. So we don't have a dedicated team of people analyzing these. Um you know the incident coordinators will work with the the sort of service teams that were contributing towards the incident. Um and then we have a you know a template for the the PS what we call them. um you know I don't I I don't think there is a sort of a set deadline um but certainly you know myself I'll be you know because I'll be involved in the SE zeros I'll be looking looking for for that report right um and you know if it hasn't shown up in maybe like two two or 3 weeks I'll be I'll be pinging people saying like you know where where is it at that point um but no no sort of like automated system that uh you know that that like pings it is

27:40 it is all linked in the um in that automation in that slack room as Uh so you know as the state moves through from you know like responding I guess like triage responding to mitigated uh to resolved um that's all getting captured as well um and and fed into that timeline. Is there like a P meeting anyone can come to like how are the reports shared internally? Yeah. So it's uh it's then published when when it is published it's it's done typically as like the last message in that Slack room um before that that then gets archived. Uh we have a a weekly meeting as well where we review uh you know the SE zeros plus like some of the some of the other uh lesser lesser severity but maybe like interesting interesting failure modes. Um and then

28:25 we publish uh uh in our sort of monthly engineering leads meeting uh we we take the the list of incidents um and we we go through them then briefly there um that that's sort of so that all the engineering leads have the the same level of visibility um and that we're we're sort of identifying common themes. Do you find that th that those reports get referenced in like planning or are they are they used do you see them being used after the fact uh you know as sort of learning pieces in terms of either yes for planning or for architecture reviews or like any of those kinds of things. Yeah. So, so the action items that come out of those reports um get created in Jira um which will then be sitting on those teams backlogs. Um and

29:11 so then you know next next planning iteration uh the team will be looking at you know looking at their their sort of like reliability backlog um and then then you know sort of like mixing that with with the other sort of various categories of work on the backlog um and then using that to prioritize uh what what uh what work occurs. And is there is there anything outside of action items in terms of like distributing what you learned? Um or is there any other activity around the incidents like that? Uh I'd say no other than the the sort of internal Slack channels where where these get published, right? And so anyone in the company is free to go and go and look at those um and and read through the reports. What was the most surprising thing to you about this incident? Yeah, probably the biggest one is that our edge provider would coalesce

29:58 requests indefinitely, right? Um, I can I can understand, you know, I can I can understand why that was the outcome, right? And I can I can imagine, you know, I can put myself in the person writing that code. Um, and it's like, oh, you know, like if if you know, fetch in flight, like wait for it to complete and then and then respond, right? Um but you know that was what allowed that thundering herd to develop right. Um and you know all of our internal RPC traffic has timeouts um which are sort of propagated and respected at each hop in that service graph. Um but you know in this particular case there wasn't there wasn't sort of like any any time out within that uh within that edge providers uh layer. Um which makes it which makes it somewhat difficult from a

30:46 you know a client code perspective to understand you know is this an error you know is this is this just like a slow fetch right or or is this like am I part of a thundering her right in which case like I I as the client would want to do something differently like you know add some back off there. Yeah. Yeah, it definitely seemed like one of the themes from this incident as well was and I I mean I see this a lot and I write about it a lot was how automation actually made it worse, right? Um and not because you're like we want the automation to make things worse, but because it can't always react to these kinds of situations and you know its model of the world is not as flexible as ours is. Uh,

31:31 and so, uh, you know, I I definitely saw like action items out of that and everything in the report, but was there what was there any reaction to that in terms of like where you had these automation surprises of of and not just like what was your response to the sort of automation making it harder and and how are you all adapting to that after the fact for the future? Yeah. So maybe maybe a good example is uh yeah I didn't I didn't mention this but uh part of what happened on that call uh was there was a person responsible for freezing the autoscaling right um because you know what can happen here is you know your traffic goes to zero um your auto especially that's that's like over a

32:18 sort of like fairly long amount of time um the autoscalers are like oh great like we can we can like downscale all of the groups uh and then the traffic comes back and then of course they can't scale up fast enough and like oh no again. Yeah. So we've we've got processes in place where you know for those particular types of incidents um we have tooling that we can run again sort of developed uh by those reliability folks where we can say like hey you like freeze all the autoscalers um and what we can do also is is go back and say like I'll actually you know set them like reset them to where they were sort of like prior to the incident occurring right so if if any have down downscaled um we can get them back. Um and so that's a that's a process that we've sort of adapted um you know given past experience and sort of thinking through what's what's like bad outcomes that could happen in these sort of events um

33:05 that you know like turns the automation off for a second and and let's let let's humans uh take control. Yeah. I mean, that's definitely a theme of of obviously my work, but like in instead of just saying, "Well, we'll add more automation to make the automation better," you all were like, "Well, let's give the humans better controls over what that might look like." I like I like seeing those kinds of of adaptations. Yeah. Like I'm uh I'm a very big fan of sort of modes within automated software where it just sort of like throws up its hands and it says you know like this this does not match my my view like what I've been modeled for and like humans please get involved right um when I've been building systems in the past uh you know I built in these these

33:50 lockdown modes right where it's like hey like the inputs don't match anymore I'm going to go into lockdown mode I'm going to flag it in my telemetry uh and and like human come and come get me out of this, right? Because rather than rather than just sort of like continuing down a path that sort of no one has prepared prepared the software for um and and possibly making things much much worse and yeah, what do you think was the biggest learning for the team out of this incident? So I think definitely practicing uh practicing those controls uh on the edge providers network. Um you know when we were putting that block in place uh we we added it with a very simple status message. Um, I can't actually remember what it said, but it was definitely not a not a user friendly status message, right? Um, because at the Yeah, all bad, good luck. Uh, yeah.

34:37 Well, it wasn't quite that bad. Um, but because at the time at the time it was like I can come up with a lot of bad error messages for you if you'd like, but probably not. Yeah. Um so so we because at the time the priority was like get the load down rather than you know debate the error message or you know like how it was how it was going to be visually presented etc. Right? Um and so now we've we've learned from that we now have sort of a canned response ready to go um in a much more uh userfriendly and visually appealing style. Uh and so that yeah that's that's sort of like one one thing that's come out of that. Um, plus also sort of our our uh ability to practice using those controls and getting more familiar with them. Um, because you you really don't want to be using something for the first time sort

35:24 of in a high pressure environment, right? Like you want to you want to have this uh you know like built up almost in your muscle memory. Um, one of the other sort of like general principles that I I try and ascribe to when um when building these systems is that any sort of emergency or or failure mode uh is exercised dayto-day, right? And that way, you know, that way you're you're prepared for it. You've been using it, you know, like like daytoday or sort of once a week and it's not a oh, you know, I've got to do this first first, you know, this thing first time in a call. You know, again, there's like a documentation page that is like four pages long, right? And I've got to execute these sequence of steps perfectly, right? Like the oven's on fire and I've never used the like fire extinguisher thing before and you kick

36:09 the Yeah, basically. Yeah. You want to be like, "Oh, yep. Do you all do do you do drills like tabletop drills, chaos engineering type stuff, or what does that preparedness look like? Is how does that definitely happens?" I'd say there's a there's a variety of drills that occur, right? Um like teams themselves will do uh will do sort of wheel of misfortune style or like role playing role playing incidents. Um but then there's also like larger drills that we do as a company. Um where we say like okay, yes, you know, we're going to we're going to do like a a sort of a a business continuity drill. Um and then our our storage teams will get involved and and and do those exercises. Thank you so much for joining me and sharing your internal process with the world.

36:55 And while I hope you don't have uh more SEZero incidents, I do hope that you continue to share your incident reports with us. And I I really appreciate all of the time and effort and cat hurting that goes into that. Um but as you said, it really is so beneficial to everyone else in the industry. So thank you for doing it. Yeah, no worries. And yeah, I just I hope I hope people can uh can can learn from it and we can get better. Uh everyone together. Absolutely. Okay. Thanks so much.