Matty: [00:00:07] Welcome to Arrested DevOps, episode 28, Incidents and Accidents: Examining Failure Without Blame. I’m your co-host, Matt Stratton, @MattStratton on Twitter.
Trevor: And I’m your co-host, Trevor Hess, @TrevorGHess on Twitter.
Bridget: And I’m your co-host, Bridget Kromhout, @BridgetKromhout on Twitter.
Matty: Arrested DevOps is brought to you by 10th Magnitude, a cloud services company that figures if you’re listening to this podcast, then you must be pretty cool. You can find out about joining their cloud services team at arresteddevops.com/xm.
Trevor: This episode is also sponsored by PagerDuty. PagerDuty eliminates the noise, chaos, and manual processes across the entire incident lifecycle to decrease resolution time. PagerDuty is trusted by companies like Etsy, Nike, and GitHub. To sign up for a free 14-day trial, visit arresteddevops.com/pagerduty.
Matty: This episode is also sponsored by Redgate Software. Redgate makes tools that bring the benefits of continuous delivery, safe releases, efficient development, and fast feedback to your database. You can find out more about database lifecycle management, you can download free trials, and even browse the database delivery learning program at arresteddevops.com/redgate.
Bridget: [00:01:25] If you’re listening or watching this live, please feel free to share your questions and comments on Twitter @ArrestedDevOps or on IRC, freenode.org, on channel Arrested DevOps.
Trevor: Joining us to talk about failure and postmortems are Dave Zwieback, VP of Engineering at Next Big Sound, and Mike Rembetsy, VP of Technical Operations at Etsy. Dave, can you give us a bit of background on yourself and your interest in this DevOps/postmortem space?
Dave: Sure, thanks for having me on the show. I have a very sordid past which included a lot of work in operations, and I eventually made my way into more general engineering management. I’m currently at a company called Next Big Sound. We do analytics for creative industries like music and books, and because of my involvement in systems and operations world, failure is always prevalent there, and dealing with failure and dealing with failure better has always been a question of how do you do it. And I’ve seen companies and organizations do it extremely well, and those were mostly the exception. And I’ve seen lots of organizations do it pretty terribly, including, you know, letting go of people and demoting them or otherwise blaming them. So When I sort of started to get interested in that area, of course, Allspaw has been— John Allspaw Ovetzi has been a huge inspiration. And I think he’s the one who kind of coined the term blameless postmortem. And he helped me, or led me, you could say, down the rabbit hole of human factors and the awesome work that folks like David Wood and Sidney Dekker and many others have been doing. And as a result of this, you know, I’m actually in the process of writing a book for O’Reilly called Being Blameless. Which is kind of a novel or a fable about how an organization might embrace this blamelessness.
Bridget: [00:03:21] Wow, I definitely want to hear more about that in a minute. But let’s introduce our other panelist. So thank you, Dave. And Mike, can I call you Mike? Because MCR seems more natural. But can you tell us about your involvement with examining and learning from failure in a DevOps context?
Michael: Sure, you can definitely call me MCR if you want. That’s what everybody at Etsy calls me. We tend to have a couple Mikes, so it’s It’s a little easy to differentiate people. Failure happens every single day here. It’s something that we deal with day in and day out. My career has been, you know, started in help desk, which is something that is riddled with failure, if you will. You know, oh, this mail client doesn’t work, this doesn’t work. So failures don’t have to be so large, but, you know, you start out small and you learn how to deal with those as a consultant. So I consulted for quite a number of years in a variety of different places, from NBCUniversal and iVillage.com and a bunch of financial folks and hedge funds that tend to be a little stressed about when things fail. And similar to what Dave had said, you know, I’ve seen organizations approach failure in the way of this guy or this woman or this person did this, and now we’re going to fix the problem by taking care of that person. And I think that, you know, as we’ve learned as engineers, that it’s not always a simple failure. Retired Lieutenant Colonel Scott Snook in Friendly Fire describes the complex system that has failed in the US military in ’93, ’94, I believe it was, when the 2 Black Hawk helicopters were shot down. So, you know, if you take that page from it and you take a look at how we as engineers approach different topics around failure, I think that the field has grown quite a bit. I have the distinct pleasure of reporting to John Allspaw and working with John every day. So, you know, he’s helped Dave down the rabbit hole. I think that I’m in the rabbit hole with John every day. So it’s a It’s quite a pleasure there. But yeah, I work at Etsy and that’s kind of what we do is we examine failure as a learning opportunity.
Bridget: [00:05:15] That’s fantastic. So Mike and Dave, we asked you on the podcast because of your experience in this space obviously. And Dave, you alluded to the book you’re writing for O’Reilly. I’m wondering if you can tell us a little bit more about both that and your workshop series on blameless postmortems that I know that you do. You have one of those coming up in February, right?
Dave: Yeah, February 12th. And I think the details will be linked to in the show notes. I’ve sort of discovered through learning more about this space and also talking to others about this space is— and by this space, I mean kind of doing postmortems well, meaning doing postmortems in such a way that they are a learning opportunity and not a blame or demotion or kind of any other thing. What I discovered is most people don’t really have an opportunity to experience what a blameless postmortem feels like. They certainly can hear, you know, John Allspaw or, you know, Etsy folks talk about it, but they’ve never had that direct experience. And that’s essentially what the workshop is about, is giving people an opportunity to experience so that they can actually kind of remember it and that when they go back to their workplaces that they can recreate that feeling. And so the workshop is really about giving folks that experience and also giving them some tools and a way to think about how to approach failure and how to approach investigation of failure. Of course, all that information comes from folks like John, John Allspaw, folks like Sidney Dekker, whose Field Guide to Understanding Human Error is probably the most important book for people like us, meaning people that are in the IT world. It’s very accessible and gives lots of examples from fields outside of IT, but they’re very relevant to what we do.
Bridget: [00:07:03] This sounds like something that people might want to attend. So it sounds like this is— you said it’s February 12th, it’s held in New York City, registration is still available. So if someone can’t possibly wait for your O’Reilly book to come out, they could come and sit with you and work through an experiential failure postmortem workshop coming up. Very cool. I wanna switch modes really quickly and ask Mike. So obviously Etsy is, I think it’s not an exaggeration to say, one of the most admired tech operations organizations out there. So can you give us the 10,000-foot view of how at Etsy you deal with failure? ’Cause you talked a little bit about it, but I’m interested in more, like how does Etsy react in the macro scale to, oh my God, something went wrong?
Michael: First, thank you for the compliment. I mean, I think there’s a lot of really awesome shops that are out there besides Etsy, so I’m humbled to hear that. Failure is going to happen, right? And it’s not a matter of if something is going to fail, it’s a matter of when something is going to fail. I’ve stood up at Velocity, and I’ve stood up at Surge, and a bunch of different conferences, and I’ve said this before, but one of the cultural things that— not even things, but one of the ideas that we try and embrace is that when something fails, you accept that failure, because it’s going to happen, and you do not lower your standards when it happens. So if I had to take a 10,000-foot view or even 20,000-foot view of how we approach failures at Etsy, we expect them to happen. We want to make sure that we know when they’re going to happen, and we also understand that we will also have lack of visibility into some failures, right? So there’s a couple different categories of failures you can have. You can have the ones that you know about, that they fail closed, and those are easy ones to detect, right? Disk space runs out on a box. Pretty simple to get an alert on that as long as you have alerts. The ones that fail open are the ones that are the surprises, and I think that a key part of understanding and accepting failure in any organization is understanding the stresses that come with surprise. Surprise reduction inside of an organization is something that I hope people would shoot for. I can walk you through a failure that happens at Etsy. It doesn’t really matter what it is. A failure could be anything from a service is degraded, like search is not working. Thankfully, our search team is pretty good, and that doesn’t really happen that often, so kudos to them. Or it could be something as major as we typo something in our network configs and the site goes down. Not saying that that’s happened, but it could, and maybe it will, or probably will, because it’s a matter of time. The approach is typically very calm. It’s not a chicken running around with your head cut off type of approach. Approach there. And people will gather into War Room. We have a War Room in our IRC channel which we use as our communication method. And if it’s serious enough, we also have Vidyo, V-I-D-Y-O, not video like E-Y, that we use as our internal communication tool for, you know, alternative to Google Hangouts. And really, the person who has discovered it tends to lead the conversation inside of War Room. And that conversation and however it goes, right, hey, maybe this is something, okay, let— I’ll take a look at that. Someone will volunteer normally. There’s no real, like, someone did this, because it’s all about fixing the problem at the time. Once the problem is fixed, however long that takes, it could take 5 minutes, it could be a deploy that did something, it could be a config that did something, it could be a server that died, it doesn’t matter what it is, right? Take the abstract from it, from a 10,000-mile view. That’s when the learning starts. We at Etsy have a couple of tools that we developed that maybe we could talk about.
Bridget: [00:10:38] We’re definitely going to talk about some of those tools, but before we get into too many details, I really appreciate that high-level view. Thank you so much.
Michael: No, it’s fine. I mean, the outcome of that high level, for the most part, besides the tooling, which we’ll definitely get into, is that we celebrate failure as much as we celebrate success here. One of the ways that we actually celebrate failure is on the Etsy website when we have a whoopsie or a problem, you’ll see someone with a 3-armed sweater. A couple years ago, we reached out to a seller on Etsy and we asked them to make us a real 3-armed sweater. We have a real 3-armed sweater in the office.
Bridget: I’ve seen it, you have it hanging from the ceiling.
Michael: Yeah, it’s actually right behind my desk, which I’m not near. The 3-armed sweater is given to the person who most spectacularly impacted the website in the year, and that could be users, and how they did that is always a fun way of figuring out who’s in the running. So we try and celebrate it as much as we also make sure that it doesn’t happen again if it’s a repeatable thing, or hopefully not.
Dave: [00:11:43] And what I’d like to kind of highlight here is until now we haven’t talked at all about tools. We talked about culture, we’ve talked about people, we talked about human factors, and that’s the kind of lens through which we’re looking at this area, at failure.
Matty: I would agree, and that’s something we talk about a lot on this show as well, is that as an industry and as technologists, you know, we like tools. Tools seem to be our savior. They seem to be our thing because tools are easy, right? They’re a thing you can understand, and they’re a thing you can install, and a thing you can be like, oh, well then I’ll just do this. So I like to talk, you know, they can get us where we need to go. What I want to ask a couple questions here is, you know, so Mike, that was awesome. You kind of talked about the how without getting into tooling. Why don’t you talk a little bit more about the why, right? You know, because I think there’s kind of this idea that, you know, sometimes we talk about the why like, well, that’s just the thing we do because that’s what Etsy does, and so therefore it must be right. So why do a blameless postmortem? I mean, what does your organization gain from this?
Michael: [00:12:47] Sure. So why do a blameless postmortem? Well, there is a real person who made a decision with the data that they had at the time. It was the best decision that they thought they could make, whatever that is, whether it’s not putting a check in place, whether it is pushing this bit of code, whether it’s hitting enter on the keyboard. And really diving deep into the thought process of what the person was thinking when they made that change results in almost a transformation of the person at the, at the end of a postmortem, a blameless postmortem. They start out, and truth be told, they feel really bad. I’ve done it myself. I’ve brought this site down many, many, many times while I’ve been here. You feel really bad about it because you’re making an impact on something that you care about. Going into a postmortem, the person feels really bad about it. The postmortem itself is a learning opportunity. Most people are— I should say, other people that I have talked to at different companies that don’t try a blameless postmortem, their only outcome that they want to have out of this is A, that it’ll never happen again, and B, that we fix the problem, whoever that is. And that is not the point of a postmortem. The point of a blameless postmortem is to learn from what we did, because it almost doesn’t matter what the person did. It matters what the outcome of what they’re going to do in the future is, or how the actual human felt afterwards and the stresses that they incurred, and how do we reduce those stresses? How do we make the environment better for them and better for the members to actually grow in the whole community? So the benefit really is on the human side of it as well as the learning side. Yes, you have to do the technical bits around it. You have to fix, you know, if you forgot a monitoring piece, if you need bigger hard drives. Like, there are technical things that will come out of these remediation items that you have, but really You know, you’ve got the technical bits that are going to happen regardless, but then you have the human factor bits of it where really you need to take a look and try and dig in to understand what those decisions were. Because at the time, if you had graphs in front of you and you see that everything is humming along great, you run your unit tests, you run the— you know, you have a peer review, you do all these different things, everything looks great, and you push that button and everything blows up, what could you have done differently As a person, what could the company have done differently to prove that that was going to happen? That’s really the learning bits, and that’s really what we try and instill on people here is like, hey, look, you broke something. Great, good luck. You know, it’s awesome that you broke something. Now let’s learn from it so we as a company, we as a culture come out better on the other end. When someone new starts in ops, it’s funny. They’ll walk in, they’ll sit down, they’ll get their computer, and they kind of look like deer in headlights, like most of us do at most new companies. You know, we’re not special. There’s no special aura in here. It’s an office.
Trevor: [00:15:37] We all work, right?
Michael: But the one thing that I’ll say to them within the first week is, hey, what have you broke this week? And they’re like, I didn’t break anything. And I’m like, why not? And they’re like, wait, he’s telling me that I should have broken something? And I say, yeah. I said, because how else are you going to learn what’s broken? Or how else are you going to learn what you can improve upon? And they’re like— it’s an amazing start because they don’t really fully understand that it’s about their own personal transformation as well as learning.
Matty: I don’t remember exactly— this is bad, I should have done my research a little bit better— but I remember the first time I was kind of exposed to this conversation was on the Food Fight episode about blameless postmortems. And Mike, it may have been actually you who said— I don’t know if I remember if you were on that one, but someone had the comment about one joining Etsy and saying like, okay, there’s this culture of blamelessness, and then fucked something up and took the site down, and boss came and was like, you know, well, so what did we learn? And you’re like, really?
Dave: What? Wait, no, no, really?
Matty: That’s how it is? You’re not gonna come and yell and scream at me? And I can, I can see how that’s really, you know, from such a history. You know, I mean, I don’t do ops now, but that’s been my whole career, and I’ve always come from a place of that it’s the— either there’s the perception or the reality that what’s gonna happen is someone’s gonna come yelling and screaming, and I remember at a previous position I was at, I had a sysadmin who worked for me who would always talk about— when we talk about certain things, he’d say, well, what happens when the CTO comes yelling and screaming and I need this right now and blah blah blah? And I told our CTO that story once and she’s like, when did I ever do that? And so it’s not always even based on reality. We kind of as an industry just assume that we’re gonna get in trouble, right? And I think that’s one of the things I’d like to understand a little bit about as well, and Dave, I’d like to get your 2 cents on the why as well, so I don’t want to cut you off, but is how do you help people through that cultural change? Because it’s really hard, you know, I mean, if you’ve spent a fair amount of your— I think someone who’s new, like in a more junior position, they don’t have 20 years of perception of getting screamed at, right?
Bridget: [00:17:47] They don’t have 20 years of emotional scar tissue.
Matty: Yeah, I mean, so—
Trevor: it doesn’t take 20 years though. It takes one bad experience to make someone shy away from doing something.
Matty: Yeah, I mean, so yeah, how do you, how do you help people understand that this really is okay other than just letting them have to experience it maybe for a little while?
Dave: The thing that you’re referring to that I think we all have like kind of muscle memory of is you walk into a postmortem and you go, yep, that was me, I fucked up, sorry about it. And in many organizations, right, let the feeding frenzy begin. Kind of situation, right? But either way, that approach from an individual often short-circuits the learning because we get to the, why did this happen? Here’s this person who is admitting, right, who is kind of doing the right thing, saying, yeah, hey, I screwed something up. I misconfigured the router or I pushed an incomplete patch out or whatever it is. And even though that’s painful, it’s also pretty comfortable because I did something, I screwed up, so I’m the cause of this outage. And it feels mentally kind of coherent, and it sounds like a good story to everybody involved. And to the extent— there’s actually a bias for this, right? To the extent that there is a good story, we feel that it’s true. The problem, of course, is it’s not true. And this is also the problem with the 5 Whys, because we get to a certain point We ask the question of, okay, this site went down. Why did it go down? Well, because we pushed a buggy version of software out. Why did that happen? Well, because we didn’t do enough testing. Why did that happen? Well, we didn’t have time. Why did that happen? And eventually, you know, you stop at some point which you arbitrarily choose. Maybe it’s 5 whys, maybe it’s 7, but you stop and you say, ah, this is the reason for this particular kind of— this is the root cause. And we as engineers, we believe in the root cause. The thing is, it doesn’t really exist.
Bridget: [00:19:48] Now this, I actually wanted to ask you about this because in the postmortem workshop materials I saw something about root cause analysis and I thought, oh, Allspaw would table flip this.
Dave: No, actually, so this is where John Allspaw and I kind of disagree a bit. So he’s got an awesome blog post, Kitchen Soap, about that there is no such thing as the root cause, and I disagree with him. So my view is that— let’s think about it for a second. Let’s say there’s an outage, and what is fundamentally present in each outage? Change, right? Somebody moved my cheese, somebody did something, and shit broke. Okay, good. Now what’s the fundamental reason for change? Right? If we start to look deeply, like way more deeper than like, you know, configuration changes and things like that. Well, it is because the systems with which we work are fundamentally changeable. Like they have the ability, the capacity to change. If they were not changeable, right, if they were kind of like permanent, static, unchangeable, then like we wouldn’t have this problem. The thing is, if we look even deeper, This impermanence, this changeable nature of systems, is the reason that they both function and malfunction. You know, we’re engineers, right? So state changes are needed and necessary for computing. If zeros aren’t changing into ones and back into zeros, we don’t have computing, period. So that change is necessary for both function, normal functioning, and also abnormal dysfunction of any system. So basically if we are looking for a root cause of any particular outage, it is this impermanence, you know. And this impermanence is also the root cause for the functioning of a system. And interestingly enough, we spend very little time like thinking about like why does our system work actually, you know. And we spend an inordinate amount of time trying to figure out why it didn’t work or it’s not working.
Bridget: [00:21:56] I don’t know, I spend some time trying to figure out how did this ever work. But that’s a— you make some very interesting points that I’ll have to mentally weigh against all spas. You know, there is no objective reality.
Michael: Yeah, I’m trying not to jump in on Dave on this. I’m like, really? Oh boy. You know, this is one of those discussions that I think is the deep, deep dark rabbit hole, Dave, that you and I certainly have conversations with John about, you know, in some ways. I am also of the other of John’s side, if you will. It’s not really a side. I guess it’s just a different approach to it. Is there really a root cause for something that failed? Is there really a root cause to say that that hard drive died? Why did the hard drive die? There’s no change. The hard drive is the same hard drive it was in the machine, right? What’s the change that caused it to die? What’s the root cause? We could definitely go down this route. I think that it’s I don’t know if you want to or not, but we could certainly have a discussion, and I think that this would have been something you and John would have argued about for the whole hour.
Dave: [00:23:05] The thing is, you’re right, and—
Michael: okay, conversation over. No, I’m just kidding.
Dave: The reason for any change is impermanence, right? That things are changeable, right? And the question I usually get is, well, how is this useful? Because I’m thinking hard drives and buffer overflows and not impermanence, right? Seems like this is very concrete and what can I do? I can’t do anything about impermanence. How is it even useful? And so the thing is, the reason it’s useful is it actually helps us, allows us to move on from looking for the root cause. Because if we’re looking for the root cause, I can tell you it’s impermanence. The question is, what’s the root cause of impermanence? Now we’re going to get to a very deep, deep philosophical rabbit hole. But let’s—
Michael: we’re going to need a lot more coffee for that. I was thinking other beverages, but Maybe after the next DevOps Days New York City.
Dave: Yes, sir.
Trevor: So you guys were talking about change kind of being an instigator for postmortem, but what is—
Michael: [00:24:09] Dave was certainly the one who said change causes outages, not me.
Trevor: Okay, I won’t put those words in your mouth. But what is the minimum criteria to cause a postmortem? What should be your your action to start one?
Michael: Dave, you want to go or you want me to? I got a bunch of stuff.
Dave: Yeah, go for it.
Michael: I think when you’re starting out with blameless postmortems, you know, and when Etsy was younger, it was easy for us, right? Site was up, site was down. It was pretty, you know, anything that was impactful to the end users on a large scale. When I say large scale, what do I mean? The site was unavailable, right? Because in the early days, like a lot of startups, we had a lot of issues. There’s no reason not to try and hide that. We had problems. As we grew and as we became better at our craft and as we became better with our systems and as we started adding more monitoring and more metrics and we started making them a bit more solid than they were in the past, what we’ve now started to notice is exactly that question. What does constitute a postmortem? What does not constitute a postmortem? If 5 servers fail because we unplugged the power PDU in the data center, but the site is still up, does that constitute a postmortem? And really what we’ve done is if it is impactful to the end users or a service degradation in sort of a P1, P2, P3, P4 fashion, anything that is impactful to our end users, we tend to have a postmortem on. So for example, if we push code and there’s a bad— let’s say we remove the buy button or something like that. We’ll probably have a quick postmortem about that. If our Hadoop cluster fails on the backend, that’s not customer impacting, we still may have a postmortem on that. But I think that each situation is now being driven— there’s no hard, fast rules of when to have a postmortem or when not to. I think if you can learn out of a failure, then it’s worth having a postmortem in order to go through the timeline of events, in order to go through what actually happened, to bring people together to share those experiences of learning and to walk out of there with some action items, or remediation items as we call them, in order to learn from them. We’re slightly revamping the remediation process, but we can kind of get into that after we talk about some of the tooling and whatnot. So that’s how we approach it in a lot of ways.
Dave: [00:26:26] Yeah, I think, I think the, the reason to have a postmortem is to learn, and if the learning opportunity is there, then let’s take it. The more we learn about the complex system with which within which we work, you know, the better we’re able to actually manage them and operate within them.
Trevor: How long is too long to do a postmortem? Or should a postmortem happen immediately following the event? Does a postmortem happen a week after the event? I want to say a year, but I can’t say that with a straight face. How long after an event should you do your postmortem?
Dave: Generally speaking, within a week or 2 weeks or so, that’s kind of the most common practice. There’s very little evidence that there’s any impact on the quality of the postmortem, like the longer you wait. I mean, without getting into sort of like years after, because the postmortem is essentially reconstructing events that happened in the past, you know, it’s constructing a coherent story about what happened. The time after the event, you know, is not of great consequence.
Michael: [00:27:30] So I agree with Dave. I mean, in terms of the timing, the one addition that I would add to this is I think it’s very important to make sure that the details of what happened, the timeline of the event, is written down almost immediately, within a day, maybe 2. You know, scheduling as you get larger becomes difficult to get everybody in a room, and rooms become unavailable. But making sure that the details are inside of that event so everyone can relive those and go through them and understand them is critical to making sure that the learning that comes out of it is maximized for efficiency. Say. Yes, within a week or two. We typically do not go beyond a week or two. If not, in a couple days we’ll have a postmortem. If something happens on a Thursday or Friday, by Tuesday or Wednesday we’ve had the postmortem in order to go through stuff.
Matty: I would think too, it’s like you need it when it’s fresh, right? You start to forget things. I would imagine too, just thinking back, not necessarily from a blameless perspective, but just from other things too, is that I think if you do it too soon, you’re a little too close to it, right? You know, you might still have a little bit of kind of some feels about what’s going on. It’s probably a little harder to be detached, right? Because as much as you might have a culture of blamelessness, if I got woken up in the middle of the night because Dave fucked something up, I’m gonna probably be a little peeved at him for at least a little bit, right?
Michael: [00:28:58] You’re talking about being hijacked, amygdala being triggered. It takes as quick as a millisecond to become hijacked, as we say. One of the things that we say here at Etsy is, are you triggered? Are you pissed off? Is something really bugging you? Did someone do something? I think that that’s a really good point, Matt, where you have this event and yes, people are going to have emotions. Why? Because we’re human beings. We feel. We understand. We get angry at things. We get sad at things. We get happy at things. Making sure that enough people’s amygdalas are no longer hijacked, that people are having clear conscious thought from their brain as opposed to that fast immediate response when they’re in this mode is important as well when you’re actually involved in a postmortem. You don’t want to have an outage in the morning, walk in in the afternoon, and then have your postmortem because you’re more likely to have emotions still running high in those rooms from people who maybe you’ve invested You might have invested hours of time in developing this feature and it just totally tanked. You’re going to feel really crappy about that. And someone else who maybe has a comment that sets you off is not going to help set the mood and the tone at all in the room. So giving it at least a day, I think, or 2 in between so people can, you know, get the right data, which is a timeline of events in order to discuss it, and have everyone walk in understanding that the egos are checked at the door and everyone is there to learn. Is a really important piece of actually having postmortems.
Trevor: [00:30:23] With that in mind, do you have a moderator in a postmortem to kind of make sure people are being responsible about checking the ego at the door and not being triggered?
Michael: Yes.
Trevor: Yes.
Dave: Yeah, and not just— I mean, the job of the moderator, especially in the beginning of each postmortem, is actually to set the context. And when we do postmortems here at Next Big Sound, we actually more or less read off or recite a series of things, reminding people that failure is inevitable, that we’re dealing with complex systems, this postmortem is about learning, it’s not about blame. And we also remind people that because they’re looking backwards in time, they will be subject to a host of cognitive biases, most prevalently hindsight bias. And we basically go, you know, I have actually a kind of a framework document that I can share with everybody here. That sets that context. And the other thing that works really well at the beginning, especially with a highly charged postmortem, is empathy and humor. When you walk into a situation and you go, you know, and this is the 3-armed sweater kind of thing of like, oh man, hey, you know, nice job on really fucking this one up, you know, like congratulations, kudos, let’s give them a round of applause, things like that, right? Something to, you know, and empathy of like, hey man, or hey woman, I’ve been there. Like, I’ve done this 100 times, no big, you know, we’re here to learn, you know, and we’re here to actually— like, it is possible to have fun during a postmortem.
Michael: [00:31:58] I think the term moderator, I would slightly adjust the verbiage to be a facilitator because I don’t think a postmortem is moderated. You don’t want to stifle opinions, you don’t want to stifle learning, you don’t want to, you know, you don’t want to stop that. You want someone who’s going to do exactly what Dave You’ve said. They’re going to come in, they’re going to make sure that people who haven’t been there— when I facilitate a postmortem, I walk in and I say, how many people is this your first postmortem? And there’s always a hand or two that goes up. People still are coming and understanding how a postmortem at Etsy works, or if they’re brand new to the company, making sure that people understand what the expectations are when they walk in and what we’re going to all do as a group. And keeping humor, absolutely, and saying high five, good job on that one. I mean, there’s always a few laughs in a postmortem, which is kind of morbid considering postmortem is the word that means after death. Anyhow, I think that you want to facilitate. Facilitation is not easy in a postmortem, in any, blameless or not. We have started to have internal— I guess I’ll call them classes. We have Etsy School here where different people within the company will go every semester and teach something. We’ve started actually teaching facilitation as one of the classes here, whether it’s learning and development or John teaching postmortem facilitation in order to actually help people learn how to facilitate a postmortem properly as opposed to just sitting on the other side. It does take some skill and it does take some practice, but for the most part, if people walk in with their egos checked, you’re going to have a pretty successful outcome, I think.
Trevor: [00:33:26] What are some techniques that you guys use to help facilitate, for example, bringing in those new people who’ve never been in a postmortem before, or people who may be introverted and not likely to kind of contribute to discussion unless you kind of prod them a bit.
Dave: Maybe I’ll answer a slightly different question, which is when you’re sort of constructing a version, a story about what happened, there’s a couple of different things that you want. Number one is actually you want pretty high diversity of opinions. Okay, because we’re dealing with human memory, recollection, even when we have logs, you know, and we have all kinds of sort of machine noise that is produced during these kinds of events, largely it relies on human interpretation and human memory of what had happened. And sort of understanding that and understanding that having more people in the room contributing to that, their version of events, their recollection of what happened, is actually super important. And so facilitating that is actually a little bit tricky because you will, you know, when you have a group of people, there’s always dynamics. You know, maybe the manager is the one who is speaking, or maybe the, you know, the more senior person is always the one kind of imposing his or her kind of version of events. And so the job of facilitator becomes like kind of soliciting that. And it’s even more interesting because we talk about sort of the timeline of event, and actually there are several timelines. There’s usually, I mean, fundamentally there are as many timelines as participants because they will have slightly different recollections. And so the job of facilitator also becomes this kind of synthesizing and putting together kind of a more coherent version of the events.
Matty: [00:35:17] So one thing I just wanted to, that’s come, there was a thought, I was thinking back earlier when talking about using the term postmortem, you know, and thinking about being able to do this type of work, this type of blameless evaluation, even when it’s not in relationship to an outage. Maybe it’s just an evaluation of how something went. And so this is kind of a little bit like off-res, you know, kind of— so I work in a sales organization at Chef now, so we do a lot of customer engagement. And I came up with this idea after about a month ago, I was like, we should start doing— I started writing what I thought were blameless postmortems. I didn’t know how to do one, but basically theoretically writing up my interpretation of saying like, here’s how this engagement went, here’s what went well, here’s what sucked, here’s what we can learn. But trying to write it in a way that wasn’t like, yeah, and then this was messed up because the sales rep like didn’t get the right information and he’s an idiot and blah blah blah and all this stuff. And then so I was calling them postmortems, but then the problem was It’s kind of like they weren’t necessarily postmortems because they weren’t necessarily that something went wrong. It was an evaluation, but kind of using that same thing. And I’m, I’m thinking, taking that from an operational or software perspective, just the way that maybe you evaluate releases or actions or anything that you do. Do you see where there’s potential value in having this type of an evaluation, maybe not having a big meeting to talk about it, But kind of— because as opposed to just talking about like, hey, let’s review when shit went wrong, what about blamelessly reviewing just in general so we can see how we’re doing? Is that something that’s within— are you doing that in a different kind of a way and just calling it something different?
Michael: [00:37:02] If I understand what you’re saying, yes, we are doing that in a different way. It’s not that things go right or wrong. Or are you talking about when things aren’t exactly right or exactly wrong? They’re kind of—
Matty: No, but just in general, just like sort of evaluating and doing a review of an action. Sure. As opposed to reacting to an outage, which is what a postmortem is, but using these same types of review techniques to say, like, hey, we released some code. And I know you guys are releasing like 35 times a minute, and so obviously you’re not going to review every single code release, but just something, right? An action that the company is taking. Evaluating, did it work, did it not? And I think even product-related stuff too.
Michael: So I mean, I think measuring the success of your product is something you should do, right? Whether you’re getting click-throughs or whether you’re getting more visitors or whatnot, right? But yes, you should be monitoring and evaluating things. You should have visibility into how things are reacting and working on your site. You should have, you know, thresholds set so that way maybe you’re trying to prevent something from happening in some way. I mean, it’s never gonna prevent What’s going to fail is going to fail. You’re never going to have this safety net that’s going to prevent everything from crashing down. But in order to have more transparency or visibility into what could potentially go wrong or when it’s going to go wrong or have some mechanism of viewing if it’s working or not is really in the monitoring realm in my mind and how we approach metrics and monitoring around stuff. If it’s a software, like, are we evaluating Chef, let’s say, or are we evaluating a new You know, we have different methodologies of how we approach that. We don’t look at a vendor or a product just as, does the product work or not. You know, we look at it as, you know, do their values and impacts align with ours as a company. How do they treat failure inside of their organization? Do they just fire and fix the person, or are they in line with how we approach failure as well? But I don’t know if I answered your question or not.
Matty: [00:38:58] I think, I guess maybe I realized as you were talking a better way to ask my question. Question, which was more about saying, like, how can we think about different parts of the organization that can use these same types of blameless techniques within their work? So I’m thinking specifically, like, again, with product people, right? To say, like, hey, when you’re evaluating, right, like, taking these same types of ideas of blamelessness and evaluation of these things and taking that same approach I guess it’s not really a question. Maybe it’s just more of a thought, you know, to say like for our listeners, if you’re not an ops person, right, like you can still get stuff out of it. You can take these same techniques, these same things.
Trevor: Absolutely.
Matty: And apply them to your particular discipline.
Michael: Yes. We actually have non-engineering postmortems at Etsy. We’ve done those before. We’ve had them in member operations. They may take 2 days because it’s a very complex problem. But we have done them before, and yes. So I think you can absolutely take these evaluation techniques, these retrospective techniques, these postmortem techniques, and take a look at how these things apply into areas that are maybe non-engineering. For instance, someone wrote into support and they asked for support on how they open a widget on site whatever. How did the support handle that request? Did it go quickly? Did it go smoothly? Was the person knowledgeable? You can make a timeline for any event that you want, and you can come up with remediation items out of it, which is why we created Morgue. Because we realized that it wasn’t just an engineering problem. We realized that multiple people across, whether it’s HR, whether it’s finance, whether it’s anywhere else within Etsy, can actually apply the same postmortem techniques, blamelessness, across their discipline.
Dave: [00:40:41] So there’s kind of a bias towards failure. And, you know, so when something goes wrong, we want to really investigate and understand it, right? And I think what we’ve been talking about is also using those same techniques like retrospectives to learn in general from working with these complex systems. So here we do, for every iteration we do a retrospective, which the way we do it is kind of stop, start, continue, and people write by themselves on little sticky notes what are the things that worked, what are the things that didn’t, what are the things they want to try next time. You know, and we sort of collate and organize this information, then we share it with the entire company. The idea is so that everybody learns from the things that are working kind of in different parts of the company. The other thing is that we can actually apply some of these techniques prospectively, not retrospectively, but for things that we’re about to do. And I know the Etsy folks do this kind of as a go/no-go or pre-op review, or what’s the term you guys use?
Michael: So we have 2 things. We do an architectural review of a feature or stack or something that we’re talking about, and that’s more to talk about the design and the architecture of it and how it’s going to work with the other systems. And once it gets past that point, we have an operability review, and that operability review talks about different things such as, is our community team aware of this? Is our communications team aware of this? Is our forums team aware of this? Do we have monitoring? Do we have metrics? What do we do if it fails at 3 o’clock in the morning? So we’ll do a full operability review around that where we do do a no-go or go on that end of it, yes.
Dave: [00:42:10] And there’s, there’s a bit of a kind of technique to doing those. Usually people do them, they say, hey, what could possibly go wrong? And the answers that come in are heavily biased because everybody who spent a lot of time working on the system wants to see it go live usually. So they have kind of an overconfidence or a kind of an optimistic, overoptimistic bias. So they think basically generally things will go pretty well. Now, if you deploy a technique called a premortem, which is something that Gary Klein and company have come up with, you structure that questioning a little bit differently. You say, imagine we deployed this new feature, or we went into this new business line, or we decided to invest a whole bunch of time and money into a particular product, or we decided to hire this person, and imagine things went really badly. How badly? Well, it wound up on the front page of whatever website or, you know, New York Times or whatever it is, that badly. You know, and we imagine that scenario 6 months or a year in advance, or even, you know, an hour in advance after we’ve deployed this feature. And then what we do is we ask, well, what went wrong? How did we wind up on the front page of the New York Times with this really bad news? And the answers that you get from the people in the room are actually a lot different. It’s all these things start to come out like, oh yeah, well, we actually didn’t think about how we’re going to communicate that to our clients, or we forgot to do this. And so that information is what essentially we’re doing is we’re harnessing hindsight bias, but like in the future by transporting ourselves mentally to the future and then working back from that.
Bridget: [00:43:49] Dave, that is a fascinating technique. And I’m also interested in hearing about a little bit more in this tool-specific space. MCR mentioned Morgue. And I remember at the DevOps Days that you and MCR ran together in 2003, seeing Bethany Macri present about Etsy’s then newly open-sourced tool, Morgue. And so I know a little about it, but I’m not sure all of our listeners will. So Mike, maybe if you want to give us the quick overview of what Morgue does for Etsy and how you use it.
Michael: Sure. I think it was 2013. I don’t know if I knew Dave in 2013.
Bridget: I think I said 2013.
Michael: Anyways, doesn’t matter. It was recent. So we document all of our postmortems, and the way that we were doing that was we had a wiki page, and that wiki page kept growing and growing and growing and growing, and it just became unmanageable, right? Because wiki is the place where things— documentation goes to die. So we decided to create an internal tool that we named Morgue, M-O-R-G-E, and we open-sourced it couple years ago, so it is available on our Etsy GitHub, our public GitHub that is up there if people want to download it. And what it was is it’s a way for us to have a more concrete way of tracking all of our postmortems. And the tool itself allows you to create a postmortem. It allows you to add things like start time, end time, the time that it was detected, so you can measure things like MTTR, MTTD, the severity— was it that P1, P2, A quick question for Dave.
Bridget: [00:45:21] You mentioned Sidney Decker’s Field Guide to Understanding Human Error, and I read it, and one thing I noticed is a lot of the discussion about human error is from the perspective of accident investigators who are trying to figure out what people were thinking by analyzing black box data. Now, a lot of the postmortems that we’re doing in this industry, usually— now, I can’t say in every case, but usually the people who were involved in the incident are actually available to discuss it with you. Can you talk a little bit about the different approaches when you do have access to the people who experienced the incident versus trying to reconstruct? Because it seems like a lot of Decker’s work is focused on, at least in the edition I read, it’s focused on the reconstruction aspect.
Dave: Yeah, a lot of Decker’s work focuses on those industries because obviously those are like super high stakes, you know. If you’ve watched, I believe, Richard Cook’s keynote at Velocity, recently, he talks about that it’s only a matter of time before the software that we all write and operate is in those life-or-death kind of situations. And it’s already happening. So it doesn’t make our investigations any less important, even though most of the time, luckily, hopefully, nobody died. There is not that much difference in my mind between reconstructing events when folks are available and folks that are when folks are not, simply because again you’re reconstructing from memory, from different fragments. And the thing that you’re trying to do, which I think Sidney Decker kind of illustrates and demonstrates really well, is what did it look like, what did it feel like at the time that this was happening from the perspective of the person kind of going through it. Not from the perspective of here we are today looking back at something. And this is kind of the Jedi mind trick really of the postmortem investigations, is how to like transport yourself mentally back in time and look at the situation from that perspective.
Bridget: [00:47:24] That is fascinating, and I can see how having effective tooling and the timeline and all the details that MCR is mentioning in Morgue would actually help with being able to reconstruct those mindsets. Mike, it looks like you’re back with us.
Michael: Yeah, sorry about that. I don’t know why I dropped, but, you know, figure it out. So I don’t know where you lost me, but it sounds like Dave filled in for me, so thanks, Dave.
Bridget: Yeah, it sounds like, well, he was talking to us a little bit about the perspective of people examining an incident after the fact versus the people who were there, and Dave made the excellent point that Even the people who are there are going to have all sorts of cognitive biases and be subject to, you know, not necessarily recollecting what happened even if they think they did. So it sounds like what you were talking about with Morgue, I think we lost you right when you were telling us about a lot of the different kinds of information that can be captured in there. But it sounds like what you’re using it for is to make sure you actually have all those details.
Michael: There certainly is a piece of data collection on the top part and then there is the piece of the actionable items of it, the remediation items on the bottom. It was the best explanation that no one ever heard, so sorry about that. But at the same time, you know, we’re also approaching remediation items a little bit differently now. We’re trying to revamp those a little bit. What we noticed as we have more and more remediation items, how are they getting done? Are they staying on track to getting done? Are they just things that don’t necessarily need such attention? Is it working? So one of the things out of this that I wanted to bring up is that people should always be reevaluating how they’re approaching, you know, getting their work done, whether it’s workflow or whether it’s remediation items, to make sure that it’s actually something that’s a work in progress. And then for us, it is a work in progress right now of how those remediation items are working or not working. It’s just a matter of us, you know, still using this format, still using the tool. It is a key component for us for everything that we do.
Dave: [00:49:14] What I also love about Bethany’s presentation at that DevOps Days was that during the Q&A, somebody asked Well, you know, how long does it take for people to do remediation things? And more importantly, does it ever happen that people just drop remediation items? And I remember the look of surprise and just like, what do you mean? And so it kind of comes back to this thing of culture and humans deciding that remediation items from these postmortems are just about the most important thing to work on, right? So of course you evaluate it to everything else that’s going on. You don’t have unlimited resources, unlimited time. But generally speaking, in companies that are successfully working with complex systems, they prioritize the work that comes out of these postmortems, prioritize essentially dealing with failure, sometimes higher than actually new features or products and things like that.
Bridget: [00:50:14] Excellent. So Dave, way earlier in this podcast that is running a little long, and I know that we could possibly do a follow-up episode, but I don’t know, Matt, do we ever actually do follow-up episodes that we promise?
Matty: We eventually might do them. I would like to point out that our episodes have all gotten super long since Bridget joined us.
Bridget: Hey, well, you know, but anyway, Dave did mention earlier that he alluded to one of Sidney Decker’s points about how storytelling, I think the quote is, is a powerful mechanism for others to learn vicariously from trouble. And I’m wondering if maybe if as we start to bring this to an eventual close, if both of you on our panel want to share a little bit about how constructing these stories and sharing them is meaningful or useful or has worked for your organizations.
Dave: Yeah, I mean, again, The point is, when you’re constructing a postmortem, the timeline and all that stuff, it is storytelling. It is constructing a story. We also do the workshop that I mentioned and you guys mentioned before privately for companies. And one of the companies that we did it at had a pretty diverse group of people, including folks from marketing, communications, and, you know, client services and so on. And the moment that we said, Like, hey, you know, we’re constructing a story of what happened. The folks that do this for a living, that write press releases and communicate to people, you know, by blogs and so on on behalf of the company, they immediately got it. They’re like, oh, okay, I know how to do that. And so the extent to which we can make the story compelling, the extent to which we can make the story true, is the extent to which we can all learn from it. And so all the sort of tips and tricks for good storytelling actually apply to writing up a postmortem. I think some of the most masterful postmortems that I’ve ever read were written by— who am I thinking of, Mike?
Michael: [00:52:10] John Allspaw?
Dave: Yes, I always think of John, very fond of him.
Michael: JB Sadney? David Woods?
Dave: No, no, the ex-Heroku, ex-Github—
Michael: Oh, Mark Embriaco?
Dave: Yes, yes, thank you.
Trevor: Yeah, Mark.
Michael: Mark is very good at those, yeah.
Dave: Yeah, and so, you know, he’s actually— I think he wants to make it his career at some point, that’s his kind of aspiration. I think he could. The ones he writes are exceptionally, I think, truthful and engaging, and also like there’s a lot of learning there. So I always use his postmortems kind of as an example of what are the good ones.
Michael: Yeah, his storytelling when it comes to publicly explaining to folks exactly what went wrong is humbling, truthful, as well as not your typical, sorry we did this, it won’t happen again. There’s a level of humanity— is the only word that I can think of right now— that is in there that is real. You know, there’s no— it doesn’t feel like there’s cruft. It doesn’t feel like it’s a request for outage, you know, type of document. It’s a truthful conversation that’s happening between him and the public. And I think that, you know, trying to maintain that and aspire to do that is something that we all can do. Public postmortems are very difficult, mostly because you have a non-technical audience potentially. You have some technical people. Writing to the public is certainly a skill set that Mark is very, very good at. He certainly is someone that I tap on the shoulder when I write publicly and say, hey, can you take a quick look at this and make sure that— can I get a buddy check? It’s always helpful to have someone peer review your work. Storytelling is important for all those reasons that I mentioned.
Dave: [00:53:48] And it’s an opportunity to actually help. Any kind of outage makes your customers kind of question, is this really— should I really be working with this company? You know, doing a postmortem that actually gets to the conditions and talking about it as publicly as open. We know that in some, like banking for instance, you can’t always do that. But talking about it even internally as widely as possible. Helps people learn and also helps people build this confidence in, okay, you know, these people really understand the system that they’re working with. And the other postmortems are, when they’re written kind of poorly, it actually basically cements this perception of like there’s a bunch of people that don’t seem to know what they’re talking about.
Bridget: All right, this has been fantastic, and I feel like we could keep talking to you gentlemen for hours, but unfortunately we can’t. But we do want to move into our checkouts where we talk about something that you may have brought that you want to share with our listeners that is of interest to you. So Dave, you want to go ahead?
Dave: [00:54:49] I would always recommend Sidney Dekker’s The Field Guide to Understanding Human Error, now in the 3rd edition. I think it’s certainly been kind of a life-changing book for me, and he’s one of the folks that I think talks about this area investigations, postmortem investigations, the best.
Bridget: Great, fantastic.
Michael: I got a couple things. So I work on the Etsy Sustainability Commission on the inside of Etsy, if you will, in some ways. My focus this year and the end of last year has been mostly around our carbon footprint and how do we actually make a more impactful and lasting world as an e-commerce marketplace. So what I checked out and what Etsy has become involved in is BSR, which is basically working with businesses to create sustainable world and visions of how they can do that. So there’s really large companies, there’s really small companies. That’s one thing that I’ve been checking out to see how they can actually help Etsy. The second thing is something that Bridget actually mentioned to me a while back that I’m finally getting around to, which is the immersion cooling from 3M around data centers and how that could potentially help with some of the heat dissipation and is that even possible from where we are? So that’s the second thing that I would check out. And thirdly, if everyone wants to check out Morgue, that would be fantastic. It’s up on our public GitHub, so github.com/etsymorgue/morgue. And that’s what I would say for my checkout.
Matty: [00:56:11] Thanks.
Bridget: Hey, thank you. My checkouts are basically— I went winter hiking in the Boundary Waters up on the Gunflint Trail. The Gunflint Lodge is a pretty great place to go even if you’re not canoeing in the winter as well. I did a lot of reading. I did read the Decker book, though I read the earlier edition, so I’m going to have to reread it now. I read Allspaw and Jesse Robbins’ Web Operations, John Cowie’s Customizing Chef, and the first 3 chapters of Jason Dixon’s Graphite book when I was not hiking out in the snow. All recommended. Nice.
Michael: Trevor?
Trevor: I was on vacation also and delightfully disconnected. It was pretty awesome. I got a new Kindle, and I’ve been reading through Game of Thrones before Matt accidentally, though honestly at this point it’s my own fault. But Matt will eventually spoil something for me, and I just know it.
Matty: Which Kindle did you get?
Trevor: Paperwhite.
Matty: Okay, yeah, I just replaced— my Paperwhite broke, I just replaced it with a Voyage.
Michael: I kind of like it.
Trevor: So far the Paperwhite’s pretty good. I now have Decker’s book waiting for me at my desk now. We also just set up a new KegBot in our office, which means we’ll have metrics about office beer and root beer consumption, which will be fun to play with.
Matty: [00:57:25] These are important things.
Trevor: Oh yeah.
Bridget: Etsy would tell you to pipe those metrics into Graphite.
Dave: Yeah, correlate them to outages.
Michael: When the keg is empty, the outages continue.
Dave: Yes.
Matty: Yeah, exactly. So I was on vacation too, so I took a couple weeks off, which was great. I was doing kind of the dad ops thing and hanging out with my kids, although I was sick for most of it, which was not delightful. But I did see Big Hero 6, which was Trevor’s pick in our last episode. And Trevor, you’re right, that was a really good movie. So I don’t know if that counts as a checkout since Trevor did it, but if you didn’t see it, you should because it was really fun. I’ve been back into audiobooks a lot lately. The one I’ve been listening to lately that I’m working my way through is The Challenger Sale by Matthew Dixon and Brent Adamson, which if you’re not involved in sales is probably not that terribly interesting to you. But if you have anything to do with product or selling product or working with customers, I really recommend it. It’s been, been pretty great. We’ve got a couple things we want to talk about, some upcoming conferences. So FOSDEM and Config Management Camp are at the end of January, beginning of February. We’ll have links to them in the show notes. Also, Agile Open Northwest is in the middle of February. I will be there. It’s actually sold out, but maybe you can get on the waitlist.
Bridget: [00:58:51] Let’s not forget February 12th, the Postmortems workshop in New York City.
Matty: That’s right.
Bridget: Another event that the illustrious Dave Zwieback and his partner in crime, Yulia, is it?
Dave: Yeah.
Matty: We will have links to all these in the show notes at arrestedevops.com/28. So the specific links to the things. If you’re into wanting to give talks, we’ve got some information about some CFPs that are open. The CFP for ChefConf will be closing January 9th. It’s quite possible that by the time you hear this, it will be after January 9th, so sorry about that. The actual ChefConf is March 31st through April 2nd. We will be having a discount code, but right now it’s too early in the morning for me to get it from Nathan, but we’ll be announcing it on the next episode. Episode. So listen to the next episode and to get your discount code. We’ll probably tweet it as well.
Bridget: And, um, I probably should mention that DevOps Days Derby’s CFP is closing January 11th. The event’s February 7th and 8th. And Ljubljana— am I pronouncing that correctly?
Dave: [00:59:56] Maybe—
Bridget: and Paris DevOps Days are also coming up in April, and their CFPs are open right now. So hey, our European listeners, you don’t have very far to go to get to your next DevOps Days.
Matty: Monitorama’s CFP was closing on February 1st, and Velocity Santa Clara will be closing on February 2nd, as well as OSCON. Yes, and also, so Agile 2015 CFP is closing February 22nd. The DevOps track chairs are Domenica De Grandis and Jennifer Davis. I actually have submitted a talk for that myself. Depending on what you think about Agile or not, actually it’s a pretty cool conference, and the DevOps track Last year was pretty amazing. So, and couple things, where are we going to be for the upcoming fortnight, Bridget?
Bridget: Let’s see, I’ll be in San Francisco speaking about Drama Fever’s Docker setup at the San Francisco Advanced AWS Meetup on January 20th, and then I’m flying back to Minneapolis on the 21st because I actually co-host our AWS meetup that is meeting that night where Donnie Burkles will be speaking.
Matty: [01:00:57] I am all over the country this month as per per usual, but if you happen to be in Chicago on January 22nd, I will be hosting the local Chef meetup where we’re going to be talking about Chef 12. I’ll have a host— a link to that in the show notes.
Trevor: And I will probably be at that also, but I will be at the Chicago Azure and Microsoft Tech Group meetup on the 8th of January in Chicago, obviously as the Chicago Azure group. We’re going to be doing an open forum type discussion about the new Azure portal.
Matty: Awesome. So, reminder, we have a newsletter, arresteddevops.com/bananastand. It is the best way to know about our upcoming podcast episodes, cool news and links with DevOps. I want to give big thanks to Mandy Moore, @therubyrep on Twitter, also at devreps.com. She is the best in the business for helping out software professionals from administration and research to event organization to podcast production. You can check out all of the available things at devreps.com.
Trevor: [01:01:58] And thanks to our sponsors. Be sure to visit them at arresteddevops.com/pagerduty and arresteddevops.com/redgate. Thanks, Dave and Mike, for joining us, and to our loyal listeners. If you enjoy Arrested DevOps, we’d appreciate it if you would visit arresteddevops.com/itunes and leave us a review in the iTunes Store.
Bridget: Be sure to check us out at arresteddevops.com or @ArrestedDevOps on Twitter. We’re always happy to get your input, ideas, or feedback at shows@arresteddevops.com. And please let us know any ideas you have for future episodes. I’m Bridget @BridgetCromhout.
Matty: I’m Matt @MattStratton.
Trevor: And I’m Trevor @TrevorGHess.
Bridget: We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand.
