Docker! Docker! Docker!

Matty: [00:00:08] Welcome to Arrested DevOps, episode 31, Docker, Docker, Docker. I’m your co-host, Matt Stratton, @MattStratton on Twitter.

Bridget: And I’m your co-host, Bridget Kromhout, @bridgetkromhout on Twitter.

Matty: Arrested DevOps is brought to you by 10th Magnitude, a cloud services company that figures if you’re listening to this podcast, then you must be pretty cool. You can find out about joining their cloud services team at 10thmagnitude.com.

Bridget: We all know that being on call sucks, but what if there was a tool out there that allowed you to route incidents to the right team, @mention specific people to ask for help, and hop into chat with your team from an easy-to-decipher incident timeline that gave you full context of what was happening? That tool is VictorOps, and they’re different. From setting up global on-call rotations to creating a postmortem report, VictorOps is there with you through every step of the incident lifecycle. Their real-time collaboration platform helps your team to solve problems faster. Sign up for a 14-day free trial to see how they’re making on-calls suck less. Visit arresteddevops.com/victorops to sign up.

Matty: [00:01:09] This episode is also brought to you by Datadog, a monitoring service for scaling cloud infrastructures that brings together data from servers, databases, apps, and other tools. Datadog monitors your Docker container’s performance and correlates these metrics with the performance of the underlying infrastructure and the applications running inside. Check it out with a 14-day free trial at arresteddevops.com/datadog31.

Bridget: Joining us here today to talk about Docker, Docker, Docker is James Turnbull, VP of Engineering at Kickstarter. James, thank you for joining us. Can you give us a bit of background on yourself and your interest in this Docker stuff?

James: Sure. So I’ve been in engineering and operations for about Well, quite a few years, more than I care to think about. Over the years, I’ve seen a lot of containerization solutions. Docker is obviously not a new thing. But when I saw them open source the Docker product, Solomon Hykes gave a talk at PyCon that was really awesome. I was finally like, my God, this is actually a solution that’s built by people to be usable by people, as opposed to some of the previous containerized solutions, which were built by engineers to be usable by a very small subset of other engineers. So I recognized pretty early on that I thought Docker was going to be sort of a pretty big change in the way people are going to do things and had the potential to sort of facilitate a bunch of DevOps-y-like stuff.

Bridget: [00:02:31] Well, we are all about DevOps-y-like stuff here. You have listed in your lower third Kickstarter, Docker. I know that you did work at Docker. Can you talk a little bit about what this Docker advisor role means? What is your ongoing involvement with the project?

James: My initial involvement with the project was obviously I was involved in the open source side of things. I wrote a book about Docker. Eventually, I worked at Docker running the services and support side of things, basically building the customer-facing business. Unfortunately, Docker’s in San Francisco and I’m in New York, and a startup at that sort of stage requires a lot of on-the-ground presence in the office. So I was on an airplane quite a lot. I was also doing a lot of evangelism. Going to conferences and talking about Docker and going to meet customers and things like that. So I spent about— I spent a frightening number of nights in hotels last year. It’s something approaching 180 to 200 nights. Needless to say, my wife and my friends were like, do you live here in New York anymore, or do you live in airport lounges? When you start to recognize the airport lounge staff and the flight attendants on the New York-San Francisco route, I start to recognize you. You realize that it might be time for a bit of a break. I parted company with the Docker guys and took a more— stepped back and took a more advisory role, which is really around— I provide a bunch of advice around the open source project and the governance there, around the business and the roadmap, and a bunch of other areas like that. Great.

Bridget: [00:03:58] Awesome. You mentioned The Docker Book. I know that you keep writing books. You say that R.B. Brown puts up with this, which is a good thing. But can you tell us a little bit about what led you to write a Docker book specifically? I have it. I’ve read it. I know what parts I find the most useful, but can you tell me what were your favorite parts to really dig in and write a book about?

James: The big thing for me around Docker was that when I first looked at it, it was much easier to use than a lot of other tools out there, but it was an entirely new technology to a lot of people, particularly to a lot of developers. It feels very infrastructure-y to developers. It feels very sysadmin-y. So, I wanted to write something that would sort of allow people of varying— a very introductory book that would allow people of varying skill levels to sort of be able to understand really quickly, you know, what is Docker, what is the value I get out of it, and how do I use it, and here’s some useful things. So, it’s very much a practical book. So, it actually focused on things like, I would like to test an application, or I would like to build a service, or I would like to use Docker with Jenkins— things that a typical developer or sysadmin would do. And those are probably my favorite chapters. Chapters 5 and 6, which sort of talk about Docker as a testing tool and Docker as a production services implementation. And it touches on sort of, you know, continuous delivery, continuous integration, and touches a bit on things like microservices and different architectures, which was really exciting to me because it sort of allowed people to sort of see this is what this might look like in production.

Bridget: [00:05:31] Here’s the million-dollar question: Are you using Docker in production at Kickstarter?

James: We are. We’ve actually been using it longer than I’ve worked here. One of my colleagues implemented it. One of the things that’s really interesting about our environment is that everybody basically has a replica of the Kickstarter environment locally on their laptops. So, we do local development there, and that’s a heavily Dockerized solution. At some point in time, we’ll probably look at doing some more Docker stuff in production. We’re a big AWS user, so there’s things like ECS that are interesting. I think, from our point of view, it presents some useful scalability advantages.

Bridget: Yeah, cool. I mean, that’s actually— and I know Matt has some questions for you about that, but I’ve got to say, what you just described is some of why we find Docker really useful where I work at DramaFever, too. Just the ability for the devs to have production on their laptop is really powerful. Here’s the exact container that’s running in production, and go.

Matty: [00:06:33] Which is a thing I wanted to kind of get a little— I wanted to kind of bring back a little bit, because as I kind of joked, I said this is going to be the episode where Matt sits around and has no idea what James and Bridget are talking about. So, I’m representing the people that aren’t as experienced with Docker or kind of work around it. I don’t mean around it, but I mean it’s part of our ecosystem. Try to get around it as this thing that gets in my way. But I want to understand. One of the things that I think about, the same thing with PaaS, I remember going through this on the Azure PaaS side, which was nobody wanted to rewrite their applications to be cloud-native, but new stuff they would do that way. Is that a true statement when we think about containerization, is that if you’re talking about true microservice, if you’re talking about something that’s built for that type of a role, Do you get more bang for your buck versus trying to take something that’s a more traditional application and saying, well, sure, that’s great that I can have a copy of everything on dev, which I think is awesome. I mean, I love config management for that reason, right? You know, I love the idea of a local VM that looks just like prod. Do you get that utilization? And Bridget, I’d like to know, like, in your experience too, where you’re doing that, are you seeing— are you talking about applications that were designed to be used in this manner, or are you taking applications that weren’t designed to be used in that manner and being able to let Docker make them easier?

Bridget: [00:08:04] We are a startup that’s been around for 5 years, and we have a giant monolithic Django Python app, and then we added a whole bunch of Go microservices later. While the Go microservices are, of course, easier to Dockerize, we do have the Django app in Docker. We did move it into containers. Before I got there. This happened in October 2013, I think. I’m pretty sure that’s about— maybe James knows, but I think that’s when Docker’s website pretty much just had giant letters in blink that said, do not use this in production.

James: October 2013, did you say? Yeah. Those letters didn’t actually come down until mid-2014 at the first DockerCon, where We actually, the night before we launched 1.0, I removed all references to the word, don’t use in production, that sentence, don’t use in production, and redeployed.

Bridget: Yeah, so we did migrate a preexisting app into Docker. From what I understand from my coworkers, Tim and Paul, who did it, who were there at the time, it was not without its challenges to Dockerize. But yeah, I mean, I think it’s certainly not impossible. You don’t have to just be greenfielding in order to use Docker. James, what is your experience on this?

James: [00:09:23] I think that’s one of the really interesting things about Docker is that the Dockerfile is incredibly simple to build. Obviously, Docker lends itself to microservices really strongly, but it’s not the only architecture you want to use. I think there’s a lot of brownfields apps that have ended up in Docker because it is inherently so easy. And it’s very easy to integrate with configuration management. So, you’ve already got those things with Chef cookbooks or Puppet modules, then it’s very easy to use those to build Docker images. Same with tools like Packer and things like that, and Vagrant, they integrate with Docker as well. If you’ve got a bunch of existing infrastructure, it’s relatively easy to introspect that and create Docker images from them. The very fact that it tends to be focused on that build-dev-test workflow means that There’s a lot less cycle time in building some of that stuff that, you know, you look at— let’s say you wanted to replicate the production environment in a test cluster in your sort of data center or locally, something like that. That’s probably a non-trivial exercise, whereas, you know, sort of building up a Docker image, you know, you can do that sort of relatively quickly. You don’t need to, you know, buy 10 new test servers, or you can actually sort of play around with that in a much more much faster than you would otherwise be able to.

Bridget: [00:10:42] And especially if you get into some of the orchestration tools that you cover in your book, like Fig, for example. Some of our devs have started using that, and we’re finding that it’s pretty helpful to put— to drop a fig.yaml file in any given repo and then say, hey, this is how you Fig up the stuff you need to actually test this project.

James: Yeah, I think multi-tier— I mean, obviously multi-tier apps are something that Docker is And Fig makes a lot of that, which is— Fig’s about to be renamed to Compose, I believe. Fig makes that a lot easier, and it allows you to do something like, you know, a sysadmin or an operations person can create a Fig YAML file that contains, like, you know, my Rails container, my NGINX frontend container, my Unicorn container, a database container, and a Redis container, and basically, you know, Hook them together, create all of the links between them. A dev runs one command and gets a functioning version of their application that they can work from on their laptop locally. I sometimes think that people forget how hard this is to do and how before Docker started talking about— people sort of forget the past and they go, oh my God, Docker’s a pain in the ass to use. Compared to what exactly? Compared to your previous build or compared to you shipping around, you know, 10 ISO files and running, you know, Vagrant and 20 VirtualBox VMs on your local machine? That’s a non-trivial thing to set up. And I’m sort of like, you know, I think the phrase I use every now and again is, I think people Docker test too much. Because it’s a new thing, as opposed to remembering the fact that it wasn’t that long ago that the Dark Ages were real. I’m not suggesting that Docker is a panacea, but it’s certainly a step in the right direction.

Bridget: [00:12:38] Yeah, that’s definitely been our experience. I actually joined Drama Fever right when we were in the process of— and this is probably backwards, maybe, I don’t know— but we had actually gone to production with Docker and had Docker on the shared dev QA environments in EC2. And then last summer, right about, I think, when James was editing the website to say it’s okay to use it in production now, at that point, we were starting to get all of the devs to actually use boot2docker. And so I came in during that part, and it was actually really— it was fun, and it was a great learning experience for me. And it was also— It basically showed me that having the devs have the ability to grab the containers, that it’s not, is this sort of like, did this get, was HEAD at the right place? Did this get checked out? Now, am I loading the right fixtures? My Vagrant box may not be exactly like what’s in QA right now. Well, what is in QA right now? And it’s like, now you can just grab the container, grab the container image for what you want, start the container you want, and you know you’re looking at the right thing.

James: [00:13:44] Yeah, I think that’s a really important step forward. I think that the fact that Docker does smooth the edges around all of those little subtle differences, like fixtures and versions of gems or pips or whatever it happens to be, you can’t underestimate how much time engineers spend actually troubleshooting shit like that. I fairly regularly have conversations with engineers who go, I lost 3 hours of time because my local machine bundler is stuffed on my local machine, or RVM is broken, or rbmv is broken. And those of you familiar with the Ruby world know that’s not an uncommon occurrence. And when it happens, it’s a pain in the ass to fix.

Matty: Yeah, that kind of ability to just sort of instantiate, you know, however that’s done, I think is, again, whatever the mechanism is, like you said, the dark ages weren’t that long ago. And I think I just, just not, you know, kind of, I make a similar statement when I talk about just having to do with the testing tools we have around configuration management. I’m like, hey, in the olden days of a year ago, here’s how we tested Chef cookbooks, right? You know, and it’s—

James: [00:14:54] do you remember the time before Test Kitchen? All of 2 and a half years ago?

Matty: Exactly right.

Bridget: Every day, every day now I say thank you, Fletcher. So we had a question from IRC. Someone named Tom Fui says, do you have databases in Docker in production? And I let him know that at DramaFever, we use RDS for what we have in the MySQL container for the dev environment— or for the local environment. Like, actually, in dev, QA, and prod, we just are pointed at RDS. But James, in your experience, do people use databases inside Docker in production?

James: They do. So we have— on local machines, we also use RDS. For production and staging, but our local development instances point to MySQL running in Docker. Same with Elasticsearch. That’s one of the things we Dockerized really early because it is a little bit of a fiddly thing to build— right version of the JVM, right version of Elasticsearch, prepping all the data and all that sort of stuff. Those sort of things are a natural fit, I think, for a lot of environments.

Bridget: [00:16:02] Yeah, I guess actually our Elasticsearch is Dockerized as well. I hadn’t really considered that in the scope of this question in terms of production just because it’s not in our request path.

James: Yeah, and Redis as well, all of that sort of stuff. Anything that we sort of think about that could be easily self-contained. Sometimes, we’re not necessarily— like in the case of dev environments, I don’t have to worry about the continuity of the data all that much. If I lose the MySQL container, with a replica, with my sanitized production replica in it, I’m like, eh, I’ll rebuild another one. Obviously, I’m not necessarily sure I’d feel the same way in production, but that’s why we use RDS.

Bridget: Something about the data in dev that is actually kind of valuable for us is we actually— James alluded to talking about using Jenkins in conjunction with Docker in his book. At Drama Fever, one of the things we do is we actually build all of the images. Build all the images and push them to our private registry on Jenkins. So we build the MySQL container and load in all the fixtures that we want people to be able to use for their tests. So they can pull down a container that actually literally has everything they need for the website to, at least in kind of a limited fashion, work.

James: [00:17:23] I think for continuous integration, I cannot emphasize as much people should Definitely look at Docker because you’ve got— you’re dealing with execution environments that live for a very short period of time, usually while the tests are run. Often the tests are destructive, or they’re likely to leave the data in an unclean state, or a state where you have to rebuild. I think Docker, just in terms of its startup speed and its build time, is a sensational combination for continuous integration. Absolutely sensational. We’ve seen people cut their build times in half.

Bridget: Yeah, it’s really the thing that takes the longest for us is running the tests, but getting the actual container doing everything we want it to do is faster.

James: That’s just it. I mean, let’s say it takes you 30 minutes to run your tests and 10 minutes of that, or even 5 minutes of that, is build and execution on a virtual machine. Taking it down to 25 minutes from 30 minutes over— and, you know, if you’re doing continuous deployment and you’re trying to deploy more than multiple times a day, those 5 minutes add up pretty quickly.

Bridget: [00:18:25] When you’re talking, and you said that you did go and travel a bunch, and I saw you at a few conferences last year talking about some of this stuff, it seems like recently on the internets or whatever, the discussions around Docker and its ecosystem and the environment around it have kind of heated up. How do you find yourself reacting to some of the controversy?

James: I guess it’s not such a— having lived with it for a long time, it’s not such a shock to me. There’s always going to be people for whom hypey things are hypey, and I don’t blame some people. There are certainly journalists out there who have simplified Docker to being like, this is a revolutionary technology that will cure world hunger. I’m fond of saying that Docker is a powerful tool to help you in your sort of in your development lifecycle, if you want to get code off developers’ laptops or out of repos into production as fast as possible, it’s one of the best tools that’s ever been developed for that. It’s not a panacea though. It won’t— not every workload in your data center is well suited to Docker right now, certainly. It certainly lends itself better to more immutable or to high-performance computing or to things that are sort of mini-node, like if you’ve got Hadoop or you’re doing sort of data analysis or anything that involves jobs or anything like that, Docker, you know, it’s like a natural fit. But, you know, if you’re running Oracle, you know, as a backend for SAP, I’m not sure yet that Docker is your ideal solution.

Bridget: [00:19:58] I would agree. I don’t think Docker is for everything. Like, we had, right before I started at DramaFever, we had a serious, like, let’s Dockerize all the things project. And some of the things that we Dockerized, like our Graphite server, I’m not convinced It’s benefiting from being Dockerized.

Matty: I’ve had people who want to say, well, can I run Chef server in a container? And I kind of say, I’m sure you could, but I don’t understand why.

James: Yeah. I look at Graphite as a really interesting example of this. So like I would probably run Carbon Relay and Carbon Cache inside Docker containers, but I’d probably point them at a physical machine with SSD disk in it to do actually burn the Whisper. Or burn the metrics down to, you know, but I can easily do things like I want to add 10 new Carbon relays. I can do that really quickly with Docker and very simply. That makes that sort of scalable ease of use stuff really simple, but takes consideration of the fact that, okay, maybe Docker isn’t the best way to write out transactional real-time metrics to a file system. I don’t know. I’ve never performance tested it. I would suggest that you need to look at each implementation and say, where does Docker fit here? Are we actually gaining stuff? Are we gaining stuff? Okay, we are. We should Dockerize that and we should leave the bits that don’t need to be Dockerized either until Docker has improved its capabilities in that area or not at all. It’s the same. It’s like use the tool that’s best for the job rather than sort of go, you know, all of the tools need to look the same.

Bridget: [00:21:34] Right. Yeah, I think that that’s exactly what we’re now to a point with our Docker use that we are reexamining it and figuring out where we can make improvements, like much smaller images, say, for the Go applications that we’re shipping that have very few dependencies. Hey, we got it down to a 10-meg container, or rather, a 10-meg container image. Excellent. Makes for pulling the image and when stuff auto-scales a lot faster. So there’s places like that where we’re iterating towards improvements. But then there’s also places where we’re kind of looking and going, do we really? As we refactor this, that, or the other, maybe Docker isn’t the right solution for absolutely everything, and that’s okay.

James: I think a lot of people respond to stuff like that and they go, I don’t like that because someone has made a sweeping generalization. Personally, I don’t base my technical architecture and my product decisions based on what tech journalists write or even what excitable people write in a 2-paragraph blog post about Docker will cure world hunger. I make them based on, you know, I try a bunch of products out, I look at a bunch of architectures, I do some testing, I collect some data, and then I make a decision. If your immediate response to something is, I hate this, I’m sort of skeptical about those sort of responses. It means someone basically hasn’t put a lot of thought into whether Docker is a fit or not and what Docker’s strengths and weaknesses are. Also, people don’t like popular things. There is a certain backlash against things that are— in Australia, we call it the tall poppy syndrome, like the flower that climbs above the rest of the other flowers, first one that gets chopped down. I think there’s probably an American equivalent of that phrase.

Matty: [00:23:15] Yeah, we would say it’s being a hipster, right? It’s like, oh man, I used Docker before it was cool, so screw that.

James: Hey, I resemble that remark.

Matty: Yeah, exactly, right?

Bridget: Does it count as a hipster if one of the reasons you decide to work at a place is because they’re using Docker in production and it’s mid-2014 and you’re like, this is a kind of fun, cool idea?

Matty: I actually think that’s— I mean, if we kind of think about it, we’ll probably talk about this in our next episode, which is going to be about starting a new DevOps job, about why you decide to work where you do. I think that things like that could totally be really legit reasons in moderation.

Bridget: What it really says to me is that they’re not afraid to try things.

Matty: Right. I’m saying it’s the level of the of the cut of the of the risk. Yeah, if it was something I was trying to think of something clever and it didn’t happen. And being funny is the only thing I’m able to possibly contribute to this episode.

Bridget: So actually, no, that’s not true because you read the thing on Reddit that I didn’t want to read because I thought it was going to do bad things to my blood pressure.

James: [00:24:15] Also, oh, that blog post. Okay.

Matty: Well, it’s not even so much the blog post, but Matt read it.

Bridget: He read it.

Matty: Matt.

Bridget: Found some information in there that he did want to talk about. So, Matt?

Matty: Yeah, so to give a little bit of background, and I’ll put the links to this in the show notes to take a look at it. So there was a post that was linked to in the r/sysadmin subreddit, and the post is called Docker is Fundamentally Flawed, Useless Hype. Or actually, I think the post is actually called Revisiting Docker Again.

James: Yeah, I think Matt also made an original blog post some months beforehand where he looked at maybe Docker 1.0 or something like that, and he was now looking at it again.

Matty: This was his revisiting, which I mean— and the thing was, I will say this, that kind of looking at the writer’s, the author’s comments much later in the thread, it was still very contentious, but, you know, kind of was doing a little backpedaling about the, hey, I’m not a journalist, I’m not a tech expert, I’m just saying my thing. But it was fairly inflammatory. But that being said, What I was more interested in, because I don’t have the chops to like even really comprehend whether or not it was valid or not, was there’s this long flame war thread where people get all angry and, you know, and I— but I kind of was picking through it and saying, well, when I take away some of the like flaminess and FUD and all this stuff that is so popular in that subreddit, there were 2 things that I said, hey, as someone who doesn’t really know this, These sound like legitimate questions in terms of, I’m actually curious, so I would like to pose them and get your thoughts. This first one, and I’m going to quote directly from the user, and it says— user is A-KO. It says that, in this person’s opinion, Docker is a DevOps solution to an operational systems problem. It’s a solution to a problem from a developer’s point of view, and someone that has little to no experience with the OS platforms they’re running on, Docker provides no security benefits at all. What it provides is an operational and developer benefit, but it makes security 100 times worse. Okay, big thing. I— the thing about that, like, to unpack it, the 2 pieces. One is there’s this belief, and again, it’s the sysadmin subreddit, so of course they hate all developers, right? So Docker is a dev tool, like, you know, whatever, blah blah blah. So I’d like your thoughts on that, and then also this security implication.

James: [00:26:33] Yeah, so this is one of the things that when I looked at that blog post, the thing that struck me most about it was that it was a very— it lacked a lot of empathy for those people that actually have to use technology. And this is something I get very strident about. I’ve been on both sides of the fence and I’m notoriously a terrible developer, but you know, I have worn that hat. I’ve been a sysadmin. I’ve been a security person. I’ve been a CTO and a VP of Engineering and an ops manager and a bunch of other things. So, you know, I consider myself somewhat of a jack of all trades. And I think one of the reasons, if I can be a little bit— one of the reasons I think I’ve been reasonably successful at what I do is that I have empathy for other people’s problems. And a lot of sysadmins, you pointed it out, you actually sort of hit the nail on the head. There’s a lot of sysadmins in our sysadmin that hate developers. And I think that’s true of a certain subset of sysadmins. Of the operations and sysadmin community. They’re basically, developers are really annoying. They want things. They want test environments. They ship you code that doesn’t work. You know, when something breaks, they’re not the ones that get woken up at 3 o’clock in the morning. They’re cowboys. They’re always cutting, writing new things with no conception of how production works. All that sort of thing. And I look at that, and the first thing that strikes me about that is, A, Have you thought about, you know, what’s going on in their world and why they’re doing that? The second thing is that you do realize that your job doesn’t exist without them. And for a lot of sysadmins, you know, you don’t manage infrastructure that sits there and is infrastructure for infrastructure’s sake. You have 2 customers. One is the business and by extension their customers, and the other is the people internal to the company that rely on the infrastructure. Who are people like DBAs and developers, business analysts, all of those people, those people pay your salaries. And I find it galling, if not outright sort of— makes me very angry, let’s put it that way, when I hear sysadmins basically lack that empathy entirely for the people that are effectively their customers. They don’t ask themselves, you know, why do developers end up shipping me this code that doesn’t work in production? Why do they build these things that don’t fit our security model or can’t be backed up or don’t fit into our logging environment? And hence, we are on a DevOps blog post. The most logical reason for that is they have no fucking idea what production looks like. And why don’t they have any idea? Because there’s this grumpy asshole who manages production, and they’re terrified to go and ask them a question.

Bridget: [00:29:16] Yeah, I used to be that grumpy asshole. Yeah, it’s so much better when the devs are not afraid to talk to you.

James: Correct.

Bridget: It makes me much happier as the former grumpy asshole.

James: It makes me much happier when the devs are willing to talk to me, and it makes the whole environment more productive because they stop doing things like shipping things that don’t match the production architecture and that don’t, you know, make assumptions about libraries and versions and operating systems and connectivity and security and all that sort of stuff. You know, I guess I look at this in response to go, well, this is a developer solution to an operational problem. And I go, so, you’re complaining because the people that you have tortured and/or ignored who actually provide the code that runs on your infrastructure and makes your company money have decided to solve this particular problem with or without you? Surely you want to get on this bandwagon before someone pushes you off the building or out of the building and replaces you. The other irony I find particularly amusing is that Solomon Hykes, who was the original architect of Docker, comes from such a strong operational background. He’s a developer for sure, but he actually ran a PaaS. He was the CTO of the ultimate in infrastructure companies.

Matty: [00:30:32] Without getting into whatever thing, but I think we’re fairly aware that there’s this perception as well that Chef is for devs. And that makes me laugh all the time for anybody who knows Adam Jacob or knows probably 90% of the people that work at Chef. So many of these things are, like you said, they’re built by sysadmins and who get it, right? And it’s— that just gets missed. We ran into this with our episode we did with Jeffrey Snover, the guy who created PowerShell, and I’m seeing people being like, oh, here’s some marketing guy and blah blah blah, and I’m like, do you guys not understand that like this thing you do that you love, like this guy created it? And because he’s done your job. So yeah, anyway.

Bridget: Also, on that security aspect, we don’t worry about containing things inside the container for security’s sake, because at least in our implementation of Docker in production, in our request path, on our production servers, we are securing the actual host instance that the containers are running on, and we’re not trying to protect the containers from each other. Like, I don’t know that I would use it were I running something like DigitalOcean. I don’t know that I would use Docker to separate customers from each other, but that’s not the use case that at least we have.

James: [00:31:47] And it’s not something Docker has ever claimed. So I didn’t answer this. I got a bit ranty. I apologize. But that security question is really interesting. So if you look at the Docker recommendations for this, we basically say you should run applications of like security risk together on Docker containers. You shouldn’t use Docker containers as security containers. They are not virtual machines. But then again, for that matter, virtual machines, a lot of cases, if you run PCI DSS, you shouldn’t share PCI DSS applications on the same virtual machine cluster despite them being far more hardened edges than, say, a Docker container is. So we make a series of recommendations about like risk profile applications or like security zone applications being run together. We strongly recommend using things like SELinux and AppArmor, and as Bridget highlighted, we strongly recommend you secure the host that it’s running on. And that’s the level of protection you look at.

Matty: Yeah, okay, awesome. I think that actually pretty much answers the second point, which was to say, you know, username Strangewill said, I wonder how various compliance regulations view this idea of run one role per server with containers with elevation exploits. So I could see them arguing the separation between containers isn’t as clear-cut. Between VMs and a Type 1 hypervisor, and that’s exactly what you just said. It’s not, so don’t treat it like that. So that kind of addresses that, right? Which is, don’t— it’s not a hypervisor. It’s like, yeah, so I think that’s probably just a question. Like I said, some of these were the things that made sense to me. I’m like, oh, that’s a good question. Yeah, to then be told, rightly so, well, no, that’s not what you should do. I’m like, oh, okay.

Bridget: [00:33:27] But it’s good that you brought those up because I think those are some of the things people get concerned about.

Matty: Yeah. That’s what I’m saying. I’m not necessarily expecting that these are like, this is why Docker bad, but it’s helping to educate, you know.

Bridget: Race conditions with device mapper is why Docker bad.

James: But the really interesting thing about security and compliance is that you ask half a dozen of these people who have an opinion about security and compliance, how many of them actually read any security guidelines or any of the compliance things? Actually, I will challenge anyone in that thread to actually tell me if they’ve read the PCI DSS standards. Now I have repeatedly because I was a security guy in a bank. I guarantee you 90% of the people I talked to who say PCI DSS needs to do X have never actually implemented PCI DSS and have no idea that X is actually a very low bar. And if you followed simply the regulations, the compliance stuff that related to things like PCI DSS, you would be running a massively insecure system.

Matty: [00:34:27] Yeah, and likewise, I think what we run into is this understanding of the, what I need to do for compliance is what I’ve been told to do.

James: Correct. Yeah, and outcomes.

Matty: This happened, we did a, one of our older episodes, so, uh, restofdevops.com/6, our Mythbusters episode. You can hear Sasha Bates like go all crazy about it. I must watch that.

James: That’s hilarious.

Matty: Like, oh, can I answer this one? And it’s totally true.

Bridget: So now that we’ve, we’ve asked you about some of the fear, uncertainty, and doubt out there, talk to us about the Open Container Spec stuff, Rocket, like anything else in the ecosystem. Is any of that like, does any of that mean that the sky is falling or has already fallen? Or maybe people should be paralyzed by indecision now because nobody has any idea what’s going on at this point? What kind of recommendations would you make around that?

James: I’m going to probably say a few reasonably controversial things in advance to the community. My view of Rocket is that CoreOS need to create a market and ecosystem, and the way to do that is to own the standard. Obviously, they identified very quickly that if Docker owned the standard, then Docker would own the ecosystem around the standard. They might not necessarily build all the tools, but they would hold the— even with the very loose governance model that Docker has, they would at least be somewhat of a gatekeeper around that ecosystem. They’d certainly gain all the buzz and the marketing, as they have, around that ecosystem. What’s the way to try and build your own buzz and your own ecosystem around that is you release a bunch of standards, or you publish an alternative standard. I personally think the CoreOS guys left it a little bit too late. I think that the run that Docker got before that, even though there were some issues around how transparent some of the early development around standards was, but broadly speaking, I think they left their run too late. I would be very surprised if they get a lot of traction around Rocket just because momentum— even if it’s a better technical solution, I see if— and I’m definitely going to be showing my age here— Betamax versus VHS. Betamax, far more elegant technical solution. Did it win? No. So I’m not sure if there’s a millennial equivalent, maybe Blu-ray and DVD or something. I don’t know. But LaserDisc, I don’t know, maybe even that one.

Matty: [00:37:02] There has to be a lot because that comparison gets made all the time. So there must be other things because I feel like we always do this. We say it a lot. It’s like Betamax and VHS, but I don’t remember what any of those things are that we’re comparing it to.

James: Yeah, so I think I’m not particularly— if I was the Docker team, I wouldn’t be particularly worried. I think Solomon’s kind of— and, you know, in his defense, he’s very passionate about this space, so he does get a little bit worked up on occasion. But rightfully so, he effectively built this whole ecosystem. Hype or not, he built this whole thing. You can say, yes, he’s standing on the shoulders of some giants, but he did actually push all this stuff out. I think he’s rightfully hurt that people are not collaborating on something. It’s not even the business stuff to him, which— Solomon’s a pretty smart guy, but at his heart, he’s an open source sort of guy. I think he’s kind of hurt that people didn’t want to collaborate on stuff and they wanted to build an alternative. He’s also really open to taking criticism about Docker. He just hates criticism that comes in the form of nobody providing solutions or answers or coming up with alternatives or coming up with alternatives that don’t involve collaboration.

Bridget: [00:38:19] All right. So, when I feel ranty about device mapper and race conditions, I should submit some pull requests.

James: You should talk to Michael Crosby. Michael Crosby is currently somewhere in San Francisco. Let’s get going then, motherfucker.

Matty: Bridget, I think you need to change your Twitter handle to Device Mapper in Race Conditions.

Bridget: Unlike some of us in this Google Hangout, I don’t change my name on Twitter every week.

James: I didn’t even know you could do that until recently, so that’s how—

Bridget: It’s really aggravating. Please don’t.

Matty: Here’s something that’s interesting to me, and James, I don’t know what— we can kind of think about this philosophically or technically. But looking at how people like Amazon and Microsoft, the way that they’re embracing Docker and providing it from a service perspective, and in both kinds of— those are, I think, two different kinds of questions. I think there’s one big question, which is the, hey, Amazon Azure saying Docker is a thing, we want you to be able to Dockerize natively. But then also thinking about Microsoft’s embracement of Docker in the first place. And kind of your thoughts and feels around these things.

James: [00:39:29] Sure. I hearkened earlier to the sort of concept of getting code off laptops and out of repos and onto production infrastructure. If you’re Amazon and Microsoft, both certainly in the AWS and the Azure space, every one of those pushes is more money, right? So every more— every time you get more of that code and faster onto more machines, Is revenue. So Docker, if it facilitates that, you know, both in the case of Amazon and Microsoft is like certainly the cloud services side of Microsoft anyway, they look at that as really attractive. And if you look at some of Azure’s considerable success in the Microsoft, more traditional Microsoft space, it’s their integration with things like developer tools that have actually allowed them to become, become so So like if you’re in, you know, if you’re in— I can’t remember what the bloody thing is called— Virtual Source Safe or whatever the developer— I can’t remember what it’s called now— the developer front end for Microsoft is called.

Matty: [00:40:34] Visual Studio.

James: Visual Studio, that’s it. I keep thinking of VSS. I knew it was VS something. Visual Studio, one of the things about that is you can click a dropdown, add your Azure credentials, and deploy your application from Visual Studio. That’s got to be driving revenue towards Azure because that’s incredibly easy, right? And if you’re a developer with no infrastructure skills whatsoever and you can just go click button, launch application in cloud, make available to customers, that’s really attractive. So if you can do the same thing with Linux-based tools and to expand some of that market out as well, the Azure guys have got to be thinking, that’s a good idea. AWS feels the same way. AWS’s initial customers were heavily developer-oriented. If you did not have operational people, Amazon was a very attractive offering because it sort of removed some of that— someone else manages up to the data center layer and sometimes even a bit forward with AMI, manages a lot of the operating system level stuff. Docker enables those people to get more workloads faster. It’s pretty attractive.

Bridget: [00:41:40] That makes a lot of sense.

James: These guys are smart business people, right? They’re not going to ignore an opportunity like that. And a lot of people have said, oh, you know, they’re just— because it’s just market hype, they want to integrate. And I’m like, no, they’ve actually got genuine business reasons for doing this. And you need to think very carefully through their business models to understand what those reasons are. And it’s not— it’s pretty transparent. Verna, most of the sort of fairly recently, has sort of said this. Listen to people like Adrian Cockcroft and people like that, you’ll get some insight into exactly why the public cloud players think Docker is extremely attractive. And Microsoft more broadly sees the same thing at the desktop level. They’re trying to enhance that market share.

Matty: That was— this came up, you know, that was one of the things Jeffrey Snover said on our last episode with the whole, you know, Microsoft loves Linux, and people like, oh, is it real? And he said, in the context of Azure, he says, we make more money if you run 10 instances of Linux on Azure than if you run 4 instances of Windows.

James: [00:42:41] Yep.

Matty: So it’s— they don’t— they’re like, we’ll meet you where you are, you know. And, and then I guess to me, thinking about, like, from a pricing perspective, I guess this is kind of the thing too, is, you know, does this help not keep people from taking advantage of— but if you’re like, oh, I guess it’s sort of like, what’s the model? And I guess the model is, again, at the end of the day, you’re selling compute. You don’t really care how it’s sliced and diced if you’re Amazon or whomever. So you’re like, hey, you want to have a whole bunch of containers, you want to have a host, you want to have a whatever, doesn’t care. It’s a certain amount of compute time, a certain amount of CPU, a certain amount of RAM, certain amount whatever, but it’s all here.

James: And every tool that drives you into the ecosystem— like Amazon is really good. I describe Amazon as a walled garden. Yeah, you— there are some pretty flowers inside the walled garden. You wander into the walled garden and The door closes behind you and you go, I can smell the pretty flowers. A bit later on you go, some of these flowers are pretty pricey, and you realize you can’t find the door, and when you do, it’s locked. This is an awesome business model. So anything that drives developers towards that walled garden and then keeps them there— if you’re an Amazon product manager, you’ve got to build that offering.

Matty: [00:43:52] Nice.

Bridget: Last question for you, James, before we move into talking about community stuff and some of our checkouts and whatnot. Is it seems like you’ve done a lot of thinking in this space, unsurprisingly. And it makes me think of like that whole Wayne Gretzky thing, like skate to where the puck is going to be, not where it’s been. Like if you were giving people advice, possibly people who are listening to this podcast, if you’re giving advice right now as to what people should be paying attention to, what people should be using or doing or giving a shit about in this container space, like right now, where is the puck going to be?

Matty: Sure.

James: So I think you can ignore all infrastructure stuff. It’s kind of boring anyway. And, and it’s, it’s, it’s only been a bit, you know, there’s been a bit of, you know, how it talks to systemd and device mapper and AUFS and all that sort of stuff. It’s kind of meaningless. What I would be looking at is the orchestration and application management capabilities. I’d be looking at the people who are building integrated SDN solutions. Software-defined networking, that is, and also software-defined data centers. All of those people who are building that sort of stuff out of Docker components. I’m not particularly— I’m kind of— I’m not hugely positive on things like PaaS and OpenStack and Cloud Foundry and stuff like that. I’m never— I think all of those things have kind of failed. Well, I wouldn’t say failed. They’re not particularly interesting to me as technology. But I look at things like Docker Compose, which is the renamed Fig, the stuff that’s going to come out, Docker Swarm, which is all of the sort of clustering technology. People who are building networking things and routers and switches and software-defined things with Docker, and people who are moving up the stack to manage whole— manage different levels of abstraction. That’s where things start to get really interesting because that’s one of the areas where configuration management is also heading. Like if you’re Chef or Puppet, then you’re clearly looking at managing up the stack towards applications. Docker is clearly doing the same thing. If you’re a developer and you can now, instead of having to run that PaaS or that OpenStack, you can take, like, you know, Fig version 3 or Fig version 4 that just automatically runs you a complex stack, both locally in development and then the same complex stack in production, that’s a really attractive offering. Like, that’s a really interesting sort of path. I have to— I can stop worrying about this clustering stuff, or I don’t have to pay as much attention to it as I did in the past, or I don’t have to rely on hugely brilliant engineers to build me this thing. It now becomes a black box commodity service for me. And this—

Bridget: [00:46:37] you make a really good point too, which is that it’s not like anyone has, you know, their data center with Docker and only Docker, and that’s the only thing that they use, and that solves all of their problems because that would be probably sort of ludicrous. And when I think of things that we’ve actually recently introduced to an ecosystem that already had Docker, we’ve started using Chef. We’ve started using Packer. We have specific use cases that those things are incredibly helpful to us for. So it’s kind of like it’s important to remember just what you said. It’s like there’s not one answer that solves all your problems, but this is a direction that is super useful.

James: And the other interesting aspect there is that people forget that there’s no secret sauce in this stuff, right? Everyone running NGINX and Apache and MySQL is probably running it 80% of the same way as everyone else. I challenge you to find a MySQL, a LAMP stack site where 85%, 90% of the configuration files are absolutely identical. So we have to remember that as sysadmins and operations people, you know, we need to be sort of looking to move up the stack ourselves to manage applications better. Because our secret knowledge of which thing to tweak is not as valuable as we think it is. We need to actually understand what is valuable, which is making applications faster, making them deliver them more quickly, offer a better level of service, offer business and customers things that will actually make them look at us and go, wow, these guys are really helping us sell widgets or provide services or whatever it is.

Bridget: [00:48:16] That, and if you rely on secret knowledge to make your life happy, that life will involve never going on vacation to the Boundary Waters and canoeing without a cell phone. And that life will also involve a lot of waking up at 4 in the morning. I recommend, you know, not going with that.

James: I would substitute sitting by the pool in Costa Rica, but okay, you can go with the outdoors.

Bridget: The pool is probably outdoors. If it’s in Costa Rica, I don’t think they need to have an indoor pool.

James: Cocktails with umbrellas in them. I’m pretty sure they do.

Bridget: Love it. Okay, so let’s, let’s move into talking about some upcoming community and event stuff. Angie from Container Camp tweeted earlier actually that I’m gonna, I’m gonna be out in San Francisco on April 17th speaking at Container Camp, and I’ll have a discount code BK15 for a 20% discount. That gave us some fits of confusion until we realized it’s 2015. I don’t know when that happened. But we also are going to have a couple of free tickets to give away. So anyone who, let’s see, we’ll pick a winner for today out of everyone who tweets I love @ArrestedDevOps and @ContainerCamp within 2 days of the live broadcast. And then again, we’ll pick a second one within 2 days when the episode goes live on iTunes. So there’s that. And then Matt, you’ve got one as well.

Matty: [00:49:41] Oh, right. So yeah, so just remember ChefConf will be March 31st through April 2nd in Santa Clara. If you go to chef.io/chefconf, the code ADO will get you a 10% discount. Bridget’s speaking, I’m speaking, Trevor’s speaking, other people are speaking but that aren’t on the show. But yeah, so come, it’ll be awesome and fun. James, what cool things are you going to be doing?

Bridget: Oh, yeah. James, because I know that you like other things besides Docker, and we didn’t even get a chance to talk about any of them. You did an awesome monitoring survey in the last year, so I know you’re into monitoring. Give us one spoiler. Tell us about something exciting that you learned from your monitoring survey or something that surprised you about it.

James: I think more that— I was both disappointed and happy with the outcome. I was disappointed that so few developers do monitoring. That made me very sad and something I’m going to try and work on over the next 12 months and try to evangelize to developers about how they can get involved with monitoring. It’s not sort of a sysadmin-centric thing. The thing that did make me pleased was that there were quite a large number of people that were actually heading towards sort of monitoring business logic instead of just like CPU, memory, disk, which is kind of, you know, it’s useful, but it’s in the context of the business, it’s kind of worthless. So, the fact that more and more people are looking at applications and the business as things they should measure was very heartening.

Bridget: [00:51:08] Awesome. So, if people want to talk to you about that kind of stuff or about Docker, where can they find you in the coming months?

James: In April, I’ll be at FluentConf talking about Docker for developers, and in June, I will be at Monitorama in Portland where I’ll be launching The 2015 version of the monitoring survey. Bigger, better, stronger. Hopefully not more complicated because it’s been a while to relearn how to use R and maths to process it all. But, um, yeah, so I’ll be, I’ll be talking about monitoring as a service and how to build a monitoring environment that sort of services your customers. And also be talking about the new monitoring survey and the results of the previous year’s monitoring survey.

Bridget: Awesome. Great. Thank you. So let’s go into our checkouts. James, what do you got for us? You got something cool to check out?

James: Yeah. So one of the things that I’ve been reading fairly eagerly at the moment is the early access version of Jason Dixon’s Monitoring with Graphite book. I think he’s about 4 chapters in. And those of you who are familiar with Graphite, it’s an amazing graphing tool. Both in terms of collecting metrics and, and the dashboard that displays them. And it integrates with a lot of other tools— Collectee, Grafana, InfluxDB, all that sort of stuff. Really cool. And if you’re thinking about looking at monitoring and looking at a more metrics-centric monitoring environment, then Jason’s book is absolutely awesome. It’s available through O’Reilly on early release. If you go to the O’Reilly website and look for Graphite or Jason Dixon, you’ll find it.

Bridget: [00:52:49] Yeah, I’ve read the first 3 chapters of it. I owe him comments on the first 3 chapters of it, but it’s really quite good.

James: Yeah, it’s very impressive. It’s a huge product, and the topic of graphs and metrics is non-trivial to cover. So, I’ve been pretty impressed by how elegantly he’s done it so far. I’ll put a pitch in as well. I’m also writing a book about monitoring. It’s going to be called, very arrogantly, The Art of Monitoring. The website is artofmonitoring.com. And at the moment you can just sign up for the mailing list and get updates when they get released. But I’m going to be building a— helping people build a new monitoring architecture. So essentially something that isn’t that sort of traditional Nagios pull-centric model, but more of an application-centric, business-centric push model, which is sort of focused really on metrics, on events. Rather than on checks and hosts.

Bridget: [00:53:49] Awesome! That’s super sweet. Okay, so let’s see, since it’s Checkout/Retro, since last time we podcasted, one of the reasons that I wasn’t on a recent one was being out of town. I actually spent a week out with my Philly and New York-based DramaFever coworkers, which was super fun, and I got to go to our 3rd annual DramaFever Awards Show. I mostly thought I was just attending for the after-party with, you know, off-key karaoke singing with coworkers, which was super fun. But the actual award show was really neat too, because there were, you know, K-drama stars who came over to the US and were completely jet-lagged and like presenting awards to one another. And there was a K-pop band called SURPRISE, spelled with a 5, all very confusing. And I was telling a friend of mine here in Minneapolis about this, and he said, Oh, surprise, the K-pop band. Yes, my 13-year-old, 14-year-old daughter loves them. And I was like, ah, this is something I don’t know anything about because I’m old. Excellent. But I got to see them. So I took some video of them, give it to him for his daughter, give it to Paul when I see him next. That was kind of fun. But then also the other thing I have Docker-related for people to check out is the Docker2Doozy Chrome plugin. So some of you may have seen this. Matt did tweet about it before, but it’s very much along the same lines as Cloud to Butt. But if you would like to see Michael Ducey all over your Docker pages instead of just the word Docker everywhere, it’s very, very funny.

Matty: [00:55:22] That inflammatory Docker post is hilarious when you have the Docker to Ducey plugin. Ducey’s like, this dude has got huge problems with me. I don’t know what I ever did to him.

James: Congratulations on 1.0 though. Docker hype or Juicy hype? I mean, either one makes me feel a little bit nauseous.

Matty: Tomato, tomato, right? You know, so— oh, he’s gonna punch me. So retro. So last week I was in Portland for the very first time ever, which is crazy because I have family out there. So I was super excited to visit my family, but I was there for Agile Open Northwest, which is an all open spaces Agile conference. That was a lot of fun. I led an open space about improv. I got a tattoo. I met the founder of Voodoo Doughnuts while winding a clock at OMSI. You know, such hipster, very, very Portland. And then the other checkouts I have is today I learned about a gem called kitty, like kitty cat. So if you just type in gem install kitty, every time you type in kitty at your prompt, you’ll get a little ASCII drawing of a cat. To cheer you up. So there’s that.

Bridget: [00:56:33] This would help Catherine if she’s saying rubby, rubby, rubby, rubby.

Matty: It would, it would. And then you can also export alias kitty to kitty if you really want to be cool. I also have finally, even though it’s been on for like 2 weeks, but I’ve finally been home to watch stuff on my DVR. So I’m getting caught up on watching Better Call Saul, which is the spinoff from Breaking Bad. It’s like the prequel to Breaking Bad. It’s awesome if you— actually, I think it would be awesome even if you didn’t watch Breaking Bad, but I had high expectations and they have been met so far. So huzzah! We have a newsletter. You can sign up for it at arrestadevops.com/bananastand. It’s the best way to know about our upcoming podcast episodes and cool news with DevOps. We’ve been sending it out regularly, twice a month, like for reals now.

Bridget: Ever since I gave you shit about it and asked if we should remove that we have a newsletter because I didn’t think we actually did.

Matty: That’s right, we totally do, and it’s awesome. I’m so excited. So I want to give big thanks to Mandy Moore. You can find her on Twitter @therubyrep or at devreps.com. She helps us out with all of our post-production of the show. She’s the best in the business for helping out software professionals from anything to administration to research to event organization to podcast production. Check her out and, and the rest of her company services at devreps.com.

Bridget: [00:57:55] Thanks to our sponsors. Please be sure to visit them at arresteddevops.com/victorops and arresteddevops.com/datadog31. Thanks, you— thank you so much, James, for joining us today. This is super fun having you.

James: No, it was awesome to be here. I apologize if I got too ranty in parts.

Matty: Um, no, if anything, not ranty enough.

Bridget: Exactly the right amount of Australian ranty.

James: Exactly. I think it was only 5 or 6 expletives, so it was a quiet night.

Bridget: I think I probably swore just about as much as you did. And to our loyal listeners, if you enjoy Arrested DevOps, we’d appreciate it if you’d visit arresteddevops.com/itunes and leave us a review in the iTunes Store. No matter what you have to say, we’d love to get your feedback.

Matty: You can check us out at arresteddevops.com on the interwebs, @ArrestedDevOps on Twitter. We’re on the Facebooks. We’re— and we’re always happy to get your input, ideas, or feedback at shows@arresteddevops.com. Please let us know any ideas you have for future episodes. I’m Matt, @MattStratton.

Bridget: [00:59:01] And I’m Bridget, @bridgetkromhout. We’re Arrested DevOps, and remember, there’s always DevOps in the banana stand.