Eating a Bureaucracy One Bite at a Time with DevOps - Mark Schwartz

My name is Mark Schwartz. I am an enterprise strategist for AWS. What that means is I'm part of a small team of people who were senior leaders in IT organizations before they joined AWS, generally CIOs and CTOs who pulled off some sort of digital transformation and moved their organizations to the cloud. What we do is we work with AWS customers, large enterprises generally, to help them get over the impediments to transformation that are typically non-technical things, almost always cultural change, organizational structure, getting the right skills for their people, figuring out the right investment models, and overcoming bureaucracy.

One of the topics that comes up a lot when I'm working with customers is they'll ask me about our transformation at USCIS, where I was the CIO before I joined AWS. US Citizenship and Immigration Services, part of the Department of Homeland Security. I tell them we moved from a release cycle time on the average of about 18 months to deploying code in some cases three or four times a day. We moved into DevOps, we moved everything to the cloud, or almost everything, we started refactoring everything into microservices, we made a lot of changes to how we did procurement and investment management. The question often is, 'How long did that all take you?' I think when I hear that question, if the person asking it is trying to justify it taking a long time in their organization, I love to tell them that actually it took us two days. Now, I know a lot of organizations can't do it in two days because they don't have the advantage of bureaucracy like we did, but if they did, they would be able to do it as well. In order to explain what I mean by that, I'm going to have to give you a little bit of context first. By the way, there are a bunch of people who worked with me in the audience here, so if I call them out by name, you'll understand.

What was the impediment to transformation for us? It was a document, a wonderful document called MD 102, Management Directive 102, a DHS policy on how essentially all IT investments were going to have to be overseen and run. It was essentially the SDLC that we had to follow. Last night I double-checked the last version of it. This document, MD 102, called for us, for each system we created, to write 87 documents, to go through 11 gate reviews, and to find 21 different oversight roles for people who could say no to things. It was an awesome thing. It called for us to do a system definition review that checked to make sure all the requirements had been written down and fleshed out, and there was a test and integration review where you had to get approval to integrate the code and test it. Clearly, this was set up to be a waterfall process. As we started our agile transformation, we had the idea that maybe it wouldn't fit too well. MD 102 was officially the policy, so we gave it a little thought. We had a lot of arguments about how we could be agile with this wonderful piece of bureaucracy. Finally, we realized something which I think is true of a lot of pieces of bureaucracy: there's usually some sort of exception process. The people who wrote it aren't dumb; they know things are going to happen. In this case, there was something called a project tailoring plan. We could use the project tailoring plan for a particular project to explain why we were going to need to follow a different process for the special needs of that project. Of course, the special needs for our project were that it was a project that should be done in an agile way. That was essentially the argument. We created this tailoring of the official process that essentially said we're going to do the opposite of everything the process says, and it's justified because we shouldn't use the process. That was a little bureaucratic move you might want to learn.

There was a little snag: somebody had to actually sign off on our plan. That was a little harder to master. The way we did it, another good learning, is we took the project that was failing the most, that nobody could figure out how to fix, and we suggested we use the tailoring for that. Since nobody else had any better ideas, we were able to get an official approval to try it out. We chose the project, if you listened to the USCIS presentation the other day, our enterprise transformation project. That was the project that had already spent about a billion dollars and gotten nothing for it, so the idea that we might try a different approach was something people bought into. With that approval, we started doing a scrum process, two-week iterations, actually I think we started with three-week iterations, standing up every day for 15 minutes, retrospectives, all of that. It was all going fine. An interesting conversation took place between me and the people in charge of this MD 102 thing. I said, 'The agile way of running IT projects is actually much better than the old waterfall way, so why don't we change MD 102 to say do things in an agile way?' They said, 'There's no need, because if you want to do things in an agile way, you can do it under MD 102, you just need to do a tailoring plan.' I said, 'Yes, but maybe it's better if the official policy doesn't say you have to do 87 documents and 11 gate reviews and endorse that kind of behavior, so why don't we just change it?' They said, 'Haha, we have to be responsible around here. No, but we don't see what you're complaining about because you can do this in an agile way.' I want you to remember this conversation because it's going to come back in a minute.

So here we are doing our little agile thing, and suddenly another snag arises. Because this project was so high-profile, we had wasted so much money on it, we were of course going to be audited by the Inspector General and GAO, the Government Accountability Office. We thought since we were now actually producing stuff every couple of weeks, they were going to say glowing things about us. I'm looking at my colleagues in the audience. It turned out they didn't say glowing things. What they did say was a bit of a surprise. Essentially, we sat down for a debrief with them, and the auditors said, 'You're not agile enough.' That required a little bit of explanation, since I had been doing agile IT initiatives since long before I joined the government, and as far as I knew, GAO didn't know anything about IT development, let alone agile approaches. I wanted to hear their reasoning. It went sort of like this: they said, 'You said in your project plan that you're going to do scrum, and we checked what the practices of scrum are.' There was a list of practices. They said, 'You're not doing some of those.' I said, 'Well, number one, don't confuse scrum with agile. Number two, you're supposed to retrospect and do continuous improvement and change your process in ways that make it better. We made a few changes, especially around product ownership, which wasn't working well for us the way it was visualized in the books on scrum.' They said, 'Well, you're not doing what it says.' I said, 'Well, agile means you change.' They said, 'Well, you said you're doing scrum, and we read in the book where Jeff Sutherland says you can't change anything in the scrum process or you're not doing scrum anymore,' which he did. We had to admit they had us on that one. It was a failure that we learned from.

If you think about it for a minute, I just want to point this out: we had been defeated by bureaucracy, but it was not government bureaucracy that defeated us, it was the bureaucracy of scrum in the end. This gave us the perfect idea. We realized that when GAO audited us, they were going to compare what we said we were going to do to what we were actually doing. All we really had to do was to say we were going to do the things we wanted to do, which is obvious if you've got bureaucracy behind you. So we decided we would write a new policy that said you have to be agile. We started to go down that route and found out that writing policy is difficult in the government. It's actually in law how you have to go about writing policy, and you have to put it out for public comment, and it takes a really long time. Our next idea was why don't we create a management directive just like MD 102? It turned out we couldn't do that either because a management directive, I don't know, somebody in our group probably knows better than me, but I think it has to come from somebody in a management directorate, the parent DHS department. So we couldn't do a management directive, but fortunately we had some good bureaucrats on our team, and they said, 'You know what you can do? A management instruction.'

We knew that for anybody to take our management instruction seriously, it was going to have to have a good title, sort of like MD 102, a compelling title like that. Since it was the first one we had ever written, we were going to have to come up with it. So we invented MI-4, Management Instruction - CIS - OIT - 001. CIS is the name of our agency, OIT is the Office of Information Technology. We figured that would make people take it seriously. We created MI-CIS-OIT-001, and it said everybody has to be agile from now on. We published it, and there was an effective date, so as of that date, everybody was agile. That part of our transformation took one day. The policy was pretty clever. We didn't want to have that same fight with GAO again, so we listed the eight practices that we were going to call agile, things like frequent delivery of value and retrospective, and then five other optional practices that were sort of advanced agile. Boom, we were agile. A little bit later on, we learned about DevOps and decided we wanted some of that, and it wasn't going to fit under that, so we had to write MI-CIS-OIT-003, which essentially said everybody's doing DevOps now. We had transformed to DevOps, and that was the second day of our transformation.

There are a couple of problems. The first problem we figured is it conflicted with MD 102, and MD 102 was written by somebody much higher in the organization, so maybe that would be a problem. My team was worried: 'Boss, you keep telling us to do things this way, but somebody is going to come in and say you're not doing what MD 102 says.' I said, 'Right, that's an impediment. My job is to remove the impediments. You follow my policy, and I'll deal with it.' I was pretty sure I could deal with it because if you remember back to that earlier conversation, the folks responsible for MD 102 had told me there's no problem doing agile under MD 102 even though it says exactly the opposite, all you need to do is tailor it. I figured, what are they going to do, tell me I can't be agile? They just told me I could. So that was an easy one. The second little challenge was as soon as I would meet with a bunch of people in the organization and talk about doing DevOps and being agile, they would go out into the hall and say, 'Oh, he's crazy. What are we supposed to do? What is this thing?' But I figured eventually if they were going to try to do it, they would figure it out. What you need to know is that my position is called Senior Executive Service, SES. Technically, an SES is supposed to be the same rank as a two-star general because they wanted to make an equivalence between military ranks and government civil service. So you have to imagine what happens when essentially the chaos monkey becomes a two-star general. That's the situation here. When I said everybody do DevOps, it was like General Chaos Monkey saying everybody do DevOps. The whole team tried their best to do what I said. We brought in some wonderful coaches, some of whom are here, and over a little bit of time, people kind of figured out how to make this work. They started to do these frequent deployments, learned how to deploy things with no downtime and how to roll back if anything went wrong, built a CI/CD pipeline, and started doing frequent deployments.

Then it was time for the GAO audit again. We knew we would pass with flying colors, even though in the history of the universe GAO has never said anything nice about anything that they audited. But we figured this time we got it, we're doing exactly what our policy said. It turned out they didn't agree. They said, 'The problem here is that even though people are doing what your policy says, you don't have a mechanism to make sure they are.' Auditors love that, you've got to be able to prove that you're actually compliant. So MI-CIS-OIT-004 is the policy that says we're going to have our independent verification and validation team look at what all the DevOps teams are doing and make sure they're doing DevOps. That was pretty brilliant. They expected us to have a policy that said if they find somebody who's not doing DevOps, they're going to take them out and shoot them or something. Instead, our policy said if they find somebody who's not doing DevOps, they'll work with them to try to help them be better at DevOps, which was a little bit of a surprise to them. We now had a policy saying everybody's got to be agile, a policy saying everybody's got to do DevOps, and a policy saying we're going to check to make sure everybody is doing DevOps. All of them were beautiful pieces of bureaucracy that had compelling titles, and we could do all of this because we had bureaucracy behind us. It was gorgeous. Obviously, it was time for me to leave the agency because my work was done.

I want to step back and take a few learnings out of this. The first point I want to make is everybody's got bureaucracy, it's not just the government. Every company I talk to, when I mention bureaucracy, they say, 'Come on, we need help with that.' It's actually going to be the topic of my next book. Every time I tell people what I'm working on, they get all excited, 'We could use that, we need that one.' So I'm going to be writing about bureaucracy, but everybody's got it. In fact, as I said, it was the scrum bureaucracy that killed us. IT organizations are great at creating bureaucracy. We make standards, and then we make a process for making sure people are following the standards, and an exception process for if they don't want to follow the standards, request it in triplicate and send it here and there. Then we've got this DevOps thing, which is in many ways the extreme of bureaucracy. Think about this: one great piece of bureaucracy we used to have is after you finish coding some new capability and you want to deploy it, you had to go to the CAB or the Change Control Board and submit all this stuff. They thought about it, but it was always a group of people that really had no way of knowing whether you should deploy this or not. What they would do is look at the paperwork and say, 'Yeah, let's check the test reports. Oh, it looks like they ran all the tests and all the tests passed, so I guess that means they can deploy it.' Beautiful bureaucracy. That was how we used to decide on whether something could deploy. What do we do in DevOps? Exactly the same thing, except it's automated. We set up the pipeline so that you can't actually deploy unless all the tests have passed and it's gone green. What's the difference? It's really the same concept: let's create rules to encapsulate good practices and let's enforce those rules in every case, no exceptions allowed, unless you fill out the form in triplicate. In a way, DevOps is all about automating bureaucracy.

I'm not saying anything negative. When I think about what it is about bureaucracy that drives us crazy, I think it's not the nature of bureaucracy itself, it's two characteristics of bureaucracy. The first one is that often we see bureaucracy that is not lean, it's wasteful. It says if you want to deploy a piece of software, you have to write 87 documents and go through 11 gate reviews. What's the problem with that? The problem isn't that you have to have some structure, the problem is that 87 documents is ridiculous, it's just extra work you shouldn't have to do. When you have to fill out the forms in triplicate and take them from window 1 to window 2 to window 3, it's just extra stuff, and it's really frustrating because it gets in your way and you keep feeling like just let me deploy. The fact that bureaucracy often is not lean is the first thing that drives us crazy. The second thing is that bureaucracy tends not to learn. You have a set of rules, and that set of rules doesn't change very fast, in fact it changes slowly. MD 102 is a perfect example. Here are a thousand books that say it's better to do things in this agile way, you're going to get better results. By the way, the waterfall way has never actually worked. If you look at your portfolio of things you're overseeing, you'll see that every single project is failing. So maybe we should change MD 102. No, bureaucracy tends not to change. But those two characteristics, the waste and the resistance to change, are not actually essential to the idea of bureaucracy. In bureaucracy, you apply rules uniformly to everybody without exceptions. That doesn't mean that the set of rules you're applying can't change, and there's no reason why you can't take a bureaucratic process like MD 102 or any other one and look at it as a value stream, look for where the waste is, and reduce that waste, either shortening steps or taking waste out of them or taking steps out entirely. Any bureaucratic process is subject to exactly that same approach as we do with any lean approach to a process. There's no reason why a bureaucracy has to be wasteful or unchanging. You can have lean bureaucracy, and you can have a learning bureaucracy.

When we set up this MI-CIS-OIT-003, we deliberately set it up to be a learning bureaucracy in the sense that the policy itself did not define what anybody should do, it defined the objectives we were looking for: frequent delivery of value, etc. Then it said the practices we want you to follow are in Appendix A, and Appendix A has what we think today are the best processes or best practices, but we expect that Appendix A is going to change over time as we learn new things. We deliberately separated these two concepts: the outcomes we were looking for and the best practices we're going to use to get there, so that we can keep changing those best practices as we learn new things. Why can't you do that in a bureaucracy? It's a little bit arrogant, but I think this was the best piece of bureaucracy ever written. This was art. I contend that you can take bureaucracy when you encounter it, and you can take advantage of the little loopholes and exceptions, but what you can also do is take that same lean approach to bureaucracy that we take to all of IT these days. Look at it as a value stream, yank waste out of it to reduce lead time, and just keep doing that. At the same time, establish a feedback mechanism so that you can learn as you go and alter the bureaucratic rules based on your learnings. In a way, this is what we're doing in DevOps, where we take what could have been a slow, wasteful bureaucracy and automate elements of it, eliminate elements of it, and set up ways of establishing compliance through automation and through artifacts that are generated automatically to demonstrate compliance. Bureaucracy to me is not an immovable impediment. It's something that sounds terrible, the words are terrible, we use the words to describe anything that's frustrating to us. But the words 'bureaucracy' and the fear of having to deal with politics and controls that are coming from outside yourself, that's just a fear. It's not an actual impediment. It can be removed as an impediment, it can be worked through, it can be altered and improved, and even used as a way to establish controls where you do want to establish controls, in cases where you want to have standardization, centralized control, shared services. There are all these applications in the world of IT where we want to put some structure and formal process around things. But when we do that, we have to be very careful to make sure that we're not creating bureaucracy that's going to resist change and that's going to add a lot of cost and lead time as people try to follow it.

My concluding message is that bureaucracy, painful as it might be and as much as you might feel like it's getting in your way, is actually something that you can tackle, you can work with, and you can use bureaucracy to get better impact, just as we used bureaucracy to actually drive a transformation, strangely. Good luck to everybody with your bureaucracies. Thanks.