What is DevOps?
Damon Edwards /
Update 1: Wikipedia now has a pretty good DevOps page
Update 2: Follow-up posts on the business problems that DevOps solves and the competitive business advantage that DevOps can provide.
If you are interested in IT management — and web operations in particular — you might have recently heard the term “DevOps” being tossed around. The #DevOps tag pops up regularly on Twitter. DevOps meetups and DevOpsDays conferences, are gaining steam.
DevOps is, in many ways, an umbrella concept that refers to anything that smoothes out the interaction between development and operations. However, the ideas behind DevOps run much deeper than that.
What is DevOps all about?
DevOps is a response to the growing awareness that there is a disconnect between what is traditionally considered development activity and what is traditionally considered operations activity. This disconnect often manifests itself as conflict and inefficiency.
As Lee Thompson and Andrew Shafer like to put it, there is a “Wall of Confusion” between development and operations. This “Wall” is caused by a combination of conflicting motivations, processes, and tooling.
Development-centric folks tend to come from a mindset where change is the thing that they are paid to accomplish. The business depends on them to respond to changing needs. Because of this relationship, they are often incentivized to create as much change as possible.
Operations folks tend to come from a mindset where change is the enemy. The business depends on them to keep the lights on and deliver the services that make the business money today. Operations is motivated to resist change as it undermines stability and reliability. How many times have we heard the statistic that 80% of all downtime is due to those self-inflicted wounds known as changes?
Both development and operations fundamentally see the world, and their respective roles in it, differently. Each believe that they are doing the right thing for the business… and in isolation they are both correct!
To make matters worse, development and operations teams tend to fall into different parts of a company’s organizational structure (often with different managers and competing corporate politics) and often work at different geographic locations.
Adding to the Wall of Confusion is the all too common mismatch in development and operations tooling. Take a look at the popular tools that developers request and use on a daily basis. Then take a look at the popular tools that systems administrators request and use on a daily basis. With a few notable exceptions, like bug trackers and maybe SCM, it’s doubtful you’ll see much interest in using each others tools or significant integration between them. Even if there is some overlap in types of tools, often the implementations will be different in each group.
Nowhere is the Wall of Confusion more obvious than when it comes time for application changes to be pushed from development operations. Some organizations will call it a “release” some call it a “deployment”, but one thing they can all agree on is that trouble is likely to ensue. The following scenario is generalized, but if you’ve ever played a part in this process it should ring true.
Development kicks things off by “tossing” a software release “over the wall” to Operations. Operations picks up the release artifacts and begins preparing for their deployment. Operations manually hacks the deployment scripts provided by the developers or creates their own scripts. They also hand edit configuration files to reflect the production environment, which is significantly different than the Development or QA environments. At best they are duplicating work that was already done in previous environments, at worst they are about to introduce or uncover new bugs.
Operations then embarks on what they understand to be the currently correct deployment process, which at this point is essentially being performed for the first time due to the script, configuration, process, and environment differences between Development and Operations. Of course, somewhere along the way a problem occurs and the developers are called in to help troubleshoot. Operations claims that Development gave them faulty artifacts. Developers respond by pointing out that it worked just fine in their environments, so it must be the case that Operations did something wrong. Developers are having a difficult time even diagnosing the problem because the configuration, file locations, and procedure used to get into this state is different then what they expect (if security policies even allow them to access the production servers!).
Time is running out on the change window and, of course, there isn’t a reliable way to roll the environment back to a previously known good state. So what should have been an eventless deployment ended up being an all-hands-on-deck fire drill where a lot of trial and error finally hacked the production environment into a usable state.
While deployment is the most obvious pain point, it is only one part of the need for DevOps. As John Allspaw points out, the need for cooperation between development and operations starts well before and continues long after deployment.
What’s the benefit of DevOps?
DevOps is a powerful idea because it resonates on so many different levels.
From the perspective of individuals toiling in hands-on development or operational roles, DevOps points towards a life that is free from the source of so many of their hassles. It’s by no means a magical panacea, but if you can make DevOps work you are removing barriers that are both a significant time-sink and a source of morale killing frustration. It’s a simple calculation to make: invest in making DevOps a reality and we all should be more efficient, increasingly nimble, and less frustrated. Some may argue that DevOps is a lofty or even farfetched goal, but it’s difficult to argue that you shouldn’t try.
For the business, DevOps contributes directly to enabling two powerful and strategic business qualities, “business agility” and “IT alignment”. These may not be terms that the troops in the IT trenches worry about on a daily basis, but they should definitely get the attention of the executives who approve the budgets and sign the checks.
A simple definition of IT alignment is “a desired state in which a business organization is able to use information technology (IT) effectively to achieve business objectives — typically improved financial performance or marketplace competitiveness” [source].
DevOps helps to enable IT alignment by aligning development and operations roles and processes in the context of shared business objectives. Both development and operations need to understand that they are part of a unified business process. DevOps thinking ensures that individual decisions and actions strive to support and improve that unified business process, regardless of organizational structure.
A simple definition of agility in a business context is the “ability of an organization to rapidly adapt to market and environmental changes in productive and cost-effective ways” [source].
Of course, developers also have their own specialized meaning of the word “agile“, but the goals are very similar. Agile development methodologies are designed to keep software development efforts aligned with customer/company goals and produce high quality software despite changing requirements. For most organizations, Scrum, the iterative project management methodology, is the face of Agile.
Agile promises close interaction and fast feedback between the business stakeholders making the decisions and the developers acting on those decisions. If you look at the output of a well functioning Agile development group you should see a steady stream improvement that is in tune with business needs.
However, when you step back and look at the entire development-to-operations lifecycle from an enterprise point of view, that Agile stream and it’s associated benefits are often obscured. The Wall of Confusion leads to a dissociation of the application lifecycle. Development works at one pace and Operations works at another. The long intervals between production deployments, in effect, turn the Agile efforts of an organization right back into the waterfall lifecycle it was trying to avoid. No matter how Agile the development organization is, it’s exceedingly difficult to change the slow and lumbering nature of a business while the Wall of Confusion is in place. Andrew Rendell has a great post that tells the anecdotal story of how an organization’s cumbersome release processes turn their agile development efforts right back into a waterfall.
DevOps enables the benefits of Agile development to be felt at the organizational level. DevOps does this by allowing for fast and responsive, yet stable, operations that can be kept in sync with the pace of innovation coming out of the development process.
If you are seeking to establish a DevOps project within your organization, be sure to keep the terms “IT alignment” and “business agility” in mind.
How do we bring DevOps to life?
Like most emerging topics, it’s easier to find a consensus about the problem than it is about the solution.
If you listen to the current DevOps conversations, there does appear to be 3 areas of focus for DevOps related solutions:
1. Measurement and incentives to change culture – Changing culture and reward systems is never easy. However, if you don’t change your organization’s culture, fulfilling the promise of DevOps will be difficult, if not impossible. When looking to influence culture in a business organization, you need to pay close attention to how you measure and judge performance. What you measure influences and incentivizes behavior. All parties across the development-to-operations lifecycle need to understand their stake in the larger business process of which they are a part. The success of both individuals and groups needs to be measured within the context of the success of the entire development-to-operations lifecycle. For many organizations this is a shift from more of a siloed approach to performance measurement, where each group measures and judges performance based on what matters to that specific group. This previous post I wrote dives deeper into the process for getting the correct end-to-end view of measurement into place.
2. Unified processes – The important theme of DevOps is that the entire development-to-operations lifecycle must be viewed as one end-to-end process. Individual methodologies can be followed for individual segments of that processes (such as Agile on one end and Visible Ops on the other), so long as those processes can be plugged together to form a unified process (and, in turn, be managed from that unified point-of-view). Much like the question of measurement and incentives, each organization will have slightly different requirements for achieving that unified process. Here is an excellent post by Six Sigma Blackbelt Ray Riescher on his experience bridging Scrum and ITIL.
3. Unified tooling – This is the area in which most of the DevOps discussion has been focused. This isn’t surprising since it seems to be the natural reflex of technologists, for better or for worse, to jump straight into tooling discussions when looking to solve a problem. If you follow the communities of tools like Puppet, Chef, or ControlTier then you are probably already aware of the significant focus on bridging development and operations tooling. “Infrastructure as code”, “model driven automation”, and “continuous deployment” are all concepts that would fall under the DevOps banner. Alex Honor wrote a good post about some of the design patterns that toolsmiths working on DevOps tools need to worry about.
Jake Sorofman does a great job with the following overview of what types of tooling is required to make DevOps a reality:
A version-controlled software library—which ensures all system artifacts are well defined, consistently shared, and up to date across the release lifecycle. Development and QA organizations draw from the same platform version, and production groups deploy the exact same version that has been certified by QA.
Deeply modeled systems—where a versioned system manifest describes all of the components, policies and dependencies related to a software system, making it simple to reproduce a system on demand or to introduce change without conflicts.
Automation of manual tasks—taking the manual effort out of processes like dependency discovery and resolution, system construction, provisioning, update and rollback. Automation—not hoards of people—becomes the basis for command and control of high-velocity, conflict-free and massive-scale system administration.
It’s essential that all individual tools be considered part of a larger toolchain that spans the entire Development to Operations lifecycle (even if tight technical integration isn’t a option). Tool choice and implementation decisions (on both the toolchain and individual tool levels) need to be made in the context of their impact on that end-to-end lifecycle. If you are wondering how that is done, take a look at this example of an open source fully automated provisioning toolchain that can be plugged into a larger Development to Operations toolchain.
What DevOps is not!
At the recent OpsCamp Austin, Adam Jacob from OpsCode/Chef railed against the idea that some system administrators were now seeking to change their job title to “DevOps”. I have to admit that, at the time, I was a bit skeptical that this was actually happening. However, I have since witnessed people on multiple occasions expressing this desire to rewrite job titles or establish DevOps as some sort of new role to be filled.
For example, Stephen Nelson-Smith wrote an excellent post about DevOps. While I agree with almost everything he said, I have to strongly disagree with the idea that DevOps should be a unique position or job title.
Turning “DevOps” into a new job title or special role sets a dangerous precedent. This makes DevOps someone else’s problem. You’re a DBA? Don’t worry about DevOps, that’s the DevOps team’s problem. You’re a security expert? Don’t worry about DevOps, that’s the DevOps team’s problem.
Think of it this way. You wouldn’t say “I need to hire an Agile” or “I need to hire a Scrum” or “I need to hire an ITIL” would you? No, you would just say I need to hire developers, project managers, testers, or systems administrators who understand these concepts and methodologies. DevOps is no different.
Why the name “DevOps”?
Probably because it’s catchy. It’s also a good mental image of the concept at the widest scale — when you bring Dev and Ops together you get DevOps. There has been other terms for this idea, such as Agile Operations, Agile Infrastructure, and Dev2Ops (a term we’ve been using on this blog since 2007). There is also plenty of examples of people arriving at the idea of DevOps on their own, without calling it “DevOps”. For an excellent example of this, read this recent post by Ernest Mueller or watch John Allspaw and John Hammond’s seminal presentation “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” from Velocity 2009.
For better or for worse, DevOps seems to be the name that is catching peoples’ imaginations. I credit the efforts of Patrick Dubois for championing the term “DevOps”, bringing the first DevOps Days conference to a (successful) reality, and maintaining the devops.info site.
Be sure to join in the DevOps conversation at the upcoming DevOps Day USA conference on June 25, 2010 in Mountain View, CA. It’s the day after O’Reilly’s Velocity 2010 conference, so be sure to hit both!
If this article raised your interest, and you’re based in or around London, don’t miss our first London Devops meetup: http://agilesysadmin.net/london-devops
Good article, and thanks for the link! II have one major concern about the "DevOps" moniker though – same concern with the "Visible Ops" book – I think what’s being lumped under "Ops" is really two different things that I usually think of as falling under "Infrastructure work."
1. System engineering
2. Operational support
I understand that in many places the devs "run the show" and even do system design and toss it over the wall to an ops team who only have any say in the runtime. But in many places, there are separate folks with Infrastructure expertise – maybe the same as the Ops team (in our case it is) and maybe not – that are different from the coders.
That system engineering function spans both product-facing concerns (what load balancer are we using, how are we distributing the pieces we need between mySQLs and memcaches, etc.) and setting up the major functional parts of an operational environment (config management, etc.) I don’t know what you call that, is that a separate "devops" thing that sits between the "dev" and the "ops"… Anyway, the devops term sometimes kinda confuses me because in a lot of places I see those three separate roles. To me, Ops is level 1/2 support and carrying a pager and execution.
Maybe I’m getting hung up on mapping the concepts there to "real" groups/job titles/responsibilities but that seems to be where other confusion’s coming from too…
Your blog is good! Generally when I visit blogs, I just come across nothing,
but this time I was really surprised when I got your blog containing wonderful information.
Thanks mate and keep this effort up.
Very nicely explained. Well done.
I’m going to send some colleagues to this blog.
Thanks
I really wish posters/bloggers would remember the rules about acronyms: Spell it out in its first iteration with the acronym following, then use the acronym to your hearts content, in the content, thereafter.
SCM … don't assume your use of the acronym is everyone's.
http://www.acronymfinder.com/SCM.html shows that yours is not the only use of acronym SCM.
Sure, I know what your use of it is, but please, I'm a professional and you're a professional. Write like a professional.
Thank you.
Hi Phil,
Thank you for your impassioned comment. I'm glad you were able to understand exactly what I meant, yet still took the time to call my writing unprofessional.
-Damon
Hi,
Is this not part of a bigger problem. Handover of work to the next link in the chain. The over the wall tossing can be all over the place. business can toss their problem over the wall to the devs the devs can trow their code over the wall to the test team and after some back and forth throwing between test and devs it ends up at ops.
In a big shop (like say microsoft) dev will have all kinds of potential walls and if kernel teams and driver team don't work together it will be crap as well. So you end up with a Ker2Driv movement 😉
In the end all parts of the organisation need to be in sync but foremost willing to listen to each other.
Most of the time it's "the other guys problem" i did my bit and if it goes wrong it because someone else messed up.
And in the end there is only one fact that matters: the business = happy customers = money = everybodys paycheck 😀
Great article, one thing bugged me though.
In "throwing over the wall" you mention that Ops might create or uncover new bugs. One of these is positive.
I think the fact that Ops might uncover bugs that Dev missed is an upside to the throwing over the wall deployment method. Using the product with a different perspective to the developer can be a useful exercise.
Hi Robert,
I would agree that looking at quality from multiple perspectives is essential, but going about that by relying on Ops to deconstruct and hack through each release to figure it out on the fly during deployment is probably the most inefficient and dangerous (not to mention demoralizing) way to go about it.
Instead I would look to how the Continuous Deployment adherents (like Wealthfront and Etsy) are going about the problem (CD is one approach to building an organization without DevOps problems). They make quality everyone's (ops and dev) responsibility through enforcing extensive automated test coverage and test-driven practices. It's a cultural shift, but the results are undeniable in terms of both quality and the speed at which their organizations move.
-Damon
I just wanna say thank you for the information that you have been shared to us readers. Thanks for posting this kind of theme.
The Real Deal on DevOps – Webinar
24th January 2012: 3:00pm – 4:00pm GMT
OVUM Events run industry-leading IT conferences that are free to attend for IT managers and personnel. We are running a DevOps webinar on the 24th of January. For more information and to register for a free pass to view the webinar (Please use Priority Code: TS03 to be eligible for a free pass) follow the link below.
http://c3162792.workcast.net/ovum_24jan.html?meid=507&pak=9047167118094919
Many thanks
I was just browsing for related blog posts for my project research and I happened to discover yours. Thanks for the excellent information!
Hi Damon,
This a simple, effective and fantastic way you have put things across. I never knew such concepts are being evangelized until very recently.Having worked in company within DevOps team, I can vouch for the benefits you mentioned. It gives to Business an edge over other companies with other models.
In my new company, I would like the management to know the benefits of this way of working. Could I use , with your permission, some of the pictures you have in here, please. I would like to have a reference your blog in the presentation too.
Do let me know, your view.
Many Thanks
N
Stackify is launching a DevOps software suite designed to fix these vary problems. Please check out what we are working on: Stackify DevOps
[…] comes together. I’ll be spending a lot of time in the coming year polishing my knowledge of DevOps, and Continuous Delivery. vFabric has a lot to offer to the organization that wants to ship more […]
[…] What is DevOps all about? […]
[…] What is DevOps? http://dev2ops.org/2010/02/what-is-devops/ […]
[…] What is DevOps? (dev2ops) […]
Damon,
This is an very well written primer. Our audience mostly look for such de-mystification before picking outsourcing vendors. One of them actually just brought devops consultants, making a significant investment.
I’d like them to read this post. May I repost this on our blog, with due credits?
I’d link them back to you!
Best,
Ashwin
[…] What is DevOps? by Damon Edwards […]
[…] Edwards, Damon – “What is DevOps?“ […]
[…] were several talks on the agile practices such as TDD, BDD, DevOps, emergent architecture and design, writing good code, etc. With multiple tracks running in […]
This all sounds waaaay too good to be true. I wonder what happens when multiple fully-“empowered” DevOps teams are all working on different products in a shared environment. Then Team A decide that their library needs an upgrade and it breaks Team B’s product in a subtle way that isn’t noticed for a week or so.
[…] Building cloud-ready applications: what changes DevOps (development-operations) Che cos’è DevOps […]
[…] What is DevOps? beschäftigt sich grundlegend mit dem Spannungsfeld zwischen Entwicklung (“I want change!”) und der Betriebsführung (“I want stability!”) und erklärt, wie durch DevOps eine Brücke zwischen Entwicklung und BF errichtet werden kann, um eine teamübergreifende, ergebnisorientierte Zusammenarbeit zu ermöglichen. […]
[…] 参考：http://dev2ops.org/2010/02/what-is-devops/ […]
[…] added that using security analytics and other techniques, such as the application of the DevOps method, offers a combined approach for improving security defences. […]
[…] added that using security analytics and other techniques, such as the application of the DevOps method, offers a combined approach for improving security defences. […]
[…] be introduced with unchecked growth. In DevOps parlance this is a physical version the infamous wall of confusion, a crude way to represent organizational, cultural or geographical separation. In the workshop, a […]
What is the future of Devops? Also, what is the career path for a Devops Engineer?
In Devops the deployment is done to the real working customer environment?
if so did it make the customer (Operations) working difficult.
if this was 2011 I’m late to the party…but istm we’ve been doing the Plan Build Deploy Operate cycle in more or less this fashion since at least the beginning of my time in say ’99 or so. great that is being credited and emulated in ways that flatter my experience; I believe my previous experiences have been in what I would certainly call agile if not Agile envts and like that prosy cat in Moliere, I was speaking DevOps and not knowing it.
In all candor, excellent article, if nothing else the googling has turned up lots of resume catnip.
[…] http://dev2ops.org/2010/02/what-is-devops/ […]
[…] and support it, proving that they care and helping out, will go a long way to breaking down the wall of confusion between operations and […]
[…] point of all this, as DevOps expert Damon Edwards explained, is that “DevOps is a response to the growing awareness that there is a disconnect between what is traditionally considered development activity and what is traditionally considered […]
[…] companies have turned to DevOps to boost the quality of delivery from their Agile development teams. They have to, given the […]
[…] is a lot of talk in the devops community about the importance of sharing principles and values, and about silo busting: breaking […]
[…] highly recommend look at What is DevOps by Damon Edwards on Dev2Ops for more […]
[…] is an older blog which explains pretty well the issue of Enterprise IT and its inability to support business at […]