View All Videos
HPDiscoverPerformance

IT stability and business innovation are not enemies

7

Damon Edwards / 

Back before the hectic end of the year I was interviewed by HP’s Discover Performance newsletter and online magazine. The questions were about applying DevOps thinking inside enterprises can enable the pace of innovation without increasing risk.

Below is the interview in full. If you like this interview, I recommend signing up for the Discover Performance newsletter. They routinely have good articles on interesting and relevant topics and avoid injecting too much self-serving HP bias (a difficult task for enterprise funded content!).

 

IT stability and business innovation are not enemies

DevOps expert Damon Edwards discusses why Ops should neither resist innovation nor be a scapegoat when things go wrong.

Innovation is a mantra in business, one that the CIO hears more and more. As IT leaders feel pressure to be more responsive, faster moving, and more innovative, Operations leaders worry that their mission—the smooth, steady delivery of high-quality IT services—may be jeopardized by rushed experimentation.

Damon Edwards

Damon Edwards, co-founder of IT consultancy DTO Solutions, has spent more than a dozen years working on web operations from both the IT and business angles. A major DevOps proponent, he recently posted about “using DevOps to turn IT into a strategic weapon.” Discover Performance reached out and asked him to talk about how Operations leaders—and IT executives in general—should approach innovation.

The (completely achievable) goal, he says, aligns IT goals with business goals by “removing all of the bottlenecks, inefficiencies, and risks between a business idea (the ‘ah-ha!’) and a measurable customer outcome (the ‘ka-ching!’).”

Q: Does there tend to be a basic disconnection between the business and IT on the subject of innovation? Why?

DE: “Disconnect” has become somewhat of an ugly euphemism inside corporations. Unfortunately it’s become code for “I’m right and you’re wrong.” In reality, a “disconnect” is usually just two people operating and making assumptions based on differing definitions. As a result, you get unfortunate infighting between people who, in all other ways, both desperately want the company to succeed.

Talk to folks in the technology roles and they tend to see innovation as being synonymous to invention. There is a rich legacy of invention in the technology world. Getting your name on a patent was an ultimate trophy. Much of the myth and lore of tech and geek culture is built on a love of tinkering and invention. Now contrast that to what you see when you visit the business folks. They see innovation as the application of new ideas to create value for their current customers and to attract new customers. Unfortunately, now that you can win a patent for what is essentially a business idea, the invention/innovation distinction is even more muddled.

If executives let the two core parts of the company operate under completely different definitions, you’re bound to have conflicts and gridlock. You have to make it clear what innovation is and isn’t for your company, and how you’re going to measure its impact.

Q: Isn’t innovation an inherently risky endeavor?

DE: There is always some level of risk with innovation because you’re operating in the unknown. You don’t know if the customers will respond. You don’t know if the response will be what you want or expect. When the revenue and health of a company are tied to getting a large number of favorable responses, there is risk.

But I should be clear that innovation, especially on the web, should be low risk from a technology perspective. You reach your customers through standard interfaces and over standard protocols. We know how to deploy safely 20 times a day. We know how to scale services to hundreds of millions of users. We know how to manage petabytes of data. If you’re running a web company, your innovation risk should almost exclusively be on the business end, not the technology end.

Q: When risks compromise IT performance, heads roll, especially on the Ops team.  How do you decrease the risk of innovation for Ops?

DE: Again, I’d ask what went wrong organizationally that put the Ops team in a position of risk. Was the business asking for something that was never done before and needed some never-thought-of-before technology to work? Doubtful. Did the developers change their underlying framework or introduce new technology that wasn’t properly vetted or Ops didn’t know how to handle? Possible. Did Ops upgrade or switch a technology component? Also possible.

My point is that, while Ops is the common scapegoat, the problem often started somewhere else and likely had nothing to do with the business being more innovative. So Ops gets blamed—which is like blaming the canary for the gas in the coal mine—and in response Ops starts saying no all the time. Suddenly “innovation” is the bad guy when it really had nothing to do with it.

Q: You say innovation is a numbers game.  How so—and how does DevOps fit in?

DE: Innovation is a numbers game because, like most things in life, business has a countdown clock. If you’re a startup, it’s a simple clock. It’s the amount of cash left in the bank. If you are an established company, it’s a bit more complicated. It could be how long until a competitor beats you to the punch. It could be how long the CEO gives you to meet a goal. The point is, one way or another, you have a finite amount of time to absolutely delight the hell out of your customers by figuring out what they want and delivering exactly that to them.

You don’t control the clock and you don’t control the customer—what do you control? You can control the number of chances you get to delight the customer before the clock runs out. That’s where DevOps comes in. DevOps aims to remove all of the bottlenecks, inefficiencies, and risks between a business idea (the “ah-ha!”) and a measurable customer outcome (the “ka-ching!”). When you remove all of that, you get a lightning-fast and highly reliable service delivery pipeline that spans from the edge of Development all the way to the datacenter. That allows you to run more experiments, get faster feedback, and take more “shots at the prize.”

Q: DevOps promises a more responsive, more collaborative IT department that can realize business ideas faster.  So what is holding back its widespread adoption?  What’s the challenge or downside?

DE: There was a movie called “Charlie Wilson’s War” that had a great line between Tom Hanks, playing a U.S. Congressman, and Philip Seymour Hoffman, playing a CIA agent. Hoffman asks, “Why is Congress saying one thing and doing nothing?” Hanks replies, “Well, tradition mostly.”

All jokes aside, tradition is a powerful thing and hard to break. Tradition, or “what we’ve always done,” in IT is no different. There was a thread on Slashdot just this past month that asked whether developers should be allowed to deploy their own applications. You should have seen the outcry. The sheer number of commenters who shot down the idea as pure heresy was shocking. And the richest part of all of their denunciation was that the mob said over and over that the idea would “never work at a real company with real revenue at stake.” I thoroughly enjoyed sending that to John Allspaw, who runs all of technology at Etsy, and Jesse Robbins, who was in charge of risk and disaster planning for operations at Amazon. Etsy does over $600 million of transactions per year, and Amazon does about $50 billion in revenue. In both companies, developers are the ones who deploy and own the uptime for their own code. John’s reaction to the thread was a simple yet priceless one: “OMG.”

Q: Cloud and SaaS services promise flexibility and value to the business, but may seem to undermine or complicate traditional Ops teams.  How do these disruptive factors fit with efforts to embrace innovation?

DE: We have a saying that we use a lot at DTO Solutions: “Moving to the cloud without changing your processes is just expensive and complicated Hosting 2.0.” The cloud gives you a new abstraction layer that provides all sorts of benefits in the form of flexibility and speed. But to take advantage of those benefits, you first must change your application lifecycle and operating procedures. Furthermore, you have to revisit the architecture and deployment model for your applications. Often you’ll find that the choices that were made in the past were based on outdated ideas like the need for hardware conservation or to fit a monolithic codebase into a waterfall project delivery cycle. The conditions have changed, so companies need to rethink how and why they do things within the context of the new conditions.

The cloud removes all sorts of infrastructure barriers that makes moving at a faster pace even possible. DevOps addresses the process and cultural issues. Agile addresses the software development process issues. Customer Development and The Lean Startup remove the business process issues. You add it all up and you are ready for your organization to move at speeds that you never thought were possible.

For more from Damon Edwards, check out DTO Solutions, their DevOps blog, and the upcoming “DevOps Cookbook.” Then check out Discover Performance’s recent DevOps issue.

************************************

 

rerun screenshot

Rerun: Making shell scripts even more useful (and a bit cool, again)

6

Damon Edwards / 

I recently made a couple of additional videos about the curiosity that is the Rerun project. You can find them below.

The conventional wisdom on shell scripts is that… well… “shell scripts suck”. But why? Shell as a language is extremely powerful and useful but shell scripts can quickly become unwieldy when trying to use amongst a team or in long-lived operations. But what if you had a framework that solved the team-level problems and the lacking of standardization while letting you use the full power and familiarity of shell? Enter Rerun, a simple tool that turns your favorite shell scripts into modular automation that has standardized options handling, command line completion, documentation generation, and a built-in test framework. Suddenly shell scripts don’t suck so bad anymore.

Why am I so interested in Rerun? Because I’ve seen Rerun have a positive effect on a very real human problem: In most non-startup organizations, the DevOps divide is made worse by a mismatch of skills, tools, and technologies.

It’s common for the Ops team to have used a tool like Puppet to automate server config and image building. But when it comes to app deployment and config, the various app teams don’t have the Puppet skills or motivation to follow suit. So each app team picks their own tooling or glue language. Of course, this just confuses Ops and makes their lives more difficult. Sometimes there will be a centralized release team (often now awkwardly rebranded as the “DevOps Team”) that will attempt to pick their own solution. But, neither Dev nor Ops ends up bring happy with the choice and the “DevOps Team” is now the bottleneck in the middle. Lots of noble DevOps intentions die in scenarios like this.

The effect of Rerun is that everyone can now come to the table on equal footing and use shell as their lingua franca. They can learn to collaborate using a simple framework for the “glue” that holds things together (of course, Ops still builds server images using a config management tool and Dev still builds their apps the way they want to). The built-in documentation generation and test automation framework makes handoffs easier. Everyone knows the simple command and options interfaces, but can also read each others code if need be (it’s just shell scripts, after all). Once you get everyone engaged and contributing to bridging the DevOps Gap, you can collaboratively start to look to other newer, specialized solutions.

I have to get Rerun’s creator, Alex Honor, to do a full post on Rerun. In the meantime you might find these videos interesting:

Video 1: Chuck Scott gives a tour of how he uses Rerun to turn his “keeper scripts” into reusable, standardized, test-driven automation

Video 2: Group discussion with Anthony Shortland, Lee Thompson, Chuck Scott that looks at an example of a DevOps toolchain automated with Rerun

 

 

screen

Automating the full management lifecycle of Jenkins using Rerun

8

Damon Edwards / 

Need a simple and self-contained* way to automate the full lifecycle of a Jenkins instance (install, uninstall, manage plug-ins, manage jobs, etc.)? Anthony Shortland shows how he gets it done with Rerun.

(*Why simple and self-contained? Many reasons… the company-wide adoption of full config management solution is proceeding at uneven pace, the need to use a lowest-common denominator language so you can have simple handoffs, you want to avoid “religious” tool wars, you need a very small footprint, you need it to be totally portable, …. and the list goes on)

 

 

Here is where you can find the Jenkins Rerun module:

https://github.com/rerun-modules/jenkins

 

RedactedValueStream

Improving Flow: Fix the Handoffs to Remove Your Worst Bottlenecks

2

Alex Honor / 

Minimizing time to market and getting faster feedback from customers are primary concern for businesses who want to stay competitive. You need to be able to go from a business idea to a customer-facing running service as quickly, reliably, and effortlessly as possible. This as a flow of work that crosses many organizational silos.

Where does this flow often bog down? Handoffs. Whether the handoffs are within a team (e.g. Dev to Dev) or between teams (e.g. Dev to Ops), there is always the need to pass work from one stage of the lifecycle to the next.

http://www.flickr.com/photos/seven13avenue/2791099838/in/photostream/

At DTO Solutions, our clients are often already aware that they have flow problems when they ask us to for help. When we use techniques like Value-Stream Mapping to learn how the work flows, handoff problems are prominent forms of waste that jump off of the page. The diagram below uses pie charts to highlight the relative time lost due to difficult handoffs during the product life cycle.

What are common reasons for difficult handoffs?

  • Conversations, email, multitudinous wikis, spreadsheets, and trouble ticket systems are used to describe, in human language, how to process work. Words are open to interpretation and documents often lag behind current operating procedure. Just imagine being the person planning or performing the work and traversing the information across these various tools.
  • Software product artifacts differ between stages of the process. Sometimes software resides in a directory on a file share and other times it’s a TAR file. The software handoff may contain the same bits, but must be handled or converted by the downstream stage of the software delivery process.
  • Work can be considered “done” yet be unfinished or in a non-working state.The lack of a test or means to verify the work was done correctly often leads to products not ready for the next person down the line. This can leave the person in the downstream stage with what is essentially scrap that has to be rejected or redone.
  • Ad hoc procedures or loose scripting often lead to different approaches and implementations for what should be standard operating procedure. This can lead to silo-specific utilities with different levels of quality and testing.

Handoff problems affect organizations, both big and small. Obviously, one answer to solving handoff problems is to minimize them. But if you are in an organization larger than just a handful of people, that just isn’t a realistic option. To decrease time to market and enable fast feedback, you are going to have to roll up your sleeves to solve the handoff problems.

Where are good places to start making handoffs smoother?

Here are a few of the top fixes that we find important for solving handoff problems at their source:

  • Consistent packaging
    • The most direct way to simplify software handoffs between Dev and Ops is using a common system package format like RPM or Debian. Using a system package format also aligns application deployment and system provisioning practices.
  • Encapsulated procedures
    • Rather than loose scripts or team-specific ones, choose a framework that enables modular automation. Using a modular approach results in a shared tool box of utilities and captured process.
  • Converting information flows into artifact flows
    • Rather than rely on human read text as the product for the downstream process to handle, formalize it as an automation product and build on the idea of encapsulated procedures.
  • Procedure verification tests
    • Verification testing should not be dominated by manual checks described in text documents. Building on the idea of converting information into artifacts, implement verification using a test automation framework. Most apps have some level of testing to verify functionality. Build a testing framework to verify an operation (eg, software deployment) procedure was successful by executing an automated test.

In subsequent posts, we’ll address each one of these fixes.

 

slide

Defining and Improving DevOps Culture (Videos)

4

Damon Edwards / 

Culture. It’s the most mentioned and the most ignored part of the DevOps conversation.

Lots of lip service has been paid to the importance of culture (“It all starts with culture”, “DevOps is a cultural and professional movement”, “Culture over tools”, etc..). But just as fast as everyone agrees that culture is supreme, the conversation turns straight to tools, tools, and more tools.

Recently, John Willis, my fellow dev2ops.org contributor and DevOps Cookbook co-author, let this tweet fly:

John has been as big of a culture warrior as anyone — constantly fighting to elevate the importance of and the discussion around DevOps Culture. He later said that this tweet was part exasperation and part challenge.

It was obvious to John that the difference between high performing and low performing companies was their DevOps culture, not the tools. But rather than be satisfied by the default explanation of DevOps Culture maturity being either that a company “gets it” or “doesn’t get it”, John was challenging the community to dive deeper into the issue.

During the week of Velocity London and DevOps Days Rome, there were finally some presentations that answered that call and were all about the culture. I did a presentation on defining DevOps Culture and what high performing companies do to reinforce it (based on the work of DTO Solutions). Michael Rembetsy and Patrick McDonnell gave a great peek behind the scenes of Etsy’s transformation to a company with a fast moving and high performing culture. Mark Burgess (CFengine) gave an interesting talk on the importance of, and science behind, relationships.


(slides were updated after the presentation)

 

 

(when you watch Mark’s video you will understand why there are no slides posted here!)

Update: John Willis knocks it out of the park talking about the importance of culture and the classic influence of Deming on this recent episode of the Food Fight Show.

WhereDevOpsProblemsLive

DevOps Transformation Workshop for Technical Managers

1

Damon Edwards / 

DevOps problems, by their very nature, are organizational issues. DevOps problems live in the “white spaces” between people and groups. Like all organizational issues, it’s ultimately the responsibility of management to solve DevOps problems.

It’s no easy task to manage a DevOps transformation initiative. Not only are you addressing people, process, and tooling issues that span multiple silos, you are also balancing current business commitments, resource constraints, and good old fashioned human dynamics.

 

Where do you start? What are the traps you’ll need to avoid? What are the best practices to adopt? How do you work across roadblocks like functional silos and entrenched culture? And ultimately, how do you avoid this becoming another failed transformation project that could put you or your business at risk?

DTO Solutions repeatedly addresses these issues across it’s diverse client base. Either as coaches, architects, or change agents, stepping up to the responsibility for making DevOps transformations successful is at the heart of DTO’s business.

I don’t normally use this form to directly promote the services of DTO Solutions. But I believe that this is a special case as it is a unique offering that I’ve seen make a positive impact for our clients.

DTO Solution’s has taken it’s best practices and methodology for managing DevOps transformations and turned it into a 2-day workshop for managers. What was previously only available to client’s within broader engagements is now available as a stand alone public class.

You can read about the details and register here:
http://www.devopstransformation.com

Page 2 of 2612345Last