View All Videos
Peanut butter in my chocolate? Convergence vs. Ad hoc Control

Peanut butter in my chocolate? Convergence vs. Ad hoc Control

2

Alex Honor / 

Recently, I’ve been in several conversations about how to reconcile two links in the management tool chain, configuration management and ad hoc control. The conversation usually revolves around conventional viewpoints about the nature and roles of these tools:

Configuration management wants to achieve and enforce a system state. The normal mode of operation is for the configuration management tool to run autonomously on each node, applying the rules of that node’s specification. In the large, the set of hosts eventually converge on the declarations made in the specification.

Ad hoc control tools are generally used to for special problems or on-demand management activities. Their use is often arbitrary and driven by management need and not on a regular periodic basis. Problem diagnosis, information gathering, deployment changes, emergency fixes across many hosts are example cases where ad hoc control tools are often used.

If these tools are used independently, one or more scenarios like the following can arise:

 

  • Changes made by ad hoc tools are reverted or partly undone by the configuration management tool
  • Configuration management tool is turned off and only run directly when the rules are needed to fire
  • Administration code is reinvented in each tool but their implementations focus on different concerns
  • One tool gets thrown out or its role and scope are severely diminished in favor of the other

 

These tools can interfere with each other and to some they can appear mutually exclusive.

Clearly, both management modes are important. On the one hand, it’s beneficial to have an ongoing compliance and enforcement loop driven by specification. On the other hand, issues of the day require near instantaneous action. Those actions may not become routine and deemed part of the desired permanent state maintained by configuration management.

Planning out how these two kinds of tools can interoperate begins with a discussion of boundaries and lines of responsibilities. It also helps interoperation when both tools share a common model and points of control.

Here are a few areas where I have seen interoperation facilitated:

  • Centralize the control loop: Initiate the control loop from a central point of administration. Configuration management loses some autonomy but the pull model is preserved which is important for scalable execution. 
  • Split Install from Startup: Separate the startup action from the installation procedures in the provisioning process. This defers startup to a centrally orchestrated authority. This is vital for multi-tier applications.
  • System control interface: Define a set of system commands that control service state on each host. Configuration management and ad hoc tools can avoid reinventing the wheel. This also gives a point of collaboration between teams.
  • Dependency driven packaging: Use packages to install files on node. Use package dependencies to install files in an order. System package utilities provide other benefits like verification and repositories.

 

 

Video: Lloyd Taylor on diagnosing and transforming your organization

Damon Edwards / 

I find Lloyd Taylor to be creating some of the most interesting and fresh DevOps related content around. He dives right to the heart of cultural issues and provides clear insights and practical strategies we can all use. Lloyd comes from an interesting background that combines senior operations experience at some of the biggest names on the web with his long-standing interest in studying interpersonal and organizational dynamics. If it sounds like I’m a fan, it’s because I am. Each time I listen to Lloyd speak, I learn something new. Hopefully you will as well.

Enjoy Lloyd’s presentation (and the ensuing discussion before my camera battery ran out) from this week’s SVDevOps meetup, it’s well worth an hour of your time:

 

 

Books from Lloyd’s recommended reading list (links go to Amazon):

Video Q&A: Adam Rosien on how Continuous Deployment relies on great testing

Damon Edwards / 

Last night, Adam Rosien of Wealthfront (formerly called KaChing) gave a presentation on Continuous Deployment at the Large Scale Production Engineering Meetup on Yahoo’s campus.

Afterwards, I grabbed Adam for a quick chat about a topic that has been troubling me: 

Newcomers to the Continuous Deployment idea often overlook the importance of fully automated testing and instead focus their attention on the number of deployments these companies are making per day.

In the video below, Adam stresses how important a role testing plays in Continuous Deployment. Testing really is the linchpin on which a successful implementation of the Continuous Deployment methodology relies. 

If you are an active reader of this blog or listen to the DevOps Cafe podcast, you probably know by now that we are big fans of the process, tooling, and culture being cultivated by the folks at Wealthfront. Previous content featuring them can be found here, here, and here.

They are not clients of any of the contributors to this blog. We just think that they are a great example of a scrappy company rethinking IT Operations to maximize business value and agility. Their engineering blog is a must read. 

 

Video Q&A: Aaron Peterson and Kevin Gray on self-healing infrastructure

1

Damon Edwards / 

At LISA 2010, I caught up with Aaron Peterson (Opscode) and Kevin Gray (Dyn) after they gave a very interesting presentation/demo called “DevOps Gameday“.

From the title, I think a number of attendees were expecting to see the standard Dev to Ops promotion/deployment of code that is so common to the DevOps discussion. Instead the presenters (Opscode, Zenoss, Dyn Inc.) focused on what happens when you have a failure after the code has been deployed. This demo was about self-healing infrastructure… breaking a multi-node system and having it heal itself.

Of course, this kind of canned demo isn’t all that new in the vendor world. However, what is very interesting about their efforts is they want to capture the best practices required to do it and share the code with the world through their combined project (hosted on GitHub). 

If they fulfill the mission of their open project, it’s exactly the kind of “here is how you can do what the big players do” sharing that is good for our industry. 

 

DevOps is not a technology problem. DevOps is a business problem.

10

Damon Edwards / 

Update: follow-up post about the competitive business advantage that DevOps enables.

 


Since Patrick Debois called for the first DevOps Days event and unleashed the term “DevOps” upon the world, there is no denying that DevOps has evolved into a global movement.

Of course, DevOps has its detractors. Negative opinions range from the misguided (“DevOps is a new name for a Sys Admin”) to the dismissive (“DevOps is just some crazy Devs trying to get rid of Ops” or “DevOps is just crazy some Ops trying to act like Devs so they will be better liked”) to the outright offended (whose arguments tend to defy logic).

I’ve spent the past nine months or so overcoming resistance to the DevOps movement in both public forums and inside client companies. During that time, I’ve begun to notice a common misconception that I think is fueling much of the negative initial reaction that some people have to DevOps ideas. I want to take a shot at clearing it up now:

DevOps is not a technology problem.

Technology plays a key part in enabling solutions to DevOps problems. However, DevOps itself is fundamentally a business problem.

What does the business have to do with DevOps?

The most fundamental business process in any company is getting an idea from inception to where it is making you money.

Within that business process there are all kinds of activity that needs to happen, some technology-driven and some human-driven. This is where all of the different functions of IT come into play. Developers, QA, Architecture, Release Engineering, Security, Operations, etc each do their part to fulfill that process.

But if you take away the context of the business process, what have you got? You’ve got a bunch of people and groups doing their own thing. You lose any real incentive to fight inefficiency, duplication of effort, conflicts, and disconnects between those groups. It’s every person for themselves, literally.

But you know what else happens if you remove the context of the business process? Your job eventually goes away. Enabling the business is why we get paychecks and why we get to spend our time doing what we do.

If there is no business to enable or we don’t do any business enabling, this all just turns into a hobby. And by definition, it’s pretty difficult to get paid for a hobby.

The whole point of DevOps is to enable your business to react to market forces as quickly, efficiently, and reliably as possible. Without the business, there is no other reason for us to be talking about DevOps problems, much less spending any time solving them.

 

 

Doesn’t this sound a lot like the goals of Agile?

If the goals of DevOps sound similar to the goals of Agile, it’s because they are. But Agile and DevOps are different things. You can be great at Agile Development but still have plenty of DevOps issues. On the flip side of that coin, you could do a great job removing many DevOps issues and not use Agile Development methodologies at all (although that is increasingly unlikely).

I like to describe Agile and DevOps as being related ideas, who share a common Lean ancestry, but work on different planes. While Agile deep dives into improving one major IT function (delivering software), DevOps works on improving the interaction and flow across IT functions (stretching the length of the entire development to operations lifecycle).

 

But I thought DevOps was all about cool tools?

Technology is the great enabler for making almost any business process more efficient, scalable, and reliable. However, we have to remember that on their own, tools are just tools.

It’s just as likely that you’ll use a new tool to reinforce bad habits and old broken processes as you will to improve your organization. It’s the desired effect on the business process you are supporting that dictates why and how a tool is best used.

When people are clear on what their DevOps problems are and what process improvements need to happen to remedy those DevOps problems, the tool discussion becomes rather straightforward (if not trivial).

Since the nascent DevOps movement is mostly made up of technologists, it’s easy to understand why there is such excitement to jump right into tooling discussions. But perhaps we need to do more to make sure that everyone is up to speed on why the tools are needed and what the desired business process improvements are before diving into our standard “Puppet vs. Chef” or “Files Centric vs. Package Centric” arguments.

 

If DevOps is about a business process then why is it called “DevOps”?

In my opinion, one fault of the early DevOps conversations is that it wasn’t immediately clear just how big the scope of this problem truly is. Now that we have a year of perspective behind us, it turns out that we have been attacking one of the biggest problems in all of business: How to enable a business to react to market forces as quickly as possible.

But alas, the conversation had to start somewhere so it gravitated to the almost universal problem of conflict and disconnect between Developer culture and Operations culture. Every org chart is different, but it’s fairly easy to cartoonishly divide the world into a Dev camp and Ops camp for the sake of having common reference points to discuss (even though we all know the world is much more complex and grey).

Within that cartoon Dev/Ops example, most of the early DevOps attention has been about improving deployment activity. Since change activity makes up the bulk of the work across an IT organization, that too was a logical and natural place to start.

Maybe Patrick should have called that first event “BizDevQASecurityOpsCloudUsers Days” or “SolvingABroaderProblemThanAgile Days”… But I doubt anyone would have shown up.

Anthony Shortland on strategies for solving DevOps problems (Video)

1

Dev2ops / 

Last night, Adrian Cole (of jclouds fame) organized a Java-oriented DevOps Meetup here in San Francisco. 

There were a number of freeform lightening talks used as an excellent device to get the conversation started. Below is an interesting talk from DTO’s Anthony Shortand. Anthony gave an excellent conceptual introduction to two strategies for solving DevOps problems, “Full Lifecycle Traceability” and “Establishing a Formal Separation of Concerns”.

 

 

 

 

Page 11 of 26First910111213Last