View All Videos

Archive for the ‘News’ Category

DTO at Velocity 2011

DTO at Velocity 2011

John Willis / 

Meet the DTO team at Velocity 2011.  Damon, Alex, myself and the rest of the gang will be attending Velocity in full force thus year.  We will have a booth again this year and we will also have a special guest @lusis (a.k.a. John Vincent) the author of Noah.  John, Alex, and myself plan on giving a killer booth demo using Rundeck, Chef, and Noah integrating command and control, orchestraion, convergence and good old fashion configuration management.  The demo will highlight multiple components of what we call the “Loosely Coupled Toolchain”.  Damon and Lee Thompson will be giving a presentation “Automating for Success: Production Begins in Development” and of course we will be attending and speaking at DevopsDays Mountainview.  Please stop by to spread some #devops love at the booth if you don’t see us in the hall.


John (a.k.a. @botchagalupe ) 

Devops Driven Demand

Devops Driven Demand

John Willis / 

I’m honored to have a guest post on the “Agile Web Development and Operations” blog this morning.  Here is a tease… 

What if DevOps created more defects, tickets, requests, and more overall work? Would that be a good thing or bad. Let’s take a look.

Read the rest here…

Peanut butter in my chocolate? Convergence vs. Ad hoc Control

Peanut butter in my chocolate? Convergence vs. Ad hoc Control


Alex Honor / 

Recently, I’ve been in several conversations about how to reconcile two links in the management tool chain, configuration management and ad hoc control. The conversation usually revolves around conventional viewpoints about the nature and roles of these tools:

Configuration management wants to achieve and enforce a system state. The normal mode of operation is for the configuration management tool to run autonomously on each node, applying the rules of that node’s specification. In the large, the set of hosts eventually converge on the declarations made in the specification.

Ad hoc control tools are generally used to for special problems or on-demand management activities. Their use is often arbitrary and driven by management need and not on a regular periodic basis. Problem diagnosis, information gathering, deployment changes, emergency fixes across many hosts are example cases where ad hoc control tools are often used.

If these tools are used independently, one or more scenarios like the following can arise:


  • Changes made by ad hoc tools are reverted or partly undone by the configuration management tool
  • Configuration management tool is turned off and only run directly when the rules are needed to fire
  • Administration code is reinvented in each tool but their implementations focus on different concerns
  • One tool gets thrown out or its role and scope are severely diminished in favor of the other


These tools can interfere with each other and to some they can appear mutually exclusive.

Clearly, both management modes are important. On the one hand, it’s beneficial to have an ongoing compliance and enforcement loop driven by specification. On the other hand, issues of the day require near instantaneous action. Those actions may not become routine and deemed part of the desired permanent state maintained by configuration management.

Planning out how these two kinds of tools can interoperate begins with a discussion of boundaries and lines of responsibilities. It also helps interoperation when both tools share a common model and points of control.

Here are a few areas where I have seen interoperation facilitated:

  • Centralize the control loop: Initiate the control loop from a central point of administration. Configuration management loses some autonomy but the pull model is preserved which is important for scalable execution. 
  • Split Install from Startup: Separate the startup action from the installation procedures in the provisioning process. This defers startup to a centrally orchestrated authority. This is vital for multi-tier applications.
  • System control interface: Define a set of system commands that control service state on each host. Configuration management and ad hoc tools can avoid reinventing the wheel. This also gives a point of collaboration between teams.
  • Dependency driven packaging: Use packages to install files on node. Use package dependencies to install files in an order. System package utilities provide other benefits like verification and repositories.



OpenStack leaves out the “extra” you’ll only find in Amazon AWS

OpenStack leaves out the “extra” you’ll only find in Amazon AWS

Noah Campbell / 

Often I’ve heard the off hand comment that it is just a matter of time before there is another rival to Amazon’s EC2. The private cloud has always been looming around the corner for those with large pocket books, and yet one rarely hears about a cloud offering the beats AWS’s EC2. Even Rich Wolski, CTO of Eucalypus, has no delusions as to the scale pre-package software can tackle out of the box.

“One of the misconceptions about Eucalyptus is that it is able to allow an org to compete with Amazon,” Wolski told us. “The Amazon AWS [Amazon Web Services] cloud is far more than a collection of software components. It operates on a gigantic scale, multiple time zones, multiple data centers, human resources that must be committed to maintain it so it can operate at that scale. It’s not likely — maybe even impossible — that you’re going to download something from the internet that is going to be able to operate at that scale. Eucalyptus can’t really be used for that purpose.” [1]

So what gives EC2 the “extra” everyone wants to replicate?

First and foremost, AWS’s EC2 is not a product of technology, but instead a product of operational excellence. It is in their culture, all the way down to their infamous door desks [2] and Jeff Bezos mediations on waste to shareholders in 2008 [3]. They took existing open source software and wrapped a service interface around it, while quietly offering some pretty amazing feats (vm startups measured in minutes from some mysterious cloud storage). Don’t let their technology stack fool you, their value is in their operational discipline. AWS keeps their operational practices under wraps because they believe it gives them an advantage, and even if they did divulge them, they’re likely nontransferable to your organization.

With the recent announcement of OpenStack, folks can consider another set of software components to implement their cloud and try to replicate the “extra” they get from AWS. However, without a solid focus on operations, they’re not going to get much further than they would have gotten with Eucalyptus.


This post is by the newest blogger, Noah Campbell. Noah is a consultant specializing in automated infrastructure and DevOps process improvement.



Scheduled Downtime vs. Seamless Conversion

Scheduled Downtime vs. Seamless Conversion


Lee Thompson / 

I fly a lot.  I did before I was a consultant, but as a consultant, my dependency on air service is very important.  American Airlines effectively fired me as a customer when they dropped the proverbial “nerd bird” flight last year which was the Austin to San Jose non-stop flight.  The last few nerd bird flights there was lots of on board discussion on Central Texas to Bay Area route options after the sudden termination notice we were given by American.  Most of the folks I spoke with had 1 to 2.5 million miles flown on AA.  That is a huge amount of butt time 500+ mph chairs!

When I’m on the planes I frequently work.  But many times I schedule some personal downtime and have a gin-n-tonic and watch a DVD.  Everyone needs scheduled downtime.  Most of you Dev2Ops readers are probably burning some of your personal scheduled downtime right now reading this post.  I assume Dev2Ops readers also don’t get a lot of scheduled downtime for the online business infrastructure they support (if ever!).  So I was blown away when I got this email from my new transportation services vendor I hired…

First off, let me say jetBlue has a great service picking up poorly serviced travel customers like the recently terminated nerd bird flock and their prices are just so reasonable.  The in flight TV is nice so you can do things like get the latest news from Haiti. JetBlue is donating their services to get personnel and supplies in which is just fantastic to see.

But now the eBusiness technologist part of my brain kicks in and says “WHAT!”.  An airline business running without a ticketing system for two days intentionally.  Scheduled downtime;  are you crazy?  I can only imagine what kind of bind jetBlue is in with their technical infrastructure.  Surely the CEO signed off on such a project plan so the issue must be nasty.  I’m left with so many questions.  Why can’t you phase in the new ticketing system by route and gradually obsolete the older system?  Why can’t you do what Facebook describes as “Dark Launches”.  Why can’t the new system run 5 minutes after the old system is powered off? 

I’m asking the wrong questions.

You have to take your technologist hat off and put on your business hat on and ask different questions.  What is the cost difference between a seamless conversion and  a scheduled outage conversion?  Would jetBlue have to raise air-fares or ask the shareholders to suffer losses?  Does the complexity of the requirements to implement a seamless conversion put the conversion out an extra year hindering the business?  Does the added complexity of a seamless conversion add tremendous risk to business operations?  Having done numerous transaction system conversions in my career, I can easily say a seamless conversion is probably 4x the size of a scheduled outage conversion (or more).  Minimizing complexity substantially reduces risk by a greater than the 4x rate as the relationship to risk and complexity is non linear!  Case in point, if the business singed up for the added expense, and schedule delay, what business impact would occur if the technology effort failed???  I would imagine given the above email, the risk was too great.

I’m about to head to the SFO airport and jump on a jetBlue flight who by that time won’t have a ticketing system.  My travel schedule makes me personally interested in jetBlue’s success in this conversion obviously!

BTW – I choose United non-stop to San Francisco and jetBlue non-stop on the way back.  Good news…  Alaska Airline service picks up mid March to San Jose!

Page 3 of 1012345Last