View All Videos

Archive for June, 2008

Looking for “cross-over people” at O’Reilly Velocity 08

Looking for “cross-over people” at O’Reilly Velocity 08


Damon Edwards / 

In the comments section of a previous post, Berkay nicely summed up why it is so difficult to solve the development to operations problem:

“There are very few people that have the crossover skills. Developers who have operations experience/knowhow and operations people who have development/deployment experience is rare. Further there are organizational silos enforcing this divide.”

The observed gap between development personnel and operations personnel is a subject we’ve touched on before. Much of the success of running an efficient business based on online services depends on closing this gap.


The first step to closing the development to operations gap is getting everyone talking and establishing a common vocabulary. Any event that promotes these types of discussions are a good thing in our book.

O’Reilly’s new conference, Velocity, should be a good forum to hold these conversations. Alex and I will be attending Velocity and will also be participating as exhibitors (with ControlTier). The bulk of the conference agenda is focused on infrastructure design and management rather than application deployment and management, but the presence of both is a good sign.

If you are attending the conference, we look forward to meeting you. If you aren’t registered yet, you can use the code “vel08js” for 20% off.

Use the word “process” and confusion ensues

Use the word “process” and confusion ensues

Damon Edwards / 

In my last post about the River of News for Monitoring concept, process played a central role. In various conversations I’ve had since writing that post, it’s become clear to me that the word “process” is a tricky and overloaded word. There are lots of processes whirling through an enterprise. There are business processes (customer transactions, supplier transactions, human resources, finance, etc.). There are application processes (pretty self-explanatory). Then there are IT processes. My view of IT processes is that they are the actions that transform your IT assets and their related environments. (Note: obviously you could use an ITIL-like definition born from a standards body at this point, but I’m looking for a simple set of buckets to use without causing further debate)

There are plenty of tools that track and examine business processes. There are plenty of tools that track and examine application processes. However, when it comes to IT process, the available tooling is quite thin. Sure there are tools (e.g. ticketing systems, bug trackers, approval workflows, etc.) that track the HUMAN aspect of your IT processes, but they give you very little visibility into what actually took place at the system or application level. To make matters worse, under their fancy dashboards, most of these systems they rely almost entirely on a human to tell them what was done or observed. In today’s complex and highly distributed environments, it’s almost impossible to get an accurate picture of what really took place using these faulty or outdated techniques.

Skeptical? Give the status quo a test. Walk into any sizable IT operations and ask them to do 2 things:

1. Show you all of the deployment activity, with the context that those activities occurred in, that took place in [pick some slice of their environment] between [pick two arbitrary points in time]. This doesn’t mean things like “Bob said he completed the steps of this process” or “Jane said she ran this set of scripts”. I’m referring to evidence of the actual technical activities that took place across the boxes.

2. Show you the entire lifecycle of [pick an arbitrary application package], from when it was built and packaged to all of its deployments throughout Dev, QA, and Prod environments.

I would be shocked if they had this information readily available. In most cases, they couldn’t produce this information at all. For many companies these systems are their source of revenue generation, their “factory floor” if you will, and they can’t answer these simple questions. In any other industry this would be wholly unacceptable.

This is the situation we are working on changing.

IT Operations Needs a “River of News”

IT Operations Needs a “River of News”


Damon Edwards / 

Over time, user interfaces for a class of tools tend to settle on a common paradigm. User expectations and vendors’ copycat nature form a self-correcting loop that settles somewhere around a comfortable middle. For a fun example, try to spot the differences between the Yahoo and AOL homepages. User interface paradigms in the systems management world have followed this same herd mentality.

Monitoring has always been the dominant feature of systems management, so it is of little surprise that the flavor of monitoring has dominated the user interfaces and dashboards of the major enterprise systems management tools. Show me my things. Show me the state of my things. Show me some rollup numbers about my things and their states. For answering questions about things (usually servers) and the state of those things, this classic monitoring paradigm tends to work quite well.

But this point of view falls short in an area that is gaining importance, visibility into IT operations processes. As systems become increasingly distributed and IT operations moves from a back of the house function to a revenue producing core competency, visibility into process is all the more important. Who has carried out what actions? What actions were performed on what machines? What’s the history of a set of packages as they have moved from development to production? How can your organization know when a complex update process has been completed? How do you know when changes have been made outside of approved change windows? The inventory and state paradigm of traditional monitoring tools doesn’t help you much when you are asking these kinds of questions.

A new kind of monitoring paradigm is needed to answer these questions. The most promising concept I’ve seen as of late comes not from the systems management world but rather from the blogosphere. Dave Winer first popularized the concept of a “River of News” and the concept has applicability here. Simply put, Dave describes the flow of information that comes into his feed reader to be more like a river of information going by his door than a set of messages being delivered to his mailbox.

The activity that takes place within IT operations can similarly be likened to a river. Today, most of that river of activity takes place in the shadows and little, if any of it, is captured by a central system. Usually, the only way to find out about these events is by word of mouth or person-to-person email chains. Anyone who is more than 1 or 2 degrees of separation away from the event is usually flying in the dark.

So how would a river of news style tool for enterprise systems management processes work?

1. You need to create the river. All of the scripts and tools you use to build your deployment artifacts and manipulate your environments need to dump events into the river. As a side benefit, creating the river is an incentive to enforce the rule that changes must only be made through change management tooling.

2. You need to create filters for the river. Filters allow you to view the river from a certain point of view (like a package, user, node, environment, etc) or only view events that happened between certain points in time. You’re going to need both common views as well as the ability to setup and share ad-hoc views.

3. You need to setup notifications. You can’t watch the river at all times and, unlike the blogosphere, there are some pieces of news that you just can’t miss. You need to be able to set traps that watch for events or a series of events that match particular patterns. When those traps are hit you then have to make sure the right people are notified and sent the relevant information.

4. You need to introduce management dashboards and auditing reports. Keep the senior managers, bean counters, and compliance auditors happy and your life will be happier.

This whole idea is still a pretty fresh one. In upcoming releases of ControlTier’s ReportCenter, we are going to be introducing these features and seeing where they take us. Any and all comments or suggestions are more than welcome.