View All Videos

Archive for the ‘News’ Category

Polling dev2ops – How do you maintain system configuration files ?

Polling dev2ops – How do you maintain system configuration files ?

4

Alex Honor / 

The system administration dev2ops folks in the trenches spend a good deal of their time maintaining environments supporting the development, test and production instances for their business services.
An essential aspect of environment management is maintaining the host’s configurations to support the application. Everyone has their preference as to the approach and toolset they use. Which one do you use and why?

Please don’t forget to leave a comment about your response!

Polling dev2ops

Polling dev2ops

Alex Honor / 

When you think about the multitude of companies that conduct business via software as a service, you’d expect to find a diverse range of underlying business models: CRM, e-tailing, social networking, advertising, etc. But if you’re not in the trenches with the dev2ops folks, you might be amazed at how diversely these companies manage their application environments and the variety of processes and mechanisms they employ.

Lots of research and mature best practice has developed in areas like security, availability and scale. Because these areas are more established and the problems better understood, one can rely on their proven methods and exploit their commodity solutions. There is much less agreement about how IT should technically manage software as a service environments, especially those that change often and are complex, multi-layer, multi-site online software systems.

Sure, there are established methods around scheduling and planning application change as evidenced by the various trouble ticket, bug trackers, and issue management tools out there. There are also emerging IT service management practices and paradigms from IT governance initiatives, such as ITIL. These tools and methodologies mostly focus on “people management” and are essential to coordinate the human activity and workflow. What is surprising, is the lack of consensus on “how” changes are technically performed and “what” is used to perform the changes.

After all, the dev2ops folks in the trenches know it boils down to a series of technical steps: people and or scripted tools distribute various files, set environment configurations, import data, alter stored procedures, synchronize content repositories, check and or modify permissions, stop and start processes, etc., etc. It is at this level of change implementation, where such an astounding variety of tools and procedures can be witnessed.

How and what implements change can even differ between groups within the same company. Developers may use developer tools to update running applications in test environments while operations uses a completely separate set of tools and procedures to maintain production environments. Even further, businesses that operate more than one application, might see this range of practices multiplied. The worst case, is one where there is a different process and set of tools for each combination of application, team, and environment.

Why do we see such a variety of approaches used to change application environments? Does each group need to invent their own procedures and toolsets? Are environments and applications so unique that there must be a customized and optimized solution for each case? Can one establish some basic approaches and paradigms that can be shared across organizational teams, even businesses?

To help answer these questions and better understand why folks choose from such a diverse array of approaches and tools, we here at dev2ops will conduct a series of polls. But, we need your help in the form of participation. We’ll come up with the first round of polls and rely on you to respond and post about your poll selections. You can also send survey ideas to polls@dev2ops.org and if that ground hasn’t already been covered, we’ll post those, too.
During the course of the polling, we’ll look for common patterns and pitfalls, some obvious to some folks while others novel to others. In the end, we can look for where there is common agreement and compare our own current practices to those.

So, look out for the polls and vote !

Change automation logs: a key tool for resolving outages

Change automation logs: a key tool for resolving outages

1

Damon Edwards / 

Todd Hoff at High Scalability wrote an interesting piece on what and how to log. His position is that you essentially need to “log everything all the time”. However, curiously missing from his list of what “everything” means is the full detail on how application release/update procedures impacted the environment.

This is a common problem we see all too often. Extensive system logging efforts but no visibility into the change management processes. Without the complete picture you are spending your time essentially studying symptoms and trying to guess at the root event, rather than quickly identifying the root event and spending your time identifying a solution. Under the pressure of a significant outage you can’t underestimate the value of having the right tools at your disposal.

From my more detailed comment to Todd’s post:

Info you can’t get from normal system and application logs:
1. When did the application change?
2. What was changed? What are all of the code, data, and content assets related to that change?
3. Exactly what procedures were run to produce the change? Who ran the commands? What variables/inputs did the procedures use?
4. What nodes did those procedures touch?
5. What commands can I run to immediately put everything back into a last known good state? (often through a “roll-forward” rather than a true “roll-back” procedure)

The common perception is that it just isn’t possible or practical to collect this kind of data in an automated and authoritative manner. It is, but it depends on the correct choice of build, deployment, and configuration management tooling.

Greenies to drive ITIL adoption in the datacenter? Or, does dev2ops need better organizational alignment?

Greenies to drive ITIL adoption in the datacenter? Or, does dev2ops need better organizational alignment?

2

Alex Honor / 

CMCrossroads news reports: Datacentres at risk from poor governance . Steven Yellen at Aperature Research Institute finds that many datacenter managers admit to using 3 to 5 disjoint configuration management systems and confess to poor configuration management. Yellen then goes on to say that increased ITIL adoption will mitigate these risks and that initiatives towards greener datacenters will encourage ITIL adoption:

…the environmental impact of the datacentre is going to encourage datacentre managers to look at ITIL.

The push to go green, whether it be decommissioning old equipment to analysing power consumption, will push organisations to have an added level of detail around change management processes, and look more closely at ITIL.

Really? So the drive to greener and denser datacenters promote ITIL presumably due to their greater complexity?

I believe the most significant risk to ITIL adoption was summed up here:

He [Yellen] pointed to an organisational divide between the IT department and the datacentre manager, with potentially damaging consequences.

I would like to know what kind of data was stored in those 3 to 5 configuration information systems. Was there any pattern to the kinds of data that lead to the datacenter managers to maintain it separately?

More on nobody caring about ITIL…

More on nobody caring about ITIL…

2

Damon Edwards / 

CMDB is probably the most tangible “thing” you can pull out of ITIL… but can anyone actually name one organization that’s actually successfully established an ITIL prescribed CMDB that actually does something meaningful across the organization?

Edit (8/31): Perhaps I was a bit too glib with my original post. What I was really looking for is pointers to empirical evidence of where a “full” CMDB implementation has provided tangible benefit from a business point of view. This could either be in the form of substantial cost savings or increased revenue production. (The business case for why you would want to attempt a CMDB is understandable… but where are the actual bottom line benefits?)

ITIL: do people in the trenches care?

ITIL: do people in the trenches care?

3

Alex Honor / 

Seems like people that dedicate themselves to making tools that help scale up and improve precision of system configuration don’t find much value in ITIL and CMDB. A recent post on the Puppet blog,
The CMDB is a Consultant’s Myth gives a scathing commentary:

“I’ve long thought that the CMDB is just a bunch of crap. What is the (usually ‘the’, not ‘a’) CMDB? Well, it stands for ‘configuration management database’, but as far as I can tell the term is entirely meaningless.”

I think dev2ops folk would benefit from some formalized practices that help promote coordination. But so far, it hasn’t translated to practical tools nor profound adoption.

Page 10 of 10First678910