View All Videos

Archive for the ‘Patterns’ Category

Deployment management design patterns for DevOps

Deployment management design patterns for DevOps

22

Alex Honor / 

If you are an application developer you are probably accustomed to drawing from established design patterns. A system of design pattern can play the role of a playbook offering solutions based on combining complimentary approaches. Awareness of design anti-patterns can also be helpful in avoiding future problems arising from typical pitfalls. Ideally, design patterns can be composed together to form new solutions. Patterns can also provide an effective vocabulary for architects, developers and administrators to discuss problems and weigh possible solutions.

It’s a topic I have discussed before, but what happens once the application code is completed and must run in integrated operational environments? For companies that run their business over the web, the act of deploying, configuring, and operating applications is arguably as important as writing the software itself. If an organization cannot efficiently and reliably deploy and operate the software, it won’t matter how good the application software is.

But where are the design patterns embodying best practices for managing software operations? Where is the catalog of design patterns for managing software deployments? What is needed is a set of design patterns for managing the operation of a software system in the large. Design patterns like these would be useful to those that automate any of these tasks and will facilitate those tools developers who have adopted the “infrastructure as code” philosophy.

So what are typical design problems in the world of software operation?

The challenges faced by software operations groups include:

  • Application deployments are complex: they are based on technologies from different vendors, are spread out over numerous machines in multiple environments, use different architectures, arranged in different topologies.
  • Management interfaces are inconsistent: every application component and supporting piece of infrastructure has a different way of being managed. This includes both how components are controlled and how they are configured.
  • Administrative management is hard to scale: As the layers of software components increase, so does the difficulty to coordinate actions across them. This is especially difficult when the same application can be setup to run in a minimal footprint while another can be designed to support massive load.
  • Infrastructure size differences: Software deployments must run in different sized environments. Infrastructure used for early integration testing is smaller than those supporting production. Infrastructure based on virtualization platforms also introduces the possibility of environments that can be re-scaled based on capacity needs.

Facing these challenges first hand, I have evolved a set of deployment management design patterns using a “divide and conquer” strategy. This strategy helps identify minimal domain-specific solutions (i.e., the patterns) and how to combine them in different contexts (i.e., using the patterns systematically). The set of design patterns also include anti-patterns. I call the system of design patterns “PAGODA”. The name is really not important but as an acronym it can mean:

  • PAtterns GOod-for Deployment Administration
  • PAckaGe-Oriented Deployment Administration
  • Patterns for Application and General Operation for Deployment Administrators
  • Patterns for Applications, Operations, and Deployment Administration

Pagoda as an acronym might be a bit of a stretch but the image of a pagoda just strikes me a as a picture of how the set of patterns can be combined to form a layered structure.

 

Here is a diagram of the set of design patterns arranged by how they interrelate.

 

The diagram style is inspired by a great reference book, Release It. You can see the anti patterns are colored red while the design patterns that mitigate them are in green.

Here is a brief description of each design pattern:

Pattern Description Mitigates Alternative names
Command Dispatcher A mechanism used to lookup and execute logically organized named procedures within a data context permitting environment abstraction within the implementations. Too Many Tools Command Framework
Lifecycle A formalized series of operational stages through which resources comprising application software systems must pass. Control Hairball Alternative names
Orchestrator Encapsulates a multi-step activity that spans a set of administrative steps and or other process workflows. Control Hairball, Too Many Cooks Process Workflow, Control Mediator,
Composable Service A set of independent deployments that can assembled together to support new patterns of integrated software systems. Monolithic Environment Composable Deployments
Adaptive Deployment Practice of using an environment-independent abstraction along with a set of template-based automation, that customizes software and configuration at deployment time. Control Hairball, Configuration Bird Nest, Unmet Integration Environment Adaption
Code-Data Split Practice of separating the executable files (the product) away from the environment-specific deployment files, such as configuration and data files that facilitates product upgrade and co-resident deployments. Service Monolith Software-Instance Split
Packaged Artifact A structured archive of files used for distributing any software release during the deployment process. Adhoc Release Alternative names

The anti-patterns might be more interesting since they represent practices that have definite disadvantages:

Anti-Pattern Description Mitigates Alternative names
Too Many Tools Each technology and process activity needs its own tool, resulting in a multitude of syntaxes and semantics that must each be understood by the operator, and makes automation across them difficult to achieve. Command Dispatcher Tool Mishmash, Heterogeneous interfaces
Too Many Cooks A common infrastructure must be maintained by various disciplines but each use their own tools to affect change increasing chances for conflicts and overall negative effects. Control Mediator Unmediated Action
Control Hairball A process that spans activities that occur across various tools and locations in the network, is implemented in a single piece of code for convenience but turns out to be very inflexible, opaque and hard to maintain and modify. Control Mediator, Adaptive Deployment, Workflow  
Configuration Bird Nest A network of circuitous indirections used to manage configuration and seem to intertwine like a labyrinth of straw in a bird nest. People often construct a bird nest in order to provide a consistent location for an external dependency. Environment Adaptation  
Service Monolith Complex integrated software systems end up being maintained as a single opaque mass with no-one understanding entirely how it was put together, or what elements it is comprised, and how they interact. Code-Data Split, Composable Service House Of Cards, Monolithic Environment
Adhoc Release The lack of standard practice and distribution mechanisms for releasing application changes. Packaged Artifact  

 

Of course, this isn’t the absolute set of deployment management patterns. No doubt you might have discovered and developed your own. It is useful to identify and catalog them so they can be shared with others that will face scenarios already examined and resolved. Perhaps this set offered here will spurn a greater effort.

 

 

“Stability anti-patterns” highlight importance of tracking non-functional requirements

“Stability anti-patterns” highlight importance of tracking non-functional requirements

9

Damon Edwards / 

Michael Nygard, author of the influential Release It!, delivered a fantastic speech at QCon London where he dives into the various flavors of “stability anti-patterns” that can (and probably will) plague your web operations.

 

The explicit lessons alone are worth watching his presentation. However, there is also a more subtle lesson delivered over the course of the speech. Whether it was intentional or not, Michael illustrates the importance (and difficulty) of tracking non-functional requirements across the application lifecycle.

 

Some of the knowledge of where your next failure could come from lives in Development. Some of it lives in Operations. Sometimes it even lives in the well-intentioned plans of your marketing department.

What are you doing about sharing and tracking that knowledge? If you are like most organizations, you are probably relying on tribal knowledge to avoid these pitfalls. Development handles what they know about, Operations handles what they know about, and everyone crosses their fingers and hopes it all works out. Unlike the well understood processes and tooling used to track business requirements, non-functional requirements all too often fly under the radar or are afterthoughts to be handled during production deployment.

CodeData Split: New OMC Design Pattern

CodeData Split: New OMC Design Pattern

21

Alex Honor / 

One everyday source of confusion seen during the dev to ops handoff is the necessary modification of files during the installation and setup during deployment. Typically, there is one or more modifications to configuration files that enable the software to work in the context of the target environment. The process can be greatly improved if these files were split away from the executable software code itself. I describe this best practice as the CodeData Split:

Separate executable software files from environment specific deployment ones, identifying files that change during installation or operation, thereby facilitating cleaner upgrade process, and the ability to manage deployments independently and separate from each other.

The concept of splitting code from data in applications is nothing new and examples can be found ranging from OS filesystem structure and application servers. The pattern emerged in those places because the operation of the software was a clear requirement so this led to the data files (both configuration and variable data files) to have their own places. This best practice should be followed generally for any application. Doing so will avoid problems like the Service Monolith

Service Monolith: Proposed OMC Anti-Pattern #2

Service Monolith: Proposed OMC Anti-Pattern #2

10

Alex Honor / 

Most software services developed and operated these days are increasingly more heterogeneous and distributed, and therefore, complex and hard to setup and manage. On top of all this, the rate of change seems to only go up. These trends can lead to an anti-pattern I’ve recently documented at the Open Management Consortium called “Service Monolith”:

Complex integrated software systems end up being maintained as a single opaque mass with no-one understanding entirely how it was put together, or of what elements it is comprised, and how they interact.

I describe some common rationale for using this anti-pattern and some of its consequences. There are positive design patterns to avoid or mitigate the anti-pattern, too. I find one interesting solution that seems to be becoming more ubiquitous and sort of side steps the deeper problem: virtualization. Virtualization is one strategy for dealing with producing these complicated systems. Using this strategy you get the whole conglomeration working on one host, cobbling the pieces together using the preferred recipe, then “freeze dry” it as an operating system image that you can instantiate on another host. With the image in hand an organization can “stamp it out” as needed. This technique seems to work for small scale deployments but I haven’t seen the approach work for maintaining large scale environments. Is this the sign of another emerging anti-pattern?

Edit (Damon 1/31/08):
This is a great quote by Kris Buytaert on his first booting of a vmware instance:
“And thus we joined the era of transferring an unmanagable image that everyone will copy around wile slightly modifying things and never placing them in version control . hence ending up one day with something nobody knows how we got there.”

Configuration Bird Nest: Proposed OMC Anti-Pattern

Configuration Bird Nest: Proposed OMC Anti-Pattern

9

Alex Honor / 

Sometimes it is more interesting (and entertaining) to talk about things not to do. Let’s face it, we have more experience doing things wrong and learning the hard way. For these reasons I decided to begin the OMC’s Design Pattern work with Anti-Patterns.

I had a clear “favorite” in mind when it came to writing the first anti-pattern. I call it Configuration Bird Nest:

A network of circuitous indirections used to manage configuration and seem to intertwine like a labyrinth of straw in a bird nest. People often construct a bird nest in order to provide a consistent location for an external dependency.

The bird nest metaphor seems so apropos as it conveys the idea that the nest cradles something important, typically something crucial to supporting an application. The pattern is so typical and manifests itself inside so many IT disciplines, I look forward to hearing about its many forms.

Where are the design patterns for software operations?

Where are the design patterns for software operations?

10

Alex Honor / 

In the world of software development, application developers are accustomed to drawing from the wealth of design patterns that address common programming problems, codify best practices, and establish proven reusable solutions. There are several well known design pattern repositories that catalog solutions into various categories from fundamental ones described by the GangOfFour to architecture specific ones like J2EE Patterns, even ones for social organization. An Anti-pattern is a pattern that tells how to go from a problem to a bad solution. Design patterns help avoid re-inventing solutions and when combined together can form the basis of a problem solving “play book.” When used effectively, design patterns become a common problem solving language and can lead to better written software.

But what happens after the code is written? For most organizations today, software operations – the acts of deploying, configuring, and operating software (and all of its related code and data artifacts) – is arguably as important as writing the software itself. If such an organization can’t efficiently and reliably operate the software, the quality of the software will not matter. But if one looks for design patterns that codify best practices for automating software operations, nothing turns up. Where is the catalog of design patterns that address the problems encountered when managing environments of software deployments and the overall life cycle of the business service?

Anyone that has managed software operations for different organizations, will recognize the same kinds of problems and will often re-invent solutions that were successful in the past. Others that work closer to the bleeding edge will encounter problems that other groups will face later. If these problems could be discussed in terms of design patterns (or failures as anti-patterns), solutions and best practices for managing software operations would be more consistent across organizations.

Here are two specific problem areas that everyone can identify with:

  • Packages: Depending on the application and infrastructure, one will find multiple package formats in use. Operating systems use their own (eg, .rpm, .deb, .pkg, .msi, etc) and so do software runtime environments (eg, java, .net). Each format has its own way (to greater or lesser extents) of being created, extracted, and described (including dependencies). These differences lead to multiple package silos and administrative gray areas (cumbersome handoffs between dev and admin groups). It would be preferable to have a common repository that can host any kind of package type, and a homogeneous interface to controlling their life cycle (creation, installation and removal).
  • Services: At a certain level, one can view applications as a set of interacting long running processes. Again, depending on the application architecture, these processes might be standalone unix-style daemons, or windows services. Each service has its own way of being started or stopped, as well as a procedure for checking its current runtime state. Often times, shutting down a service is not a simple matter of just invoking a single command. Things go wrong at shutdown requiring other logic to figure out the next course of action. Besides coping with these differences, the deployment process is also difficult because change of runtime state and software package installation is intertwined. Software operations would benefit from a body of design patterns that described proven strategies to managing runtime state and a common model for describing these states.

Here is a sampling of general recurring problems in the world of software operations:

  • Complex application deployments: Applications are based on technologies from different vendors, are spread out over numerous machines in multiple environments, and use different architectures
  • Inconsistent management interfaces: Every application component and supporting piece of infrastrucure has a different way of being managed. This includes both how components are controlled and how they are configured.
  • Hard to scale administrative management: As the layers of software components increase, so does the difficulty to coordinate actions across them. This is especially difficult when the same application can be setup to run in a minimal footprint while another can be designed to support massive load and redundancy.
  • Incoherent life cycles: Applications are typically multi-tiered, where each tier may be on its own development track, uses its own release paradigm and requisite tools.
  • Generally, these problems are found in combination which means coping with them on the whole is a difficult challenge.

What’s needed: Domain specific patterns for software operations

 

The body of existing design patterns can and should be used to analyze and solve some of the above problems. To make the design patterns more readily useful to software operations, we need a set of domain specific patterns. These patterns would be expressed in terms of concepts familiar to software operations groups (eg, package, service, process, node, etc) and would be geared to coping with typical problems they face (eg, various startup, shutdown strategies for services among many others). Ideally, these patterns can be composed into a system of patterns that help solve larger scale problems.

Developing patterns is a bit of an organic process but the most durable patterns are ones that have been proven over and over again in different contexts. The first step is to establish a repository to which various patterns can be contributed and a supporting forum where their merits can be discussed. Ultimately, the software operations community will find consensus about some of these patterns, thus establishing some common vocabulary and a basis for framework development.

External links:
PortlandPatternRepository
Hillside

Page 2 of 3123