Peanut butter in my chocolate? Convergence vs. Ad hoc Control

2

Alex Honor / February 16th, 2011

Recently, I’ve been in several conversations about how to reconcile two links in the management tool chain, configuration management and ad hoc control. The conversation usually revolves around conventional viewpoints about the nature and roles of these tools:

Configuration management wants to achieve and enforce a system state. The normal mode of operation is for the configuration management tool to run autonomously on each node, applying the rules of that node’s specification. In the large, the set of hosts eventually converge on the declarations made in the specification.

Ad hoc control tools are generally used to for special problems or on-demand management activities. Their use is often arbitrary and driven by management need and not on a regular periodic basis. Problem diagnosis, information gathering, deployment changes, emergency fixes across many hosts are example cases where ad hoc control tools are often used.

If these tools are used independently, one or more scenarios like the following can arise:

 

• Changes made by ad hoc tools are reverted or partly undone by the configuration management tool
• Configuration management tool is turned off and only run directly when the rules are needed to fire
• Administration code is reinvented in each tool but their implementations focus on different concerns
• One tool gets thrown out or its role and scope are severely diminished in favor of the other

 

These tools can interfere with each other and to some they can appear mutually exclusive.

Clearly, both management modes are important. On the one hand, it’s beneficial to have an ongoing compliance and enforcement loop driven by specification. On the other hand, issues of the day require near instantaneous action. Those actions may not become routine and deemed part of the desired permanent state maintained by configuration management.

Planning out how these two kinds of tools can interoperate begins with a discussion of boundaries and lines of responsibilities. It also helps interoperation when both tools share a common model and points of control.

Here are a few areas where I have seen interoperation facilitated:

• Centralize the control loop: Initiate the control loop from a central point of administration. Configuration management loses some autonomy but the pull model is preserved which is important for scalable execution. 
• Split Install from Startup: Separate the startup action from the installation procedures in the provisioning process. This defers startup to a centrally orchestrated authority. This is vital for multi-tier applications.
• System control interface: Define a set of system commands that control service state on each host. Configuration management and ad hoc tools can avoid reinventing the wheel. This also gives a point of collaboration between teams.
• Dependency driven packaging: Use packages to install files on node. Use package dependencies to install files in an order. System package utilities provide other benefits like verification and repositories.