View All Videos

Peanut butter in my chocolate? Convergence vs. Ad hoc Control


Alex Honor / 

Recently, I’ve been in several conversations about how to reconcile two links in the management tool chain, configuration management and ad hoc control. The conversation usually revolves around conventional viewpoints about the nature and roles of these tools:

Configuration management wants to achieve and enforce a system state. The normal mode of operation is for the configuration management tool to run autonomously on each node, applying the rules of that node’s specification. In the large, the set of hosts eventually converge on the declarations made in the specification.

Ad hoc control tools are generally used to for special problems or on-demand management activities. Their use is often arbitrary and driven by management need and not on a regular periodic basis. Problem diagnosis, information gathering, deployment changes, emergency fixes across many hosts are example cases where ad hoc control tools are often used.

If these tools are used independently, one or more scenarios like the following can arise:


  • Changes made by ad hoc tools are reverted or partly undone by the configuration management tool
  • Configuration management tool is turned off and only run directly when the rules are needed to fire
  • Administration code is reinvented in each tool but their implementations focus on different concerns
  • One tool gets thrown out or its role and scope are severely diminished in favor of the other


These tools can interfere with each other and to some they can appear mutually exclusive.

Clearly, both management modes are important. On the one hand, it’s beneficial to have an ongoing compliance and enforcement loop driven by specification. On the other hand, issues of the day require near instantaneous action. Those actions may not become routine and deemed part of the desired permanent state maintained by configuration management.

Planning out how these two kinds of tools can interoperate begins with a discussion of boundaries and lines of responsibilities. It also helps interoperation when both tools share a common model and points of control.

Here are a few areas where I have seen interoperation facilitated:

  • Centralize the control loop: Initiate the control loop from a central point of administration. Configuration management loses some autonomy but the pull model is preserved which is important for scalable execution. 
  • Split Install from Startup: Separate the startup action from the installation procedures in the provisioning process. This defers startup to a centrally orchestrated authority. This is vital for multi-tier applications.
  • System control interface: Define a set of system commands that control service state on each host. Configuration management and ad hoc tools can avoid reinventing the wheel. This also gives a point of collaboration between teams.
  • Dependency driven packaging: Use packages to install files on node. Use package dependencies to install files in an order. System package utilities provide other benefits like verification and repositories.



44 Responses

  1. Alex,

    This is one of the reasons I'm creating Noah. While Chef and Puppet both can handle application level configuration perfectly well (for instance see Chef's data-driven configuration stuff), the requirement to perform a new Chef or Puppet run is pretty much a pain in the butt just to change a logging level or update a list of memcache nodes.

    I would wager that 95% of users of either tools run them in an ad-hoc fashion because of concerns about changes applied to the CM repo going out "before they're tested".

    Look at Wealthfront (formerly KaChing!) and how they use ZooKeeper (my inspiration for Noah) as a coordination tool for the entire infrastructure.

    Noah/Zookeeper/whatever can happily coexist with existing CM tools as long as the system of record is determined to be the CM tool. Bootstrap a new memcached node via Chef? The base config file for your application gets updated but memcached startup (or some CM task) also registers the new node with Noah so that existing and running nodes can be instantly aware and reconfigure themselves.

    Excellent post and something that I've been meaning to write about myself!

  2. Alex Honor says:


    I agree there is a missing link the chain. Having something like Noah/Zookeeper that can be used to store and access both configuration and operational state is extremely useful.


  3. Testopac provides environmental and testing services to meet your needs.package testing

  4. Excellent in terms of males and females proceeding once more directly below &happy to discover a satisfying write-up. Could very well help quite a buddy’s to discover forms articles. Go on executing the actual stop good perform!

  5. Truth be told there might actually be various main features in addition you should be honestly modern-day in addition to learning. We now have might honestly generally watch as well as watch the write-up in fact you may have discover wonderful total joy hiring receiving normally normally persons great.

  6. Kampfsport says:

    Currently was amazed. You may be in essence informed along with clever. Everyone posted a problem which will men and women may track down and therefore made selected challenging made for people. Currently was reducing this kind of suitable for difficult enjoy some great benefits of.

  7. Concerning publish telly exhibit. Many specifics by way of folks at times will be amazing.

  8. Become grateful relating to picking out the moment go over these kinds of, I need to say when i perception incredibly over it and in addition enjoy knowing additionally in this supposed style. Whenever probable, as you obtain information, does indeed just one thoughts altering your blog together with more information? It’s very a good solution pertaining to professionally.

  9. Kavin Marcoo says:

    I just want to tell you that you are the best article writer. You have the shared great information. It’s really important in my work.
    Rubber gym flooring and tiles from

  10. Simply built you might look at along with an abundance of admiration related to each of the excellent implement any individual typically produce. Care for these kinds of robust implement.

  11. Offered in addition to people checklist the most popular world-wide-web report Fantastic.. I like your own matter on the web page at present, plus states that lots of the content articles by itself web page.

  12. achom says:

    Numerous people commonly using such a using ought to enable up-to-date substantial thumbs far more strategy process strategy means strategy continuously in place in terms of excellent files you might have as well as caused by write-up. Working with are remaining expanding becoming every time they might special re-occurring on-line web site gained regarding a lot more fundamentally!

  13. Supplements can help your dreams of having your own product come alive. Intermountain Supplements offers a wide variety of custom formulation services including liquids, capsules, sprays, powders, and more.

  14. When i have a look at this site often and I adore evaluating web sites submitted in this article. We have this site formerly additional within my personal web browser. Love exclusively for this specific publish.

  15. 192.168.l.l says:

    Its a good posts..keep posting and update the information.selfie stick….

  16. really an awesome job you have done till now

  17. you should really checkout this

  18. thanks for sharing this info..

  19. Xender is among one of the most popular file sharing programs available on the Google play store, it has been already downloaded few millions of times and has an overall great standing among people that have attempted it so Far

  20. Wian Ruthers says:

    This is indeed a really nice and eye catchy headline that I have come across recently. I am pretty sure most of you would have just opened this up to see what the connection is btw peanut butter and IT tech. medical billing and coding companies

  21. Sex Contacts says:

    The precise a true very gratifying circulating really. Relation created for communicating around this kind of gratifying circulating close to.

  22. We all thus need to get pleasure from the thought to article topics some people build on the other hand extending getting well-versed a very write-up. Right now was getting searching for to have the most beneficial goal over the total exact possessing some people inside the medial place trusted.

  23. Steven Smith says:

    Not much people would agree on this but I can see a future when people will be regretting saying we did have a change with this back then and we failed to support it. I am sure such a day would come if it is avoided now. kosher restaurants la

  24. There exists more amount of places wherever people would want to travel with themselves and loved ones everywhere in the year. We should understand that you have more amount of travel businesses is operating across the world.

  25. You can find more wide variety of places when people would wish to travel with themselves and close family on year. We really need to understand there presently exists more wide variety of travel specialists is operating globally.

  26. There does exist more availablility of places the place people would love to travel with their loved ones and relations in your year. We will need to understand there presently exist more availablility of travel services is operating around the world.

  27. There is always more lots of places at which people choose to travel with themselves and relatives on the year. We want to understand that there is more lots of travel providers is operating internationally.

  28. Latest movies watch here,really awesome site

  29. This is really the top notch review. Should make respected you will identify with absolute minimum troublesome substance material web powers There are presently obtained bought check. Offering value identifying with spot such a convey.

  30. thanks for share best best video appfacetime is develop by apple.

  31. When i has been extremely impressed having all your content. All this provides extra information personally. Hopefully your site more lucrative.

  32. We are just considering which a significant part of the time could which consistently doesn’t don’t succeed anyone paying little mind to this specific web site page.

  33. Basic printed material. The force is frequently suitable held see to locate this sort of specifics. have an exceptional time in this article identifying with showcasing people which routinely could profitable exploration material.

  34. Cinema Box app is the latest version of PlayBox HD app which works on any device, nice application

  35. mp3songspk says:

    great post sir keepp

  36. nikky says:

    Railway Recruitment Control Board is a government organisation in India. It was set up in 1998 in the Ministry of Railways (Railway Board), New Delhi. RRB Result 2016

  37. facetime for android : The human race is known to be greedy and wanting everything that the other person has. This era is the era of technology. Everything that we see is modernizing.
    facetime for pc

Post a new comment