View All Videos

Companies have plenty of monitoring, what they don’t have is control

50

Lee Thompson / 

I was honored to be asked to speak at DevOps Days in Manila and just got off stage. I was blown away when I found out over 400 people signed up to attend. Speaking gives me a chance to unload a bunch of baggage I’ve been carrying around years.

We all bring a lot of baggage with us into a job. The older you are, the more you bring. The first part of my career I did 10 years of real-time industrial control software design, implementation, and integration way way back before the web 1.0 days. Yes, I wrote the software for the furniture Homer Simpson sat in front of at the nuclear plant that was all sticky with donut crumbs…

I took that manufacturing background baggage to E*TRADE in ’96 where I ran into fellow dev2ops contributor Alex Honor who brought his Aimes Research Laboratory baggage of (at the time) massive compute infrastructure and mobile agents. We used to drink a bunch of coffee and try to figure out how this whole internet e-commerce thing needed to be put together. We’d get up crazy early at 4:30AM, listen to Miles, and watch the booming online world wake up and trade stocks and by 9:00AM have a game plan formulated to make it better.

My manufacturing background was always kicking in at those times looking for control points. Webserver hits per second, firewall MBits/sec, Auth success or fail per second, trades per second, quotes per second, service queue depths, and the dreaded position request response time. I was quite sure there was a correlation algorithm between these phenomena and I could figure it out if I had a few weeks that I didn’t have. I also knew that once I figured it out, the underlying hardware, software, network, and user demand would change radically throwing my math off. Controlling physical phenomena like oil, paper, and pharmaceutical products followed the math of physics. We didn’t have the math to predict operating system thread/process starvation and it took us years to figure out OS context switches per second has a huge kernel scaleability issue not often measured or written about.

One particularly busy morning in late ’96 Alex was watching our webserver, pointed at a measurement on the screen and said, “I think we’re gonna need another webserver”. With that, we also needed to figure out how to loadbalance webservers. As usual for the era, two webservers was a massive understatement. Within a year, there was more compute infrastructure at E*TRADE supporting the HTTPS web pages then the rest of the trading system and the trading system had been in place for 12 years by this time… Analytics of measurements (accompanied by jazz music) became an important part of our decision making.

 

Alex and I were also convinced in early ’97 that sound manufacturing principles used in the physical world made a ton of sense to apply to virtual online world of the internet. I’m still surprised the big control systems vendors like Honeywell and Emerson haven’t gotten into data center control. No matter, the DevOps community can make progress on it as its so complimentary to DevOps goals and its what the devops-toolchain project is all about.

Get a bunch of DevOps folks together and the topic of monitoring comes up every time. I always have to ask “Are you happy with it?” and the answer is always “no” (though I don’t think anyone at Etsy was there). When you drill into what’s wrong with their monitoring, you may find that most companies have plenty of monitoring, what they don’t have is control.

Say your app in production runs 100 logins/sec and you are getting nominally 3 username/password failures a second. While the load may go up and down, you learn that that the 3% ratio is nominal and in control. If the ratio increments higher, that may be emblematic of a script kiddie running a dictionary attack or the password hash database is offline or a application change making it harder for users to properly input their credentials. If it drops down, that may indicate a professional psyber criminal is running an automated attack and getting through the wire. Truman may or may not of said “if you want a new idea, read an old book”. In this case, you should be reading about “Statistical Process Control” or SPC. It was heavily used during WWII. With our login example, the ratio of success to failed login attempts would be “Control Charted” and the control chart would evaluate weather the control point was “in control” or “out of control” based on defined criteria like standard deviation thresholds.

Measurement itself is a very low level construct providing the raw material for the control goal. You have to go through several more toolchain layers before you get to the automation you are looking for. We hit upon this concept in our talk at Velocity in 2010…

Manufacturing has come a long long way since WWII. Toyota built significantly on SPC methodologies that eventually became the development of “Lean Manufacturing”; a big part of the reason Toyota became the worlds largest automobile manufacturer in 2008. A key part of lean is Value Stream Mapping which is “used to analyze and design the flow of materials and information required to bring a product or service to a consumer” (wikipedia).

Value Stream Mapping a typical online business through marketing, product, development, qa, and operations flows minimally will help effectively communicate rolls, responsibilities, and work flows through your org. More typically it becomes a tool to get to a “future state” which has eliminated waste and increase effectiveness of the org, even when nothing physical was “manufactured”. I find agile development, devops, and continuous deployment goals all support lean manufacturing thinking. My personal take is that ITIL has similar goals, but is more of process over people approach instead of a people over process approach and it’s utility will be dependent on the organizations management structure and culture. I prefer people over process, but I do reference ITIL every time I find a rough or wasteful organizational process for ideas on recommending a future state.

I was lucky enough to catch up with Alex, Anthony, and Damon over dinner and we were talking big about DevOps and Lean. Anthony mentioned that “we use value stream mapping in all of our DevOps engagements to make sure we are solving the right problem”. That really floored me on a few levels. First off, it takes Alex’s DevOps Design Patterns and DevOps Anti-Patterns to the next level similar to SPC to Lean adding a formalism to the DevOps implementation approach. It also adds a self correcting aspect to a companies investment into DevOps optimizations. I’ve spoken with many companies who made huge investments in converting to Agile development without any measurable uptick in product deployment rates. While these orgs haven’t reverted back to a waterfall approach as they like the iterative and collaborative approach, they hit the DevOps gap head on.

“We use Value-Stream Mapping in all of our DevOps engagements to make sure we are solving the right problem”
                                                 -Anthony Shortland (DTO Solutions)

Practicers of Lean Manufacturing see this all the time. Eliminating one bottleneck just flows downstream to the next bottleneck. To expect greater production rates, you have to look at the value stream in its entirety. If developers were producing motors instead of software functions, a value stream manager would see huge inventory build up of the motors which produce no value to the customer and identify the overproduction as waste. Development is a big part of the value stream and making that more efficient is a really good idea. But a measurement of the release backlog growing is seldom measured or managed. If you treat your business as a Digital Information Manufacturing plant and manage it appropriately to that goal, you can avoid the frequent mistake Anthony and other Lean practitioners are talking about where you solve a huge problem without benefiting the business or the customer.

To sum up, DevOps inspired technology can learn quite a bit from Lean Manufacturing and Value Stream Mapping. This DevOps stuff is really hard and you’ll need to leverage as much as possible. Always remember that “Good programmers are lazy” and its good when you apply established tools and techniques. If you don’t think your working in a Digital Information Manufacturing plant, I bet your CEO does.

50 Responses

  1. Elmer.Rivera says:

    Hi Lee,

    It's my/our pleasure to listen to your keynote back there. We always learn a lot from it. Hope to hear from you soon. Btw, nice meeting you 🙂

    Regards

  2. Jordan Curzon says:

    Sounds like http://en.wikipedia.org/wiki/Theory_of_constraints. We had to read Eli Goldratt's "Critical Chain" in college and that really shaped my my views of project planning.

  3. John Allspaw says:

    Ha! I'd say that Etsy's happiness with monitoring exists on a spectrum, somewhere between miserable and ecstatic. 🙂

  4. Sophia Dizon says:

    Hi Sir Lee, it's my pleasure to meet you during DevCon Summit 2011 days (december 2, 2011), i am so honored that you told me that your going to hire me, i was so lucky, I am from City of San Fernando Pampanga, Philippines… I wish i could be like you someday.. Thank you so much for your kindness.

  5. I think many have learned from your DevOps Days. I'm very unfortunate to have missed it.

  6. Great post Lee, and welcome back to the fold! I'm just starting to wrestle with applying Lean to what we do, so this is a great topic. If y'all wanted to share a sample "real" value stream mapping I bet people would find it insanely interesting (I know I would…)

  7. quel fantastique post ça a été. Dans aucun cas vu ce genre associé à poste utile. Vous trouverez quelque chose d’utile dans mon site:

  8. robert says:

    If you delay even a brace weeks best you’ll acquisition that you can get up to 50% off of the aboriginal price, which is a abundant accord and can save you hundreds of dollars at http://www.lucybags.co.uk

  9. robert says:

    Whether you use it for traveling to the bank or the malls, the ladies backpack at http://www.gailunhandbags.co.uk will consistently be alleged an “accessory” of your accustomed outfits.

  10. newyear2017 says:

    New Year wishes for boyfriend are the best form of expressions of love and … to see me happy and smiling and I know 2017 https://happynewyear2017.tips/new-year-2017-wishes-for-boyfriend/ will become more beautifu

  11. canada post tracking
    canada post courier tracking
    tracking canada post

  12. ewrkje io hof soe oisdjoiodwerijho jwoejhfo dogoicv osoegfoii sedogn oweng qofosdng wnfgoin f oen r

  13. Thanks it helped me in ranking my christmas blog… http://merrychristmas2016wish.net

  14. Jacke Armon says:

    Best New year 2017 Wishes sites :http://www.happynewyearmessage.org/
    | http://www.newyearpictures.org/ You may visit here.

  15. every time when i visit your site i found uniqe contect which i love to read and share, plus one for you

  16. The information you have posted above is very informative, it helps me a lot thanks.

  17. New Year is the best event for everyone so celebrate this New Year 2017 with us and send some wonderful and awesome New Year Wishes, Greeting, Quotes and many more for your Love ones.

  18. idm android says:

    thanks for sharing this kind of great information with us really loved it man.

  19. thanks for sharing this amazing and useful information with us. and your blog is very nice and I really appreciate. Thanks again.

  20. Raees Movie Download, Raees full movie download, Raees full movie free download 720p,
    Raees torrent download, raees movie online,

  21. To sharing your love to your Important person in this special day, you need to have the clear concept about its like How to Celebrate Valentines Day, Valentines Day requirements and more other. As our valuable visitor, we are Very Happy to inform you that here you will get all of your Answer according to the Happy Valentines Day which will help you to celebrate the day with traditional activities. Happy Valentines Day 2017

  22. SSC Result 2017 BY SMS is The Topic of This Post. So, Through This Article, We Discus About The System of Getting SSC Result 2017 by Mobile SMS. In The Modern Technology Age, it is the new and Popular way to get SSC Exam Result 2017 by Send SMS SSC Result 2017 by SMS

  23. thanks for sharing this kind of great information with us really loved it man.

  24. track colombia post
    track postal colombia
    colombia post tracking

  25. thanks for sharing this awesome information man really appreciate it. hipstore pokemon go ios and tubemate for ios apk

  26. Amit says:

    thanks, man the content is damn good and sexy at the same time. snaptube IOS and terrarim TV for IOS

  27. itubeappp download says:

    iTube is an incredible application that lets you download and play your preferred videos, audio and music albums on your Android device
    itube apk

  28. kk says:

    I like this very much replique hublot

  29. rajesh says:

    Apple, the giant software company, had brought the most expensive devices in the world – iPhone. Everyone wishes to possess an iPhone, and for some people, it is like a dream to purchase iPhone.
    for more information check this :- iphone 8 rumors

  30. parcel tracking says:

    Welcome to Parcel Tracking Track Parcel Couriers & Parcels By Entering the Parcel Tracking Number Track Parcel Courier Online
    Enter Your Package Number :- bluedart tracking

  31. Great post! I am actually getting ready to across this information, is very helpful my friend. Also great blog here with all of the valuable information you have. Keep up the good work you are doing here.
    9CBSE NEET 2017

  32. juegos friv says:

    Very good article. I just found your blog and wanted to say that I really enjoyed reading your recent articles. Good job!

  33. Easter 2017 says:

    Good Friday images After I initially commented I seem to have clicked on the -Notify me when new comments are added- checkbox and now every time a comment is added I receive 4 emails with the exact same comment. There has to be an easy method you are able to remove me from that service? Kudos! Happy Easter Sunday

  34. Hope all are coming to Easter Bunny Party Tonight!

  35. NEET PG 2017 says:

    Great post! I am actually getting ready to across this information, is very helpful my friend. Also great blog here with all of the valuable information you have. Keep up the good work you are doing here.

  36. Great post! I am actually getting ready to across this information, is very helpful my friend. Also great blog here with all of the valuable information you have

  37. Great post! I am actually getting ready to across this information, is very helpful my friend. Also great blog here with all of the valuable information you have. Keep up the good work you are doing here

  38. Happy Easter says:

    A blog is all about Happy Easter

Post a new comment