10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
Alex Honor /
The Flickr guys, John Allspaw and Paul Hammond gave an entertaining and validating presentation at OReilly Velocity (slides).
The talk began with a brief description about how Flickr’s technology group enabled the business to deliver features and update their online service frequently (10+ deploys per day) but it really turned out to be a success story about how Dev and Ops can align and work together without falling into age-old traditional cross organizational conflicts.
Here’s a few (paraphrased) quotes:
Ops’ job is to enable the business. (not to keep the site stable and fast)
The business requires change… so lower the risk of change through tools and culture. Make more frequent, but smaller changes … through push button (and hands off) deployment tools.
Plan fire drills to make sure everyone (junior guys included) knows how to solve production problems because failure will happen.
Ops who think like devs. Devs who think like ops
The talk really boiled down to two ingredients to enable the close dev and operations collaboration (tools + culture):
Tools
1. Automated infrastructure
2. Shared version control
3. One step build and deploy
4. Feature flags
5. Shared metrics
6. IRC and IM robots
Culture
1. Respect
2. Trust
3. Healthy attitude about failure
4. Avoiding blame
I think for some, the real validation was hearing that it’s just as much making a cultural shift as it is a mixture of choosing and using the right kind of tools. Anybody who has worked in the trenches will realize that of course.
It sounds so easy. But it's really hard to do. The cultural shift is foundational to get everyone willing to do the hard word. It's the automate-everything part that's quite hard (and essential) to do.