View All Videos

Archive for the ‘DevOps’ Category

Videos from DevOps Day 2010 panels!

3

Damon Edwards / 

 

InfoQ.com has posted the videos they recorded at DevOps Day USA 2010. You can watch six of the seven panels now on the InfoQ.com site. There was a production problem with the seventh panel (“DevOps outside of WebOps”) that, if it can be fixed, will be posted as well. InfoQ decided that the lightening talks didn’t fit into their format so they have sent my co-organizer, Andrew Shafer the raw video and he’s going to look into posting them himself.

You can also download audio only versions (.mp3)

Here are the links to the 6 panels…

Your mileage may vary: Experiences and lessons learned facing DevOps problems in the IT trenches (even if they weren’t calling it DevOps!). The good, the bad, the surprises, and ideas for the future.
Stefan Apitz – LinkedIn
Ernest Muller – National Instruments
Dan Nemec – SilverPop
Burzin Engineer – Shopzilla
Kevin Rae – PowerReviews
moderator: Andrew Shafer
http://www.infoq.com/presentations/your-mileage-may-vary

 

Infrastructure as code: Automation is essential to DevOps. The infrastructure as code concept drives many of today’s cutting edge automaton techniques. What is it all about? Where are its limitations?
Theo Schlossnagle – OmniTI
Luke Kanies – Puppet Labs
Adam Jacob – Opscode
Erik Troan – rPath
moderator: Patrick Debois

Changing culture to enable DevOps: Changing tools is easy when compared to changing people and processes. How can we cultivate an organization’s culture to identify and solve DevOps problems?
John Allspaw – Etsy
Lee Thompson – DTO Solutions
Israel Gat – The Agile Executive
Lloyd Taylor – Netelder Associates
moderator: Andrew Shafer
http://www.infoq.com/presentations/changing-culture-to-enable-DevOps

Does the Cloud needs DevOps? Does DevOps need the Cloud?: Examining the role that cloud technologies can play in solving DevOps problems and the role that DevOps solutions can play in getting the most out of cloud technologies.
James Urquhart – Cisco
Adrian Cole – Jclouds
Justin Dean – Shopzilla
Joe Arnold – Cloudscaling
moderator: John Willis
http://www.infoq.com/presentations/does-Cloud-need-DevOps

DevOps requires visibility: monitoring, testing, and performance: Examining the (often overlooked) role of monitoring and testing techniques in solving DevOps problems.
Jyoti Bansal – AppDynamics
Gareth Bowles – Appscio
Matt Ray – Zenoss
Eishay Smith – kaChing
Javier Soltero – SpringSource
moderator: Damon Edwards
http://www.infoq.com/presentations/DevOps-requires-visibility

Making the business case: We know that solving DevOps problems improves your business operations and improves the bottom line, but how do you do you explain that to your CEO or CFO? How do you get the executives to buy in and invest in DevOps solutions?
Kurt Milne – IT Process Institute
Jay Lyman – The 451 Group
Rolf Andrew Russell – ThoughtWorks
Jody Mulkey – Shopzilla
moderator: Damon Edwards
http://www.infoq.com/presentations/Making-the-business-case

 

EDIT: The recording for seventh panel was rescued from technical oblivion and is now live!…

DevOps outside of Web Operations: Much of the public discussion about DevOps focuses on Web Operations. This panel is about taking the lessons of DevOps to other types of IT.
Adam Fletcher – ITA Software
Gene Kim – Tripwire
Michael Stahnke –
James Turnbull – Puppet Labs
moderator: Patrick Debois
http://www.infoq.com/presentations/DevOps-outside-Web-Operations

 

 

DevOps (live) at OSCON

DevOps (live) at OSCON

Dev2ops / 

Early reports from OSCON are that DevOps is a topic of much discussion. My fellow dev2ops.org contributor Alex Honor and I are headed to Portland this morning to give DevOps related talks at OSCON. If you are there Wednesday or Thursday, please come by and say hello!

 

Wednesday (7/21) 1:40pm in room Portland 251 is Alex’s presentation…
Open Source Tool Chains for Cloud Computing 

Thursday (7/22) 10:40am in room D135 is Damon’s presentation… 
The IT Philharmonic: How Out of Tune Are Your Operations? 

 

Both talks feature lots of new content (even though the titles and outdated descriptions on the OSCON site are similar to our Velocity talks) 

 

 

Velocity and DevOps Day US was a success, now what?

Velocity and DevOps Day US was a success, now what?

2

Damon Edwards / 

 

Now that DevOps Day US and Velocity are in the past, where can you go for in person DevOps discussions?

Below are the DevOps Meetups that I know about:

In the San Francisco Bay area…
Tonight (!) Tuesday July 6 is the Silicon Valley DevOps Meetup

In the Los Angeles area…
Wednesday July 14 is the SoCal DevOps Meetup

In the Boston area…
Tonight (!) Tuesday July 6 is the Boston DevOps Meetup 

Across the pond in the London area…
The London DevOps Group meets regularly and has a great site.

And of course don’t forget about the second installment of DevOps Days 2010 Europe October 15 – 16 in Hamburg Germany.

If you are looking for some online discussion, you can check out the following lists/groups:
DevOps Toolchain Google Group
Agile Systems Administration Google Group
DevOps LinkedIn Group
DevOps Google Group

Don’t forget #devops on Twitter.

 

and don’t forget to subscribe to the DevOps Cafe podcast!

 

Bonus: Here’s an interesting video interview with Patrick Debois (instigator of the DevOps Days conferences) by Daniel Cukier at the DevOps Day 2010 after-party.

The (Comedic) Truth about DevOps

The (Comedic) Truth about DevOps

2

Damon Edwards / 

Many serious words have been written about DevOps, including my own “What is DevOps?” post. But sometimes a bit of comedy helps drive home some fundamental truth.

Behold Adam Jacob (CTO of Opscode): 

Note: this was part of Adam’s “Choose Your Own Adventure” talk at Velocity. Be sure to check out the other videos from his show talk. 2 drink minimum not required.

DevOps Cafe Podcast now available on iTunes!

Damon Edwards / 

John Willis (johnmwillis.com, @botchagalupe on Twitter, and VP of Services at Opscode) and I (@damonedwards on Twitter and President of DTO Solutions) have started a new podcast. We call it the DevOps Cafe. The name is a take on the popular Cloud Cafe series that John used to do.

Our primary goal for the DevOps Cafe podcast? Explore the emerging fields of DevOps and Agile Operations. While you couldn’t stop John and I from adding our own commentary and experiences if you tried, this will primarily be an interview driven show. We are going to seek out the people in the trenches who are pioneering these new trends and bring them directly to you on a regular basis.

Our secondary goal? To have some fun.

The first two episodes are now available:

Episode 1 – Guest: Lindsay Holmwood (DevOps Days Down Under organizer; Cucumber-Nagios and Flapjack developer)

Episode 2 – Guest: John Allspaw (VP of Technical Operations at Etsy; Frequent public speaker and author)

 

 

http://devopscafe.org

 

 

Q&A: Continuous Deployment is a reality at kaChing

Q&A: Continuous Deployment is a reality at kaChing

5

Lee Thompson / 

Update: KaChing is now called Wealthfront! Their excellent engineering blog is now http://eng.wealthfront.com/

kaChing invited me over to their Palo Alto office last week and I sat down with Pascal-Louis Perez (VP of Engineering & CTO) and Eishay Smith (Director of Engineering) to talk shop.  

I learned about continuous deployment, business immune systems, test randomization, external constraint protection, how code can be thought of as “inventory”, and that the ice cream parlor across the street from kaChing has great cherries! Below is a transcript of our chat.

Note: Tomorrow night (May 26) , kaChing will be presenting on Continuous Deployment at SDForum’s Software Architecture & Modeling SIG at LinkedIn’s campus!

 

                  
Pascal-Louis Perez
VP of Engineering & CTO
                Eishay Smith
                Director of Engineering

 

Lee:
Eric Ries, possibly Continuous Deployment’s biggest advocate, is a tech advisor for kaChing and he’s done a few startups himself! Eric mentions Continuous Deployment (CD) within a concept called the Lean Startup. It’s possibly the best business reason I’ve heard to do continuous deployment, giving you more iteration capability to get the product right in the customer’s perspective.  You’re doing a startup with kaChing and blogged about your CD a few weeks ago.  It’s great to be here today to learn a bit more about your CD implementation!

Pascal:
Yeah, and I think that Eric really made it popular at IMVU, which has probably one of the most famous continuous deployment system kind of 20-minute fully automated commit to production cycle. Very early on Eric and I started spending some time together.  He was kaChing’s tech advisor almost from the start.  He helped drive the vision of engineering and clearly conveyed why you would care about continuous deployment and why it’s such a natural step from agile planning to agile engineering. I think many people confuse agile planning with agile engineering.  Agile engineering is the ability to ensure your code is correct quickly and achieve very quick iteration, like three minutes or five minutes.  You commit; You know the system is okay.

Lee:
Before you forget about what you just changed. 

Pascal:
Right.  Exactly.  So continuous deployment in that sense is just one more automation step that builds on top of testing practices, continuous testing, continuous build, very organized operational infrastructure, and then continuous deployment is kind of the cherry on top of the cake, but it requires a lot of methodology at every level.

Lee:
Testing, you talked about that a lot in the blog that testing is a major, major part of continuous deployment.

Eishay:
Without fast tests, and a lot of them, continuous deployment is very hard to achieve in practice.

Lee:
And unit versus integration testing both obviously I would think?

Eishay:
Yeah.  We focused mostly on the unit.  We test almost every part of our infrastructure, so integration tests are also important but not as much as unit tests.

Lee:
We were talking offline a little bit about the large number of tests and that you randomize the order in which they are ran.

Eishay:
We randomize them after every commit; And we have small commits.

Pascal:
And we’re trunk stable, so everybody develops on trunk and the software is stable at every point in time.

Lee:
And the unit tests are with the implementation?

Pascal:
Absolutely.

Lee:
And then the integration tests, or is that a separate package or application?

Pascal:
No, it’s all together. Some of the integration tests are part of the continuous build, and I think the other integral part to continuous deployment is what Eric Ries calls the immune system, which is basically automated monitoring and automated checking of the quality of the system at all times. One of the things in our continuous deployment system is we release one machine, we let it bake for some time, then we release two more, we let that bake for some time, and then we release four more.

Lee:
So the software is compatible with being co-deployed with new version versus old version?

Pascal:
Yeah. Forward/Backward compatibility is a must at every step.

Eishay:
We need to do self-tests of the services. One service starts and has a small self-test. It checks itself. If it fails its own self-test it means it’s not configured properly, it can’t talk with its peers or for any other reason, then it will rollback.

Lee:
So you have a bunch of assert statements that once the software boots it has these things that are critical for it to run and it checks it?

Eishay:
Right.

Pascal:
For instance, the portfolio management system starts. Can it get prices for Google, Apple, and IBM? If it can’t, it shouldn’t be out there.

Lee:
Yeah. If IBM changed symbol, which it hasn’t since it’s been public, but if it did then you’d have to go in there and change that assertion, but that you would know that almost immediately.

Pascal:
Yeah. We actually have – digressing a bit into the the realm of financial engineering – our own symbology numbering scheme to avoid those kind of dependencies on external references. Our own instrument ID. It’s kaChing’s instrument ID. Everything is referenced that way. At a high level, we really try from the start to protect ourselves from external constraints and having external conversions between the external domains and our view of the world at the boundary, at the outsests of the system so that within our system everything can be as consistent and as modern as we want. I’ve seen many systems where external constraints were impacting the core and making it very hard to iterate.

Lee:
Which makes for a tightly coupled system. So you’re trying to make the coupling looser between computing entities.

Pascal:
Yeah.

Lee:
There’s just a lot of stuff we’ve talked about just in three minutes. Two simple words: continuous deployment, and a lot of investment in technical capabilities underneath that. A lot of investment in build automation. A lot of investment in how the application is architected such that it can be co-deployed and co-resident with multiple versions. You have integration testing, unit testing, scale and capacity in running those tests a lot and randomizing, and then you went into monitoring, right? So I mean to do a continuous deployment system in an organization that is pretty large, that would probably be a pretty big transform, but since you guys are doing it right from the get-go, right from the start…

Pascal:
I think it would be extremely hard for an architecture, for a company culture that is not driven by test-driven development to then shift gears and decide “In one year we’re going to do continuous deployment. We need to hit all those milestones to get there.” It’s a very difficult company culture to shift. Everything in our engineering processes are geared towards having a system at every version, having multiple versions in production, having no down time, database upgrades that are always forward and backward compatible. It’s really ingrained in many things, and to be able to help engineers work at every level, to be able to achieve all the different little things that require seamless continuous implementation is quite hard, which is why obviously doing that from the start is much easier. I think it would be very hard to get a large project and shift it to doing CD.

Eishay:
TDD or test development is the fundamentals. It’s at the core. If continuous deployment is the cherry on the top, the TDD is the base. Without that you just can’t…

Pascal:
And something that one of our engineers, John Hitchings, commented about was our view of testing. He was saying, “At my previous company I would be doing testing to make sure I didn’t do any blatant mistake”, but at kaChing I do testing to make sure that I’ve really documented all of my feature and protecting my feature from people who are coming next week and changing it.

Lee:
Is that behavior driven development?

Pascal:
It’s really writing specs as tests. If I write a feature and then I go on vacation, anybody in the company should be able to go and change it. I need to be able to document it in code sufficiently well to enable my peers to come and change it in a safe way. The tests aren’t there to help me. The tests are there to protect my little silo of codeIt’s a very different approach on testing.

Eishay:
We don’t have any dark code areas in our system. Anyone can get into our system and do major re-factoring with the confidence that if the tests pass then it’s okay. Of course he has to test anything he adds, but I can be pretty confident that I can move classes from one place to the other or change behaviors and if the test passes it’s good. In other places I know in other companies I used to work at, there’s a lot of places that the developer will have a piece of code, left the company or is working on something else, and this part of code is locked. Nobody can touch it, and it’s very scary.

Lee:
Fragile. Yeah. Let’s talk about confidence. So I walked in, it looked like you kicked off a build and that was running. Do you think it’s probably already in production right now while we’re sitting in the conference room talking about it?

Eishay:
Oh yeah. We say five minutes, but in practice it’s probably more like four minutes since I commit something and it’s out there. It could be very typical that I do three or four commits, change 20 lines of codes here and there, and oh, it’s in production. It’s not uncommon that we have 20, 30 releases of services a day.

Lee:
That’s incredible. I think most organizations are happy with a week or every other week and you’re doing it 20 times a day. That’s something to really be proud of, and by the way, I was really glad to see David Fortunato’s blog. There’s a lot to be proud of. That’s just a great post.

Eishay:
Thank you.

Lee:
It’s really good.

Pascal:
We’re talking a lot about the technical aspects, which are a lot of fun, but at the business level we’ve been able to launch kaChing Pro from idea to having clients using it in one month, and kaChing Pro is essentially Google Analytics meets Salesforce for investment managers. It has full stats. It allows the manager to have a kind of CRM system to manage all of his clientele, private store front for them to be able to on board the new clients and basically a little bank interface where their customers can come in, log in, and see their brokerage statements, trade. Being able to turn the gears so quickly on a product is really key.

Lee:
Which is the key part of the lean startup idea that Eric Ries was bringing up. The business value that the continuous deployment does, that was the best written documentation of why to do this. I’ve been thinking about continuous deployment type principles from a technical perspective to help engineers do their job. Lean Startup documents why to do CD from a business perspective. I’m glad you brought that up. That’s a really good point.

Eishay:
For instance, one of our customers contacted Jonathan, one of our business guys, and told us that something didn’t make sense in our workflow and it felt like we needed to change it now. He called the customers after 15 minutes or half an hour and asked them how it looks like right now. We can immediately change things and deploy them and check how the market reacts or the customers react.

Lee:
The way I’ve usually seen this done is that the customer has a discussion with a business person and a developer, and the developer runs a prototype next version and shows it to the customer and the customer likes it and then it takes four weeks or longer to get it into the production system. The way you’re doing it where it’s checked in, looks good to you, you put it out there and the customer is going on to a live site and seeing it.

Eishay:
Sometimes we have “experiments”, which is a term coined by Google. We’ll test full features with a select group, just like you would A/B test parts of the site… except it is for full features!

Pascal:
I’ll give you an example. I think some of the misunderstandings of pushing code to production is pushing code is equivalent to a release, when really those two things are completely disconnected. Pushing code is, well, I have inventory in my subversion. I need to get that inventory out in the store in front of customers, versus releases, unveiling a new aisle. The aisle can be there in the store, it’s just not going to yield.

So what we’ll very often do is we’ll have the next generation of our website but you need to have a little code to be able to see it, or your user needs to be put in a specific experiment and we showed that website to you, very similar to Google’s homepage being shown to 2% of its user base. You basically have the two versions of the website running at the same time and just showing it selectively. So we can, before a PR launch, have only the reporters look at the live website on KaChing.com with all of the new hype, but everybody else doesn’t see it and then when the marketing person is happy they just flip the switch and it goes public.

Lee:
Is that on separate servers and separate application or is it the same server?

Pascal:
No, same server. It’s just selectively decides user 23, you’re in that experiment. Here, I’ll show you that website.

Lee:
There’s a book called Visible Ops and it says that 80 percent of your problems in operations are changes. This can cause a conflict between your operations staff and your development staff because the change is met with great suspect. Maybe resistance, but it’s definitely going to be a point of scrutiny. Testing is probably the biggest thing that gives you the confidence that the change isn’t going to kill you. And if it did fail, you are missing a test.

Eishay:
And we’re not this type of company. We don’t have ops operations.

Lee:
I would argue you’re ops, but yeah. [Laughs]

Eishay:
We also don’t have a standard QA. Since we deploy all the time, there’s no person that looks at the code after every point.

Lee:
I think you built it – the console that you showed me functions as QA. QA runs the test and signs off on the results and you’re basically automating that function.

Pascal:
QA is classically associated with two functions. There’s making sure you’ve built per spec, specs being human readable, only humans can do that part. But then once the spec is fully understood and disambiguated then that part can be automated. I think many people kind of mix the two and don’t automate QA. You should clearly separate the two in attaining a spec that is fully understood and then making sure this is fully automated. Then you can have humans do the interesting thing, like the product manager saying, “This flow does look like what I had in my mind”, and that viewpoint was not encoded into a test. So this part will never be able to automate because from a human brain, it needs to be understood by a human brain.

Lee:
What I think I’ve learned today is in order to do continuous release and continuous deployment you have to have continuous testing. It sounds to me like if you don’t have that you’re not gonna have the confidence.

Eishay:
It drives a fundamental culture of thinking, engineering culture; which means that the engineer who writes the code, he knows that there’s no second tier of QA persons who will check that the small feature change is now good. He has to know that he must write all the tests himself to fully cover any feature change in places. At places that have formal QA, I’ve seen people change something they didn’t fully test because they know someone will later on have a second look at this feature or this change.

The first feature when it’s released you need to have a person, probably the product manager, to look at it and see if it’s right, but afterwards they’ll never look at it again ‘cause they’ll assume the engineer wrote the features. Of course the software changes all the time because we re-factor, we extract services to another machine. We do all sorts of stuff, but having no other person to do the QA makes us as the engineers do better testing, and better tests.

Page 9 of 14First7891011Last