Andie Nordgren recaps the CCP Games internal DevOps conference
Damon Edwards /
The following is a guest post by Andie Nordgren, Technical Producer at CCP Games in Reykjavik, Iceland. Andie recently wrote this post as a recap of their internal DevOps conference (for which she was the primary organizer). I thought it was a great overview of what an internal DevOps conference is all about and asked her to share it with our community.
-Damon
Andie Nordgren
CCP Games
Our first ever CCP Games Service Delivery & Devops Conference took place last week. It gathered people from many disciplines to talk about how we get our software into customers’ hands. Alex Honor and Anthony Shortland of DTO Solutions gave a number of well attended and appreciated talks, and led workshops and hack sessions.
What did we do and how did it go?
Day 1 – Introductions, lectures, and constructive complaining
Day 1 started with introductions over breakfast, and went into the “Setting the Bar” talk, that introduced DTO solutions and current industry trends in releasing software on a fast and lean cycle – and building up a “business immune system” of tests and monitoring that enables that.
The afternoon was spent with a goal setting session, where Andie and Halldor outlined the business reasons that we are looking at how we deliver our service right now – our growing number of services, a much more rapid release cadence, and bigger environments with the introduction of new battleservers and new components.
We talked through capabilities we want to improve as a company, and the opposing forces that hinder us from having those capabilities.
Day 2 – Lectures and tentacles
Day two started with talks from Alex and Anthony about value stream mapping, service delivery, and a model for how to build successful service delivery toolchains.
The afternoon was spend in a “show and tell” where a large group of CCP-ers helped draw up what services are part of our live game, how they are related in terms of versioning and data dependencies, and what tasks they perform. As you can see below the picture had tentacles in it, which is mostly a sign of how central our database is.
Day 3 – Let’s do something!
Day 3 started with Anthony and Alex putting 2 stakes in the ground: We would prototype/hack a service delivery tool chain in a HACK sandbox, using Jenkins as a build console, and Rundeck as a deploy console (two key components in the DTO approach for identifying crucial roles in such a toolchain).
We would also attempt to explicitly cover each stage along the top in the Quadrant-diagram from DTO below – develop-commit-build-package-register-promote-install-configure for 1 isolated service in our backend.
After the problem space was talked through, 3 teams were formed to tackle parts of this toolchain:
- Team Build
- Team Repo
- Team Deploy
These teams were various combinations of developers, release managers, build engineers, QA engineers and virtual world sys-admins and leads.
We watched a deployment and kept talking about aspects of our service delivery over lunch. In the afternoon the 3 teams kicked off with their tasks organized from a shared Kanban wall.
A number of standup meetings were held during the day to synch up on progress, and the day ended with many components in place, a refactored build system, a newfound love for Jenkins and all the Jenkins plugins, and a decision for how to integrate Rundeck with our tools using a clever approach that let the tools continue to function in a standalone fashion, and developers write their deployment descriptors in familiar python, but Rundeck could take care of distributed orchestration and having a pretty deploy console web UI.
Day 4 – Hacking, demo and beers
Day 4 was spent continuing the work of getting the full process through for the server, and the day ended with the 3 teams demoing the working components they had each delivered, and how they fit into the Service Delivery Platform, to the wider group of conference participants.
We ended up with server build artifact being created through multiple build steps orchestrated by Jenkins, these artifacts being registered in the Build repository, good progress into using the Deploy commands to spin up a slice and start up the server and CREST as an extended startup-test, before automatically moving the build into a “deployable” area. The Repo served up its contents as options that Rundeck could understand and present to the user, for integrating with the deployment tasks on top of the slice tools, that could now be orchestrated through Rundeck. With a couple of hacky corners cut, as it’s supposed to be in a 1,5 day hack session, but also some solid progress on things we wanted to do anyway, we got a great sense for what a full service delivery platform would look like, and how we can now build towards that kind of environment.
We concluded that some of this work would continue over the summer, and that we will pick it up again in August.
And then of course there was some beer drinking, and all the conversations that didn’t fit during the conference itself, as we made a devops invasion of the bar Úrilla Górillan.
What’s next?
- Parts of the work done in the hack sessions will continue and be or is already integrated into MAIN.
- We will gather up in August after vacations to re-evaluate our tool chain and the stuff we did in the Hack Sessions.
- We will keep identifying opportunities for continuous improvement, because we now have a language and high level goal in common across teams and disciplines.
Thank you to everyone who participated and to all who continue to carry the service delivery platform and Devops torch in CCP, and a big thank you to Anthony and Alex from DTO for engaging with our tech and teams and sharing their wealth of experience in the form of concrete tools and models.
If you read this far, you’ve earned one Devops dinosaur.
Over and out
/Andie, Team Rex, Halldor and Thorir B
