View All Videos

Andie Nordgren recaps the CCP Games internal DevOps conference


Damon Edwards / 

The following is a guest post by Andie Nordgren, Technical Producer at CCP Games in Reykjavik, Iceland. Andie recently wrote this post as a recap of their internal DevOps conference (for which she was the primary organizer). I thought it was a great overview of what an internal DevOps conference is all about and asked her to share it with our community.

Andie Nordgren
CCP Games

Our first ever CCP Games Service Delivery & Devops Conference took place last week. It gathered people from many disciplines to talk about how we get our software into customers’ hands. Alex Honor and Anthony Shortland of DTO Solutions gave a number of well attended and appreciated talks, and led workshops and hack sessions.


What did we do and how did it go?

Day 1 – Introductions, lectures, and constructive complaining
Day 1 started with introductions over breakfast, and went into the “Setting the Bar” talk, that introduced DTO solutions and current industry trends in releasing software on a fast and lean cycle – and building up a “business immune system” of tests and monitoring that enables that.

The afternoon was spent with a goal setting session, where Andie and Halldor outlined the business reasons that we are looking at how we deliver our service right now – our growing number of services, a much more rapid release cadence, and bigger environments with the introduction of new battleservers and new components.

We talked through capabilities we want to improve as a company, and the opposing forces that hinder us from having those capabilities.

Day 2 – Lectures and tentacles
Day two started with talks from Alex and Anthony about value stream mapping, service delivery, and a model for how to build successful service delivery toolchains.

The afternoon was spend in a “show and tell” where a large group of CCP-ers helped draw up what services are part of our live game, how they are related in terms of versioning and data dependencies, and what tasks they perform. As you can see below the picture had tentacles in it, which is mostly a sign of how central our database is.

Day 3 – Let’s do something!
Day 3 started with Anthony and Alex putting 2 stakes in the ground: We would prototype/hack a service delivery tool chain in a HACK sandbox, using Jenkins as a build console, and Rundeck as a deploy console (two key components in the DTO approach for identifying crucial roles in such a toolchain).

We would also attempt to explicitly cover each stage along the top in the Quadrant-diagram from DTO below – develop-commit-build-package-register-promote-install-configure for 1 isolated service in our backend.

After the problem space was talked through, 3 teams were formed to tackle parts of this toolchain:

  • Team Build
  • Team Repo
  • Team Deploy

These teams were various combinations of developers, release managers, build engineers, QA engineers and virtual world sys-admins and leads.

We watched a deployment and kept talking about aspects of our service delivery over lunch. In the afternoon the 3 teams kicked off with their tasks organized from a shared Kanban wall.

A number of standup meetings were held during the day to synch up on progress, and the day ended with many components in place, a refactored build system, a newfound love for Jenkins and all the Jenkins plugins, and a decision for how to integrate Rundeck with our tools using a clever approach that let the tools continue to function in a standalone fashion, and developers write their deployment descriptors in familiar python, but Rundeck could take care of distributed orchestration and having a pretty deploy console web UI.

Day 4 – Hacking, demo and beers
Day 4 was spent continuing the work of getting the full process through for the server, and the day ended with the 3 teams demoing the working components they had each delivered, and how they fit into the Service Delivery Platform, to the wider group of conference participants.

We ended up with server build artifact being created through multiple build steps orchestrated by Jenkins, these artifacts being registered in the Build repository, good progress into using the Deploy commands to spin up a slice and start up the server and CREST as an extended startup-test, before automatically moving the build into a “deployable” area. The Repo served up its contents as options that Rundeck could understand and present to the user, for integrating with the deployment tasks on top of the slice tools, that could now be orchestrated through Rundeck. With a couple of hacky corners cut, as it’s supposed to be in a 1,5 day hack session, but also some solid progress on things we wanted to do anyway, we got a great sense for what a full service delivery platform would look like, and how we can now build towards that kind of environment.

We concluded that some of this work would continue over the summer, and that we will pick it up again in August.

And then of course there was some beer drinking, and all the conversations that didn’t fit during the conference itself, as we made a devops invasion of the bar Úrilla Górillan.

What’s next?

  • Parts of the work done in the hack sessions will continue and be or is already integrated into MAIN.
  • We will gather up in August after vacations to re-evaluate our tool chain and the stuff we did in the Hack Sessions.
  • We will keep identifying opportunities for continuous improvement, because we now have a language and high level goal in common across teams and disciplines.

Thank you to everyone who participated and to all who continue to carry the service delivery platform and Devops torch in CCP, and a big thank you to Anthony and Alex from DTO for engaging with our tech and teams and sharing their wealth of experience in the form of concrete tools and models.

If you read this far, you’ve earned one Devops dinosaur.

Over and out
/Andie, Team Rex, Halldor and Thorir B

66 Responses

  1. Trading cards available at Chaos Cards. Many collectable trading cards to be bought online, complete your collection of trading cards now.

  2. Cool says:

    Thanks For Share Your Blog with Us.

  3. susan says:

    yeah i think these artifacts being registered in the Build repository, good progress into using the Deploy commands to spin up a slice and start up the server and CREST as an extended startup-test, before automatically moving the build into a “deployable” area helper

  4. Alan says:

    tennis court lights OAK LED Co ,Limited is one of the top level China LED tennis court light manufacturers and suppliers with one of the famous LED tennis court light brands, as a professional company, we have our own factory, which is able to produce high quality products with reliable quality tennis court lights

  5. Alan says:

    led stadium lighting  OAK LED Co ,Limited specialized in high power led lighting for stadium We offer 100 2000w LED Football Stadium Light With one of the top football stadium lighting manufacturers and suppliers, we produce high quality products with reliable quality led stadium lighting 

  6. smart says:

    If you are looking for mobile phones, laptops and tablets – you came to the right address. online shop offers original products, with official warranty from the manufacturer. Now it has become easier to buy a smartphone in Chisinau.

  7. Merlin John says:

    I think I am very lucky to find this article that is really worth reading and helpful in all extend. I would like to say thanks for providing such great information and such articles are really very useful to improve our business. Hope you will share more like this.
    directv family package

  8. icloud giris says:

    Love this article, great articel information for buisness people. by admin of
    icloud giris

  9. Love this beautifull article..must to be shared for buisness people.

    hotmail aç

  10. Thesto Bero says:

    I simply couldn’t leave your website before recommending that I must say I loved the most common info a person resource for your friends?
    Mens Life Line

  11. The article is very detailed. Thank you for giving it to us who read it. It’s very inspiring and useful
    paket umroh Februari

  12. Great post, please keep on sharing amazing article like this! It makes me happy reading your post.

  13. Jio TV app says:

    I think I am very lucky to find this article that is really worth reading and helpful in all extend. I would like to say thanks for providing such great information and such articles are really very useful to improve our business. Hope you will share more like this. Jio cinema for PC

  14. this is the great post. this post is very helpful for youth and students. students can check jee main 2017 admit card

  15. thanks for sharing such a great article. Jee Main 2017 Answer key

  16. nice article thanks fro sharing

  17. too good article, thanks again

  18. haiminhco says:

    Cong ty haiminhco chuyen cung cap va phan phoi cac dong san pham may photocopy chinh hang hien nay tren thi truong. Neu ban dang ban khoan khong biet phai tim mo dia chi uy tin cung cap cac dong may photocopy nhu the nao thi cac ban co the xem tai day

  19. Moi chi tiet ve khoa hoc lai xe nang hang cac ban lien he truc tiep voi Hien de duoc sap xep lich hoc cung nhu cho o de co the hoan thanh khoa hoc nhanh chong nhe. Minh cam ket se co chung chi trong vong 3 ngay cho cac ban. Chi tiet vui long xem tai website cua trung tam nhe:

  20. I really like so bad your own posting! I study most short article is good. I find your current content taking Yahoo search. Verify my site is good as you: free paypal money instantly, I work many to write down this. Over again give thanks you and stay with article writer! Best of luck!

  21. Tessa says:

    Really glad to go through the recap of this CCP Games done by the Technical producer Andie Nordgren. She explained it with diagrams and we can get a clear overview of this game while going through this. I think lots of people are gathered there and share their ideas for its improvement. facial mask

  22. Thanks for Sharing this article for Details about the Board Exam Results, Check own details about RBSE Results

  23. Rajasthan BOard 12th Result

  24. Haryana Board 12th Result

Post a new comment