Friday, 17 October 2014

Change the characteristics of your complexity

I was lucky enough to attend a talk by John Turner from Paddy Power as part of London Continuous Delivery meetup.  The talk was very informative and gave the audience some great insight into rolling out PaaS and continuous delivery in the real world.

In order to innovate faster, Paddy Power made a number of changes to "re-orient" the company and put the engineer team first.  To help speed up engineering, where possible, decisions on architecture were removed and replaced with convention.


An abstract (no tools specified) pipeline was defined with the following stages:

1. Code commit

Trunk based commits with feature switches.

2. Build

Component and Integration tests
Code quality rules (enforced by sonar)

3. Acceptance Tests

Run against the application/service as delpoyed in a VM as provisioned by Gigaspace's Cloudify product.  Config for the VM's dependencies (e.g. java installation) is defined in a blueprint.  Acceptance tests in BDD style with JBehave and Thucydides.

4. Performance Test

The goal here is to see if there has been a significant degradation in performance.  The goal is not to try and measure absolute performance.  This is performed by JMeter.

5. Stability Test

At this stage, the acceptance are run with some deliberate random failures added to various dependencies.  This has been achieved by an in-house implementation of Netflix's SimianArmy.

6. Manual Test

This is some times required for new features, legal issues and also as John put it "that warm fuzzy feeling"

7. Production deploy

No Rollbacks

I have debated about this a few times with various people (see here for my thoughts) and got into the same debate with John after his talk.  John had a great analogy... rollbacks are like car airbags, they encourage drivers to take risks.  On the other hand, a spike on the steering wheel is your roll-forward only approach and encourages a lot more care :)

Other snippets

"You can't remove complexity from an inherently complex system.  You can however change the characteristics of the complexity, by distributing it" - This is the goal of moving to a microservices architecture.

Getting your organisation to change can be hard.  A technique John mentioned was to trial out the change on a small scale, prove it works, and then to try and convince others... sound advice!