I’ve been working as a guest on another codebase recently and it has been very painful for one reason only. The code itself is absolutely fine but a run of the pipeline for a feature branch takes in excess of an 45 minutes and master takes an hour – way too long to be productive.

My specific task this week has been to work on deployment, for which I generally needed to run the whole pipeline to evaluate some very small changes. In fact, realistically I was achieving 4 deploys per day into the development environment.

Developer productivity is all about reducing idle cycles without introducing context switches. Sure I could go and work on another project in that 1 hour downtime, but there’s a spin-up cost on the context switch, both cognitive and in tooling, so you don’t get the whole hour back.

A good rule of thumb

• If the pipeline runs in less than 5 minutes then developer will remain productive. That’s enough time to quickly check slack and email and then get back to see the results.
• If the pipeline runs < 20 minutes then there’s a good chance the developer will get up to go get coffee/snack/have some kind of break. Obviously that’s fine sometimes but if that ends up turning every pipeline run in 30 minutes of inactivity then it’s costly.
• If the pipeline runs > 20 mins then the developer is going to get absorbed in another task, quite likely not directly work related, to avoid the context switch. They will get back to checking the pipeline when they’re done with that task. It could be 45 minutes or it could be 3 hours. Cycle time drops off a cliff here.

How do I avoid the trap?

Tiered pipelines

Strategy one is to tier the pipelines. Not every test should run in every pipeline. Things that should run on every commit are linting, unit testing, dependency security checks. Generally these should be run in parallel where possible. This should give a good balance between correctness and feedback time.

When you get to a PR you can add some more expensive tests – integration testing, basic performance testing, etc. You should have high confidence here that the merge to master will be successful. It’s OK for PR pipeline to run a bit longer.

Against master branch additional tests should be run. Typically this will validate a bit more of the deployment. Failures should be rare and fixed immediately, and you still want pipelines to run in < 30 minutes ideally.

Finally, heavy tests, like deep integration tests that could take an hour to run, should be run on an automated cycle outside of normal developer workflow and only report on any issues. There should never be a developer waiting 1 hour for feedback.

Reduce test coverage

This may seem counter intuitive but one of the biggest problems of testing is too much testing. If the tests are too slow and brittle then inevitably people will start finding workarounds. You shouldn’t need to test every possible failure scenario in your unit tests, consider whether they’re realistic failure scenarios that should be handled, and whether you can run those tests less frequently. This is where code coverage tools are somewhat of a blunt tool.

Parallelise everything

Compute time is less than developer time, so spin up the resources to run as many tests in parallel as possible. This is also why I’m a fan of serverless CI environments, where compute resource can be spun up and down exactly according to direct developer need.

Shift left to the local machine

In many organisations I see a worrying number of tests failing on linting, in some cases pretty much the start of every PR cycle. This should be a rare scenario. Most editors can be configured to enforce linting rules and have static analysis plugins that allow a fix-while-you-type approach. A green pipeline is always a faster pipeline.