An odd thing has started to happen to us in the course of our consulting work. If it were just once or twice, we’d note it, and move on. Instead, it’s frequent enough that it’s worth a mention.
It starts simply enough. We’re on a consulting assignment, someone is describing their process. We talk about something more traditional. The person we are interviewing has a startled look and says “oh, no, we are a CI/CD shop.” That’s fair enough; it makes many things easier. We move on.
Two, or four, or six months later, we are working with the same company, not in a consulting capacity but doing contracting. Or perhaps we meet the employees at an event. In this case, we’ve gone from describing the process to talking about a specific repository and build.
The workers are frustrated because it takes so long to get anything done.
“Continuous Delivery” in the wild
All of it for a good reason. When the code change is submitted to version control, it needs a pull request to go into the repository; that pull request needs code review. That process takes a couple of days. Then you have to create a custom test environment and run integration tests. It is common that tests pass on a local machine but fail in the simulated real environment. That simulated environment is an artifact built by a build pulling from a repository, so you need another PR to get the test to run. That leads to a back and forth. The test environment might not always be stable, but if it is, the code still needs to be merged onto the deploy branch. At this point, we are four to seven days after first push. It’s possible someone else’s change invalidates the assumptions, or there is a merge change. If that happens, the code will fail the next level tests. And on, and on, and on.
Doing the math, a typical change might average ten days (counting weekends) from first push to version control to production. That is typical; a few take more, some less. We are not talking about huge changes here. Significant database and code library changes take longer.
Seven to ten days for median change? The one in the middle?
Ten years ago, when I interviewed the staff at Etsy about how they do continuous delivery … that was not what they meant.
The in-practice definitions of CI/CD might need a tune up.
A CI/CD Tune Up
The original case studies for CI/CD were relatively simple. Etsy, for example, was a PHP shop. The code was interpreted, which meant a re-compiled every time the .php file changed on the server. Every web page could be its own .php file. If you broke a web page, you only broke the page you were touching. That mean if you didn’t change data in the database, didn’t touch a code library, and didn’t touch private or financial data, you could just make your change, test it, and deploy it. In the worst case, you broke your isolated little change. The company had graphical displays of errors (and load time) so you could deploy and change and the watch the monitor for problems. They also had feature flags, so you could roll a change out, perhaps to employees or beta users, before rolling it out to everyone. That’s a wonderful combination, a company created from the ground up to run CI/CD on the technology side. There’s good reason that John Allspaw, former VP than CTO of Etsy, is mentioned a sort of folk hero in the IT classic The Phoenix Project.
Then there is everyone else.
Or, at least, there are the groups that tacked on Jenkins, TeamCity, CircleCL, GitLab, BuildKite, or something else to an existing non-trivial delivery process. As it turns out, the “Continuous Delivery” part is more than a tool, it is a process. Processes involve tradeoffs, tough decisions, and change management.
Over the next few months we’d like to start talking more about CI/CD, and how to tune it up, in a way that balances human staff, time, and processing power. If you’ve got budget and would like a tuneup, that’s a service we offer right now. And, of course, if you think we are misguided and have misunderstood the state of the practice, we’d like to hear from you.
Let’s keep talking.
