How Uber halved Golang monorepo CI build time

Before the year 2021, Uber™ engineers would have to take quite a taxing journey to make a code change to the Golang Monorepo. First, the engineer would make their changes on a local branch and put up a code revision to our internal code review system, Phabricator.

Next, Uber™’s infrastructure would see the request and initiate a number of validation jobs on our CI. Those jobs would run build and test validation using the Bazel™ build system, check the coverage, do some other work, and report back to the user a red light (i.e., tests failed or some other issues) or green light. Next, the user, after seeing a “green light,” would get their code reviewed and then initiate a “land” request to the Submit Queue.

After receiving this request, the queue would patch their changes on the latest HEAD of the main branch and re-run these associated builds and tests to ensure their change would be valid at the current state of the repository. If everything looked good, the changes would be pushed and the revision would be closed.

This sounds easy, right? Make sure everything is green and reviewed, and then let the queue do the work to push your change!

Well… what if the change is to a fundamental and widely used golang library? All packages depending on the library will need to be validated, even though the change itself is only a few lines of code. This can slow down the “build and test” part. Validating this change could sometimes take several hours. Internally, we call such changes big changes