Although we have explained the reasoning behind our Intelligent Cache and the need for faster cores, there is nothing quite like seeing the result.

Rather than use one of our internal verification projects or build a facsimile of a complex project, we benchmarked Grafana. Grafana provides enough complexity to demonstrate a real world caching bottleneck while also utilizing two languages and a container build.

While Grafana does not utilize Gitlab CI, we used the existing CI definitions as a guide to build a simple, but realistic Gitlab CI setup. The result was 6-9X faster than Gitlab.com.

Building containers is a ubiquitous activity with a wide range of facets and tricks. Everyone builds containers and everyone wants them to build faster.

Most efforts to improve build performance are focused around layer caching. Steps are carefully ordered and broken apart to optimize the number of steps before a cache miss occurs. All steps prior to the miss can be hydrated from cache. Next, the layer cache performance can be tuned since acquiring the layer takes the place of performing the step. If downloading the layer, or reading it from a slow disk, takes longer than the step the cache is moot.

Although the various techniques, employed in CI, for providing responsive access to container layers are effective, they fail to optimize the portions specific to the application source code.

As anyone familiar with Continuous Integration (CI) knows, CI is repetitive and so are its problems. The repetitive nature of CI makes it a tempting target for caching. Unfortunately, the common CI caching mechanisms are relatively brute force in nature and do not accommodate the complexity of real-world use-cases.

Overhead#

The compress, upload, download, and extract steps, used by almost all caching mechanisms, result in overhead that scales proportionally to cache size. The overhead works against the cache which results in diminishing returns. In many cases, full caching is slower than no cache. Optimizing the performance of a CI job tends to be a balance between theoretical efficiency gains and cache size.

Caches are often used multiple times in a pipeline, such as a dependency cache. The overhead cost of the download and extract steps is paid on each job. It is not uncommon for the whole process to take on the order of minutes. The job that updates the cache pays the cost on both ends (extract and compress).

Any increase in parallelism multiplies the overhead cost in compute time, since it must be performed on each job. The higher the concurrency the larger the percentage of a job becomes just overhead.

Contrary to the prevailing wisdom, cores aren't everything. For many workloads, peak single-core frequency has the biggest impact on overall performance.

CI jobs tend to be composed of a combination of tooling with glue code holding everything together. Individual tools might take advantage of all available cores, but they tend to taper towards a single-core as they complete their task. Combining multiple tools, back-to-back, with serial scripts and waiting for large artifact transfers can result in something like the example above. Despite 8 cores being available, most of them are barely utilized.

Increasing the CPU frequency would have the biggest impact, even when coupled with a reduction of core count. Gitlab only offers different core counts, all running at mediocre frequencies. By contrast, Cedar CI utilizes purpose built hardware to deliver the fastest core speeds. Our CPUs offer 2-3 times the performance of leading CI providers. Coupled with the fastest cache around, our results are hard to match.

Our cores will end up reducing your CI cost while saving engineer time, which is far more valuable. Realize just how fast your CI can be today by leveraging Cedar CI.

Since Cedar CI provides twice as many cores, as Gitlab.com, for a given rate, we decided to demonstrate the performance difference to emphasize the value. For a simple and direct comparison we created a CI job to calculate π to the first 4,000 digits, repeated 32 times. Each calculation is enough to saturate a CPU core.

We ran the job on Gitlab.com's saas-linux-small-amd64 and Cedar CI's 4-core cedarci-4-core since they're the same price. The job definition is as follows.

While applications may start with a language or application specific base container it does not take long before additional distro packages are required. The packages may be needed by the application itself or its support tooling. Many times this begins by installing something like curl and grows over time. The package installation is a source of intermittent network failures and adds overhead to each job.

A general trick to improve performance is to avoid doing things. Creating a bespoke container for CI is a common pattern, but can create a lot of friction and is tricky to get right. The following is an approach for creating a conditional CI container build that avoids unnecessary execution while also automatically incorporating updates.

Automation via CI/CD primarily minimizes mistakes and increases efficiency. Instead of remembering to run various test suites and build steps, and having to deploy to a review environment, engineers can simply push changes and wait for CI to complete. The larger the coverage area of CI, the higher the potential value.

However, the more that is added to CI, the more compute time is required, and the more time an engineer traditionally has to wait for the result. In some cases, an engineer can avoid waiting for CI, but this generally means context switching, which can have its own impact on overall productivity. For this reason, wait time directly correlates to engineer time and causes fatigue from repeatedly paused development cycles.

As the number of CI operations increases, so too does the likelihood of transient failures. Depending on the configuration, failures may be automatically retried or may require manual intervention. Each failure increases the overall duration. A transient network failure that is retried locally may only add a few seconds, but a flaky test run as part of a suite may require a rerun of an expensive test job. It isn't unrealistic for several failures to result in a doubling of the overall duration.

Over the last two decades developing software, we've had the opportunity to work on CI/CD from just about every angle.

• built distributed CI system from the ground up
• optimized performance and reliability in various environments
• migrated between CI systems without disruption
• utilized centrally controlled CI systems from the outside

During that time, the landscape has changed quite a lot, but the core challenges remain the same. What caused transient failures a decade ago still does, and computation that could be avoided is performed endlessly on repeat. Mechanisms have been developed to improve efficiency, but generally require configuration, sometimes extensive, and come with unintuitive performance curves that are not always understood.

We have decided to embark on a journey to drastically improve the baseline CI/CD experience for all, by taking the best we've seen and combining it with some tech we've always wanted to build. Our plans cover the full span of CI/CD from novel performance and reliability improvements to cost optimization and observability. We've decided to start by directly targeting performance and cost via a low overhead business model. To that end, we've launched an initial offering that is compatible with Gitlab.com SaaS runners, but at a lower price point to enable better performance for less.

We look forward to demonstrating just how optimized your CI/CD experience can be and how much further we can improve it going forward.