While applications may start with a language or application specific base container it does not take long before additional distro packages are required. The packages may be needed by the application itself or its support tooling. Many times this begins by installing something like curl and grows over time. The package installation is a source of intermittent network failures and adds overhead to each job.

A general trick to improve performance is to avoid doing things. Creating a bespoke container for CI is a common pattern, but can create a lot of friction and is tricky to get right. The following is an approach for creating a conditional CI container build that avoids unnecessary execution while also automatically incorporating updates.

Simple Application#

Consider a simple application with a build and test job executed in Gitlab which both have the same two prerequisite steps.

# .gitlab-ci.yml
app:build:
  stage: build
  image: language/base
  script:
    - manager install pkg1 pkg2 pkg3
    - make internal_tool
    - make

app:test:
  stage: test
  image: language/base
  script:
    - manager install pkg1 pkg2 pkg3
    - make internal_tool
    - make test

Generally, most changes in the repository will manifest in the final step, but automatically incorporating modifications to prerequisite steps and validating that they don't break later steps is important.

Extract Container#

Assuming the majority of development takes place in the last step the prerequisite steps can be extracted into a container built purely for use in CI.

# ci/Dockerfile
FROM language/base

RUN manager install pkg1 pkg2 pkg3

COPY . .
RUN make internal_tool

Many applications may only require additional package installation which would simplify example above.

Naive Approaches#

The naive approaches are generally to build:

• always
• in the target branch
• manually

Always building relegates the cost to once per pipeline which effectively removes it from one pipeline stage.

Building in the target branch optimizes feature branch builds, but is essentially an always build in the target branch and does not make it easy to test updates in a feature branch.

Manually building fully optimizes CI, but suffers from human error and is cumbersome to test.

Conditional Build#

A fully optimized CI experience can be achieved while maintaining full automation for a quality test workflow. The CI image can be built conditionally for changes that impact it and be used for the rest of that pipeline while falling back to the target branch version if unchanged.

Project Image#

Define a variable for the CI image to be referenced by job definitions. The CI_REGISTRY_IMAGE predefined variable points to a container registry associated with the project.

variables:
  CI_IMAGE: ${CI_REGISTRY_IMAGE}/ci

app:build:
  stage: build
  image: ${CI_IMAGE}
  script: make

Conditional Build#

Next, the CI image will need to be built while avoiding unnecessary builds. A changes stanza can be used to only run the job when relevant. To ensure it runs before anything else the default .pre stage can be used.

ci_image:build:
  stage: .pre
  rules:
    - changes:
        paths: 
          - ci/**/*

Override Image#

Since the image is being conditionally built a method for injecting the image from current pipeline into subsequent jobs is needed. That can be accomplished by writing the desired image URL to a dotenv file and exporting as an artifact.

ci_image:build:
  script:
    echo "CI_IMAGE=${CI_REGISTRY_IMAGE}/ci:${CI_COMMIT_SHA}" > ci_image.env
  artifacts:
    reports:
      dotenv: ci_image.env

The dotenv artifact will be applied to the variables of subsequent jobs which will effectively update the app:build as follows.

app:build:
  stage: build
  image: ${CI_REGISTRY_IMAGE}/ci:${CI_COMMIT_SHA} # ${CI_IMAGE}
  script: make

Complete Pipeline#

Pulling it all together, the following would conditionally build a CI image when inputs are modified and utilize the image for subsequent app:build and app:test jobs. For changes to the app, that do not impact the CI image, the ci_image:build job is not invoked which avoids the transient download failures and overhead.

variables:
  CI_IMAGE: ${CI_IMAGE_LATEST}
  CI_IMAGE_BASE: ${CI_REGISTRY_IMAGE}/ci
  CI_IMAGE_LATEST: ${CI_IMAGE_BASE}:latest
  DOCKER_LIBRARY: ${CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX}/library
  
ci_image:build:
  stage: .pre
  image: ${DOCKER_LIBRARY}/docker:stable
  script: |
    cd ci/
    if [ "${CI_COMMIT_REF_NAME}" != "${CI_DEFAULT_BRANCH}" ] ; then
      CI_IMAGE="${CI_IMAGE_BASE}:${CI_COMMIT_SHA}"
      echo "CI_IMAGE=${CI_IMAGE}" > image.env
    fi
    echo "${CI_REGISTRY_PASSWORD}" | docker login "${CI_REGISTRY}" --username "${CI_REGISTRY_USER}" --password-stdin
    DOCKER_BUILDKIT=1 docker build --cache-from "${CI_IMAGE_LATEST}" --build-arg BUILDKIT_INLINE_CACHE=1 --tag "${CI_IMAGE}" .
    docker push "${CI_IMAGE}"
  artifacts:
    reports:
      dotenv: ci/image.env
  rules:
    - changes:
        paths:
          - ci/**/*
  services:
    - name: ${DOCKER_LIBRARY}/docker:stable-dind
      command: ["--tls=false"]
      alias: docker
  variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""

app:build:
  stage: build
  image: ${CI_IMAGE}
  script: make

app:test:
  stage: test
  image: ${CI_IMAGE}
  script: make test