Built-in commercial CI/CD on heptapod.host with Clever Cloud

As of December 2021, a Shared Runner going by the name of clever-cloud-docker is available on heptapod.host, our commercial service. It will take only jobs that require it explicitely.

Intended audience: project maintainers familiar with GitLab CI/CD.

All examples in this document can also be found in the corresponding demo project.

Key properties

These are explained in greater detail in the following subsections.

  • on-demand provisioning and billing
  • opt-in activation based on job tags
  • usable like any Docker-based GitLab Runner
  • privacy: each job is provided with its own dedicated Docker host.
  • vertical scalability: host sizes (flavors) can be selected from the job definitions.
  • works for Git and Mercurial projects

Provisioning and billing

With the Clever Cloud Runner, you may run a great number of parallel jobs, paying for your actual consumption without any threshold effect.

Each time it takes a job, the Clever Cloud Runner spawns a virtual machine within your Clever Cloud Organization.

While the job is running, the virtual machine is visible in the Clever Cloud console, in which each job corresponds to an Application having exactly one Instance attached. The Instance is the virtual machine running the job.

The type of the Instance is "Heptapod Docker Runner", and is identified by a Heptapod icon. The job number is included of the Application name, which looks like hpd-job-kvzWdEMo-275833.

Both Application and Instance are removed as soon as the job completes, whatever end status was returned to the coordinator. You will be billed for the time the Instance was up, with a 10 seconds granularity.

Activation with tags

The Clever Cloud Runner only takes jobs bearing the clever-cloud tag.

If you maintain your own Group or Project runners, this makes sure the Clever Cloud Runner does not steal jobs from them, as chances are high that they would require a specific setup. Also, this avoids billing users for a service that they did not ask for and may not be aware of.

To flag a job so that the Clever Cloud Runner can take it, use the clever-cloud job tag.

Example (.gitlab-ci.yml extract):

myjob:
  tags:
    - clever-cloud
  script:
    - ./run-all-tests

See also this example live on heptapod.host, with its resulting pipeline.

It is also possible to set the tag in all the jobs in the CI/CD configuration in one stroke, thanks to the default global keyword:

default:
  tags:
    - clever-cloud

lint:
  script:
    - ci/lint

tests:
  script:
    - ci/run-all-tests

This example is also available on heptapod.host. Here is the corresponding pipeline (the failure of the lint job is intended).

Usable like any Docker-based Heptapod or GitLab Runner

Except for provisioning, everything happens as if the job was launched by a standard Docker executor, with only a handful of differences (see the dedicated subsection).

Note: for Git projects, there is no difference between Heptapod Runner and GitLab Runner.

So typically, you would create a Docker image with all the needed dependencies, perhaps host it on registry.heptapod.host (why not?), and use it as the base context for your job.

Notably, the job can make use of services – these will run inside the same virtual machine as the main job.

Here's a full example making use of a PostgreSQL database, relying on the base image built by our Demo Project for CI image to provide the PostgreSQL client utilities (psql, createdb, etc.)

default:
  tags:
    - clever-cloud

tests:
  image: registry.heptapod.host/heptapod/demos/ci-image:latest
  services:
    - name: postgres:14  # official Docker image for PostgreSQL on Docker Hub
      alias: pgserver  # (optional) service host name to use from the job
  variables:
      # All job environment variables are also set in service containers.
      # This one has the effect that postgres will blindly authenticate any
      # existing user without any password.
      # (the `postgres` image has many more authentication options,
      #  this one being good enough for our purposes)
      # Note: GitLab (Runner) 14.5 will allow setting variables on a
      # per-service basis, see https://docs.gitlab.com/ce/ci/services/
      POSTGRES_HOST_AUTH_METHOD: trust
  script:
    - createuser -h pgserver -dSR -U postgres db_owner
    - createdb -h pgserver -U db_owner mydb
    # Now use the database.
    # This query just lists schemas (namepaces) for the sake of example.
    - psql -h pgserver -U db_owner -c "SELECT nspname FROM pg_catalog.pg_namespace" mydb

This example is also available on heptapod.host. Here is the corresponding pipeline.

Differences with a Heptapod Runner using the standard Docker executor.

Automatic Dependency Proxy

The Dependency Proxy is a standard GitLab Free feature that provides transparent caching of Docker Hub images to minimize bandwidth and avoid rate limiting problems.

With the Clever Cloud Runner, all Docker images from Docker Hub get automatically retrieved from heptapod.host's Dependency Proxy, By contrast, this is an opt-in feature with the standard Docker executor.

The automatic Dependency Proxy should be mostly transparent to users, except in case of services using a namespaced Docker image and not having an explicit alias. For these, we provide a single service network name whereas the standard executor provides two (replacing forward slashes either by double underscores or dashes). We chose the dash-based RFC compliant one. In short, use octobus-heptapod to reach a service whose Docker image is octobus/heptapod:sometag, or better, set your own explicit alias.

Job cache

Currently, the job cache is discarded at the end of each job.

This is because we don't have offloading capabilities for the cache yet, and the entire virtual machine gets decommissioned at the end of the job.

We're working on it and should have a solution soon (see the Persistent cache upcoming feature).

Isolation

Because each job runs in its own transient virtual machine, your jobs are effectively pretty private.

Even if a malicious job were to break the sandboxing provided by Docker itself, it would find nothing to spy on within its Docker host.

Even if your projects are public, this can be an important factor, for instance to protect package signing keys and upload credentials when doing Continuous Delivery.

Docker host flavors

While most of the public CI offering seems to have converged on running jobs on comparatively small systems (e.g., 2 virtual cores with 7GB of RAM), and many testing frameworks have ways to partition big test suites in several jobs to run in parallel (sharding), this does not fit all needs:

  • Jobs performing big compilation tasks (OS kernels, desktop applications) may benefit from in-job parallelization and may not be practical to cut in several parallel jobs.
  • At the opposite side of the spectrum, jobs for unit tests or fast linting of small projects may content themselves with a tiny system.

In other words, one size does not fit all.

Jobs executed by the Clever Cloud Runner can specify the size (flavor) of the Docker host it needs, using the CI_CLEVER_CLOUD_FLAVOR variable.

Example (.gitlab-ci.yml extract):

default:
  tags:
    - clever-cloud

small-job:
  variables:
    CI_CLEVER_CLOUD_FLAVOR: XS
  script:
    - ci/display-specs
    - ci/lint

bigger-job:
  # our base image provides /usr/bin/make
  image: registry.heptapod.host/heptapod/demos/ci-image:latest
  variables:
    CI_CLEVER_CLOUD_FLAVOR: XL
  script:
    - ci/display-specs
    - make all
  artifacts:
    paths:
      - out/binaries

This example is also available on heptapod.host. Here is the resulting pipeline (the failed status of the lint job is intentional).

While choosing a flavor, keep in mind that the main job and all services are running in the same host.

As of this writing, the available flavors range from XS (1 core, 2GB RAM) to 3XL (16 cores, 32GB RAM), the default being M (4 cores, 8GB of RAM).

For the full list of flavors with their specification and prices, please refer to the "Heptapod runner" section of the Clever Cloud pricing page (or "Jenkins runner" in the early beta days of this commercial CI/CD offering if the table for Heptapod runners are not yet published. The initial values are identical).

Upcoming features

Persistent cache using Clever Cloud Cellar

In the near future, the Clever Cloud Runner will get the ability to do persistent S3 caching using a Cellar addon tied to your Organization.

This will work without any effort on the users' side. If your job configuration uses the cache: keyword, it will start relying on Cellar as soon as we deploy this feature.

The storage will be billed as any Cellar storage. There should not be any outbound bandwidth costs, since all communication will happen within the same Clever Cloud zone.

Running jobs for other instances

If you're interested on using the elastic properties of the Clever Cloud Runner on another Heptapod or GitLab instance than heptapod.host, this is already possible with an ad-hoc installation or configuration, depending on what you want to do.

Please contact us for more details.

Warm instances

We are thinking of providing the option to also have persistent instances of the Clever Runner, that could be reused at the Group or Project level.

These would provide faster job start up, by reusing clones of the repositories (fetch strategy) and the needed Docker images. On some projects, this can make a substantial difference, with the drawback for users to be billed for idle time.

We are not at this point entirely sure of what would be the best way to provide this, so we'd love to hear your feedback.

Some possibilities:

  • Allowing instances to persist once a job is finished: if the CI is busy enough, they will get reused for new jobs
  • A pool of instances, growing as much as needed, but not shrinking below a certain size, and giving priority to the oldest ones.
  • preprovisioning in the same way as GitLab's Docker Machine executor does

(there is much in common betwen these possibilities).

In any case, the virtual machines would stay private and the system would be opt-in, so that without user intervention, the billing would still be based on actual use time only.

Not supported GitLab Runner features

We don't plan to support Docker-in-Docker (DIND) any time soon, for security reasons.

Note that building Docker images can easily be done without DIND, using Kaniko or similar tools. The Docker image for Heptapod itself is built by our CI with Kaniko, as well as all the images involved in our pipelines. A self-contained heptapod.host example is provided by the Demo Project for CI image.