The Colden pipeline

What the GitLab pipeline runs in each stage, what the runner and the project need, how a project describes its environments and how a deployment and a rollback happen.

GitLab CI is the kit's pipeline of record. It runs on every merge request, on main, on tags and when started by hand, in six stages: build, test, scan, publish, deploy and e2e.

• Builds, tests and scans run on their own, in every merge request and on main.
• Publishing the images and deploying are manual jobs on main and on tags, so unmerged code never reaches a registry or a cluster.

.gitlab-ci.yml holds the stages, the pinned images and the includes. Each stage's jobs live in .gitlab/ci/<stage>.yml and the shared templates in .gitlab/ci/templates.yml. Every job runs a command a developer can run locally (the commands table in README.md).

Jobs

The rollback job, failing image scans and the published chart are planned; until then a rollback uses the commands under Rolling back.

What the runner and the project need

• A runner that allows privileged services, for the Docker-in-Docker service of dotnet:test and publish:images.
• The project's container registry enabled, or another registry (see Images and registry).
• Network access to:
  • NuGet and the npm registry;
  • GitHub (the chart check downloads kubeconform and helm-unittest with verified checksums);
  • the Kubernetes release site (kubectl);
  • the Trivy databases;
  • Docker Hub and the Microsoft Container Registry (the job images).
• For the deploy jobs, a runner that can reach the cluster's API server.

Images and registry

publish:images pushes to COLDEN_IMAGE_REGISTRY with COLDEN_PUSH_USER and COLDEN_PUSH_PASSWORD. When those are not set, it pushes to the project's own registry ($CI_REGISTRY_IMAGE) with the job's credentials. The deploy jobs set the chart's global.image.registry to the same registry, and its global.image.tag to this pipeline's sha-<commit>.

Deploying

A project describes its environments with these variables; the deploy jobs exist only when COLDEN_DEPLOY is true.

Mark every secret Protected and Masked, and scope the application Secret's keys to their environment (nonprod or prod). Protected variables reach only main and protected tags, which is why the deploy jobs exist only there.

A deploy job, in its environment's resource_group so two deploys of one environment never overlap:

1. Checks its inputs: the kubeconfig is not empty, and the namespace, the host and the values file are set.
2. Runs tools/bootstrap-namespace.sh <namespace>. It checks the variables (a missing required key or a key of the wrong length stops the job without printing a value), then applies the namespace (or checks it exists), the pull secret and the application Secret colden-secrets, with server-side apply.
3. Runs helm upgrade --install colden dist/colden-*.tgz -n <namespace> -f <values> --set global.image.registry=<registry> --set global.image.tag=sha-<commit> --wait --timeout 10m --atomic. The migrations run first as hooks, then the pods roll; a failure rolls the release back to the previous revision.
4. Restarts the release's Deployments when the Secret changed: the pods read it only when they start.
5. Shows helm history and kubectl get deploy,job,ingress in the job log.

A developer prepares a namespace the same way with the variables exported in their shell (tools/bootstrap-namespace.sh <namespace>), or checks them first with --dry-run, which lists the keys and needs no cluster.

Rolling back

A failed deploy rolls itself back (--atomic). To return to an earlier image, run the deploy job of the older pipeline whose image you want: it deploys that pipeline's sha-<commit>. To return to an earlier revision of the release, helm -n <namespace> history colden lists them and helm -n <namespace> rollback colden <revision> restores its Deployments and ConfigMaps. Migrations are not undone, so each release keeps the database usable by the previous one.

The cloverheap group

In projects under cloverheap/, .gitlab-ci.yml includes .gitlab/ci/cloverheap.yml, which refines the kit's pipeline for the operator's setup described in cloverheap/ci-cd:

• Every job extends .use-local-runners from cloverheap/ci-cd: it runs on the operator's self-hosted runners and reuses the images already there (pull_policy: if-not-present). The SAST analyzers come from GitLab's template and keep its runner choice.
• The images go to the group's registry (CH_GITLAB_DOCKER_REGISTRY) with the group's registry credentials (CH_RW_REGISTRY_USER, CH_RW_REGISTRY_TOKEN), which also make the cluster's pull secret gitlab-docker-credentials.
• The deploy jobs use the group's kubeconfig (CH_K8S_LOCAL_KUBECONFIG). They check the namespace the operator created, and describe Cloverheap's environment prod, the live demo. Cloverheap has no nonprod yet: deploy:nonprod is replaced by a stand-in that prints success, so deploy:prod keeps its place after it.
• The application Secret's keys are not GitLab variables there. .gitlab/ci/cloverheap.yml includes cloverheap/ci-cd's resolver of Azure Key Vault references (gitlab-templates/azurekv/main.yml, .ch-azurekv), and before step 2 the deploy job resolves the environment's references file (COLDEN_PROD_SECRETS, infra/helm/environments/cloverheap/references/prod.yaml) into its environment, where tools/bootstrap-namespace.sh reads the keys as it reads variables.

docs/deployment.md lists those environments, their secrets and the one-time setup.

Pinned images and tools

.gitlab-ci.yml pins every image by tag and digest:

• the .NET SDK;
• Node.js 22 on Alpine;
• Docker 29 with Docker-in-Docker;
• alpine/helm 3.22.1;
• aquasec/trivy 0.76.0;
• Playwright 1.64.1 on Ubuntu.

kubectl 1.37.1 and, for chart:smoke, kind 0.33.0 are downloaded and checked against their SHA-256; keep kubectl within one minor version of the clusters it deploys to, and kind's node image (kindest/node 1.34, by digest) at the Kubernetes version tools/helm-check.sh validates against. tools/helm-check.sh pins kubeconform and helm-unittest the same way.

Changing the pipeline

• A new job goes into the file of its stage and runs a command from README.md, so it can be run locally first.
• It extends .colden-job, or .colden-check for a build, test or scan job, directly or through a template, so a runner setup reaches it.
• A new image is pinned by digest in .gitlab-ci.yml and named in this guide.
• A new secret key of the application Secret goes into the same change in four places:
  • tools/bootstrap-namespace.sh;
  • the chart's README;
  • the chart's list of secret keys;
  • this guide's table.