Eve library

Build-Once Promotion with Argo CD

Guide

Revision 1

On this page

Lilith's delivery pipeline adheres to a "build once, promote everywhere" standard. Container images are compiled exactly once inside the CI workflow, attested, and then promoted unchanged across the dev → staging → prod environments. In short, we build once and promote the artifact across dev → staging → prod without ever rebuilding it. Argo CD consumes Git-hosted configuration that advertises the committed release manifest and environment-specific overrides, guaranteeing that every promotion reuses the identical digests that passed the previous gate.

Release Manifests#

During the .github/workflows/ci.yaml run on main, the docker-build matrix captures the digest for every service image it pushes. The assemble-release-manifest job aggregates those digests into deploy/releases/release-${GITHUB_SHA}.yaml with the schema:

text
release:
  sha: <commit SHA>
  generatedAt: <ISO timestamp>
  services:
    - name: <service workspace name>
      image: ghcr.io/lilith/<service>@sha256:...

That manifest is uploaded as the release-manifest-${GITHUB_SHA} artifact and committed to the dedicated releases branch so promotion jobs can fetch it later without rebuilding images. The manifest doubles as the source of truth for environment overrides and is preserved for auditability.

Promotion Workflow#

The GitOps Deployment Promotion workflow accepts three inputs: the target environment, the release SHA (used to locate the release manifest artifact), and an optional skip_tests toggle. Promotion proceeds as follows:

  1. Fetch deploy/releases/release-${release_sha}.yaml from the releases branch (for audit history).
  2. Invoke scripts/deploy/promote-release.sh which applies the manifest digests to deploy/helm/environments/<env>/*-values.yaml, ensuring Helm charts reference repository@digest.
  3. Commit the updated values back to main so Argo CD detects the change and syncs deploy/argocd/apps/<env>.yaml (all pinned to targetRevision: main).
  4. Trigger Argo CD sync and wait for health checks before publishing a deployment summary.

Because the workflow only mutates Git-hosted configuration, every environment simply reconciles from Git history, preserving determinism and provenance.

Environment Gates#

Promotion is strictly sequential:

  • dev: smoke validation plus optional integration tests.
  • staging: executed after dev succeeds; runs the same artifact with production-like configuration.
  • prod: only promotable after the release manifest has passed staging validation. The workflow enforces the same digests so prod cannot diverge.

Each promotion writes the applied release SHA into deploy/helm/environments/<env>/values-<env>.yaml under global.releaseSha, making gates observable to downstream tooling and compliance audits.

Rollback Strategy#

Rolling back is symmetric: re-run the promotion workflow with a previously published release SHA. The script reapplies the older manifest digests, Argo CD reconciles to the prior Git commit, and the immutable containers guarantee binary parity. Cosign attestations remain valid because the images are never rebuilt between environments.

Verification#

Playwright compliance coverage (tests/ci.build-once-promotion.spec.ts) ensures the documentation, workflows, and scripts stay aligned. CI additionally uploads the manifest artifact, and deploy-time automation downloads it, so any regression that breaks the build-once guarantee fails fast.