Status#
Accepted
Context#
We deploy a growing set of Node.js/Fastify microservices to Kubernetes. Previous deployments used blue-green without pre-promotion signal and lacked automated rollback based on live health during rollout. As service count and change velocity increase, we need a safer, incremental promotion path and a post-deploy guardrail to contain risk.
Decision#
We will:
- Introduce a canary stage (10% traffic) for production releases using dedicated Helm values and canary hostnames.
- Hold a 5-minute soak window with health sampling across all targeted services before full promotion.
- Require successful canary completion before the main production rollout job proceeds.
- Add an automated rollback guard that continuously samples production health post-promotion; any regression triggers Helm rollback to the previous revision and fails the pipeline.
- Keep blue-green as the underlying deployment primitive for fast cutover while layering canary gating and rollback triggers above it.
Consequences#
- Safer releases: regressions detected in canary or immediate post-promotion are rolled back automatically.
- Longer pipeline runtime due to soak/validation steps, but improved reliability for user-facing traffic.
- Requires maintaining canary-specific ingress/Helm values and ensuring observability endpoints remain fast and stable.
- Future work: incorporate SLO/error-budget checks from observability backends to replace simple health sampling, and refine traffic weights per service risk profile.