Kustomize for Kubernetes Deployment Configuration¶

Einordnung ins Overtime-Projekt

Overtime läuft in mehreren Umgebungen (Test und Produktion) mit leicht unterschiedlicher Konfiguration. Dieses ADR begründet, warum Kustomize-Overlays statt vollständiger Manifest-Duplizierung oder Helm-Templating genutzt werden.

1. Kontext & Problemstellung (Deutsch)¶

Wenn dieselbe Applikation auf mehrere Cluster ausgerollt wird (z. B. overtime-staging für Tests und overtime-prod für den Live-Betrieb), erfordert jede Umgebung spezifische Anpassungen: unterschiedliche Image-Tags, abweichende Ressourcen-Limits (CPU/RAM), andere Replica-Anzahlen, umgebungsspezifische Ingress-Hostnamen und individuelle Geheimnisse (Sealed Secrets).

Diese Abweichungen müssen im GitOps-Repository abgebildet werden, ohne dass die vollständigen YAML-Manifeste pro Umgebung kopiert und dupliziert werden (DRY-Prinzip).

2. Architecture Decision Record (English)¶

ADR-008: Kustomize for Kubernetes Deployment Configuration¶

Status¶

Accepted

Context and Problem Statement¶

Deploying the same application to multiple clusters requires per-environment differences: different image tags, resource limits, replica counts, ingress hostnames, and sealed secrets. These differences must be expressed without duplicating the entire manifest set.

Decision Drivers¶

DRY (Don't Repeat Yourself): Shared manifest structures should not be copy-pasted per environment.
Plain YAML Output: What ArgoCD applies must be inspectable as standard Kubernetes YAML without rendering a template first.
Native Tooling: Should preferably work with native kubectl without requiring cluster-side plugins or heavy CLI wrappers.
CI-Friendly Updates: The deployment pipeline must be able to update image tags programmatically after a successful build in a deterministic, diff-friendly way.

Considered Options¶

Plain YAML per environment — Full duplication of all manifests. Environment-specific changes risk getting lost in copy-paste noise, making drift hard to detect.
Helm — Offers powerful templating with values.yaml files. However, it introduces the Go template language on top of YAML.
Kustomize — Uses layered overlays over a common base via pure YAML patches. The kustomize build output is plain Kubernetes YAML. It has native kubectl support (via kubectl apply -k) and first-class native integration in ArgoCD.

Decision Outcome¶

Chosen: Kustomize with a base + overlays layout.

Since the author had experience in the past with ArgoCD and Helm Charts (worked well enough), he was curious to learn how the big alternative kustomize works, so it was chosen here.

A base directory defines the complete, functional shape of the application resources. Each environment gets an overlay directory containing a kustomization.yaml that applies strategic merge patches or JSON patches for environment-specific values and pins the image tags natively.

Consequences & Trade-offs¶

Positive Consequences¶

Local Verification: Running kustomize build overlays/overtime-prod | kubectl apply --dry-run=client works without any cluster connection, which is highly useful for local inspection and CI pre-checks.
Transparent GitOps: The ArgoCD diff view shows exactly which field changed between syncs, as the final output is pure YAML.
Easy Environment Scaling: Adding a new environment only requires creating a new overlay directory and adding a row to the CI deployment matrix — no changes to the base manifests are needed.
Independent Secrets: Sealed Secrets are strictly per-overlay. Each environment's secrets are managed and encrypted independently.

Negative Consequences / Trade-offs¶

Patching Complexity: Kustomize strategic merge patches can sometimes behave surprisingly on complex nested fields (e.g., container environment lists). JSON patches must be used for precise overrides.
Versioning: If the base template introduces breaking changes, the overrides would break, so one has to be careful.
Not for External Distribution: Kustomize is not suitable for highly parameterized chart-style packaging intended for external consumers, but this is not a requirement at the moment for an internal product like Overtime.

3. Implementation & Code Snippets (English)¶

The implementation relies on strict directory separation in the central gitops repo. The base provides the full application definition, while the overlay overrides only the specific fields needed for that environment.

Directory Layout¶

apps/django-backend/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml       # The full, complete deployment manifest
│   ├── service.yaml
│   └── ingress.yaml
└── overlays/
    ├── overtime-staging/     # Test environment
    │   ├── kustomization.yaml
    │   └── sealedsecret-django.yaml
    └── overtime-production/  # Production environment
        ├── kustomization.yaml
        └── sealedsecret-django.yaml

How Overlays Work (Example)¶

The Base Deployment (base/deployment.yaml) contains the fully configured Kubernetes object with defaults. Notice the generic image tag and the default replica count of 1:

# base/deployment.yaml (Abbreviated)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: django-backend
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: django-backend
          image: registry.gitlab.com/alexb-overtime/overtime-django-backend:latest
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
# ... readiness probes, security contexts, etc. are fully defined here

The Production Overlay (overlays/overtime-production/kustomization.yaml) imports this base and applies specific changes.

Kustomize has built-in support for overriding images without writing raw patches. For deeper structural changes (like increasing replicas to 2 or changing the Ingress hostname), standard JSON patches are used:

# overlays/overtime-production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: django-backend

resources:
- ../../base
- sealedsecret-django.yaml

# Native Kustomize feature to easily override the image and tag
images:
- name: registry.gitlab.com/alexb-overtime/overtime-django-backend
  newName: registry.gitlab.com/alexb-overtime/overtime-django-backend
  newTag: v1.1.8

patches:
# Override Ingress Hostname
- patch: |-
    - op: replace
      path: /spec/rules/0/host
      value: api.overtime.example.com
  target:
    kind: Ingress
    name: django-backend
# Override Replica Count for Production
- patch: |-
    - op: replace
      path: /spec/replicas
      value: 2
  target:
    kind: Deployment
    name: django-backend

Image Tag Updates in CI¶

.update-gitops:
  stage: deploy
  image: registry.gitlab.com/alexb-overtime/overtime-ci-templates/gitops-tools:5.8.1@sha256:35cd40a053946a5a130da9d1360e5872d1b2e3b002d58b2ebfa6bbc7e019eae5
  before_script:
    - git config --global user.email "ci-bot@overtime.local"
    - git config --global user.name "Overtime CI Bot"
    ...
  script:
    - EFFECTIVE_TAG="${IMAGE_TAG:-$CI_COMMIT_SHORT_SHA}"
    - git clone --depth=1 "https://gitlab.com/alexb-overtime/overtime-argocd.git" /tmp/gitops
    - TARGET_DIR="/tmp/gitops/${OVERLAY_PATH}/${TARGET_ENV}"
    - |
      if [ ! -d "$TARGET_DIR" ]; then
        echo "ERROR: overlay ${TARGET_DIR} does not exist in overtime-argocd."
        exit 1
      fi
    - cd "$TARGET_DIR"
    - kustomize edit set image "${IMAGE_NAME}=${IMAGE_NAME}:${EFFECTIVE_TAG}"
    ...
    - git commit -m "deploy: update ${IMAGE_NAME##*/} to ${EFFECTIVE_TAG}"
    - git push origin main
    ...

This creates a clean, deterministic git commit modifying only the newTag value. ArgoCD detects this commit and syncs the new image to the cluster automatically. (Kustomize version 5.8.1 is bundled in the pipeline and kept up to date via Renovate).

4. Compliance Einordnung (Deutsch)¶

Das detaillierte Mapping auf BSI IT-Grundschutz, CRA und CIS Benchmarks wird später ergänzt.