Quick Start

From zero to first PR in under an hour. Follow this sequence — no archaeology of the repo required.

bash scripts/env-checker.sh

make hooks
# equivalent to:
pre-commit install
pre-commit install --hook-type pre-push

make dev
# Creates kind cluster "devops-playbook"
# Starts local registry at localhost:5001
# Installs ingress-nginx via Helm
# Applies dev Kustomize overlay

git checkout -b feat/add-service-name
make lint          # run all pre-commit hooks
git commit -m "feat(api): add health endpoint"
git push origin feat/add-service-name

Repository Structure

Organized by functional concern first, then by platform and technology — so you ask "I need to deploy to AKS" and navigate directly there, not "what type of file is this".

Design Principles

Dev Environment

Two modes: the devcontainer (recommended for onboarding and pair programming — guarantees identical toolchain) or local machine with manual tool installation.

# All tools are pre-installed in the container:
# kubectl, helm, kind, terraform, pre-commit, ruff, mypy
# hadolint, checkov, gitleaks, cosign, velero, yq, jq

# Config: .devcontainer/devcontainer.json
# Dockerfile: .devcontainer/Dockerfile
# Post-create: .devcontainer/scripts/post-create.sh

# 1. Verify all tools
bash scripts/env-checker.sh

# 2. Start local Kubernetes cluster
make dev
# → kind cluster "devops-playbook" with registry + ingress

# 3. Install git hooks
make hooks

# 4. Run all linters
make lint

# 5. Deploy to local cluster
make deploy-dev

# 6. Check status
make k8s-status

# 7. Tear down
make teardown

# Open: .devcontainer/gpu/devcontainer.json
# Requires: NVIDIA drivers + Docker GPU pass-through

# Validate GPU visibility
nvidia-smi

# Start Jupyter
python3 -m jupyter lab --ip=0.0.0.0 --no-browser

Golden Paths

Golden paths are opinionated, end-to-end workflows with exact file references and enforced guardrails. They eliminate decision fatigue and encode hard-won production experience. Pick your scenario and follow the steps — the right templates, security gates, and operational hooks are already connected.

Kubernetes Microservice — End to End

The most complete path: backend API or microservice from local dev to production on any cloud-managed Kubernetes cluster. Every step names the exact file to copy or edit.

File Map by Step

Kubernetes Security Baseline (Kyverno-Enforced)

The following fields are required on every pod. Kyverno will block deployments missing any of them in production namespaces:

FinOps Checkpoints (Pre-Production)

Three checkpoints are embedded into this path and are not optional before promoting to production:

# Checkpoint 1: Verify cost labels are present
python finops/scripts/validate-cost-tags.py --namespace <your-ns>

# Checkpoint 2: Review VPA rightsizing (after 24h in staging)
python finops/scripts/analyze-rightsizing.py --namespace <staging-ns>

# Checkpoint 3: Confirm budget headroom
# Check: Grafana → FinOps — Budget Tracking dashboard

SLO-Driven Development

Translate reliability requirements into measurable targets, automated alerts, and an engineering process that balances feature velocity with operational risk.

Core Concepts

Burn-Rate Alert Tiers

Four alert tiers are required. The two-window approach (fast + slow detection per tier) reduces both false positives and missed gradual degradations:

Implementation Steps

# 1. Define your SLO (copy + fill the schema)
cp observability/prometheus/slos/slo-schema.yaml \
   observability/prometheus/slos/my-service-slo.yaml

# 2. Write recording rules (replace "api-gateway" with your service)
cp observability/prometheus/recording-rules/slo-burn-rates.yaml \
   observability/prometheus/recording-rules/my-service-slo-burn-rates.yaml
kubectl apply -f observability/prometheus/recording-rules/my-service-slo-burn-rates.yaml

# 3. Deploy burn-rate alerts
cp observability/prometheus/alerts/slo-burn-rate-alerts.yaml \
   observability/prometheus/alerts/my-service-slo-burn-rate-alerts.yaml
kubectl apply -f observability/prometheus/alerts/my-service-slo-burn-rate-alerts.yaml

# 4. Deploy the SLO dashboard ConfigMap (auto-discovered by Grafana)
kubectl apply -f observability/prometheus/dashboards/slo-status-configmap.yaml

# 5. Validate naming convention
make slo-validate

SLO Target Selection Guide

Most teams default to 99.9%. That is often wrong. Use these questions first:

• What is the user impact of 1 minute of downtime?
• What is your current measured baseline? (never set target above baseline)
• Can your team sustain the on-call burden of a tighter target?

Incident Response

A 5-phase ops runbook that applies to any service using this playbook. Start here when PagerDuty fires or a user reports production degradation.

Severity Levels

Phase 1 — Acknowledge & Assemble (0–5 min)

# Acknowledge PagerDuty within 5 minutes to stop escalation

# Open dedicated Slack channel: #inc-YYYY-MM-DD-<service-name>

# Post immediately:
Service: <name>
Severity: SEV-X
Started: HH:MM UTC
IC: @name
Impact: <what users see>

# Verify ownership via service catalog
cat catalog/services/<service-name>.yaml
# → spec.oncall.pagerduty_service, spec.oncall.slack_channel

Phase 2 — Triage (5–15 min)

# Layer 1: Is the service returning errors?
curl -I https://my-service.example.com/health

# Layer 2: Kubernetes health
kubectl get pods -n <namespace> -l app=<service>
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -20

# Layer 3: Recent changes (most common root cause)
kubectl rollout history deployment/<name> -n <namespace>
git log --oneline --since="1 hour ago" main

# Layer 4: Logs
kubectl logs -n <namespace> -l app=<service> --previous | tail -100

Phase 3 — Mitigate Options

Phase 5 — Post-Incident Review

Mandatory for SEV-1 and SEV-2. Complete within 48 hours. Include: timeline, root cause, contributing factors, action items with owners and due dates. Write or update the runbook: cp docs/runbooks/template.md docs/runbooks/<service>-<symptom>.md

Platform Onboarding

New team setup — get every foundation in place before any application path. Completing this path means the application paths work on first attempt.

Production Readiness Checklist

• bash scripts/env-checker.sh passes on all engineers' machines
• pre-commit install run on all engineers' clones
• Branch protection enabled with required CI checks
• OIDC federation configured — no long-lived credentials in GitHub Secrets
• Remote Terraform state backend configured
• Namespace with correct RBAC and NetworkPolicy applied
• External Secrets store configured and syncing
• Kyverno policies installed — no failures in kubectl get policyreport
• At least one alert rule deployed and routing verified
• PagerDuty on-call rotation configured
• Runbook written for top 2 failure modes with runbook_url in alert
• Velero backup schedule applied to production namespace

Mobile Backend (BFF)

Backend for Frontend pattern for iOS/Android apps — covering the concerns that make mobile different from a standard API: versioning, PKCE auth, push notifications, and aggressive client retry protection.

Key Mobile-Specific Requirements

Mobile-Specific Prometheus Metrics

Multi-Tenant SaaS

Namespace-per-tenant isolation model. Scales to ~100 tenants per cluster before control plane overhead becomes significant. Beyond that, provision additional clusters.

Isolation Layers

Tenant Onboarding Flow

Add a YAML record to tenants/<slug>.yaml with 6 required fields. CI detects the new file and creates exactly 6 resources per tenant: namespace, RBAC, NetworkPolicy, ESO store config, database schema, and ArgoCD Application. The job is idempotent — re-running is safe.

Tenant Offboarding (Order Matters)

Never automate schema drops. The required sequence:

1. Set status: offboarding in tenant YAML → merge PR
2. Export billing metrics → take Velero namespace snapshot
3. Delete ArgoCD Application
4. Delete namespace
5. Manual DBA step: Drop tenant schema (requires explicit approval)
6. Remove tenant secrets from secrets store

Service Catalog Registration

Every production service must be registered in the Git-native catalog with owner, on-call routing, runbook link, SLO file, and cost center. CI validates metadata integrity. Kyverno can block deployments from unregistered services in production namespaces.

Required Service Fields

# catalog/services/<service-name>.yaml
metadata:
  name: api-gateway          # must match Deployment app label + filename
  owner: platform-team       # must exist in catalog/teams/
  tier: tier-1               # tier-1 | tier-2 | tier-3
spec:
  oncall:
    pagerduty_service: "..."
    slack_channel: "#alerts-api"
  slo:
    definition_file: "observability/prometheus/slos/api-gateway-slo.yaml"
  cost_center: engineering   # must exist in finops/config/budgets.yaml
  runbook: "docs/runbooks/api-gateway-crashloop.md"

Validation & Automation

# Validate locally (strict mode, skip URL checks)
python catalog/scripts/validate-catalog.py --strict --skip-url-check

# Regenerate .github/CODEOWNERS from team ownership metadata
python catalog/scripts/generate-codeowners.py --output .github/CODEOWNERS
make catalog-codeowners

# Export to Backstage format (when service count approaches 50)
python catalog/scripts/migrate-to-backstage.py --output-dir backstage/catalog

CI Core Concepts

Every CI pipeline in this repo follows the same conceptual model regardless of platform — build once, test in isolation, scan in parallel, deploy immutable artifacts. Platform-specific syntax differs; the stages and contracts do not.

Universal Pipeline Stages

Reusable vs Standalone Workflows

OIDC vs Static Credentials

CI Templates

Copy the workflow file for your stack into .github/workflows/ and add security scanning as a parallel job. All pipelines emit the same stage contract regardless of language.

# Minimal wiring example: build → security scan
jobs:
  build-test:
    uses: ./.github/workflows/reusable-docker-build.yml
    with:
      image-name: my-service
      dockerfile: docker/python/Dockerfile.fastapi
    permissions:
      id-token: write   # Required for OIDC
      contents: read

  security-scan:
    needs: build-test
    uses: ./.github/workflows/reusable-security-scan.yml
    with:
      image: my-service:${{ github.sha }}

Deployment Targets

Choose your deployment target based on whether you need Kubernetes features. If unsure, start with the decision tree:

GitOps & ArgoCD

GitOps treats Git as the single source of truth for cluster state. A controller running inside the cluster continuously reconciles actual state with desired state from Git — no push-based deployments from CI, no manual kubectl apply in production.

GitOps Principles

ArgoCD Application Patterns

Promotion Flow

# 1. CI builds image, tags with Git SHA
# 2. CI updates dev overlay: image tag → new SHA
#    cd/kubernetes/_overlays/dev/kustomization.yaml
# 3. ArgoCD syncs dev automatically
# 4. After testing: PR to update staging overlay
# 5. After staging approval: PR to update prod overlay
# 6. ArgoCD syncs prod (with manual sync gate)

# Verify sync status
kubectl get applications -n argocd
kubectl describe application <app-name> -n argocd | grep -A 20 "History:"

Kubernetes Concepts & Patterns

Core workload patterns and the decision logic for choosing between them. All base manifests are in cd/kubernetes/_base/; reusable patterns for specific scenarios are in cd/kubernetes/_patterns/.

Workload Type Selection

Autoscaling Decision

Helm vs Kustomize

Deployment Strategies

Base Manifests & Overlays

Start from the base manifests and layer environment differences with Kustomize overlays. Never duplicate full manifests per environment — only patch what differs.

Base Manifest File Map

Kustomize Overlay Structure

cd/kubernetes/
  _base/                    # Shared defaults (all environments)
  _overlays/
    dev/kustomization.yaml  # replicas: 1, relaxed resources, mutable tags
    staging/kustomization.yaml
    prod/kustomization.yaml # replicas: 3, tight PDB, pinned image SHA

# Preview what Kustomize will generate (validate before applying)
kubectl kustomize cd/kubernetes/_overlays/dev

# Run policy checks locally
conftest test cd/kubernetes/_base/ --policy policy/conftest/kubernetes

Kyverno Policy Engine

Kyverno enforces governance at Kubernetes admission time — before a resource is admitted to the cluster. It complements static analysis (Checkov, Conftest) which runs earlier in CI. These operate at different lifecycle stages and are not alternatives.

Policy Modes

Installed Policies

# Check violations in your namespace
kubectl get policyreport -n <your-namespace>
kubectl describe policyreport -n <your-namespace> | grep -A 5 "fail"

# Check cluster-wide (requires cluster-admin)
kubectl get clusterpolicyreport

Terraform

All cloud infrastructure is provisioned declaratively via Terraform modules. Each cloud target is a self-contained module with its own state key. Bootstrap the remote state backend once before using any other module.

Bootstrap Remote State (Required First)

# Run ONCE per cloud account, by a human with admin access (not CI)

# AWS — creates S3 bucket + DynamoDB lock table
cd terraform/_bootstrap/aws
terraform init && terraform plan -out=tfplan && terraform apply tfplan

# Azure — creates Storage Account + Blob container
cd terraform/_bootstrap/azure
terraform init && terraform apply

# GCP — creates GCS bucket
cd terraform/_bootstrap/gcp
terraform init && terraform apply

# Then uncomment the backend block in your workload module's main.tf
# and run: terraform init -migrate-state

Cloud Modules

Database Backup Modules (Include with Cluster)

State Strategy

This repo uses separate state keys per module (e.g., prod/eks/terraform.tfstate) rather than Terraform workspaces. Workspaces share provider configuration — this makes it harder to use different cloud accounts per environment, which is the recommended security posture.

Environment Strategy

Three environments with distinct purposes, triggers, and data policies. Never bake environment config into images — externalize via Kubernetes ConfigMaps, Helm values files, or Kustomize overlays.

Production Access Controls

• No direct kubectl exec to production pods
• All changes via Git PR (GitOps) — ArgoCD applies, never CI push
• Break-glass procedures documented and audited in secops/runbooks/
• RBAC: least-privilege service accounts — use cd/kubernetes/_base/rbac/ci-deployer.yaml
• Separate Kubernetes namespaces per environment

OIDC / Keyless Cloud Authentication

OpenID Connect federation lets GitHub Actions authenticate directly to cloud providers with short-lived, auto-rotating tokens — no secrets stored in GitHub, no rotation to manage, full audit trail per workflow run.

Setup by Cloud

# Step 1: Create OIDC provider (once per AWS account)
aws iam create-open-id-connect-provider \
  --url "https://token.actions.githubusercontent.com" \
  --client-id-list "sts.amazonaws.com"

# Step 2: Create IAM role with trust policy (restrict to your repo)
# Condition: "token.actions.githubusercontent.com:sub": "repo:YOUR-ORG/YOUR-REPO:*"

# Step 3: Add GitHub secret
# AWS_DEPLOY_ROLE_ARN = arn:aws:iam::123456789012:role/github-actions-deploy

# Step 4: Use in workflow
permissions:
  id-token: write    # Required for OIDC
  contents: read
steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}
      aws-region: us-east-1

# Step 1: Create app registration + service principal
az ad app create --display-name "github-actions-deploy"
az ad sp create --id $APP_ID

# Step 2: Add federated credential (restrict to main branch)
# subject: "repo:YOUR-ORG/YOUR-REPO:ref:refs/heads/main"

# Step 3: Add GitHub secrets
# AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_SUBSCRIPTION_ID

# Step 4: Use in workflow
permissions:
  id-token: write
steps:
  - uses: azure/login@v2
    with:
      client-id: ${{ secrets.AZURE_CLIENT_ID }}
      tenant-id: ${{ secrets.AZURE_TENANT_ID }}
      subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

# Step 1: Create Workload Identity Pool
gcloud iam workload-identity-pools create "github-actions-pool" \
  --location="global"

# Step 2: Create OIDC provider with attribute condition
# --attribute-condition="assertion.repository_owner == 'YOUR-ORG'"
# Critical: without this, ANY GitHub repo could request tokens

# Step 3: Grant Service Account access to pool
# --member="principalSet://iam.googleapis.com/${POOL_ID}/attribute.repository/YOUR-ORG/YOUR-REPO"

# Step 4: Use in workflow
permissions:
  id-token: write
steps:
  - uses: google-github-actions/auth@v2
    with:
      workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
      service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}

Common Troubleshooting

Shift-Left Security Scanning

Security checks run at three stages: locally via pre-commit hooks, in CI on every push, and at cluster admission via Kyverno. Each layer catches different classes of issues — they are complements, not alternatives.

Scanner Selection Guide

When a Scanner Fires

Secrets Management

Secrets must never be stored in Git — even in private repos. Every secret needs a source of truth in a cloud-managed secret store, a sync mechanism into Kubernetes, a rotation policy, and an offboarding procedure.

Where Secrets Live

External Secrets Operator

ESO watches ExternalSecret resources in the cluster and syncs values from your cloud secret store into Kubernetes Secrets automatically. The source of truth stays in the cloud store — Kubernetes Secrets are ephemeral replicas.

# 1. Apply SecretStore for your cloud (edit annotations first)
kubectl apply -f secrets/external-secrets/aws-secret-store.yaml
# or: azure-secret-store.yaml | gcp-secret-store.yaml

# 2. Create your ExternalSecret
cp secrets/external-secrets/example-external-secret.yaml \
   secrets/external-secrets/my-app-secrets.yaml
kubectl apply -f secrets/external-secrets/my-app-secrets.yaml

# 3. Verify sync (STATUS should be "SecretSynced")
kubectl get externalsecret my-app-secrets -n <namespace>

Secret Lifecycle Guides

Anti-Patterns to Avoid

• ❌ .env files committed to Git
• ❌ Secrets in docker-compose.yml
• ❌ Long-lived service account keys / personal access tokens
• ❌ Same secret in dev, staging, and prod
• ❌ Secrets in container image layers
• ❌ Secrets in Kubernetes ConfigMaps (base64 is not encryption)

Runtime Security

Shift-left scanning catches known vulnerabilities before deployment. Runtime security detects threats and anomalous behavior in live environments — attacks that only appear after a workload is running.

Falco

Falco watches kernel syscalls and Kubernetes audit events for suspicious behavior. Custom rules are in secops/runtime/falco/rules/custom-rules.yaml. Alerts route to Slack or PagerDuty via secops/runtime/falco/rules/alerts.yaml.

Audit Logging

Kubernetes API server audit logs are shipped to Loki via Promtail (secops/runtime/audit-logging/loki-shipper.yaml). The audit policy (secops/runtime/audit-logging/audit-policy.yaml) records all resource modifications and secret access at the metadata level.

SecOps Runbooks

Compliance as Code

Machine-readable control libraries map specific Kyverno policies and cluster checks to compliance framework controls. Evidence is collected automatically and scored weekly — turning audits into continuous engineering practice.

Supported Frameworks

Evidence Pipeline

# Run evidence collection manually
bash secops/compliance/scripts/collect-evidence.sh --output-dir ./evidence

# Generate compliance report (fails if score < 85%)
python secops/compliance/scripts/generate-compliance-report.py \
  --control-library-dir secops/compliance/control-library \
  --evidence-dir ./evidence \
  --fail-below 0.85 \
  --format markdown

# CI gate equivalent
make compliance-report

Observability Stack

The three pillars of observability — metrics, logs, and traces — each answer different operational questions. Instrument all three from day one; correlate them with shared labels and trace IDs to cut mean-time-to-diagnosis by orders of magnitude.

Signal → Question Matrix

Install Order

Install in this order — each layer builds on the previous:

OpenTelemetry Language Env Vars

Trace Sampling Rates

Override with OTEL_TRACES_SAMPLER_ARG in your Helm values or Kustomize overlay. See observability/opentelemetry/env-vars/ for language-specific env files.

SLOs & Error Budgets

File Layout

Error Budget Policy Template

# docs/slo-policies/<service>-error-budget-policy.yaml
service: api-gateway
slo_target: 99.9%
error_budget_monthly_minutes: 43.8

policy:
  above_50_percent_remaining:
    action: normal operations, proceed with planned changes
  below_25_percent_remaining:
    action: pause non-critical feature work, prioritize reliability
    owner: engineering manager
  below_10_percent_remaining:
    action: halt all deployments except rollbacks and critical fixes
    owner: engineering director approval required
  exhausted:
    action: incident response, all hands on reliability
    escalation: VP Engineering

Alert Routing

Alerts route to different channels based on severity. Configure receivers in the Alertmanager config in observability/prometheus/values.yaml under alertmanager.config.

Available Notification Integrations

# Test alert routing before going live
kubectl -n monitoring port-forward svc/kube-prometheus-stack-alertmanager 9093:9093
curl -XPOST http://localhost:9093/api/v1/alerts \
  -H 'Content-Type: application/json' \
  -d '[{"labels":{"alertname":"TestAlert","severity":"warning","team":"my-team"}}]'

FinOps — Cost Governance & Optimization

A production-grade FinOps system that integrates cost visibility, governance, and optimization into the CI/CD lifecycle — from Terraform PRs through to monthly chargeback reports. Cost control is an engineering discipline, not a monthly finance surprise.

Required Cost Labels (Enforced at Admission)

# All pods must carry these labels — Kyverno blocks any missing them
metadata:
  labels:
    app: my-service
    finops.org/costcenter: "engineering"     # Must match finops/config/budgets.yaml
    finops.org/environment: "production"     # dev | staging | production
    finops.org/team: "platform"             # Optional but recommended
    finops.org/project: "my-service"        # Optional

FinOps Quick Start

# 1. Install cost monitoring (Kubecost or OpenCost)
./finops/scripts/install-cost-monitoring.sh --tool kubecost

# 2. Deploy Kyverno policies in audit mode first (non-blocking)
./finops/scripts/deploy-policies.sh --audit-mode

# 3. Import all 9 Grafana dashboards
export GRAFANA_API_KEY=your-key
./finops/scripts/deploy-dashboards.sh

# 4. Validate cost label compliance
python finops/scripts/validate-cost-tags.py --all-namespaces

# 5. Find rightsizing opportunities
python finops/scripts/analyze-rightsizing.py --all-namespaces

FinOps Optimization Loop

The optimization loop closes the gap between cost alert and verified savings with a repeatable, automated workflow. Every step produces artifacts — patches, PR descriptions, advisor reports — so engineers don't start from scratch each time.

Alert Response Guide

Safety Guardrails in Optimization Scripts

• Never reduce memory limits > 50% in one change (use --allow-aggressive to override with justification)
• Never recommend limits below VPA lower bound
• Warn on single-replica workloads with strict PDB before reducing resources
• Never commit reserved capacity for workloads < 3 months old
• Reserved capacity commitments > $10,000/year require leadership approval (documented in finops/docs/optimization-runbook.md)

Scripts Reference

Database Migration Patterns

All migrations must be backwards-compatible, idempotent, and tested locally before production. These three golden rules prevent the most common zero-downtime deployment failures.

Pattern Selection

Expand / Contract Pattern

Never drop or rename a column in the same release that removes the code using it. Old pods are still running when new pods start:

Zero-Downtime Checklist

• Migration is backwards-compatible with the old app version
• Migration is idempotent (safe to run twice)
• Rollback migration script written and tested
• activeDeadlineSeconds set on the Job or init container
• Database backup taken within the last 24 hours
• Migration tested on staging with production-scale data volume

Versioning Strategy

Two automated release tools are included. Both require Conventional Commits format — commit messages are parsed to determine version bumps and generate changelogs automatically.

Semantic Versioning (SemVer)

Format: MAJOR.MINOR.PATCH — recommended for libraries and APIs.

Image Tagging Rules

# Tag with Git SHA (always unique, traceable)
docker build -t my-app:sha-$(git rev-parse --short HEAD) .

# Also tag with semver on release
docker tag my-app:sha-abc1234 my-app:1.2.3
docker tag my-app:sha-abc1234 my-app:1.2

Branching Strategy

Branch Protection (Required on main)

• Require pull request with at least 1 reviewer
• Require status checks to pass (CI workflow names)
• Require branches to be up to date before merging
• No force push; no deletion

Runbooks

Action-oriented operational procedures for incidents and routine operations. Every alert in production must link to a runbook via the runbook_url annotation on the PrometheusRule.

Included Runbooks

Write a New Runbook

# Copy the template
cp docs/runbooks/template.md docs/runbooks/<alertname>.md

# Name the file to match the alert name exactly (lowercase, no spaces)
# e.g.: docs/runbooks/highlatencyp99.md

# Link it from the PrometheusRule
annotations:
  runbook_url: "https://github.com/your-org/repo/blob/main/docs/runbooks/<alertname>.md"

# Commit to docs/runbooks/ so it appears in PR review

Architecture Decision Records

ADRs capture why decisions were made — not just what was decided. They prevent repeated debates and preserve context when team members change. All ADRs live in docs/decisions/.

Makefile & Taskfile Targets

All common commands are abstracted behind make targets. Run make help to see the full list at any time.

Supply Chain Security

Keyless signing, SBOM attestation, SLSA provenance, and Kyverno admission verification — wired as a standard delivery path. Zero private keys are stored anywhere.

Prerequisites

Implementation Steps

# Step 1: Configure OIDC first (prerequisite for keyless signing)
# See docs/guides/github-actions-oidc.md

# Step 2: Apply supply chain policies
kubectl apply -f secops/supply-chain/cosign-verify-policy.yaml
kubectl apply -f secops/supply-chain/sbom-policy.yaml
kubectl apply -f secops/supply-chain/slsa-verify.yaml

# Step 3: Wire the reusable workflow (copy from dotnet example)
# ci/github-actions/dotnet/supply-chain-integration.yml

# Step 4: Start in audit mode — non-blocking migration
kubectl label namespace <ns> policy.kyverno.io/supply-chain=audit --overwrite

# Step 5: Verify signature
cosign verify <image> \
  --certificate-identity-regexp 'https://github.com/YOUR-ORG/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

# Step 6: Verify SBOM attestation
cosign verify-attestation \
  --certificate-identity-regexp 'https://github.com/YOUR-ORG/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --type spdx <image>

# Step 7: Graduate to enforce (after all violations resolved)
kubectl label namespace <ns> policy.kyverno.io/supply-chain- --overwrite

# Step 8: Monitor compliance
kubectl get policyreport -A

Policy Guardrails

FinOps Optimization

Close the loop from a cost alert to a merged PR to verified savings with a repeatable, automated workflow. Every step produces concrete artifacts — no starting from scratch each time.

python finops/scripts/normalize-cloud-costs.py --by team

python finops/scripts/analyze-rightsizing.py --namespace <ns>

python finops/scripts/generate-optimization-pr.py \
  --namespace <ns> --min-savings 20

kubectl apply -k optimization-patches/<ns>/

Reserved Capacity (Quarterly Review)

# Risk-scored reserved instance recommendations
python finops/scripts/reserved-capacity-advisor.py \
  --min-savings 500 --max-risk medium --term 1yr

# Safety rules:
# - Never recommend 3-year for workloads < 3 months old
# - Commitments > $10k/year require leadership approval
# - Always test in staging 24h before production changes

Disaster Recovery

Recovery procedures for the three most common disaster scenarios. Use the decision tree to identify which section applies and follow the steps in order.

RTO / RPO Summary

Section 1 — Cluster Recovery

# 1. Provision new cluster from existing Terraform
terraform -chdir=terraform/aws-eks apply \
  -var="project=<project>" -var="environment=production"

# 2. Install Velero (pointing to same backup bucket)
bash backup/velero/aws-install.sh

# 3. Restore latest scheduled backup
velero backup get
velero restore create \
  --from-schedule daily-full-backup \
  --restore-volumes=true
velero restore describe <restore-name> --details

# 4. Verify ESO re-syncs secrets automatically
kubectl get externalsecret -A

# 5. Validate cluster health
kubectl get nodes
kubectl get pods -A
curl -f https://my-app.example.com/health

Section 2 — Database Recovery (AWS RDS PITR)

# Find last good timestamp before corruption event
aws rds describe-db-instances \
  --db-instance-identifier rds-<project>-production

# Restore to new instance
aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier rds-<project>-production \
  --target-db-instance-identifier rds-<project>-production-restored \
  --restore-time "2026-01-15T14:30:00Z"

# Wait for availability (~10-20 minutes)
aws rds wait db-instance-available \
  --db-instance-identifier rds-<project>-production-restored

# Update connection string in AWS Secrets Manager, then restart pods

Post-Recovery Checklist

• All pods are Running (no CrashLoopBackOff)
• Ingress endpoints respond with HTTP 200
• Database connection strings point to recovered instance
• ExternalSecrets are Synced (kubectl get externalsecret -A)
• Monitoring/alerting is active (Prometheus targets healthy)
• On-call acknowledged — incident ticket updated with timeline
• Post-mortem scheduled within 48 hours

Concepts & Glossary

Canonical terminology used consistently across all templates, guides, and review language in this repository.