pal-e-docs

Deployment Lessons Learned

deployment-lessons Sop

sop active deployment
Project

Hard Shutdown Survival — Layered Fault Tolerance (2026-03-06)

PC was unplugged with no graceful shutdown. Cluster came back fully operational — all pods running, all data intact, tofu plan showed no changes. Zero manual intervention required.
Why it survived — six layers:
  • systemd (Salt-managed): Salt states ensure k3s, tailscaled, and NetworkManager are enable: True. On boot, systemd starts them automatically. nftables is also enabled — loads firewall rules from /etc/nftables.conf before traffic flows (Type=oneshot).
  • k3s embedded DB: k3s stores cluster state in SQLite at /var/lib/rancher/k3s/server/db/state.db. Survives hard shutdowns because it's a file on disk. On start, k3s reads the DB and knows full desired state.
  • Kubernetes reconciliation loop: Every Deployment has restartPolicy: Always (default). When k3s starts and the controller sees 0 pods vs N desired, it recreates all pods. This is core Kubernetes — the reconciliation loop does the heavy lifting.
  • Persistent Volumes (local-path-provisioner): k3s's built-in provisioner stores PVC data at /var/lib/rancher/k3s/storage/. Plain files on disk — survive reboots. pal-e-docs SQLite, Loki data, Harbor DB, Forgejo repos, Postgres — all intact.
  • Tailscale operator: Tailscale operator pod comes back (layer 3), re-establishes the Tailscale connection, re-registers all funnels. External access restored automatically.
  • Helm release state: Helm stores release metadata as Kubernetes secrets. Since k3s state survived (layer 2), Helm/Tofu see no drift — tofu plan shows no changes.
What did NOT survive cleanly: Grafana was already in CrashLoopBackOff before the shutdown (pre-existing config bug, not shutdown-related). See todo-fix-grafana-duplicate-default-datasource.

Memory Limits

64Mi is NOT enough for Python FastAPI apps. Use 256Mi minimum. Both basketball-api and pal-e-docs needed this fix after OOM kills. Note: the SERVICE_ONBOARDING.md template still shows 64Mi as the default limit — update it when onboarding a Python service.

K8s Secrets Trailing Newlines

When creating secrets from files, \n gets included. Fix: use tr -d '\n' or --from-literal without file redirection. The SERVICE_ONBOARDING.md uses --from-literal with $(cat ...) which avoids this.

Woodpecker Variable Syntax

Use $CI_COMMIT_SHA (without curly braces) in plugin settings. Curly braces (${CI_COMMIT_SHA}) conflict with Woodpecker's compiler. This is documented in SERVICE_ONBOARDING.md.

Woodpecker Repo Activation

Activate via the Woodpecker UI (Add repository). Must activate BEFORE first push to trigger the pipeline. API alternative: POST /api/repos?forge_remote_id=N (query param, not body).

Postgres PVC Reinitialization

POSTGRES_PASSWORD only sets password on first init. To change: scale down deployment → delete PVC → scale up.

Forgejo Push Auth

For Woodpecker CI pushing to Forgejo repos, the remote URL needs user:pass embedded (HTTPS, no SSH). This is separate from ArgoCD Image Updater write-back, which uses the git-creds k8s secret.

SQLite Alembic Migrations — DANGER

SQLite cannot do transactional DDL. Each ALTER TABLE auto-commits immediately. Alembic expects transactional DDL — all steps succeed or none do. With SQLite, a multi-step migration that fails mid-way leaves the DB in a partial state: some columns added, alembic_version not stamped. Every restart retries the migration and crashes on "duplicate column name."
This has caused two production outages:
  • PR #29 (2026-02-26): is_public + page_note_id migration. Manual fix via sqlite3.
  • PR #61 (2026-03-02): note_type + status + parent_note_id + position migration. Manual fix via sqlite3 in litestream sidecar.
Fix: Migrate to Postgres. Until then, every Alembic migration on pal-e-docs is a deployment risk. See incident-2026-03-02-sqlite-migration-crash-pr61.

Helm Chart Value Keys — Verify Before Implementing

Helm charts from different maintainers use different value keys for the same concepts. Bitnami charts use extraVolumes, extraVolumeMounts, extraEnvVars. otwld (ollama-helm) uses volumes, volumeMounts, extraEnv. Community charts vary further. Using the wrong keys silently creates duplicate volumes/mounts that fail at the K8s API layer with cryptic errors like Duplicate value or must be unique.
Rule: Always run helm show values {repo}/{chart} --version {ver} before writing Terraform helm_release values. Include this in Forgejo issue specs for Helm-related work. Discovered during F12 (PR #90 → hotfix PR #91): dev agent used Bitnami conventions for an otwld chart, required a production hotfix to correct.

Service Onboarding — Port + Registry + Realm Validation (2026-03-16)

Three bugs hit in sequence during mcd-tracker-app deployment, each requiring a separate fix-push-wait cycle. All were preventable with validation checks in the issue spec or CI.
  • Harbor project name ≠ image_repo path: Terraform creates Harbor project mcd-tracker-app (from service key). Robot account scoped to that project. But pipeline pushed to mcd-tracker/app (different project). Rule: image_repo in pipeline MUST match the terraform service key. If service is mcd-tracker-app, image is mcd-tracker-app/app, not mcd-tracker/app.
  • Dockerfile EXPOSE ≠ kustomize port: Dockerfile served on port 80 (nginx default). Kustomize overlay, service, probes, and ingress all said 3000. Pod passed readiness probe on wrong port → ImagePullBackOff masked by port mismatch. Rule: Forgejo issue spec MUST state the Dockerfile EXPOSE port. QA must verify port consistency across Dockerfile, kustomize containerPort, service port, probe port, and ingress backend port.
  • Keycloak realm wrong: Dev agent guessed realm: 'pal-e' instead of realm: 'mcd-tracker'. Keycloak returned 404 on init. Rule: Forgejo issue spec MUST state the exact Keycloak realm name and client ID. Never let the agent guess realm names.
  • Ingress port stale after targeted tofu apply: tofu apply -target didn't recreate the ingress when port changed in tfvars. Required manual kubectl patch ingress. Rule: port changes require full tofu apply (not targeted) or manual ingress patch.
  • First deploy needs :latest tag: Kustomize overlay starts with newTag: latest. Pipeline only pushed :SHA. ArgoCD couldn't find the image. Rule: pipeline must push both latest and SHA tags. The image updater will switch to SHA-only after first deploy.

keycloak-js check-sso Redirect Trap (2026-03-16)

keycloak.init({ onLoad: 'check-sso' }) WITHOUT silentCheckSsoRedirectUri does a full-page redirect to Keycloak — not a hidden iframe check. The page navigates away, Keycloak checks for a session, then redirects back. If the realm/client is misconfigured, the user sees a Keycloak error page. Fix: Either provide a silentCheckSsoRedirectUri pointing to a static callback HTML file (for iframe-based silent check), or remove onLoad entirely and trigger auth on demand via keycloak.login(). For SPAs with public landing pages, on-demand auth is the correct pattern.

Terraform State Lock — Stale Lock from Crashed Apply (2026-03-17)

Pipeline #80 on pal-e-platform crashed mid-tofu apply (OOM or timeout), leaving a stale state lock on the Kubernetes backend secret in tofu-state namespace. Every subsequent merge to main triggered CI, which immediately failed with "the state is already locked" — blocking ALL platform deployments for ~2 hours. Root cause: The Kubernetes state backend has no lock TTL — locks persist until explicitly released. A crashed process never releases its lock. Fix: (1) Immediate: tofu force-unlock -force <LOCK_ID> after confirming no active apply. (2) Preventive: Added lock-aware retry logic to CI apply step (Phase 17b.1) — detects lock errors, extracts lock ID, auto-unlocks, retries once. (3) Future: Consider remote backend with native lock TTL when project count exceeds 3. Key lesson: Any CI pipeline that runs tofu apply on merge MUST handle stale locks, or a single crash blocks the entire deployment pipeline.

ArgoCD Image Updater — Don't Manually Deploy (2026-05-29)

Attempted to manually deploy palinks by editing kustomization.yaml image tag in pal-e-deployments and running kubectl apply -k. This failed because (1) the SOPS-encrypted secret can't be applied without decryption — only ArgoCD with the kustomize-sops CMP plugin can do that, and (2) ArgoCD Image Updater already handles image tag updates automatically. Root cause: Didn't know the platform had ArgoCD Image Updater configured (in pal-e-services/terraform/main.tf and services.tf). It polls Harbor for new commit-SHA tags matching regexp:^[0-9a-f]{7,40}$, uses newest-build strategy, and writes back to the kustomization via git. Correct deploy workflow: (1) Merge PR to main. (2) CI builds and pushes image to Harbor. (3) Image Updater detects new tag and syncs automatically. (4) Only manual step: run kubectl -n {namespace} exec {pod} -- bin/rails db:migrate if the release includes migrations. Key lesson: For any service registered in k3s.tfvars with an image_repo field, ArgoCD Image Updater owns the deploy pipeline. Never manually edit image tags in pal-e-deployments or kubectl apply -k prod overlays — it will either fail on SOPS or conflict with the automation.

SOPS Secrets — kubectl apply Fails on Encrypted Overlays (2026-05-29)

Running kubectl apply -k on any overlay that includes a secrets.enc.yaml (SOPS-encrypted) will fail with strict decoding error: unknown field "sops". The non-secret resources (deployment, service, PVC) may partially apply, but the overall command errors out. Root cause: SOPS-encrypted secrets contain a sops: metadata block that Kubernetes doesn't understand. Only ArgoCD with the kustomize-sops CMP plugin can decrypt and apply these. Workaround if you must apply manually: Either (1) decrypt first with sops -d secrets.enc.yaml | kubectl apply -f -, or (2) skip the secret if it already exists in the cluster and apply other resources individually. Key lesson: Any overlay in pal-e-deployments that references a .enc.yaml file is ArgoCD-only territory. Don't kubectl apply -k it from the CLI.