pal-e-docs

TODO: pal-e-docs Deployment Reliability — Zero-Downtime Deploys

todo-pal-e-docs-deployment-reliability Todo

deployment sre todo done
Project

TODO: pal-e-docs Deployment Reliability — Zero-Downtime Deploys

Problem

pal-e-docs is the single point of failure for the entire AI agency. Every Claude Code session depends on it (session-start hooks, plan slug enforcement, MCP tools). When it goes down, all agent workflows crash — not gracefully, just 502s everywhere.
We have now had two production outages from the same root cause: SQLite auto-commits DDL, Alembic migrations crash mid-way, the DB is left in a partial state, and the pod enters CrashLoopBackOff. Both times required manual intervention via kubectl.
  • Incident 1: PR #29 (2026-02-26) — is_public + page_note_id migration
  • Incident 2: PR #61 (2026-03-02) — note_type + status + parent_note_id + position migration. See incident-2026-03-02-sqlite-migration-crash-pr61.
The current deployment strategy is a simple rolling update via ArgoCD. The old pod is terminated before the new pod proves it can serve traffic. There is no readiness probe, no health check, no rollback mechanism. Any deployment failure = downtime.

Three Things We Need

1. Postgres Migration (eliminates the root cause)

Postgres supports transactional DDL. Alembic migrations are atomic — all steps succeed or all roll back. The entire class of "partial DDL + unstamped version" bugs is eliminated.
Existing plan: plan-2026-02-26-tf-modularize-postgres (currently DEFERRED). Phase 1 (TF modularization) was deferred, but Phases 2-4 (deploy Postgres, migrate pal-e-docs, backups) don't depend on it. The plan explicitly says "Postgres can be added directly to main.tf when needed." It's needed now — two incidents is the trigger.
What to do: Un-defer the Postgres plan. Skip Phase 1 (modularization). Execute Phases 2-4 directly.

2. Blue-Green Deployments with Readiness Probes (eliminates downtime from ANY deployment failure)

Even with Postgres, a bad deployment can crash a pod. Blue-green ensures the old pod keeps serving until the new pod passes its readiness probe. Zero-downtime for any deployment, not just migrations.
Implementation:
  • Add a readiness probe to pal-e-docs (HTTP GET on a health endpoint, only passes after migration completes and app is serving)
  • Switch from ArgoCD rolling update to Argo Rollouts blue-green strategy
  • Old ReplicaSet stays alive until new one is Ready
  • Traffic switches atomically
Existing plans: plan-2026-02-26-kustomize-service-bases touches k8s manifests and deployment patterns. Blue-green could be a phase there, or a standalone effort under pal-e-platform. No existing plan covers Argo Rollouts specifically.

3. Migration Testing in CI (catches the problem before it hits prod)

Run alembic upgrade head against a copy of the production schema in CI before deploying. Would have caught "duplicate column name" before the image ever reached ArgoCD.
Existing TODO: todo-migration-testing-ci-pal-e-docs (open). This has been a known gap since the first incident.
Implementation: Add a Woodpecker CI step that: (a) starts a Postgres container (or SQLite for now), (b) applies all migrations up to current HEAD, (c) verifies clean state. Blocks deploy on failure.

Priority Order

  • Postgres — eliminates root cause. Two incidents from the same bug is unacceptable. Un-defer the plan.
  • Blue-green + readiness probes — eliminates downtime from any future deployment failure. Defense in depth.
  • Migration testing in CI — belt to blue-green's suspenders. Catches problems earlier in the pipeline.

Relationship to Current Work

The active decomposition plan (plan-2026-03-01-note-decomposition) has no more Alembic migrations in Phases 3-5. Phase 3 (MCP tools) and Phase 4 (browse rendering) are code-only changes. So the immediate SQLite risk is low for current work. But the next plan that touches the schema will hit this again.
  • incident-2026-03-02-sqlite-migration-crash-pr61 — the incident that prompted this TODO
  • plan-2026-02-26-tf-modularize-postgres — Postgres plan (deferred, needs un-deferring)
  • plan-2026-02-26-kustomize-service-bases — k8s deployment patterns
  • todo-migration-testing-ci-pal-e-docs — existing CI migration testing TODO
  • deployment-lessons — SQLite DDL danger documented
  • plan-2026-03-01-note-decomposition — current active work (no more migrations remaining)