TODO: Deployment Safety — Rollback, Migration Guards, Dev Environment

todo-deployment-safety Todo

resolved todo

Project

Incident: pal-e-docs down due to Alembic migration crash (2026-02-26)

What happened

PR #29 (Schema Entity Links Phase 1) merged and deployed. The Alembic migration added is_public and page_note_id columns to the projects table. The DDL executed successfully (columns exist in SQLite), but the alembic_version stamp was never updated. On pod restart, Alembic tried to re-run the migration and crashed with duplicate column name: is_public. CrashLoopBackOff — site down.

Root cause

SQLite DDL is auto-committed outside transactions. ALTER TABLE ADD COLUMN takes effect immediately and cannot be rolled back. If anything interrupts the migration after DDL but before the version stamp, you get a split-brain state: columns exist but Alembic doesn't know. This is a known SQLite + Alembic footgun — Postgres doesn't have this problem because DDL is transactional.

How it was fixed

Manually stamped alembic_version to c3d4e5f6a7b8 via sqlite3 in the litestream sidecar. Deleted the crashing pod. New pod came up clean. Total downtime: ~10 minutes (until discovered and fixed in session).

What we need

1. Idempotent Alembic migrations

All migrations that run DDL against SQLite must check before acting. Wrap ALTER TABLE ADD COLUMN in a "column exists?" check. This is the immediate fix — prevents this exact failure mode from ever happening again.

2. Rollback mechanism

We had no way to quickly revert to the previous image. Need:

ArgoCD rollback procedure documented as an SOP
Image tag history (know what the previous working image was)
One-command rollback (e.g., argocd app rollback or k8s deployment revision history)

3. Deployment protection / health checks

The migration crash should have prevented the rollout from completing. Need:

Proper startup probes that gate readiness on "migrations ran successfully"
Rolling update strategy that keeps old pods alive until new pods are healthy
Alert when a pod enters CrashLoopBackOff

4. Dev/staging environment

This migration was never tested against a real database before hitting production. Need a staging environment where migrations run against a copy of prod data before deploying to prod.

5. Migration testing in CI

CI should run alembic upgrade head against a seeded database (not just empty) to catch migration failures before merge.

Priority

Items 1 (idempotent migrations) and 3 (health checks) are the highest priority — they prevent the same class of failure. Items 2 and 4 are the next tier. Item 5 is a nice-to-have that catches issues earlier.

Plan: Platform Observability Foundation (plan-2026-02-25-platform-observability) — overlaps with alerting and health checks.