pal-e-docs

Incident: pal-e-docs CI migration-test failure (Alembic drift)

incident-paledocs-alembic-drift-2026-03-14 Doc

resolved incident
Project

Incident: pal-e-docs CI migration-test failure

Discovered

2026-03-14, during Phase 7a (Kustomize migration). Blocked Image Updater write-back verification.

Impact

  • pal-e-docs CI pipeline cannot build-and-push new images (migration-test step blocks it)
  • Phase 7a Image Updater verification blocked
  • All pal-e-docs deploys require manual docker build+push workaround

Root Cause

alembic check in migration-test detects model-vs-DB schema drift. Exit code -1 (255 unsigned). The SQLAlchemy models evolved but no Alembic migration was generated for:
  • modify_nullable on blocks.created_at, blocks.updated_at, board_items.created_at/updated_at, boards.created_at/updated_at, compiled_pages.html/compiled_at
  • remove_index ix_board_items_board_column
  • remove_constraint boards_slug_key + changed index ix_boards_slug unique flag
  • remove_index ix_notes_parent_note_id_position
  • remove_index ix_notes_search_vector
  • remove_column notes.search_vector

Diagnosis Path

  • Woodpecker API returned empty logs (stream: not found — separate cosmetic bug)
  • K8s events showed postgres service pod killed prematurely (red herring — Woodpecker k8s backend lifecycle issue)
  • Direct DB query on Woodpecker log_entries table revealed actual error: alembic check detected drift

Fix Required

Generate an Alembic migration in pal-e-docs that resolves all detected drift. This is a code fix, not a platform fix.
Repo: forgejo_admin/pal-e-docs

Secondary Issue

Woodpecker log API returns empty lines instead of actual log content. Logs ARE stored in log_entries table (15K+ rows). The stream: not found error in server logs suggests the log streaming layer doesn't work with the Postgres backend. This is a known Woodpecker issue (separate from the Alembic drift).
  • phase-pal-e-platform-kustomize — Phase 7a where discovered
  • plan-pal-e-platform — parent plan