pal-e-docs

Review: CrashLoopBackOff — Alembic migration chain forked, DB in partial state

review-445-2026-03-26 Review

review needs-refinement
Project

Verdict: NEEDS_REFINEMENT

Template Completeness

  • [x] Type — Bug
  • [x] Lineage — Standalone, discovered during westside landing site validation
  • [x] Repo — forgejo_admin/basketball-api
  • [x] What Broke — Detailed with error messages, pod status, and root cause analysis
  • [x] Repro Steps — 3 concrete kubectl/psql commands
  • [x] Expected Behavior — Present
  • [x] Environment — Cluster, image SHA, DB details, Alembic stamp
  • [x] Acceptance Criteria — 5 verifiable criteria with curl/kubectl commands
  • [x] Related — References project-westside-basketball and ArgoCD drift
  • [x] Fix Plan (bonus) — 4-step recovery procedure referencing sop-db-migration-recovery
  • [x] DB State table (bonus) — Per-migration audit of what is/isn't applied

Traceability

  • [ ] story:X label — MISSING. No user story label on board item #445. This is a production outage affecting /teams dynamic content on the westside landing site. Should reference a westside user story (e.g., story:WS-S26 if teams display is part of that scope).
  • [x] arch:postgres label — Present, though arch:basketball-api may be more precise since the fix is at the Alembic migration layer, not Postgres infrastructure.
  • [x] Forgejo issue — forgejo_admin/basketball-api#183, open

File Targets

This is a DB operations fix (stamp + upgrade), not a code change. No file modifications expected. Migration files referenced in the issue body were all verified against the codebase:
  • [x] alembic/versions/018_add_groupme_and_outbox.py — verified: exists, revision 018, down_revision 017
  • [x] alembic/versions/019_player_teams_junction.py — verified: exists, revision 019, down_revision 018
  • [x] alembic/versions/020_add_custom_notes_to_player.py — verified: exists, revision 020, down_revision 019. Column custom_notes also confirmed in models.py
  • [x] alembic/versions/021_add_oauth_tokens_table.py — verified: exists, revision 021, down_revision 020
  • [x] alembic/versions/022_merge_heads.py — verified: exists, revision 022, down_revision 021. Now a no-op passthrough (chain was linearized)
  • [x] alembic/versions/023_backfill_player_jersey_from_orders.py — verified: exists, revision 023, down_revision 022
  • [x] alembic/versions/024_add_is_public_to_players.py — verified: exists, revision 024, down_revision 023
Migration chain is correctly linearized: 018 -> 019 -> 020 -> 021 -> 022 -> 023 -> 024. The Dockerfile confirms alembic upgrade head runs at boot.

Repo Placement

OK. Issue filed on basketball-api, fix is kubectl exec into basketball-api namespace. Downstream validation on westsidekingsandqueens.tail5b443a.ts.net is correctly identified as acceptance criteria, not a separate repo fix.

Dependencies

  • Duplicate/overlap with #184 (board item #449): Forgejo issue #184 ("Alembic migration chain stuck — 020 applied but not stamped, app down") is CLOSED but board item #449 is still in in_progress. This is either (a) #184 was an incomplete fix and #183 restates the remaining work, or (b) they are duplicates and one should be cleaned up. This must be clarified before moving to next_up.
  • Board item #445 title is null: The board item was created but the title was not synced from the Forgejo issue. Should be set to match the issue title.
  • No other blocking dependencies found. No items in in_progress that this depends on (other than the potentially-duplicate #449).

Acceptance Criteria

All 5 criteria are verifiable by an agent:
  • [x] Pod Running — kubectl get pods -n basketball-api
  • [x] Health endpoint 200 — curl command provided
  • [x] /public/teams returns data — curl command provided
  • [x] westside landing /teams shows rosters — requires browser/screenshot verification
  • [x] Alembic stamped at 024 — psql query against alembic_version table
All criteria are concrete and testable. The fix plan (stamp to 020, upgrade head) aligns with sop-db-migration-recovery guidance for schema conflict recovery.

Blast Radius

  • mcd-tracker-api also uses Alembic with Postgres (4 migrations, simple linear chain 001 to 004). No fork risk currently, but the same pattern (migrations running on boot via Dockerfile CMD) exists there. If mcd-tracker-api ever has a forked chain, the same failure mode would occur.
  • westside-app /teams page is the downstream consumer — confirmed in acceptance criteria. No other downstream services depend on basketball-api /public/teams currently.
  • ArgoCD image tag drift is noted as related. If ArgoCD deploys an old image after this fix, the problem could recur. This is not addressed in the acceptance criteria but is noted in the Related section.

Recommendation

Three items need attention before this ticket moves to next_up:
  • Clarify relationship with closed #184 / board item #449: If #184 was fully resolved, #449 should move to done and #183/#445 needs to explain what remains. If #184 was only partially resolved, the issue body should reference it. Currently the two issues look like near-duplicates.
  • Add story:X label to board item #445 for traceability (likely story:WS-S26 based on westside landing site context).
  • Sync board item #445 title from the Forgejo issue — currently null.
The issue itself is high quality — detailed root cause, DB state audit, concrete fix plan, verifiable acceptance criteria, and a valid SOP reference. The refinement needed is board/traceability hygiene, not scope quality.