Review: basketball-api DB migration stuck -- app DOWN

review-449-2026-03-26 Review

review ready

Verdict: READY

Template Completeness

  • [x] Type -- Bug
  • [x] Lineage -- discovered during westside deploy session
  • [x] Repo -- forgejo_admin/basketball-api
  • [x] What Broke -- detailed migration state table, clear symptom description
  • [x] Repro Steps -- 5-step crash loop sequence
  • [x] Expected Behavior -- idempotent migration or correct stamping
  • [x] Environment -- cluster, stamped version, actual DB state
  • [x] Acceptance Criteria -- 4 verifiable criteria
  • [x] Related -- project and incident context linked
  • [x] Test Expectations (bonus) -- 555 tests, pytest command, fresh DB and pre-applied scenarios
  • [x] Constraints (bonus) -- no manual stamp, no direct prod ALTER, deploy pipeline only
  • [x] Checklist (bonus) -- PR, tests, no unrelated changes
All required bug template sections present. Three useful bonus sections included.

Traceability

  • [ ] story:X label -- missing. Production outage bug, not feature work. Foundational/operational -- acceptable.
  • [x] arch:database label -- database component correctly identified
  • [x] Forgejo issue -- forgejo_admin/basketball-api#184, open

File Targets

  • [x] alembic/versions/020_add_custom_notes_to_player.py -- verified: op.add_column("players", sa.Column("custom_notes", sa.Text(), nullable=True)) with NO idempotency guard. This is the exact line that causes the crash.
  • [x] alembic/versions/019_player_teams_junction.py -- verified: down_revision = "018", creates player_teams table. Chain link confirmed.
  • [x] alembic/versions/022_merge_heads.py -- verified: passthrough migration, chain linearized 018->019->020->021->022. Confirms issue's claim about linearization.
  • [x] Migration chain 018->019->020->021->022->023 -- all files exist, revision chain is correct and linear.

Repo Placement

Correct. Issue filed on forgejo_admin/basketball-api. Fix is in basketball-api's alembic/versions/020_add_custom_notes_to_player.py. Single-repo fix, no cross-repo concerns.

Dependencies

  • Board item #352 (Alembic forked history -- 019 and 020 both descend from 018) -- DONE. That fix linearized the chain, which created the precondition for this bug (020 was applied during the fork era but stamping was lost during linearization).
  • Board item #433 (Alembic migration crash -- app down) -- DONE. Prior similar incident. Pattern recurrence confirms the fix approach (idempotency) is the right one.
  • No items currently in_progress. No blocking dependencies. This ticket can proceed independently.

Acceptance Criteria

All 4 acceptance criteria are agent-verifiable:
  • "Migration 020 is idempotent" -- verifiable by code inspection (check for column existence guard)
  • "Alembic runs cleanly from 018 through 023" -- verifiable with alembic upgrade on test DB
  • "App starts without crash-loop" -- verifiable post-deploy via kubectl/health check
  • "All 5 migrations applied and stamped" -- verifiable via alembic current
Test expectations are clear: pytest tests/ -x (555 existing tests), plus fresh DB and pre-applied DB scenarios. The constraint "do NOT stamp the DB manually" correctly scopes the fix to code, not ops workaround.

Blast Radius

  • basketball-api: 30+ op.add_column calls across the migration history, NONE have idempotency guards. If any future stamping mismatch occurs, the same crash pattern will repeat. Consider a follow-up ticket to add guards to all add_column migrations (or establish a convention for new ones).
  • mcd-tracker-api: Same pattern -- 2 non-idempotent op.add_column calls in alembic/versions/. Same vulnerability exists but lower risk (simpler migration chain, no prior fork incidents).
  • No other repos with Alembic migrations found (westside-contracts has no alembic directory).

Recommendation

No action needed -- ticket is READY to move to next_up. Scope is tight, file targets verified, acceptance criteria testable, constraints well-defined. The fix is a single-file change (add column existence check to migration 020). Consider creating a follow-up ticket for idempotency guards across all migrations as a hardening measure.