Review: P0 pal-e-services tf state drifted — prod postgres in blast radius

Verdict: NEEDS_REFINEMENT

Template Completeness

• [x] Type (Bug) — present
• [x] Lineage — excellent, cites pal-e-services#58/#59 and plan numbers
• [x] Repo — present (forgejo_admin/pal-e-platform)
• [x] What Broke — exceptionally thorough, per-resource table
• [x] Why This Is P0 — clear, 3 distinct justifications
• [x] Repro Steps — concrete, commands are runnable
• [x] Expected Behavior — zero-diff plan gate, clear
• [x] Environment — full namespace list
• [x] Acceptance Criteria — 6 criteria, per-resource decisions required
• [x] Scope Boundary — present ("Does NOT include" section)
• [x] Interim Safety Protocol — present and actively scoped
• [x] Related links — 7 references

Traceability

• [x] Forgejo issue — #297 open on forgejo_admin/pal-e-platform
• [ ] story:platform-bootstrap label — STORY KEY DOES NOT EXIST on project-pal-e-platform user-stories table. Valid keys: story:superuser-deploy, story:superuser-observe, story:superuser-recover, story:superuser-onboard-service, story:superuser-remote-access. The best fit is story:superuser-deploy (the story that owns the "tofu plan/apply succeeds without manual intervention" success metric — that metric is exactly what is broken here) or story:superuser-onboard-service (drift-detection clause ties into service-onboarding-sop).
• [ ] arch:pal-e-services — pal-e-services IS a documented architecture component (appears in project-pal-e-platform Domain Model + Deployment mermaid diagrams as TF_S / tf_services box). Label is structurally valid. No standalone arch-pal-e-services note exists, but foundational bootstrap repos typically live in the project-page architecture section rather than as individual arch notes. Acceptable without a separate arch note — this is bootstrap/foundational.
• [ ] phase reference — ticket cites phase-platform-28-keycloak-declarative-onboarding; that slug does not exist. Phase 28 in plan-pal-e-platform is phase-pal-e-platform-28-keycloak-smtp. A much better-fit phase exists and is not referenced: phase-platform-17b-tf-state-governance (status: not-started) — this ticket is essentially a P0 expression of that phase.

File Targets

• [x] cnpg.tf:62 — kubernetes_manifest.cnpg_cluster resource exists as stated. Manages pal-e-postgres in postgres ns.
• [x] cnpg.tf:158 — kubernetes_manifest.cnpg_scheduled_backup manages pal-e-postgres-daily.
• [x] services.tf:94 — kubernetes_secret_v1.harbor_creds (for_each over services).
• [x] services.tf:21 — harbor_robot_account.service_ci.
• [x] services.tf:123 — argocd_application.service (for_each includes pal-e-mail).
• [x] k3s.tfvars — exists, gitignored as sop-platform-tf-changes documents.

• [x] pal-e-postgres CNPG cluster — 49d old, healthy, primary pal-e-postgres-1 serving traffic. Matches ticket.
• [x] pal-e-postgres-daily ScheduledBackup — exists, last backup ~63min ago (ticket said 53min; minor drift but within operational noise).
• [x] pal-e-mail ArgoCD app — Synced + Healthy despite being "archived" per memory. The contradiction is real.
• [x] Parallel app of interest: westside-email Synced + Healthy — suggests pal-e-mail may have been partially replaced but the ArgoCD app was never cleaned up. Out of scope but noted.

Repo Placement

Dependencies

• Blocks: pal-e-services#58 / #59 (pal-e-docs Keycloak realm onboarding). Those PRs are currently workaround-pattern (-target apply), which is not the canonical sop-platform-tf-changes flow.
• Related: Any future service onboarding that touches pal-e-services.
• Parallel phase: phase-platform-17b-tf-state-governance (not-started) is scope-adjacent — this ticket may be the P0 trigger that finally kicks off that phase. Decision pending: is this ticket "an expression of 17b" or "standalone, 17b stays separate for remote-state-backend work"? Phase 17b scope should be checked.
• Out-of-scope deferrals: remote tf state backend, manual-tool-managed resource migration — correctly bounded in ticket.

Acceptance Criteria

• AC #1 (per-resource create decisions): good, but "Decision recorded per resource" needs a target — where? In the ticket comment thread? A drift-investigation note? Needs explicit location.
• AC #2 (per-resource update decisions): same ambiguity as #1.
• AC #3 (import plan executed): no dry-run gate — executing tofu import on a CNPG cluster has non-zero risk if the manifest spec diverges from cluster reality. Should require: "After import, tofu plan shows zero diff on the imported resource before proceeding to the next import." Import-and-verify, one resource at a time.
• AC #4 (zero-diff plan): the verification gate — good.
• AC #5 (convention/SOP update): "Platform convention note added (or service-onboarding-sop updated)" — ambiguous. Needs a specific target. Recommended: extend service-onboarding-sop → Pre-Deploy Validation Checklist with a new "Plan-diff check" row, AND extend sop-platform-tf-changes → What NOT to Do with a drift-response bullet. Both, not one-or-the-other.
• AC #6 (re-run #58/#59 plan): good verification step.
• Missing AC: "No prod postgres data loss during reconciliation" — should be stated explicitly as a go/no-go. Any step that would cause pod restart on pal-e-postgres-1 is a blocker until a backup is confirmed restorable via sop-postgres-restore.

Blast Radius

• [P0 gap] CNPG operator reconcile semantics: a kubernetes_manifest that imports cleanly can still cause churn if the CNPG operator reconciles the imported spec against its internal defaults. The header comment at cnpg.tf:58-60 already acknowledges this ("CNPG operator manages all other fields... including them here would cause perpetual plan drift"). The ticket should explicitly call out: "Import of kubernetes_manifest.cnpg_cluster must be tested against CNPG operator reconcile — expect non-trivial plan output until the terraform manifest precisely matches operator-projected state for the managed fields, or until we accept perpetual no-op drift on operator-managed subtrees." This is the technical nuance that distinguishes "import went fine" from "we didn't destroy data but we now have a forever-drift loop."
• [Flagged for awareness] woodpecker-db CNPG cluster: kubectl get clusters.postgresql.cnpg.io -A also shows woodpecker-db in the woodpecker namespace (37d old, healthy). If woodpecker-db is managed by pal-e-platform (not pal-e-services), it may not be part of this drift — but the ticket's table only lists pal-e-postgres. Verify during investigation that woodpecker-db is NOT declared in pal-e-services and is NOT in the drift list.
• [Scope-boundary clarifier] ticket says scope is "read-only investigation + import/cleanup PRs" but AC #3 is "Import plan executed." That's not read-only. Reword scope boundary as: "investigation + import/cleanup — no schema changes, no backend migration, no manual-tool-managed-resource migration."

Decomposition Assessment

• Repos touched: 2 (pal-e-platform for SOP update, pal-e-services for imports)
• File targets: ~5 (cnpg.tf, services.tf, potentially main.tf, plus service-onboarding-sop + sop-platform-tf-changes SOP updates)
• Acceptance criteria: 6 (at threshold)
• Estimated agent work: >5 min, clearly needs decomposition. Import of a live CNPG cluster is not 5-min work — it's careful, verify-after-each-step work.

• Drift investigation note — a per-resource decision matrix (adopt/remove/recreate) posted as a pal-e-docs note. Dev reads the 5 create-drifts + 11 update-drifts, records a decision row for each with rationale. Output = input to the import tickets.
• Import CNPG cluster + scheduled backup (highest risk, lowest throughput — do first, alone). Includes backup-restore dry-run via sop-postgres-restore BEFORE import, zero-diff verification after.
• Import/remove harbor_creds + argocd_application drift (medium risk, bulk). Handle the 10× label-strip churn loop at the same time since it's the same ArgoCD-fights-tf pattern.
• Remove pal-e-mail from configuration OR resurrect it canonically (depends on #1 outcome — this needs a Lucas/Ava call before the dev can act).
• SOP updates — extend service-onboarding-sop Pre-Deploy Validation Checklist with a plan-diff row; extend sop-platform-tf-changes with drift-response clause. Small, separate ticket.
• Zero-diff verification + lift interim safety protocol — capstone ticket.

Recommendation

• [LABEL] Fix story label on board item #1064: replace story:platform-bootstrap with story:superuser-deploy (or story:superuser-onboard-service — Ava's call; superuser-deploy is the stronger fit given the "tofu plan/apply succeeds without manual intervention" success metric).
• [BODY] Fix phase reference: replace phase-platform-28-keycloak-declarative-onboarding (does not exist) with phase-pal-e-platform-28-keycloak-smtp AND add phase-platform-17b-tf-state-governance to the Related section — the latter is the canonical home for this kind of work.
• [BODY] Clarify repo placement in the ticket body: add an "Execution Repos" subsection under Repo: SOP/convention updates land in pal-e-platform; code (tofu import, config removal) lands in pal-e-services. Branch naming should reflect this.
• [BODY] Tighten AC #1 and #2: specify where per-resource decisions are recorded. Recommend: "Create a pal-e-docs note drift-investigation-2026-04-20 (note_type: doc) with a per-resource decision table before opening import PRs."
• [BODY] Strengthen AC #3: require zero-diff verification per resource, one at a time, not batch. Language suggestion: "After each tofu import, run tofu plan and confirm the imported resource shows zero diff OR only operator-managed-field drift before moving to the next import."
• [BODY] Split AC #5 into two explicit deliverables: (a) service-onboarding-sop → Pre-Deploy Validation Checklist gets a "Plan-diff check" row; (b) sop-platform-tf-changes → What NOT to Do gets a "No apply when plan diff includes unintended resources" bullet. Both SOPs need the clause; don't leave it as "or."
• [BODY] Add an explicit AC: "No pod restart on pal-e-postgres-1 during reconciliation. If any step would cause restart, pause and confirm backup restorability via sop-postgres-restore dry-run first."
• [BODY] Add CNPG-operator-reconcile nuance to the "Why this is P0" section: call out that kubernetes_manifest resources against operator-reconciled CRDs (CNPG clusters) have non-trivial import semantics because the operator re-projects fields the terraform manifest does not declare. This is the technical source of the perpetual-drift risk that the ticket alludes to but does not name.
• [BODY] Reconcile Interim Safety Protocol vs sop-platform-tf-changes: add a sentence saying "This protocol supersedes sop-platform-tf-changes for pal-e-services until the zero-diff gate is restored." Otherwise readers will follow the SOP's plan-and-apply-before-merge pattern and light the fuse.
• [BODY] Flag woodpecker-db CNPG cluster in the scope section: confirm during investigation that woodpecker-db is NOT in pal-e-services drift list (it should be pal-e-platform-managed, but verify).
• [DECOMPOSE] After [BODY] fixes above are applied, decompose into a child board (board-1064-tf-drift-reconcile) via skill-decompose-ticket. Recommended 6 sub-tickets listed in Decomposition Assessment above. The parent ticket tracks the overall P0; sub-tickets are the executable units.
• [SCOPE] Ava to decide: is this ticket the P0 expression of phase-platform-17b-tf-state-governance, or does 17b remain as the remote-state-backend epic? If the former, update phase-17b to reference #297 as its blocker/trigger.