SOP: harbor-creds Migration (SOPS-Overlay → Terraform-Managed)

SOP: harbor-creds Migration (SOPS-Overlay → Terraform-Managed)

Purpose

Pre-flight Checks

• Service in tfvars: grep for {service-name} in ~/pal-e-services/terraform/k3s.tfvars services map. If not present, this SOP does not apply — that is a separate "onboard service" task.
• Dockerfile runs non-root: if the service's namespace is PSA-restricted (kubectl get ns {service} -o jsonpath='{.metadata.labels}' shows pod-security.kubernetes.io/enforce: restricted), verify the service's Dockerfile contains a USER directive. Without it, pods will fail at runtime with CreateContainerConfigError even with a working Secret.
• Harbor admin credentials: verify HARBOR_ADMIN_PASSWORD exists in ~/secrets/pal-e-services/secrets.env per feedback_check_local_secrets_first.
• SOPS age key (optional): verify ~/.config/sops/age/keys.txt exists if you want to decrypt the existing file for content confirmation. Not required to delete the file.
• Existing overlay structure: confirm overlays/{service}/prod/kustomization.yaml references harbor-creds.enc.yaml in its resources: list, and that harbor-creds.enc.yaml exists in the same directory. If neither, this SOP does not apply (already migrated or never SOPS-managed).

Steps

• Open a Forgejo issue in pal-e-deployments using template-issue-bug. Title pattern: Migrate {service} harbor-creds from SOPS-overlay to terraform-managed (under #144). Reference parent ticket forgejo_admin/pal-e-deployments#144. Mirror the AC list from forgejo_admin/pal-e-deployments#143 (10 ACs covering kustomization edit, file deletion, Harbor robot existence, valid base64 dockerconfigjson, deployment ready, no errors, no-drift tofu plan, ArgoCD Synced, external HTTP 200).
• Add a board item via mcp__pal-e-docs__create_board_item linking the new issue. Labels: story:{service-story-slug},arch:harbor,type:bug,blocks:deploy-chain. Column: backlog. Board: typically board-{service}; if no service-specific board exists, file the Forgejo issue alone and surface to user for board placement.
• Run /review-ticket board-{service}#{item-id}. Expect APPROVED on first pass if the ticket body mirrors the #143 template.
• Move board item: backlog → todo → next_up → in_progress via mcp__pal-e-docs__update_board_item. Each transition is a separate call (never skip columns per feedback_kanban_column_flow).
• Clone pal-e-deployments to /tmp/pal-e-deployments-fix-{N} using the Forgejo token from ~/secrets/pal-e-services/forgejo.env. Create a branch {N}-migrate-{service}-harbor-creds. Set .current-issue to {N} in the working tree.
• Edit overlays/{service}/prod/kustomization.yaml: remove the line - harbor-creds.enc.yaml from the resources: list. Use the Edit tool (Read first to satisfy the contract).
• Delete the encrypted file: git rm overlays/{service}/prod/harbor-creds.enc.yaml.
• Commit with a message that explains the architectural why (cite parent #144, cite #143 as the working template) and includes Closes #{N}.
• Push the branch and open the PR via mcp__forgejo__submit_pr. PR body must use template-pr-body structure (Summary, Changes, Test Plan, Review Checklist, Related Notes). Include the coordinated terraform-apply sequence inline in the PR body — see #145 for the template.
• Run /review-pr forgejo_admin/pal-e-deployments#{pr-number}. Expect APPROVED with at most cosmetic nits.
• Get explicit user approval to merge. Merge via mcp__forgejo__merge_approved_pr with method=squash, delete_branch=true.
• Run /update-docs immediately — the post-merge hook blocks other actions until this completes. Move the board item in_progress → qa as part of this step.
• Wait ~3 minutes for ArgoCD auto-sync, or force-refresh: kubectl annotate application {service} -n argocd argocd.argoproj.io/refresh=hard --overwrite. Verify ArgoCD synced to the merge commit and the placeholder Secret was pruned: kubectl get secret harbor-creds -n {service} returns NotFound. Pod's image-pull error transitions from illegal base64 data to no basic auth credentials — this is the diagnostic confirmation that ArgoCD pruned cleanly.
• Run targeted terraform plan: cd ~/pal-e-services/terraform && tofu plan -var-file=k3s.tfvars -target='harbor_robot_account.service_pull["{service}"]' -target='kubernetes_secret_v1.harbor_creds["{service}"]' -lock=false. Review the diff. Expected: 1 create (the harbor-creds Secret). Possible incidental: other for_each peers that have drifted (e.g., new services added to tfvars but never applied); these are benign-but-eventual scope. Anything destructive in the plan warrants pause.
• Get explicit user approval to apply. Run tofu apply with the same flags plus -auto-approve. Expect "Apply complete! Resources: N added, M changed, 0 destroyed."
• Force pod recreation so kubelet retries image-pull immediately: kubectl rollout restart deployment {service} -n {service}. Without this, kubelet's existing exponential backoff will eventually retry on its own, but restart shortens the wait from potentially minutes to seconds.
• Verify deployment ready: kubectl get deployment {service} -n {service} shows expected READY count. Pod events show Successfully pulled image, no ImagePullBackOff or CreateContainerError.
• Verify the in-cluster Secret has real credentials: kubectl get secret harbor-creds -n {service} -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq. The auth field must be valid base64 (not the literal PLACEHOLDER_REPLACE_AFTER_PAL_E_SERVICES_HARBOR_PROJECT_PROVISIONED string).
• Verify external funnel returns HTTP 200 if the service has one: curl -sk -o /dev/null -w "HTTP %{http_code}\n" https://{service}.tail5b443a.ts.net/.
• Run /validate-ticket board-{service}#{item-id}. Expect PASS. Skill auto-moves the item to done on PASS verdict.

Rules

• Never apply without -target= the service's robot + Secret. A bare tofu apply -var-file=k3s.tfvars reconciles every for_each entry across all of pal-e-services state — much larger blast radius than needed.
• Always include -lock=false on terraform commands per feedback_tofu_lock_false. State lock blocks CI and other concurrent operations.
• Never merge the PR without explicit user approval per feedback_no_merge_without_approval. The merge triggers ArgoCD's prune — it is a prod-state change.
• Never tofu apply without explicit user approval for the same reason — it writes to the cluster.
• Always run /update-docs after merge — the post-merge hook enforces this as a blocking requirement.
• Always file discovered scope per feedback_discovered_scope_always_tracked. If tofu plan shows incidental scope (e.g., a new service added to tfvars but never applied — like notion-mcp-remote during the westside-admin migration), file a tracking ticket. Do not silently apply and forget.
• Never modify bases/standard/deployment.yaml as part of this SOP. That is a separate platform-wide hardening concern (parent ticket forgejo_admin/pal-e-deployments#140). harbor-creds migration is per-overlay only.
• Do not rotate the Harbor robot password as part of this SOP. Terraform's harbor_robot_account.service_pull[service] resource may already exist in state from a prior partial apply — terraform will use the cached .secret. If the robot does not exist in Harbor, terraform creates it on apply with a new secret.
• Document the brief image-pull-down window in the PR body. Between merge (when ArgoCD prunes the placeholder Secret) and tofu apply (when the real Secret is written), the pod's image-pull fails with "no basic auth credentials." This is observable but no worse than the pre-migration state.
• One service per PR. Per feedback_smaller_scopes_parallel, never batch multiple services into one migration PR. Each service has its own kustomization edit + its own validation + its own potential failure mode.

Recovery

• Pod stuck in ImagePullBackOff after tofu apply: verify the in-cluster Secret has real credentials (Step 18). If it does, run kubectl rollout restart deployment {service} -n {service} to force kubelet retry. Kubelet's exponential backoff can otherwise stretch to multiple minutes.
• tofu apply fails with "no value for required variable": always include -var-file=k3s.tfvars. The terraform module requires multiple secret variables (Harbor admin, SOPS age key, Keycloak admin, postgres passwords) that all live in k3s.tfvars.
• ArgoCD shows OutOfSync after merge: the kustomize build may be failing post-merge. Check ArgoCD app events: kubectl describe application {service} -n argocd. Most common cause: a SOPS-encrypted file referenced in kustomization.yaml that was not removed from the resources list when the file was deleted.
• Pod fails with CreateContainerConfigError after pull succeeds: the Dockerfile lacks a USER directive but the namespace is PSA-restricted. Add USER to the Dockerfile in a separate PR (in the service's repo, not pal-e-deployments) before re-validating.
• tofu plan shows unexpected destroys: abort. Investigate state drift before applying. Never apply a plan with destroys you did not intend.
• ArgoCD never prunes the Secret after merge: verify the merge actually landed (git log origin/main), then force-refresh the Application. If still not pruned, check ArgoCD's auto-sync policy includes prune: true.

Related

• forgejo_admin/pal-e-deployments #144 — parent ticket tracking the platform-wide migration of remaining 13 services
• forgejo_admin/pal-e-deployments #143 — first concrete migration (westside-admin) that validated this SOP's pattern
• forgejo_admin/pal-e-deployments #145 — the PR that demonstrated the kustomization edit + file deletion
• validation-143-2026-05-03 — validation note proving end-to-end correctness of the pattern
• sop-harbor-robot-import — sister SOP for Harbor robot lifecycle (out-of-band rotation, recovery)
• sop-validation — referenced by Step 20 (the /validate-ticket gate)
• feedback_tofu_lock_false — convention requiring -lock=false on tofu operations
• feedback_check_local_secrets_first — verify ~/secrets before assuming credential re-issuance
• feedback_no_merge_without_approval — convention requiring explicit user approval for merges
• feedback_validate_before_done — convention requiring validation note before moving to done
• feedback_kanban_column_flow — never skip columns when moving items
• feedback_discovered_scope_always_tracked — file incidental scope as separate tickets
• template-issue-bug — issue template for the per-service migration ticket
• template-pr-body — PR body template