pal-e-docs

Review: Bug: Harbor robot account expired — blocks all deployments

review-761-2026-04-03 Doc

review needs-refinement
Project

Verdict: NEEDS_REFINEMENT

Template Completeness

Checked against template-issue-bug required sections:
  • [x] Type — Bug
  • [x] Lineage — discovered during #30 validation
  • [x] Repo — listed as forgejo_admin/pal-e-services
  • [x] What Broke — detailed with error messages and symptoms
  • [x] Repro Steps — clear 4-step reproduction
  • [x] Expected Behavior — stated
  • [x] Environment — cluster, namespace, Harbor project listed
  • [x] Acceptance Criteria — 4 criteria, all testable
  • [x] Related — project and blocking issue referenced
  • [ ] File Targets — not present (bug template does not require them, but the fix path is ambiguous without them)

Traceability

  • [x] story:read-ops label — Admin Read Operations via GroupMe
  • [x] story note verified — story-westside-ai-assistant-read-ops exists in project-westside-ai-assistant user-stories section
  • [x] arch:A4 label — infrastructure component
  • [ ] arch note MISSING — [SCOPE] Create architecture note arch-A4 for the infrastructure component
  • [x] Forgejo issue — forgejo_admin/westside-ai-assistant#33, open

File Targets

No specific file paths listed in the issue body. Verified the claimed repo placement:
  • [ ] pal-e-services/terraform — ISSUE: westside-ai-assistant does NOT appear anywhere in the pal-e-services terraform codebase. The service was never onboarded into var.services. The harbor-creds secret was provisioned through a different mechanism (likely manual or one-off).
  • [x] harbor-creds secret exists in westside-ai-assistant namespace — confirmed via kubectl. Username is robot$westside-ai-assistant+pull, but this robot no longer exists in Harbor.
  • [x] ImagePullBackOff confirmed live — pod westside-ai-assistant-dd855f577-c94mp actively failing with 401 Unauthorized.

Repo Placement

MISMATCH. The issue says the fix repo is forgejo_admin/pal-e-services, but westside-ai-assistant is not in pal-e-services var.services. Two possible fix paths:
  • Proper onboarding: Add westside-ai-assistant to pal-e-services var.services, which auto-provisions Harbor project, robot accounts (with duration = -1, no expiry), namespace, and harbor-creds secret. This is the correct long-term fix.
  • Manual re-creation: Create a new robot account in Harbor manually and update the k8s secret. This is a bandaid.
The issue body must clarify which path and update the Repo field accordingly.

Dependencies

  • [x] Blocks #29 (Nemo app SDK swap) — documented in issue
  • [x] Discovered during #30 validation (Nemo qwen infra) — documented
  • [ ] Board item #599 (GroupMe go-live, issue #7) is in_progress — not documented but would also fail on next deploy
  • [ ] Board item #759 (email_blast tool, issue #31) is in_progress — not documented but same failure on next deploy
  • This is a namespace-wide blocker: ALL westside-ai-assistant deployments fail until resolved

Acceptance Criteria

4 criteria from the issue, all verifiable by an agent:
  • [x] "Valid Harbor robot account exists" — verifiable via Harbor API or curl
  • [x] "harbor-creds secret updated" — verifiable via kubectl
  • [x] "New pod can pull image" — verifiable via kubectl rollout restart + watch
  • [x] "No ImagePullBackOff events" — verifiable via kubectl get events
Missing AC: If fix path is pal-e-services onboarding, add AC to verify tofu plan shows the service in state with duration = -1 robots.

Blast Radius

Checked all other namespaces with harbor-creds (basketball-api, westsidekingsandqueens, mcd-tracker, pal-e-docs, pal-e-mail, gcal-scheduler, mcd-tracker-app, pal-e-production, platform-validation). No ImagePull failures found in any other namespace. All other services are provisioned through pal-e-services with duration = -1 robot accounts. Only westside-ai-assistant is affected because it was provisioned outside the standard pipeline. Rollback is straightforward — the old pod with cached image is still running.

Decomposition Assessment

Apply three-thing limit and five-minute rule:
  • Single repo fix (pal-e-services terraform vars OR manual Harbor operation)
  • 4 acceptance criteria — under the limit
  • Estimated agent time: under 5 minutes if fix path is clarified
  • No independent subtasks to parallelize
No decomposition needed.

Recommendation

  • [BODY] Fix Repo field: clarify whether fix is (a) onboarding into pal-e-services var.services or (b) manual Harbor robot re-creation. If (a), update file targets to include terraform/terraform.tfvars (or equivalent services var file). If (b), note this is a manual operation with no code PR.
  • [BODY] Add AC: "westside-ai-assistant appears in pal-e-services var.services with duration = -1 robot accounts" (if onboarding path chosen).
  • [BODY] Document that this blocks all in_progress items (#7, #31) — not just #29.
  • [SCOPE] Create architecture note arch-A4 for the infrastructure component.