pal-e-docs

Review: Fix: pal-e-docs-app pod ImagePullBackOff from Harbor

review-613-2026-03-28 Doc

review needs-refinement
Project

Verdict: NEEDS_REFINEMENT

Template Completeness

  • [x] Type — present but wrong value (says "Feature", should be "Bug"; ImagePullBackOff is broken behavior)
  • [x] Lineage — present (board, story, arch, blocking info)
  • [ ] Repo — present but inaccurate: says "pal-e-platform (Harbor config)" but Harbor config lives in pal-e-services (see Repo Placement)
  • [x] User Story — clear: platform operator needs pod running for portfolio demos
  • [x] Context — good background: pod name, image reference, namespace age, root cause hypothesis
  • [x] File Targets — 3 targets listed (Harbor UI, deployment overlay, woodpecker config)
  • [x] Acceptance Criteria — 4 testable conditions
  • [x] Test Expectations — explicit kubectl and curl commands
  • [x] Constraints — 3 constraints listed
  • [x] Checklist — 5 discrete steps
  • [x] Related — links to Harbor incident and live URL
Missing bug-template sections: Repro Steps, Expected Behavior, Environment (structured). These are embedded loosely in Context but not per bug template format.

Traceability

  • [x] story:reader-browse — pal-e-docs-app frontend serves the documentation browse experience
  • [x] arch:k8s-deploy — deployment/pod infrastructure
  • [x] Forgejo issue — forgejo_admin/pal-e-platform#234, open
All three traceability legs present.

File Targets

  • [x] pal-e-deployments/overlays/pal-e-docs-app/ — verified: directory exists with kustomization.yaml (image: harbor.tail5b443a.ts.net/pal-e-docs-app/app:e23a1d8c...), harbor-creds.enc.yaml, deployment-patch.yaml, ingress.yaml
  • [x] .woodpecker.yaml in app repo — verified: exists at ~/pal-e-app/.woodpecker.yaml, has build-and-push step pushing to pal-e-docs-app/app Harbor project with Kaniko + update-kustomize-tag step
  • [x] Harbor admin UI — operational check, not a file target (acceptable)
Targets are specific enough for agent execution. The kustomize overlay image tag e23a1d8c... is the key artifact to verify against Harbor.

Repo Placement

ISSUE: The Forgejo issue is filed on forgejo_admin/pal-e-platform and says the repo is "pal-e-platform (Harbor config)". But pal-e-platform has zero references to pal-e-docs-app anywhere in the codebase. The Harbor project, robot accounts, namespace, pull secrets, and ArgoCD application for pal-e-docs-app are all managed by pal-e-services/terraform/k3s.tfvars (lines 137-144). The correct repo for Harbor config is forgejo_admin/pal-e-services, not forgejo_admin/pal-e-platform.
The deployment overlay is correctly identified as forgejo_admin/pal-e-deployments. The Woodpecker pipeline lives in the app repo (forgejo_admin/pal-e-docs-app, checked out locally as ~/pal-e-app).
The fix likely involves: (1) verifying the image was pushed to Harbor (operational), (2) possibly triggering a Woodpecker build if missing (pal-e-docs-app repo), (3) verifying pull secrets (pal-e-services terraform). None of these touch pal-e-platform.

Dependencies

  • [ ] Item #510 "Rename pal-e-app to pal-e-docs-app" (backlog) — status: unknown/pending. If the rename has already happened, the old image SHA from pre-rename may not exist. This could be the root cause and should be investigated.
  • [x] Item #513 "Validate: pal-e-app (4 PRs, clone failure)" (backlog) — independent, same story:reader-browse but no blocking relationship.
No hard blockers on the board. The rename item (#510) is a potential root cause contributor that should be investigated during execution.

Acceptance Criteria

4 criteria, all operationally verifiable by an agent:
  • [x] "Image exists in Harbor at the expected tag" — verifiable via Harbor API or kubectl describe pod events
  • [x] "Image pull secret exists in pal-e-docs-app namespace" — kubectl get secret -n pal-e-docs-app
  • [x] "Pod transitions from ImagePullBackOff to Running" — kubectl get pods -n pal-e-docs-app
  • [x] "curl returns 200" — curl -sI https://pal-e-docs-app.tail5b443a.ts.net
All criteria are testable and specific. No ambiguous language. No missing criteria detected.

Blast Radius

All 10 services in pal-e-services use the same Harbor pull pattern (harbor.tail5b443a.ts.net/{project}/{image}:{sha}). If the root cause is a credential or Harbor project configuration issue, the same problem could affect other services. However, since the issue is specifically about a missing or wrong image tag for one service, blast radius is limited to pal-e-docs-app only. Rollback is straightforward — revert the kustomize image tag to a known-good SHA.

Decomposition Assessment

Applying three-thing limit and five-minute rule:
  • Discrete changes: 2-3 (diagnose root cause, fix image/secret/pipeline, verify). Under the 3-thing limit.
  • Repos touched: primarily operational (kubectl, Harbor API). Code changes likely limited to pal-e-deployments kustomize tag or triggering a Woodpecker build.
  • Estimated agent time: <5 minutes for diagnosis + fix.
  • No independent subtasks that need parallelization.
No decomposition needed.

Recommendation

  • [BODY] Fix Type header: "Feature" to "Bug" (ImagePullBackOff is broken behavior, not new functionality). Board item label type:bug is correct.
  • [BODY] Fix Repo line: replace forgejo_admin/pal-e-platform (Harbor config) with forgejo_admin/pal-e-services (Harbor config via terraform). pal-e-platform has zero references to pal-e-docs-app.
  • [BODY] Add structured Environment section per bug template: Cluster: prod, Namespace: pal-e-docs-app, Image: harbor.tail5b443a.ts.net/pal-e-docs-app/app:e23a1d8c..., Kustomize overlay: pal-e-deployments/overlays/pal-e-docs-app/prod/kustomization.yaml
  • [BODY] Add Repro Steps section: 1. kubectl get pods -n pal-e-docs-app 2. Observe ImagePullBackOff on pod 3. kubectl describe pod shows image pull failure
  • [SCOPE] Investigate whether board item #510 (pal-e-app rename to pal-e-docs-app) has already executed — if so, the old SHA may reference a pre-rename build that doesn't exist in Harbor.