Review: Harbor connectivity timeout from Woodpecker CI agent

Verdict: NEEDS_REFINEMENT

Template Completeness

• [x] Type -- present (Bug)
• [x] Lineage -- present (standalone, discovered during CI monitoring)
• [x] Repo -- present (forgejo_admin/pal-e-platform)
• [x] Context -- present as "What Broke" (detailed error output, retry behavior)
• [x] Repro Steps -- present (4 steps)
• [x] Expected Behavior -- present
• [x] Environment -- present (pods, services, IPs, pipeline numbers)
• [x] Acceptance Criteria -- present (3 items)
• [x] Related -- present (cross-references issues, memory, SOP)
• [ ] User Story -- MISSING. Bug template should still state the affected persona (e.g., "As a CI pipeline, I want to push images to Harbor...")
• [ ] File Targets -- MISSING. No specific files identified for the fix. Key candidates: terraform/network-policies.tf, terraform/main.tf (Harbor helm values, Woodpecker agent env), basketball-api/.woodpecker.yaml.
• [ ] Test Expectations -- MISSING. No verification commands. Should include kubectl connectivity test, pipeline retry command, or at minimum a manual curl from a woodpecker-namespace pod.
• [ ] Constraints -- MISSING. Should note: don't change Harbor to expose 443 (design is HTTP-only ClusterIP), don't alter network policies without tofu plan, reference prior fix from issue #135.
• [ ] Checklist -- MISSING (PR opened, tests pass, no unrelated changes).

File Targets

• [x] terraform/network-policies.tf -- verified: Harbor network policy exists (line 83), woodpecker namespace ingress is explicitly allowed (line 101). Policy is correctly configured.
• [x] terraform/main.tf -- verified: Harbor helm release exposes ClusterIP on port 80 only (line 916), Woodpecker agent env includes HARBOR_REGISTRY_INTERNAL=harbor.harbor.svc.cluster.local (line 784).
• [x] basketball-api/.woodpecker.yaml -- verified: build-and-push step uses Kaniko plugin connecting to harbor.harbor.svc.cluster.local with insecure: true (lines 41-53). Configuration looks correct.
• [x] westside-app/.woodpecker.yaml -- verified: identical pattern to basketball-api (same registry, insecure flag, kaniko plugin).
• [x] salt/pillar/mac-agent.sls -- verified: Mac agent uses backend: local, connects via Tailscale subnet router. Mac agent cannot run container images natively -- potential root cause if job was misrouted.

Repo Placement

Dependencies

• Board item #254 (done): "#135: Harbor unreachable from CI pods" -- nearly identical symptoms. Prior fix addressed TLS hairpin and internal URL standardization. Current issue may be regression or different root cause.
• Board item #399 (done): "Fix Woodpecker agent secret duplication" -- recently completed. Agent secret changes could affect connectivity if agent misconfigured.
• Board item #401 (backlog): "Remove non-functional gRPC funnel" -- the gRPC funnel for Mac agent was recently added (#173). If Mac agent is now active and receiving jobs, builds could route to an agent that cannot reach ClusterIP services.
• Board item #394 (done): "Tailscale Connector -- k8s subnet router" -- just completed. Subnet routing changes could affect how external agents reach cluster services.
• Dependencies are NOT documented in the issue scope.

Acceptance Criteria

• "Identify whether this is network policy, DNS, or agent resource issue" -- investigative, not testable by an agent. Should be: "Root cause documented in PR description."
• "CI pipeline successfully builds and pushes image to Harbor" -- testable but needs specific command (e.g., "Re-run basketball-api pipeline on main, verify build-and-push step succeeds").
• "basketball-api deploys with migrations 022 + 023" -- out of scope for Harbor connectivity fix. Split to separate basketball-api issue.

Blast Radius

• All repos with build-and-push steps affected: basketball-api, westside-app, pal-e-docs, pal-e-app, mcd-tracker-api, mcd-tracker-app, minio-api all use identical Kaniko pattern with harbor.harbor.svc.cluster.local.
• Mac agent routing risk: The recently enabled Mac agent (board item #391, next_up) uses backend: local. If Woodpecker routed a build-and-push job to the Mac agent, Kaniko would fail because Mac cannot run container images. Issue should investigate which agent ran pipeline #145.
• Resource pressure signal: Postgres service container failure on pipeline #146 retry suggests possible node resource pressure affecting ALL CI pipelines.
• SOP gap: sop-ci-pipeline-recovery lists "Push step FAILURE" but only covers auth/project/disk causes. The connectivity timeout failure mode (port 443 on HTTP-only service) is undocumented. SOP update needed after root cause found.

Recommendation

• Add File Targets section: List terraform/network-policies.tf, terraform/main.tf, and basketball-api/.woodpecker.yaml as investigation targets.
• Scope the acceptance criteria: Remove "basketball-api deploys with migrations 022 + 023" (separate deploy concern). Replace investigative criterion with "Root cause documented in PR description." Add test command: "Re-run basketball-api pipeline, verify build-and-push completes."
• Add investigation hypothesis about Mac agent: The issue should determine whether the Mac agent (recently enabled, cannot run containers) received the job. Check Woodpecker pipeline #145 agent assignment. If Mac agent received a k8s-backend job, the fix is label-based routing, not network policy.