Review: Harbor unreachable from CI pods

Verdict: NEEDS_REFINEMENT

Template Completeness

• [x] Lineage
• [x] Repo
• [ ] User Story — missing. Uses "What Broke" narrative instead of As a / I want / So that format
• [x] Context — partially present, embedded in "What Broke" and status table
• [ ] File Targets — partially present. Mentions network-policies.tf:89 inline but no structured File Targets section listing files to modify and files NOT to touch
• [x] Acceptance Criteria
• [ ] Test Expectations — missing entirely. No test commands, no verification steps
• [ ] Constraints — missing. No patterns to follow, no dependency guidance
• [ ] Checklist — missing (PR opened, tests pass, etc.)
• [x] Related

File Targets

• [x] terraform/network-policies.tf — verified exists. Harbor netpol block at lines 75-95. Woodpecker-to-harbor ingress rule at line 90 (ticket says line 89, off by one)
• [ ] terraform/network-policies.tf — ISSUE: Harbor NetworkPolicy is NOT currently deployed in the cluster. kubectl get networkpolicies -n harbor returns empty. The ticket assumes it is active ("woodpecker -> harbor is explicitly allowed") but it is not applied
• [x] westside-app/.woodpecker.yaml — verified: uses harbor.harbor.svc.cluster.local + insecure: true
• [x] basketball-api/.woodpecker.yaml — verified: uses harbor.harbor.svc.cluster.local + insecure: true
• [x] pal-e-docs/.woodpecker.yaml — verified: uses harbor.harbor.svc.cluster.local + insecure: true
• [x] pal-e-app/.woodpecker.yaml — verified: uses harbor.harbor.svc.cluster.local + insecure: true
• [x] mcd-tracker-api/.woodpecker.yaml — verified: uses harbor.tail5b443a.ts.net, no insecure flag
• [x] mcd-tracker-app/.woodpecker.yaml — verified: uses harbor.tail5b443a.ts.net, no insecure flag
• [ ] minio-api/.woodpecker.yaml — MISSING FROM TICKET. Also uses harbor.tail5b443a.ts.net without insecure flag. Same pattern as mcd-tracker repos. Must be included in scope

Repo Placement

Dependencies

• #127 (kube-router ipset sync stale) — POTENTIAL BLOCKER, NOT DOCUMENTED AS SUCH. This issue is open. Pipeline logs prove the Harbor push failure is intermittent (westside-app pipeline #73 failed with dial tcp 10.43.131.178:443: i/o timeout, pipeline #74 succeeded with the same commit). This intermittent pattern is the exact signature of kube-router not adding short-lived pod IPs to ipsets. If #127 is the root cause, migrating URLs alone will not fix the problem.
• #133 (CI clone broken) — done, resolved. Clone step now works for repos that adopted alpine/git.
• #138 (split-horizon DNS) — done, resolved. But only fixes host-level DNS, not CoreDNS inside the cluster.
• #110 (westside-app Harbor auth) — done, resolved.

Acceptance Criteria

• "Identify root cause (auth vs network vs kube-router ipset vs service resolution)" — Good, but the ticket does not provide a diagnostic runbook. An agent would need to blindly probe. The pipeline logs already show the error (dial tcp 10.43.131.178:443: i/o timeout) which points to network/ipset, not auth or DNS. This evidence should be in the ticket.
• "Service repo pipelines can push images to Harbor reliably" — Testable, but "reliably" is undefined. Should specify: N consecutive successful pushes, or success rate threshold.
• "mcd-tracker repos migrated from external to internal Harbor URL" — Clear and testable. But missing minio-api.
• "All repos use consistent harbor.harbor.svc.cluster.local + insecure: true pattern" — Clear and testable.

Blast Radius

• minio-api is affected but not listed in the ticket's table. Uses harbor.tail5b443a.ts.net without insecure — identical pattern to mcd-tracker repos.
• The DORA exporter image in terraform/variables.tf:119 uses harbor.tail5b443a.ts.net as its default value. This is a Terraform variable (not a CI push target), but if pods pull this image at runtime, they would also be affected by the same DNS/network issues.
• If the root cause is kube-router ipset stale (#127), then ALL CI steps that connect to ANY NetworkPolicy-protected namespace are affected — not just Harbor. Fixing Harbor URLs alone would mask the systemic issue.
• Harbor namespace currently has NO NetworkPolicy deployed despite being defined in Terraform. When tofu apply eventually runs, the policy will be created and the woodpecker-to-harbor rule will activate. This is a latent state drift that could cause a surprise outage if the ipset issue is not resolved first.

Recommendation

• Add the actual error message to the ticket body. Pipeline logs show: dial tcp 10.43.131.178:443: i/o timeout (westside-app pipeline #73). The "Exact error message TBD" must be replaced.
• Add minio-api to the affected repos table. It has the same external URL pattern as mcd-tracker repos.
• Document the #127 dependency explicitly as a potential blocker. The intermittent failure pattern (pipeline #73 fails, #74 succeeds on same commit) is the kube-router ipset signature. If #127 is the root cause, URL migration alone will not fix this.
• Add a File Targets section listing each file to modify (network-policies.tf, plus each repo's .woodpecker.yaml) with specific changes.
• Add Test Expectations — at minimum: "trigger a pipeline on each affected repo and verify build-and-push step succeeds."
• Clarify multi-repo strategy — state whether this single issue covers all repos or whether child issues will be created per repo.
• Note the Harbor NetworkPolicy state drift — the policy is defined in Terraform but not deployed. The ticket should acknowledge this and decide whether to apply it as part of this fix or defer.