pal-e-docs

Review: Bug: Blackbox probe TLS failure on pal-e-app funnel

review-385-2026-03-26 Review

review needs-refinement
Project

Verdict: NEEDS_REFINEMENT

Template Completeness

  • [x] Type — present (bug)
  • [x] Lineage — present (standalone)
  • [x] Repo — present (forgejo_admin/pal-e-platform)
  • [x] Context — present (What Broke + Repro Steps + Expected Behavior + Environment cover this well)
  • [ ] File Targets — MISSING. The fix is in terraform/main.tf lines 454-457 (change pal-e-app probe URL from external funnel to internal service). This must be stated explicitly.
  • [x] Acceptance Criteria — present (3 criteria, all verifiable)
  • [ ] Test Expectations — MISSING. Should specify: run tofu plan -lock=false to verify only the probe URL changes; after apply, query probe_success{instance=~".*pal-e-app.*"} == 1 in Prometheus.
  • [ ] Constraints — MISSING. Should note: match the existing pattern from issue #117 (Keycloak probe fix commit 4213fde); must use internal http://pal-e-app.pal-e-app.svc:3000 URL format.
  • [ ] Checklist — MISSING. Standard PR/test checklist needed.
  • [x] Related — present (project, story, arch labels)

File Targets

  • [x] terraform/main.tf line 454-457 — verified: pal-e-app probe target uses external URL https://pal-e-app.tail5b443a.ts.net. This is the file that needs modification.
  • [x] Internal service endpoint confirmed: pal-e-app Service exists in namespace pal-e-app on port 3000 (verified via pal-e-deployments/overlays/pal-e-app/prod/kustomization.yaml).

Repo Placement

OK. The Forgejo issue is filed on forgejo_admin/pal-e-platform and the fix is in terraform/main.tf within this repo. Single repo fix.

Dependencies

  • No blockers found on the board.
  • Issue #138 (split-horizon DNS) is in done — it fixed the Woodpecker OAuth hairpin but did NOT deploy CoreDNS rewrites. No in-cluster DNS rewrite exists.
  • Issue #117 / commit 4213fde (Keycloak probe fix) is the exact precedent — switched Keycloak probe from external funnel URL to internal service URL. Same pattern applies here.
  • No dependencies need to be documented.

Acceptance Criteria

All three criteria are verifiable after apply:
  • "Blackbox probe succeeds" — query probe_success{instance=~".*pal-e-app.*"} in Prometheus. Testable.
  • "EndpointDown alert clears" — check AlertManager UI. Testable.
  • "No regression on other probes" — query probe_success for all 13 targets. Testable.
Missing: specific tofu plan validation command and Prometheus query strings. An agent could figure this out but explicit commands reduce ambiguity.

Blast Radius

WARNING: Three other probes use external funnel URLs with the same hairpin risk:
  • pal-e-docs — https://pal-e-docs.tail5b443a.ts.net/healthz (line 451)
  • westside-app — https://westsidekingsandqueens.tail5b443a.ts.net (line 466)
  • westside-dev — https://westside-dev.tail5b443a.ts.net (line 471)
If pal-e-app is failing due to hairpin TLS, these three targets may also be failing or intermittently failing. The ticket should either (a) scope all four fixes together, or (b) explicitly note the other three as discovered scope for separate tickets.

Recommendation

Before moving to next_up, the issue needs:
  • Add File Targets section — terraform/main.tf lines 454-457. Change URL from https://pal-e-app.tail5b443a.ts.net to http://pal-e-app.pal-e-app.svc:3000.
  • Add Test Expectations — tofu plan -lock=false shows only probe URL change; post-apply Prometheus query confirms probe_success=1.
  • Add Constraints — follow pattern from issue #117 / commit 4213fde.
  • Address blast radius — decide whether to fix all 4 external-funnel probes in this ticket or create separate tickets for pal-e-docs, westside-app, westside-dev.
  • Add Checklist — standard PR opened / tests pass / no unrelated changes.