pal-e-docs

Review: Bug: Blackbox probe TLS failure (expanded to 4 probes)

review-385-2026-03-26c Review

review ready
Project

Verdict: READY

Template Completeness

  • [x] Lineage — present (standalone, discovered during AlertManager triage)
  • [x] Repo — present (forgejo_admin/pal-e-platform)
  • [x] Context / What Broke — present (TLS hairpin routing, 5-day firing alert)
  • [x] File Targets — present (comment #2 + #4: exact URLs per probe)
  • [x] Acceptance Criteria — present (comment #2: 4 probes switched, 14 total pass, tofu plan shows only 4 changes)
  • [x] Test Expectations — present (comment #4: tofu plan -lock=false + PromQL probe_success{job="blackbox"})
  • [x] Constraints — present (comment #2: follow precedent #117 / commit 4213fde)
  • [ ] Checklist — missing (standard PR/test checklist not present). Minor: agent will follow standard PR workflow regardless.
  • [x] Related — present (project, user story, arch components)

File Targets

  • [x] terraform/main.tf line 483-485 — verified: pal-e-docs probe uses external URL https://pal-e-docs.tail5b443a.ts.net/healthz
  • [x] terraform/main.tf line 488-489 — verified: pal-e-app probe uses external URL https://pal-e-app.tail5b443a.ts.net
  • [x] terraform/main.tf line 498-499 — verified: westside-app probe uses external URL https://westsidekingsandqueens.tail5b443a.ts.net
  • [x] terraform/main.tf line 503-504 — verified: westside-dev probe uses external URL https://westside-dev.tail5b443a.ts.net
Note: Line numbers in scope comments (~451, ~456, ~466, ~471) are ~30 lines off from actual (483, 488, 498, 503). This is cosmetic — the probe names and URLs are correct and unambiguous.

Internal URL Verification (cross-checked against pal-e-deployments)

  • [x] http://pal-e-docs.pal-e-docs.svc.cluster.local:8000/healthz — confirmed: namespace pal-e-docs, base service port 8000, /healthz endpoint
  • [x] http://pal-e-app.pal-e-app.svc.cluster.local:3000 — confirmed: namespace pal-e-app, service port overridden to 3000 in kustomization
  • [x] http://westside-app.westsidekingsandqueens.svc.cluster.local:3000 — confirmed: namespace westsidekingsandqueens (NOT westside-app), service port overridden to 3000
  • [x] http://westside-dev.westsidekingsandqueens.svc.cluster.local:80 — confirmed: namespace westsidekingsandqueens (NOT westside-dev), service port 80 (targetPort 5174)

Repo Placement

OK. All 4 probe URLs are in terraform/main.tf within helm_release.blackbox_exporter values in pal-e-platform. Single-repo, single-file fix. Forgejo issue filed on correct repo.

Dependencies

None. No board items block this work. No in_progress items touch blackbox exporter config. NetworkPolicies are all commented out (disabled due to kube-router ipset bug), so no cross-namespace ingress rules needed for the internal probes.

Acceptance Criteria

All criteria are agent-verifiable:
  • tofu plan -lock=false — agent can run and verify exactly 4 URL changes
  • probe_success{job="blackbox"} == 1 for all 14 targets — verifiable via Prometheus API after apply
  • AlertManager EndpointDown clearing — verifiable via AlertManager API

Blast Radius

Scope already covers full blast radius. All 4 external-URL probes are included. The remaining 10 probes already use internal URLs and are unaffected. Changing from external HTTPS to internal HTTP means:
  • Probes test service health, not funnel/TLS path — acceptable tradeoff (same pattern as Keycloak fix #117)
  • If a Tailscale funnel breaks, the probe will not detect it — this is a known limitation, consistent with existing probe strategy
No downstream consumers affected. Comment at main.tf line 481 ("Application services (external URLs — validates full funnel path)") should be updated to reflect the new strategy.

Recommendation

READY for next_up. The missing Checklist section is minor — agents follow standard PR workflow regardless. The line number offsets in scope comments are cosmetic; probe names make targets unambiguous. All 4 internal URLs have been independently verified against pal-e-deployments kustomization overlays. The namespace trap (westside services in westsidekingsandqueens namespace) is correctly documented in comment #4. One suggested addition for the implementing agent: update the code comment at line 481 from "external URLs" to "internal URLs" after the change.