pal-e-docs

Review: Bug: Blackbox probe TLS failure on pal-e-app funnel (expanded to 4 probes)

review-385-2026-03-26b Review

review needs-refinement
Project

Verdict: NEEDS_REFINEMENT

Template Completeness

  • [x] Lineage — standalone bug, discovered during AlertManager triage 2026-03-26
  • [x] Repo — forgejo_admin/pal-e-platform
  • [ ] User Story — missing (acceptable for bug type; "What Broke" substitutes)
  • [x] Context — root cause documented: hairpin TLS from inside cluster to external funnel URL
  • [x] File Targets — added in scope correction comment: terraform/main.tf lines ~451, ~456, ~466, ~471
  • [x] Acceptance Criteria — updated in scope correction: 4 probes switched, alert clears, 14 probes pass, tofu plan shows only 4 changes
  • [ ] Test Expectations — partially present. First review comment mentions tofu plan -lock=false + probe_success query, but scope correction omits the actual test commands
  • [x] Constraints — precedent documented: issue #117 / commit 4213fde
  • [ ] Checklist — missing standard PR/test checklist
  • [x] Related — project, user story, and architecture labels present

File Targets

  • [x] terraform/main.tf:451 — verified: pal-e-docs probe uses https://pal-e-docs.tail5b443a.ts.net/healthz (external funnel URL)
  • [x] terraform/main.tf:456 — verified: pal-e-app probe uses https://pal-e-app.tail5b443a.ts.net (external funnel URL)
  • [x] terraform/main.tf:466 — verified: westside-app probe uses https://westsidekingsandqueens.tail5b443a.ts.net (external funnel URL)
  • [x] terraform/main.tf:471 — verified: westside-dev probe uses https://westside-dev.tail5b443a.ts.net (external funnel URL)
  • [x] Other probes (forgejo, woodpecker, grafana, alertmanager, harbor, argocd, keycloak, minio, basketball-api, platform-validation) — confirmed already using internal svc.cluster.local URLs

Repo Placement

OK — all changes are in forgejo_admin/pal-e-platform, which is where the blackbox exporter Helm release and probe config live. Single repo, single PR.

Dependencies

  • No items in in_progress column on the board — no blocking dependencies
  • No NetworkPolicies exist on target namespaces (pal-e-app, pal-e-docs, westsidekingsandqueens) — monitoring pods can reach internal services without NetworkPolicy changes
  • Precedent commit 4213fde confirmed in git history: "fix: add monitoring ingress to Keycloak NetworkPolicy + use internal probe URL (#117)" — same pattern
  • Board item #385 is in todo column, no blockers documented

Acceptance Criteria

Updated criteria are testable and verifiable. Each criterion maps to a concrete check: tofu plan diff, AlertManager state, Prometheus probe_success metric. However, the scope correction should include the actual test commands:
  • tofu plan -lock=false — verify only 4 probe URL changes
  • PromQL: probe_success{job="blackbox"} — verify all 14 probes return 1

Blast Radius

Scope correction already expanded from 1 to 4 probes — good. After this fix, zero blackbox probes will use external funnel URLs. No downstream consumers affected. No NetworkPolicy changes needed (target namespaces have no default-deny ingress).

Recommendation

Two issues must be addressed before moving to next_up:
  • Specify exact internal URLs in the scope correction. The scope correction says "change to internal service URL" for pal-e-docs, westside-app, and westside-dev without providing the actual URLs. Critically, the westside-app and westside-dev services live in the westsidekingsandqueens namespace (not westside-app or westside-dev). An agent would likely guess wrong. Verified internal URLs:
  • Add Test Expectations and Checklist sections to the Forgejo issue body or scope correction. Include tofu plan -lock=false and the PromQL verification query.