pal-e-docs

Review: Monitoring: golden signals + blackbox probe

review-792-2026-04-03 Doc

review needs-refinement
Project

Verdict: NEEDS_REFINEMENT

Template Completeness

  • [x] Type — Feature
  • [x] Lineage — references #1 and #3 as prerequisites
  • [x] Repo — forgejo_admin/pal-e-platform
  • [x] User Story — present, well-formed (platform operator wants golden signals + probes)
  • [x] Context — clear motivation (stream uptime = revenue + trust)
  • [x] File Targets — 3 files listed with clear intent
  • [x] Acceptance Criteria — 4 criteria
  • [x] Test Expectations — 3 expectations including tofu plan -lock=false
  • [x] Constraints — present (follow existing patterns, tofu plan -lock=false, dependency ordering)
  • [x] Checklist — present
  • [x] Related — present with cross-references to project page, arch note, and prerequisite issues

Traceability

  • [x] story:observability label — "Platform Observability"
  • [x] story note verified — story-twitch-2k-wager-observability exists in pal-e-docs (id: 1166)
  • [ ] story NOT listed on project page — [SCOPE] The observability story is missing from the project-twitch-2k-wager user-stories table. Only challenger-auth, challenger-pay, game-status, and winner-payout are listed. Add observability row to the table.
  • [x] arch:monitoring label present
  • [ ] arch note MISSING — [SCOPE] No arch-monitoring note exists in pal-e-docs. This work uses pal-e-platform's existing monitoring module, not a new twitch-2k-wager-specific component. Recommend either: (a) create arch-monitoring note describing the platform monitoring module, or (b) change label to arch:deployment since arch-deployment-twitch-2k-wager already exists and covers the deployment topology this monitoring extends.
  • [x] Forgejo issue — forgejo_admin/twitch-2k-wager#5, state: open

File Targets

  • [x] terraform/modules/monitoring/main.tf — verified: file exists (844 lines). Blackbox exporter targets list at lines 342-422. New probe entry follows existing pattern (name/url/labels object). Dashboard ConfigMap pattern at lines 646-660 shows how to wire a new JSON file.
  • [x] terraform/dashboards/ — verified: directory exists with 4 JSON files. pal-e-docs-golden-signals.json is the template to copy. New file twitch-2k-wager-golden-signals.json would be created here.
  • [ ] terraform/modules/ops/main.tf — ISSUE: ticket says "add service to ops module if applicable." File exists (294 lines) but contains NVIDIA device plugin, Ollama, embedding worker metrics, and TF state backup CronJob — none related to service monitoring. This target is not needed. [BODY] Remove or replace the ops/main.tf file target. All monitoring work belongs in monitoring/main.tf and dashboards/.

Repo Placement

Issue filed on forgejo_admin/twitch-2k-wager but all work targets forgejo_admin/pal-e-platform. The issue body correctly declares Repo: forgejo_admin/pal-e-platform. Cross-repo pattern is acceptable — project issues live on the project repo, implementation may touch platform. Single target repo (pal-e-platform), so no additional Forgejo issues needed.

Dependencies

  • [ ] twitch-2k-wager#1 (service onboarding) — pending (backlog). Required for service to exist in cluster.
  • [ ] twitch-2k-wager#3 (kustomize overlay) — pending (backlog). Required for ArgoCD deployment.
Both prerequisites are in backlog. The blackbox probe URL (https://playme2k.tail5b443a.ts.net/health) requires the service to be deployed. tofu plan will succeed but the probe will fire EndpointDown alerts until dependencies are satisfied. This ordering concern is documented in the ticket Constraints section — acceptable.

Acceptance Criteria

4 AC, assessment:
  • [x] "Blackbox probe configured" — testable via tofu plan output showing new target in blackbox_exporter helm values.
  • [x] "Grafana dashboard shows golden signals" — testable: agent can verify JSON file has latency/traffic/errors/saturation panels.
  • [ ] "Probe alerts on downtime (5xx or timeout)" — ambiguous. The existing EndpointDown PrometheusRule (line 428-476) already fires on probe_success == 0 for ALL blackbox targets. Agent does NOT need to create a new alert rule. [BODY] Rephrase to: "Existing EndpointDown alert covers the new probe target (no new alert rule needed)." This prevents the agent from creating a redundant PrometheusRule.
  • [x] "Dashboard accessible in Grafana" — manual post-apply verification, reasonable.

Blast Radius

Low risk. All changes are additive:
  • New entry in blackbox exporter targets list (no modification to existing probes)
  • New dashboard JSON file (no modification to existing dashboards)
  • New ConfigMap resource in monitoring/main.tf
Rollback: remove the three additions and tofu apply.
Note on probe URL pattern: the ticket specifies the external Tailscale funnel URL (https://playme2k.tail5b443a.ts.net/health) whereas all existing application probes use internal cluster URLs (e.g., http://pal-e-docs.pal-e-docs.svc.cluster.local:8000/healthz). Internal URLs are faster and avoid funnel round-trips. [BODY] Consider changing to internal URL: http://twitch-2k-wager-app.twitch-2k-wager.svc.cluster.local:3000/health — or document why external URL is intentionally preferred (e.g., testing the full funnel path).

Decomposition Assessment

Apply three-thing limit and five-minute rule:
  • Discrete changes: 3 (add probe target, create dashboard JSON, add ConfigMap resource). At the limit but tightly coupled — all part of one monitoring setup.
  • Repos: 1 (pal-e-platform)
  • AC count: 4
  • Estimated agent time: under 5 minutes
  • Independent subtasks: none — dashboard and probe are logically coupled
No decomposition needed.

Recommendation

  • [SCOPE] Add observability story row to project-twitch-2k-wager user-stories table (story note exists but is not listed on the project page).
  • [SCOPE] Either create arch-monitoring note for the platform monitoring module, or change board item label from arch:monitoring to arch:deployment (which has a backing note).
  • [BODY] Remove terraform/modules/ops/main.tf from file targets — ops module is unrelated to monitoring work.
  • [BODY] Clarify AC "Probe alerts on downtime" — existing EndpointDown alert already covers all blackbox targets. Rephrase to prevent agent from creating redundant PrometheusRule.
  • [BODY] Consider changing probe URL from external funnel URL to internal cluster URL to match existing app probe patterns, or document why external is preferred.