pal-e-docs

Review: Observability: blackbox probe + golden signals

review-826-2026-04-03 Doc

review ready
Project

Verdict: READY

Template Completeness

  • [x] Type — Feature
  • [x] Lineage — "Required for production. Marcus needs uptime assurance during streams."
  • [x] Repo — forgejo_admin/pal-e-platform
  • [x] User Story — clear operator story with motivation
  • [x] Context — explains monitoring pattern and why it matters for playme2k
  • [x] File Targets — 2 specific paths with descriptions
  • [x] Acceptance Criteria — 4 testable conditions
  • [x] Test Expectations — tofu plan + Prometheus verification
  • [x] Constraints — pattern matching, internal URL, lock flag
  • [x] Checklist — PR, tests, no unrelated changes
  • [x] Related — links to story note, arch note, project page

Traceability

  • [x] story:observability — Platform Observability. Story note story-twitch-2k-wager-observability (id 1166) exists. Key "observability" listed in project-twitch-2k-wager user-stories table.
  • [x] arch:deployment-twitch-2k-wager — Deployment topology. Arch note arch-deployment-twitch-2k-wager (id 1156) exists in pal-e-docs.
  • [x] Forgejo issue — forgejo_admin/twitch-2k-wager#57, open

File Targets

  • [x] terraform/modules/monitoring/main.tf — verified exists. Blackbox exporter targets array at line 342. Dashboard ConfigMap pattern at line 644. Agent adds a new target entry and a new ConfigMap resource.
  • [x] terraform/dashboards/ — verified directory exists. pal-e-docs-golden-signals.json available as copy template. playme2k-golden-signals.json does not yet exist (expected — ticket creates it).

Repo Placement

OK. Issue filed on forgejo_admin/twitch-2k-wager (service repo) for traceability. ### Repo field correctly identifies forgejo_admin/pal-e-platform as the target repo for code changes. Single-repo change — no cross-repo coordination needed.

Dependencies

  • [x] pal-e-platform repo access — satisfied (working directory)
  • [x] Existing monitoring patterns — satisfied (blackbox exporter + dashboard ConfigMaps already deployed)
  • [ ] playme2k service running — pending (board item #825, issue #56 "Fix: prod pod" is in backlog). However, Terraform resources can be applied independently. Probe will report "down" until service is live, matching existing patterns. Not a blocking dependency for implementation.

Acceptance Criteria

4 AC, all testable by an agent:
  • "Blackbox probe configured" — verifiable via tofu plan -lock=false output showing new target entry
  • "Probe appears in Prometheus targets" — verifiable post-apply via Prometheus UI (post-merge validation, not agent-gated)
  • "Grafana dashboard shows golden signals" — verifiable via ConfigMap creation in plan output
  • "Alerts fire on downtime" — covered automatically by existing EndpointDown PrometheusRule (line 445, probe_success == 0) which fires for all blackbox targets. No new alert rule needed.
Test expectations are realistic: tofu plan -lock=false for pre-merge, Prometheus target check for post-apply.

Blast Radius

  • 2 files touched in 1 repo (monitoring/main.tf + new dashboard JSON)
  • No changes to existing probes or dashboards
  • Existing EndpointDown alert automatically covers the new target
  • No downstream consumers affected
  • Rollback: remove target entry + ConfigMap + delete dashboard JSON. Straightforward.

Decomposition Assessment

  • 2 file targets in 1 repo — under the 3-file limit
  • 4 acceptance criteria — under the 5 AC limit
  • Estimated agent time: 2-3 minutes (copy dashboard template + add probe target + add ConfigMap) — under the 5-minute rule
  • No independent subtasks that need parallelization
  • No decomposition needed.

Recommendation

  • No action needed. Scope is solid, all file targets verified, traceability complete, fits in a single agent pass.
Implementation note: The probe URL in the Constraints section (http://twitch-2k-wager-app.playme2k.svc.cluster.local:3000/) uses namespace playme2k while the arch diagram says twitch-2k-wager namespace. The agent should verify the actual k8s namespace at implementation time by checking the pal-e-deployments kustomize overlay. This is an implementation detail, not a scoping issue.