pal-e-docs

Phase: Synthetic Monitoring + Uptime Checks

phase-pal-e-platform-14-synthetic-monitoring Phase

Project

Phase: Synthetic Monitoring + Uptime Checks

Status: COMPLETED 2026-03-14. PR #67 merged.

Lineage

plan-pal-e-platform → Phase 14

Deliverables

  • DONE: Deployed Prometheus Blackbox Exporter via Helm (chart v9.1.0) in monitoring namespace
  • DONE: Configured 13 HTTP probe targets — 8 platform (internal URLs), 5 app (external Tailscale funnels)
  • DONE: Created PrometheusRule: EndpointDown (critical, 2m) + EndpointSlowResponse (warning, 5m)
  • DONE: Created Grafana uptime dashboard: overview stats, UP/DOWN matrix, latency timeseries, probe history
  • DONE: Dashboard auto-discovered by Grafana sidecar via grafana_dashboard: "1" label

Probe Targets

Name URL Tier
forgejo http://forgejo-http.forgejo.svc (internal) platform
woodpecker http://woodpecker-server.woodpecker.svc (internal) platform
grafana http://kube-prometheus-stack-grafana.monitoring.svc (internal) platform
alertmanager http://kube-prometheus-stack-alertmanager.monitoring.svc (internal) platform
harbor http://harbor-core.harbor.svc/api/v2.0/health (internal) platform
argocd http://argocd-server.argocd.svc (internal) platform
keycloak http://keycloak.keycloak.svc:8080/health/ready (internal) platform
minio-api http://minio.minio.svc:9000/minio/health/live (internal) platform
pal-e-docs https://pal-e-docs.tail5b443a.ts.net/api/health (external) app
pal-e-app https://pal-e-app.tail5b443a.ts.net (external) app
basketball-api https://basketball-api.tail5b443a.ts.net/api/health (external) app
westside-app https://westsidekingsandqueens.tail5b443a.ts.net (external) app
platform-validation https://platform-validation.tail5b443a.ts.net (external) app

PR

PR #67: forgejo_admin/pal-e-platform#67 — merged 2026-03-14 (squash)

Files Changed

  • terraform/main.tf — Blackbox Exporter Helm release, PrometheusRule, uptime dashboard ConfigMap
  • terraform/dashboards/dora-dashboard.json — fixed Lead Time panel queries + repo variable
  • terraform/dashboards/uptime-dashboard.json — new Grafana uptime dashboard
  • dora-framework — synthetic monitoring closes MTTR gap for endpoint-level failures
  • sop-incident-response — runbooks reference Blackbox Exporter alerts (EndpointDown, EndpointSlowResponse)
  • phase-pal-e-platform-15-dora-rebaseline — DORA dashboard fixes in same PR