Observability Baseline Audit (2026-03-13)
Observability Baseline Audit (2026-03-13)
Summary
Phase 2 of
plan-pal-e-platform. Baseline verification of the monitoring stack: what's collecting, what's alerting, what's broken.Prometheus
26 total targets. 23 UP, 3 DOWN.
| Scrape Pool | Status | Notes |
|---|---|---|
| kube-prometheus-stack (12 targets) | ALL UP | apiserver, coredns, grafana, kube-state-metrics, kubelet (3), node-exporter, operator, prometheus (2), alertmanager (2) |
| harbor (4 targets) | ALL UP | core, jobservice, registry, nginx |
| cnpg-system | UP | CNPG operator |
| pal-e-postgres | UP | Postgres instance metrics |
| basketball-api | UP | App metrics |
| pal-e-docs | UP | App metrics |
| platform-validation | UP | Smoke test service |
| dora-exporter | UP | DORA metrics |
| gmail-mcp-remote | DOWN | Connection refused on :8000/metrics |
| linkedin-scheduler-remote | DOWN | Connection refused on :8000/metrics |
| notion-mcp-remote | DOWN | Connection refused on :8000/metrics |
Verdict: Core platform and all healthy services are scraped. 3 DOWN targets are MCP remote services with broken pods — not a Prometheus configuration issue.
Grafana
3 datasources configured:
- Prometheus (default) — scrape data
- Loki — container logs
- Alertmanager — alert state
Default kube-prometheus-stack dashboards are deployed (node exporter, k8s resources, pod metrics). No custom dashboards yet. Admin password: from k8s secret
kube-prometheus-stack-grafana.Loki
Ready. 17 namespaces ingesting logs.
Namespaces: argocd, basketball-api, cnpg-system, forgejo, harbor, kube-system, monitoring, ollama, pal-e-app, pal-e-docs, pal-e-frontend, palworld, platform-validation, playground, postgres, tailscale, woodpecker.
10 label keys available: app, component, container, filename, instance, job, namespace, node_name, pod, stream.
Sample query
{namespace="pal-e-docs"} returned healthz logs confirming ingestion is working.Retention: 7 days (configured in Helm values).
Alertmanager
22 active alerts. No notification channels configured.
The Alertmanager is running with the default kube-prometheus-stack config template. Global config references Slack, PagerDuty, and OpsGenie URLs but these are template defaults — no actual receivers are wired up. The
Watchdog alert fires continuously as expected (proves Alertmanager is functional).Alert Inventory
| Category | Count | Namespaces | Root Cause |
|---|---|---|---|
| MCP Remote Services | 12 | gmail-mcp-remote, linkedin-scheduler-remote, notion-mcp-remote | Pods not ready / connection refused. Services onboarded but images broken or missing config. |
| westside-app | 4 | westsidekingsandqueens | App not yet deployed — deployment exists but image not available. 2 pods stuck. |
| basketball-api-dev | 3 | basketball-api-dev | Dev namespace with stale/broken deployment. Container waiting > 1 hour. |
| NodeClockNotSynchronising | 1 | monitoring (host) | NTP not configured on host. Salt state needed. |
| Watchdog | 1 | monitoring | Expected — certifies Alertmanager is alive. |
Gaps Identified
- No notification channels — Alertmanager has no real receivers. 22 alerts firing into the void. Phase 3 (Alerting) should configure at least one channel.
- No custom dashboards — Only kube-prometheus-stack defaults. No service-level dashboards (pal-e-docs request latency, basketball-api throughput, etc.). Phase 4 should address this.
- 3 broken services generating 12 alerts — MCP remotes need fixing or scaling to 0. Alert noise makes real problems harder to spot.
- Dev namespace pollution — basketball-api-dev has a broken deployment generating 3 alerts. Should be cleaned up or scaled to 0.
- westside-app not deployed — 4 alerts from a deployment that was never completed.
- NTP not configured — NodeClockNotSynchronising. Host-level fix via Salt.
- No SLOs defined — User stories reference MTTR and SLO breach alerting, but no SLOs exist yet.
Baseline Summary
| Component | Status | Health |
|---|---|---|
| Prometheus | Running, 15d retention | 88% targets UP (23/26) |
| Grafana | Running, 3 datasources | Functional, default dashboards only |
| Loki | Running, 7d retention | 17 namespaces ingesting |
| Promtail | Running (DaemonSet) | Collecting from all nodes |
| Alertmanager | Running, no receivers | 22 alerts firing undelivered |
| Node Exporter | Running | Host metrics scraped |
| kube-state-metrics | Running | K8s object metrics scraped |
Related
plan-pal-e-platform— Platform Hardening planphase-observability-2-verify-baseline— this phasephase-observability-3-alerting— next phase (configure notification channels)project-pal-e-platform— project page