Observability Audit: pal-e-platform
Observability Audit: pal-e-platform (2026-02-25)
Honest assessment of what we have deployed vs what we're actually using.
What's Deployed
| Component | Chart/Source | Namespace | Status |
|---|---|---|---|
| Prometheus Operator | kube-prometheus-stack 82.0.0 | monitoring | Running |
| Prometheus | (included) | monitoring | Running, 15d retention, 15Gi storage |
| Alertmanager | (included) | monitoring | Running but unconfigured |
| Grafana | (included) | monitoring | Running (fixed 2026-02-25, was CrashLoopBackOff) |
| kube-state-metrics | (included) | monitoring | Running |
| node-exporter | (included, DaemonSet) | monitoring | Running on all nodes |
| Loki | loki-stack 2.10.3 | monitoring | Running, 7d retention, 10Gi storage |
| Promtail | (included, DaemonSet) | monitoring | Running, ships logs to Loki |
What We're Using
| Capability | Available | Using? | Gap |
|---|---|---|---|
| Host metrics (CPU, memory, disk) | Yes (node-exporter) | Collected, not dashboarded | Need custom Grafana dashboard for archbox resource usage |
| K8s object metrics | Yes (kube-state-metrics) | Collected, default dashboards | Should review default dashboards, verify they're useful |
| Service metrics via /metrics | Yes (ServiceMonitor CRD) | Some services have ServiceMonitors | MCP services not yet onboarded, need to verify existing ones scrape |
| Log aggregation | Yes (Loki + Promtail) | Collected automatically | No LogQL queries, no log-based dashboards, no log alerting |
| Alerting | Yes (Alertmanager + PrometheusRules) | Not at all | No rules defined, no alert routing, no notification channels |
| Custom dashboards | Yes (Grafana sidecar auto-discovers) | Not at all | No service-specific dashboards |
| Tracing | Not deployed | No | Would need Jaeger/Tempo + OpenTelemetry instrumentation |
The Four Golden Signals (Google SRE)
Google's SRE book defines four signals that every service should monitor. Here's our status:
| Signal | What It Measures | Our Status |
|---|---|---|
| <strong>Latency</strong> | Time to serve a request (distinguish success vs error latency) | Not measured — services don't expose latency histograms |
| <strong>Traffic</strong> | Request rate (HTTP requests/sec, transactions/sec) | Not measured — no request counters exposed |
| <strong>Errors</strong> | Rate of failed requests (5xx, timeouts, application errors) | Not measured — no error rate metrics |
| <strong>Saturation</strong> | How full the service is (CPU, memory, queue depth) | Partially — node-exporter and kube-state-metrics give us pod/node saturation, but not application-level (e.g., connection pool usage) |
What Prometheus Is Scraping Today
To check:
kubectl port-forward svc/kube-prometheus-stack-prometheus -n monitoring 9090:9090 then visit http://localhost:9090/targets. This shows every scrape target and whether it's UP or DOWN.Known targets: kubelet, kube-state-metrics, node-exporter, Prometheus itself, Alertmanager, any ServiceMonitors in any namespace.
Recommendations (Ordered by Impact)
- Verify Prometheus targets are UP — before adding more, confirm what we have actually works
- Add basic PrometheusRules — pod restart alerts, OOM alerts, node disk pressure. These catch real problems.
- Configure Alertmanager routing — even if just to a Slack webhook or email. Alerts that nobody sees are useless.
- Build one service dashboard — pick one service (e.g., pal-e-docs), add request rate + latency + error rate, prove the pattern
- Then onboard MCP services with the proven pattern
Resource Bounds Reference
When diagnosing performance issues, identify which resource is the bottleneck:
| Bound | Bottleneck | Symptoms | k8s Signals |
|---|---|---|---|
| <strong>CPU-bound</strong> | Computation | CPU at 100%, slow responses, high latency | CPU throttling in cAdvisor metrics, high <code>container_cpu_usage_seconds_total</code> |
| <strong>Memory-bound</strong> | Available RAM | OOMKilled, swap thrashing, GC pauses | <code>container_memory_working_set_bytes</code> near limit, OOMKilled events in pod describe |
| <strong>IO-bound</strong> | Disk or network | CPU idle but slow, high iowait | High <code>node_disk_io_time_seconds_total</code>, slow PVC operations |
| <strong>Network-bound</strong> | Bandwidth or latency | Timeouts on external API calls, slow inter-service calls | High <code>node_network_transmit_bytes_total</code>, TCP retransmits |
Race conditions are a concurrency problem, not a resource bound. They happen when multiple threads or processes access shared state without synchronization. They can manifest in any bound — two processes writing the same file (IO), two threads modifying the same variable (memory), two pods writing the same database row (network+IO). The root cause is always unsynchronized concurrent access, regardless of which resource is involved.