Project: Landscaping Observability
Vision
Full Datadog-equivalent observability for the landscaping-assistant app and pal-e platform using the open-source Grafana ecosystem. Everything stores in MinIO, queries through Grafana, alerts through Alertmanager. The Datadog capability matrix is the benchmark — every row moves from NOT STARTED to COMPLETE. Detailed rollout in
docs/observability-roadmap.md (landscaping-assistant repo).User Stories
| Key | Role | Summary | Success Metric |
|---|---|---|---|
| tracing | Platform Operator | Trace requests end-to-end across services | Click a slow request in Grafana, see the full trace with span-level latency |
| database-visibility | Platform Operator | Identify slow queries and database bottlenecks | Grafana dashboard shows top queries by time, calls, rows; slow queries in Loki |
| profiling | Platform Operator | See where CPU/memory time goes per endpoint | Flame graphs in Grafana linked to specific traces |
| cluster-viz | Platform Operator | Visual cluster overview without kubectl | Web UI showing namespaces, pods, resource usage, log streaming |
| frontend-monitoring | Platform Operator | See what users experience in the browser | Core Web Vitals, JS errors, frontend-to-backend trace correlation |
| slo-tracking | Platform Operator | Measure reliability against targets with error budgets | Grafana dashboard shows budget remaining and burn rate per service |
| runtime-security | Platform Operator | Detect anomalous container behavior | Falco alerts for unexpected shells, file access, outbound connections via Alertmanager |
Architecture
Single deployment target: pal-e k8s cluster (single-node k3s). All observability components deploy via Terraform Helm releases in pal-e-platform. Service instrumentation ships via app-level SDK changes + kustomize overlay env vars in pal-e-deployments.
The stack is the Grafana LGTP ecosystem:
- Loki — logs (COMPLETE, 7d retention, MinIO backend)
- Grafana — dashboards, explore, alerting (COMPLETE, 26 dashboards)
- Tempo — traces (NOT STARTED, MinIO backend)
- Pyroscope — profiles (NOT STARTED, MinIO backend)
Supporting components: OpenTelemetry Collector (trace ingestion), Faro (frontend RUM), Headlamp (cluster UI), Pyrra (SLO engine), Falco (runtime security).
Key decisions:
- Tempo over Jaeger — native Grafana integration, MinIO backend matches Loki pattern
- Pyroscope over standalone pprof — trace-to-profile linking, Grafana-native
- OTel Collector as DaemonSet, not sidecar — simpler, matches Promtail pattern
- Tempo in monolithic mode — single-node cluster doesn't need microservices mode
- Pyrra over Sloth — better Grafana integration for SLO dashboards
Datadog Benchmark
The scorecard. Every Datadog capability mapped to our open-source equivalent.
| Datadog Capability | Our Equivalent | Status | Issue |
|---|---|---|---|
| Infrastructure Monitoring | kube-prometheus-stack + node-exporter | COMPLETE | — |
| Log Management | Loki + Promtail + Grafana Explore | COMPLETE | — |
| Application Metrics | yabeda-rails + yabeda-prometheus | COMPLETE | #19 (done) |
| Dashboards | Grafana (26 dashboards + golden signals) | COMPLETE | #16 (done) |
| Alerting | Alertmanager → Telegram + Slack | PARTIAL | #17 |
| CI Visibility | DORA exporter + Woodpecker API | COMPLETE | #20 (done) |
| Synthetic Monitoring | Blackbox exporter (13 probes) | COMPLETE | #21 (done) |
| Container Scanning | Harbor Trivy | COMPLETE | — |
| APM / Distributed Tracing | OpenTelemetry + Grafana Tempo | NOT STARTED | #85 |
| Database Monitoring (query-level) | pg_stat_statements + Grafana dashboard | NOT STARTED | #86 |
| Continuous Profiling | Grafana Pyroscope | NOT STARTED | #87 |
| Infrastructure Visualization | Headlamp + Grafana NodeGraph | NOT STARTED | #88 |
| RUM (Frontend) | Grafana Faro | NOT STARTED | #89 |
| SLOs / Error Budgets | Pyrra | NOT STARTED | #90 |
| Runtime Security | Falco | NOT STARTED | #90 |
Board
Primary kanban:
board-landscaping-observability9 items: 1 done (#83 roadmap doc), 8 in backlog. 40 total points. All issues filed in
ldraney/landscaping-assistant repo, parent: #43.Dependency chain: #85 (Tempo) is the keystone — #87 (Pyroscope) and #89 (Faro) depend on it for trace linking. #86 (pg_stat_statements) and #88 (Headlamp) are independent.
Status
8 of 15 Datadog capabilities are COMPLETE (pre-existing platform infrastructure). 6 are NOT STARTED (the new work). 1 is PARTIAL (#17 PrometheusRule alerts, needs issue refinement). The foundation — Prometheus, Loki, Grafana, Alertmanager — is solid. The gaps are tracing, profiling, database query analysis, frontend monitoring, infrastructure visualization, SLOs, and runtime security.
Milestones
2026-06-03: Observability roadmap doc merged (PR #84). Datadog gap analysis, mermaid target architecture diagram, 6-phase rollout plan. Six child issues created (#85-#90).
Repos
| Repo | Platform | Role | Status |
|---|---|---|---|
| <a href="https://forgejo.tail5b443a.ts.net/ldraney/landscaping-assistant">landscaping-assistant</a> | Forgejo | App instrumentation (OTel SDK, Pyroscope gem, Faro JS) | Active |
| <a href="https://forgejo.tail5b443a.ts.net/ldraney/pal-e-platform">pal-e-platform</a> | Forgejo | Helm releases (Tempo, Pyroscope, Headlamp, Pyrra, Falco) | Active |
| <a href="https://forgejo.tail5b443a.ts.net/ldraney/pal-e-deployments">pal-e-deployments</a> | Forgejo | Kustomize overlays (env vars, CNPG config) | Active |