Landscaping Observability

Verdict: NEEDS_REFINEMENT

Third scope review of this ticket. Two prior reviews (2026-06-01, 2026-06-02) flagged wrong repo and wrong metric names. Neither issue has been fixed. The issue body is unchanged since original filing. This ticket cannot be dispatched to an agent in its current state.

Template Completeness

• [x] Type -- Feature
• [x] Lineage -- Standalone, observability gap audit
• [ ] Repo -- WRONG. Says ldraney/pal-e-deployments. Should be ldraney/pal-e-platform. All existing PrometheusRules are Terraform kubernetes_manifest resources in terraform/modules/monitoring/main.tf, not Kustomize overlays. Verified: blackbox_alerts, embedding_alerts, payment_pipeline_alerts, gmail_oauth_expiry_alert all live there.
• [x] User Story -- Clear operator perspective, well-formed As/I want/So that
• [x] Context -- Good background, mentions ServiceMonitor dependency
• [ ] File Targets -- WRONG. Points to overlays/landscaping-assistant/prod/prometheusrule.yaml and overlays/landscaping-assistant/prod/kustomization.yaml in pal-e-deployments. Should point to terraform/modules/monitoring/main.tf in pal-e-platform (append new kubernetes_manifest resource blocks).
• [ ] Acceptance Criteria -- INCOMPLETE. References "5xx error rate" and "p95 latency" without specifying actual metric names or PromQL. Agent will guess wrong. See Acceptance Criteria section below for details.
• [ ] Test Expectations -- WRONG. Says kustomize build overlays/landscaping-assistant/prod/. Should say tofu plan against the monitoring module.
• [x] Constraints -- Good: for: 5m, API version monitoring.coreos.com/v1, tuning caveat
• [x] Checklist -- Present
• [x] Related -- References project-landscaping-assistant

Traceability

• [x] story:observability -- present on board item labels
• [x] arch:k8s-deploy -- present on board item labels
• [x] Forgejo issue -- exists and is open (landscaping-assistant#17)
• [ ] Board item title -- says "(needs refinement)" which is accurate, but the board item URL still points to landscaping-assistant/issues/17 despite the work targeting pal-e-platform

File Targets

• [ ] overlays/landscaping-assistant/prod/prometheusrule.yaml -- This file does not exist and should not be created here. The platform convention is to define PrometheusRules as Terraform kubernetes_manifest resources in pal-e-platform.
• [x] overlays/landscaping-assistant/prod/kustomization.yaml -- This file exists in pal-e-deployments, but should NOT be modified for this ticket. It is the wrong location for alerts.
• [ ] Missing correct target: terraform/modules/monitoring/main.tf in ldraney/pal-e-platform -- this is where the new kubernetes_manifest resource blocks should be appended, following the pattern of existing alert resources.

Repo Placement

The issue is filed on ldraney/landscaping-assistant (Rails app) but the work targets infrastructure code. Two options:

1. Preferred: Re-create the issue on ldraney/pal-e-platform where the Terraform code lives. Update the board item URL. Close this issue as "moved."
2. Acceptable: Keep the issue here but correct the ### Repo field to ldraney/pal-e-platform and update file targets. The implementing agent branch would be created on pal-e-platform.

Currently the issue, the repo field, and the file targets all disagree about where the work lives. This is the fundamental blocker.

Dependencies

• [x] ServiceMonitor -- SATISFIED. Base template bases/standard/servicemonitor.yaml exists. Prod kustomization.yaml patches it for landscaping-assistant namespace and label selectors.
• [x] /metrics endpoint -- SATISFIED. Gemfile includes yabeda-rails, yabeda-prometheus, yabeda-puma-plugin. Initializer at config/initializers/yabeda.rb confirms auto-configuration.
• [x] Prometheus scraping -- SATISFIED. ServiceMonitor scrapes /metrics on port http every 30s.

Acceptance Criteria

The AC says "high 5xx error rate (>5% over 5m), p95 latency spike (>2s over 5m), and zero available pods" but does not specify the actual PromQL. The app uses yabeda-rails which exposes:

• Counter: rails_requests_total with labels: controller, action, status, format, method
• Histogram: rails_request_duration (seconds) -- Prometheus exposes as rails_request_duration_bucket, rails_request_duration_sum, rails_request_duration_count

Common wrong guesses an agent would make: http_requests_total (generic Prometheus convention), http_request_duration_seconds_bucket (Go/generic convention). Neither metric exists on this app.

The AC should include explicit PromQL or at minimum the metric names. Suggested expressions:

• 5xx rate: sum(rate(rails_requests_total{namespace="landscaping-assistant", status=~"5.."}[5m])) / sum(rate(rails_requests_total{namespace="landscaping-assistant"}[5m])) > 0.05
• p95 latency: histogram_quantile(0.95, sum(rate(rails_request_duration_bucket{namespace="landscaping-assistant"}[5m])) by (le)) > 2
• Zero pods: kube_deployment_status_replicas_available{namespace="landscaping-assistant", deployment="landscaping-assistant"} == 0

Note: the status label in yabeda-rails is a string (e.g., "200", "500"), so regex matching status=~"5.." is correct.

Blast Radius

• One file modified: terraform/modules/monitoring/main.tf (append-only, 3 new resource blocks)
• Low risk: adding PrometheusRule resources is additive. Does not modify existing alerts or application code.
• Rollback: tofu apply with the resource blocks removed, or kubectl delete prometheusrule.
• Failure mode: wrong PromQL means alerts never fire (silent) or always fire (noisy). Neither breaks the application.

Decomposition Assessment

• 3 discrete changes: one resource block per alert (error rate, latency, availability). Exactly at the three-thing limit.
• All three belong in the same file and same PrometheusRule resource (or separate resources in the same file). No need to split.
• Estimated agent time: well under 5 minutes once the ticket is correctly scoped.
• No parallelization needed.

Recommendation

1. Fix the Repo field: Change from ldraney/pal-e-deployments to ldraney/pal-e-platform.
2. Fix the File Targets: Replace kustomize overlay paths with terraform/modules/monitoring/main.tf. Note this is an append to an existing file following the pattern of blackbox_alerts, embedding_alerts, etc.
3. Add metric names to AC: Specify that alerts must use rails_requests_total (counter, labels: controller/action/status/format/method) and rails_request_duration_bucket (histogram). Include the suggested PromQL expressions or at minimum the metric names.
4. Fix Test Expectations: Replace kustomize build with tofu plan (or tofu validate) against the monitoring module.
5. Clarify Pyrra relationship: The observability roadmap (docs/observability-roadmap.md) places PrometheusRules under Pyrra in Phase 6, where Pyrra would auto-generate multi-window burn rate alerting from SLO CRDs. Clarify whether these are interim manual rules that Pyrra will eventually replace, or intended to coexist.

Verdict: NEEDS_REFINEMENT

Board item #1309 | Forgejo issue ldraney/landscaping-assistant#86 | Reviewed 2026-06-04 by agent-qa

Template Completeness

• [x] Type -- Feature
• [x] Lineage -- Child of #43, references Phase 2 of observability roadmap
• [ ] Repo -- Lists forgejo_admin/pal-e-platform and forgejo_admin/pal-e-deployments. Both repos are actually ldraney/pal-e-platform and ldraney/pal-e-deployments. Verified via Forgejo API. Wrong owner will cause agent lookup failures.
• [x] User Story -- Clear "As a platform operator / I want query-level database visibility / So that I can identify slow queries"
• [x] Context -- Good motivation, explains the gap (PodMonitors give connection-level but not query-level visibility)
• [ ] File Targets -- Two of three targets lack concrete file paths. Only terraform/dashboards/postgres-query-analysis.json is specific. The CNPG cluster spec and Postgres config targets say "(pal-e-deployments)" without naming the actual file. See File Targets section below.
• [x] Acceptance Criteria -- 4 criteria, all checkboxed. One has an internal contradiction (see below).
• [x] Test Expectations -- 3 expectations: SQL query check, dashboard load, Loki log check
• [ ] Constraints -- Contains a contradiction. States "Must not cause CNPG cluster downtime" but also acknowledges "pg_stat_statements requires restart." See Acceptance Criteria section below.
• [x] Checklist -- PRs for both repos listed
• [x] Related -- References project-pal-e-platform and parent #43

Traceability

• [x] story:observability -- present on board item labels
• [x] arch:platform, arch:k8s-deploy -- present on board item labels
• [x] Forgejo issue #86 -- exists, open, linked from board item
• [x] Parent #43 -- exists, open, confirmed as observability tracking issue

File Targets

• [x] terraform/dashboards/postgres-query-analysis.json (pal-e-platform) -- NEW file. Parent directory verified: terraform/dashboards/ exists with 8 dashboard JSON files (e.g., landscaping-assistant-golden-signals.json, dora-dashboard.json). Naming convention is consistent.
• [ ] "CNPG cluster spec (pal-e-deployments)" -- NO PATH SPECIFIED. The pal-e-postgres CNPG Cluster manifest is NOT in pal-e-platform (confirmed via code search -- zero results for pg_stat_statements, shared_preload_libraries, or any Cluster kind in the database module). It must be in pal-e-deployments, but that repo is private and the exact path (likely a Kustomize base or overlay YAML) is not stated. An agent cannot act on this without the path.
• [ ] "CNPG Postgres config (pal-e-deployments)" -- NO PATH SPECIFIED. Same problem. The log_min_duration_statement parameter would go in the same Cluster manifest's postgresql.parameters section, but no file is named. Additionally, no threshold value is specified for log_min_duration_statement.

Assessment: Targets are NOT specific enough for an agent to act without guessing. The Grafana dashboard target is good. The two pal-e-deployments targets are too vague.

Repo Placement

This issue is filed in ldraney/landscaping-assistant (the Rails app), but ALL file changes are in pal-e-platform and pal-e-deployments. Zero files in landscaping-assistant are modified.

This is acceptable because the parent tracking issue #43 lives in landscaping-assistant and all observability children (#85-#90) are filed here for consistency. However, this means the issue's Repo section is the ONLY way an agent knows which repos to target -- making the wrong owner (forgejo_admin/) a higher-severity problem than usual.

Multi-repo scope: YES (2 repos). The checklist correctly lists separate PRs for each repo. This is structured appropriately as a single coordinated ticket rather than needing child issues, since the changes are tightly coupled (extension enable + dashboard that reads the metrics).

Dependencies

• [x] CNPG operator deployed -- satisfied (confirmed in terraform/modules/database/main.tf, CNPG chart v0.27.1)
• [x] PodMonitor infrastructure -- satisfied (monitoring module sets podMonitorSelectorNilUsesHelmValues = false for cluster-wide discovery; CNPG database module enables podMonitorEnabled = true)
• [x] Grafana sidecar pattern -- satisfied (grafana_dashboard: "1" label used across 8 dashboards in monitoring module)
• [x] Prometheus scraping -- satisfied (existing PodMonitors working)
• [x] Phase 1 (Tempo/OTel, #85) -- NOT a dependency. The ticket correctly does not depend on it. However, the roadmap notes that log-trace correlation (slow query -> trace) will only work after Phase 1 ships. This is informational, not blocking.

No unresolved dependencies that would block execution.

Acceptance Criteria

• AC1: pg_stat_statements extension enabled -- Testable. SELECT * FROM pg_stat_statements LIMIT 1 is a direct verification. Note: enabling the extension requires both (a) adding to shared_preload_libraries in the CNPG Cluster spec AND (b) running CREATE EXTENSION pg_stat_statements. The ticket does not mention step (b). For an existing cluster, this needs a one-time SQL command or CNPG's managed extension list.
• AC2: Grafana dashboard shows top queries -- Testable. Dashboard load with real data is verifiable.
• AC3: Slow queries logged and searchable in Loki -- Testable. But the log_min_duration_statement threshold value is not specified. Agent will have to guess a value (common defaults: 500ms, 1000ms).
• AC4: "No CNPG cluster restart required (or planned maintenance window if needed)" -- Contradicts the Constraints section which says "pg_stat_statements requires restart." The parenthetical escape hatch "(or planned maintenance window if needed)" is ambiguous. This should be rewritten to state the reality: a restart IS required, and either (a) CNPG handles it as a rolling restart (multi-instance) with zero downtime, or (b) brief downtime is accepted for single-instance clusters during a maintenance window.

Blast Radius

• Files touched: ~3 files across 2 repos (CNPG Cluster manifest, possibly a custom queries ConfigMap, Grafana dashboard JSON)
• Services affected: pal-e-postgres (restart required), Prometheus (new metrics), Grafana (new dashboard)
• What could break: The Postgres restart is the main risk. If pal-e-postgres is single-instance (like the Woodpecker cluster), there will be brief downtime affecting all services using this database. The Grafana dashboard and Prometheus scraping are additive and low-risk.
• Rollback: Straightforward. Remove shared_preload_libraries entry and restart (same risk), delete dashboard ConfigMap. CNPG handles rollback gracefully.

Decomposition Assessment

Three-thing limit: 3 discrete changes (enable extension, create dashboard, configure slow query logging). Fits within the limit.

Five-minute rule: The Grafana dashboard JSON will be the bulk of the work (PromQL queries, panel layout). The CNPG config changes are small. Overall scope is reasonable for a single ticket -- estimated 3 story points on the board matches.

Parallelization: The dashboard JSON could be developed independently of the CNPG config, but it needs real pg_stat_statements data to verify. Sequential execution is appropriate.

Recommendation

1. Fix repo owner (BLOCKER): Change forgejo_admin/pal-e-platform to ldraney/pal-e-platform and forgejo_admin/pal-e-deployments to ldraney/pal-e-deployments.
2. Add concrete file paths for pal-e-deployments targets (BLOCKER): Specify the exact path to the CNPG Cluster manifest (e.g., bases/postgres/cluster.yaml or overlays/production/cnpg-cluster.yaml -- whatever the actual path is).
3. Resolve restart contradiction: Rewrite AC4 and Constraints to explicitly state: (a) how many instances pal-e-postgres has, (b) whether brief downtime is acceptable, (c) whether a maintenance window is needed.
4. Specify log_min_duration_statement threshold: Add a concrete value (e.g., 500ms or 1000ms) so the agent does not have to guess.
5. Add CREATE EXTENSION step: Note in File Targets or Constraints that the extension must be created after enabling shared_preload_libraries, and specify the mechanism (CNPG managed extensions list, postInitSQL, or manual SQL).
6. Clarify pg_stat_statements metric exposure: Specify whether CNPG's default metrics exporter covers pg_stat_statements after the extension is loaded, or whether a custom queries ConfigMap is needed.

Verdict: NEEDS_REFINEMENT

Board item: #1311 on board-landscaping-observability (backlog, 2 points)
Forgejo issue: ldraney/landscaping-assistant #88
Labels: type:feature, arch:platform, story:observability, parent:43, devops

Template Completeness

• [x] Type -- Feature
• [x] Lineage -- Child of #43, Phase 4 of observability roadmap
• [ ] Repo -- Says forgejo_admin/pal-e-platform but actual owner is ldraney (forgejo_admin 301-redirects to ldraney). Minor inaccuracy.
• [x] User Story -- "As a platform operator / I want a visual web UI / So that I can see pod health... and provide cluster access via RBAC"
• [x] Context -- References 26 Grafana dashboards, CNCF status, explains the gap clearly
• [ ] File Targets -- Three files listed but paths do not match the actual repo structure (see File Targets section below)
• [x] Acceptance Criteria -- Four testable criteria
• [x] Test Expectations -- Two smoke checks (acceptable for 2-point infra ticket)
• [x] Constraints -- Keycloak OIDC, Tailscale funnel, read-only RBAC
• [x] Checklist -- Standard PR/tests/scope items
• [x] Related -- References project-pal-e-platform and parent #43

Traceability

• [x] story:observability -- present on board item labels
• [x] arch:platform -- present on board item labels
• [x] Forgejo issue #88 -- exists and is open
• [x] Parent #43 -- exists and is open (Observability & DORA metrics stack [PARENT])
• [x] Roadmap alignment -- confirmed as Phase 4 in docs/observability-roadmap.md

File Targets

Issue lists:

• [ ] terraform/headlamp.tf -- DOES NOT EXIST. No flat .tf files at terraform root beyond main.tf, variables.tf, providers.tf, outputs.tf, versions.tf, network-policies.tf.
• [ ] terraform/tailscale.tf -- DOES NOT EXIST. Tailscale funnels live in terraform/modules/networking/main.tf.
• [ ] terraform/keycloak.tf -- DOES NOT EXIST. Keycloak resources live in terraform/modules/keycloak/main.tf.

Actual repo structure uses modules:

• terraform/modules/monitoring/ -- Prometheus, Grafana, Loki, blackbox, DORA dashboards
• terraform/modules/networking/ -- Tailscale operator, ACLs, 9 existing funnels (Grafana, Forgejo, Woodpecker, Harbor, MinIO, Keycloak, etc.)
• terraform/modules/keycloak/ -- Keycloak k8s deployment (Deployment, Service, PVC, ConfigMap only -- no OIDC client resources)
• terraform/main.tf -- Module orchestration with ~80 moved blocks from monolith refactor

Assessment: File targets are wrong. An agent following them will create files in the wrong directory. Correct targets depend on a design decision: does Headlamp go into the monitoring module or get its own headlamp module? The funnel goes in networking. The OIDC client approach needs clarification (see Dependencies).

Repo Placement

Correct repo: ldraney/pal-e-platform. This is a platform-level Helm deployment with Tailscale networking and Keycloak auth -- all concerns live in pal-e-platform. Single-repo scope is appropriate. No cross-repo work needed (Headlamp reads the k8s API directly, no app instrumentation required).

The ### Repo field should be corrected from forgejo_admin/pal-e-platform to ldraney/pal-e-platform.

Dependencies

• [x] Headlamp Helm chart -- available at https://kubernetes-sigs.github.io/headlamp/, chart name headlamp/headlamp, version 0.42.0. Note: the roadmap references headlamp-k8s/headlamp which is the old GitHub org; chart is now under kubernetes-sigs.
• [x] Tailscale funnel pattern -- 9 existing funnels in networking module using kubernetes_ingress_v1 with tailscale.com/funnel annotation. Pattern is well-established and trivially extensible.
• [x] Keycloak deployment -- running (v26.0.7, k8s Deployment in keycloak module).
• [ ] Keycloak OIDC client management -- UNRESOLVED. The keycloak module uses only the kubernetes provider (k8s Deployment/Service/PVC). There are no keycloak_openid_client resources using the Keycloak Terraform provider. OIDC clients for existing services (Grafana, Forgejo) appear to be configured manually via the Keycloak admin UI. The Headlamp chart supports OIDC natively via config.oidc values (clientID, clientSecret, issuerURL, scopes, usePKCE), so the chart side is ready -- but the ticket must clarify how the Keycloak-side client is created.

Acceptance Criteria

• "Headlamp deployed and accessible via Tailscale funnel" -- testable: curl the funnel URL, expect HTTP 200.
• "Visual overview of all namespaces, pods, and resource usage" -- testable only via manual UI check. Acceptable for a UI deployment ticket.
• "Log streaming per pod works from the UI" -- manual verification. Acceptable.
• "OIDC login via Keycloak (or service account for initial deploy)" -- the parenthetical fallback is good but creates ambiguity about whether OIDC is in scope or deferred. Should pick one path for this ticket and make the other a follow-up.

Blast Radius

• Files touched: 2-4 files in pal-e-platform (Helm release, funnel ingress, optionally OIDC client, module wiring)
• Services affected: None -- Headlamp is additive. It reads the k8s API; it does not modify existing services.
• Failure modes: Bad RBAC could expose cluster-admin via web UI (the constraint says read-only, which mitigates this). Broken Tailscale funnel would only affect Headlamp access.
• Rollback: tofu destroy -target=module.monitoring.helm_release.headlamp (or equivalent) -- straightforward Helm uninstall.
• Risk: LOW. Purely additive infrastructure with no dependencies from existing services.

Decomposition Assessment

Three-thing limit: This ticket has 3 discrete changes (Helm release, Tailscale funnel, OIDC client). At the limit but acceptable if OIDC scope is clarified.

Five-minute rule: An agent with correct file targets can execute this in under 5 minutes. The Tailscale funnel is a copy-paste of an existing pattern. The Helm release follows established module patterns.

Parallelization: Not needed -- all changes are in one repo, one PR.

Assessment: Scope is appropriate for a 2-point ticket. No decomposition needed.

Recommendation

1. Fix File Targets. Replace the three flat paths with the correct modular paths. Decide whether Headlamp goes in terraform/modules/monitoring/main.tf (alongside Grafana) or gets a new terraform/modules/headlamp/ module. The monitoring module is the natural fit given Headlamp is in the observability roadmap.
2. Clarify OIDC implementation path. Three options: (a) Add the keycloak Terraform provider and define the OIDC client as IaC (sets a new pattern for all future OIDC clients), (b) document manual Keycloak admin UI steps in the PR, (c) defer OIDC to a follow-up ticket and use a ServiceAccount + ClusterRoleBinding (read-only) for initial deploy. Pick one and update the AC accordingly.
3. Fix Repo owner from forgejo_admin/pal-e-platform to ldraney/pal-e-platform.
4. (Optional) Update the Helm chart repo reference in docs/observability-roadmap.md from headlamp-k8s/headlamp to kubernetes-sigs/headlamp to reflect the current ownership.

Verdict: NEEDS_REFINEMENT

Forgejo Issue: ldraney/landscaping-assistant #85 -- Deploy Grafana Tempo + OpenTelemetry for distributed tracing
Board Item: #1308 (8 points, backlog, labels: type:feature, arch:platform, arch:rails-app, arch:k8s-deploy, story:observability, parent:43, devops)
Reviewed: 2026-06-04 by agent-qa

Template Completeness

• [x] Type -- Feature
• [x] Lineage -- Child of #43 (Observability & DORA metrics stack), Phase 1 of observability roadmap
• [x] Repo -- Three repos listed (but owner is stale, see Recommendation #2)
• [x] User Story -- Clear "As a platform operator / I want distributed tracing / So that I can trace requests end-to-end"
• [x] Context -- Thorough. Explains zero tracing today, MinIO backend, correlation goals, relationship to observability roadmap
• [ ] File Targets -- Six targets listed across three repos, but Terraform paths follow wrong pattern (see Recommendation #1)
• [x] Acceptance Criteria -- Seven criteria, all testable and specific
• [x] Test Expectations -- Three items (infrastructure-level only; see Recommendation #4)
• [x] Constraints -- Four constraints, all reasonable and actionable
• [x] Checklist -- Per-repo PR checkboxes
• [ ] Related -- References project-pal-e-platform but ticket lives on board-landscaping-observability (see Recommendation #5)

Traceability

• [x] story:observability -- present on board item
• [x] arch:platform, arch:rails-app, arch:k8s-deploy -- present on board item
• [x] Forgejo issue #85 -- exists and is open
• [x] parent:43 -- label present, parent issue exists and is open

File Targets

• [ ] terraform/tempo.tf (pal-e-platform) -- WRONG PATH. pal-e-platform uses modular Terraform structure under terraform/modules/. The monitoring module lives at terraform/modules/monitoring/ with its own main.tf/variables.tf/outputs.tf/versions.tf. There are NO top-level .tf files for individual Helm releases. An agent following this target literally would create orphaned files outside the module structure.
• [ ] terraform/otel-collector.tf (pal-e-platform) -- Same issue. Should be inside a module.
• [ ] terraform/grafana.tf (pal-e-platform) -- Same issue. Grafana config modifications should go in the existing monitoring module.
• [x] Gemfile (landscaping-assistant) -- Verified exists. Currently has yabeda-rails, yabeda-prometheus, yabeda-puma-plugin, lograge. Adding opentelemetry-sdk and opentelemetry-instrumentation-all here is correct.
• [x] config/initializers/opentelemetry.rb (landscaping-assistant) -- New file. Parent directory exists. Existing config/initializers/yabeda.rb is a separate file, no conflict.
• [?] overlays/production/patches/env.yaml (pal-e-deployments) -- Cannot verify; repo is private. Path pattern is consistent with Kustomize conventions.

Assessment: Three of six file targets point to the wrong location in the Terraform repo. An agent cannot execute reliably without corrected paths.

Repo Placement

This is a multi-repo ticket spanning three repositories:

1. pal-e-platform (Terraform) -- Helm releases for Tempo and OTel Collector, Grafana datasource config
2. landscaping-assistant (Rails) -- Gem additions, OTel initializer
3. pal-e-deployments (Kustomize) -- OTLP endpoint env var

The issue is filed on landscaping-assistant, which is reasonable as the primary user-facing repo, with the checklist tracking PRs across all three repos. This pattern works -- the issue serves as a tracking issue with per-repo PRs.

Repo owner mismatch: Issue references forgejo_admin/pal-e-platform and forgejo_admin/pal-e-deployments. The canonical owner is ldraney (forgejo_admin 301-redirects to ldraney). Agent API calls using the old owner could fail depending on Forgejo API behavior with redirects.

Dependencies

• [x] MinIO -- satisfied. Already deployed per project-pal-e-platform status and observability roadmap.
• [x] Grafana -- satisfied. 26 dashboards deployed, monitoring module exists.
• [x] Loki -- satisfied. 7d retention, Promtail DaemonSet running. Needed for trace-log correlation.
• [x] Lograge -- satisfied. Already in Gemfile. Needed for TraceID injection into structured logs.
• [x] yabeda metrics -- satisfied. Constraint says "must not break existing yabeda Prometheus metrics" -- coexistence is possible since OTel and yabeda operate on different telemetry signals (traces vs metrics).
• [ ] Tempo Helm chart repo -- ATTENTION. The Grafana Tempo chart migrated from grafana/helm-charts to grafana-community/helm-charts as of January 2026. OCI path is now oci://ghcr.io/grafana-community/helm-charts/tempo. Issue does not mention this. An agent using the old repo would pull a stale chart.

Acceptance Criteria

All seven AC are testable and specific:

1. "Tempo is deployed and healthy in the monitoring namespace" -- verifiable via kubectl
2. "OTel Collector DaemonSet is running and receiving OTLP" -- verifiable via kubectl + logs
3. "Rails requests produce traces visible in Grafana Explore -> Tempo" -- verifiable via Grafana UI
4. "ActiveRecord, ActionPack, Net::HTTP, Puma spans auto-instrumented" -- verifiable via trace detail in Grafana. The opentelemetry-instrumentation-all gem (v0.94.0) bundles all four of these instrumentations.
5. "TraceID appears in structured log output (Lograge)" -- verifiable via log inspection
6. "Clicking a TraceID in Loki opens the trace in Tempo" -- verifiable via Grafana Explore. Requires derived fields config in Loki datasource.
7. "Grafana NodeGraph panel shows service dependency map from trace data" -- verifiable via Grafana. Note: with only one service instrumented initially, the dependency map will be minimal.

Assessment: AC are well-crafted. No "works correctly" ambiguity.

Blast Radius

• Files touched: ~6 files across 3 repos
• Services affected: Monitoring namespace (new pods), landscaping-assistant (new gems + initializer), Grafana (new datasource)
• What could break: (a) Rails boot failure if OTel initializer has errors. (b) Yabeda /metrics endpoint regression if gems conflict. (c) Monitoring namespace resource pressure from new pods (Tempo + OTel Collector DaemonSet).
• Rollback: Straightforward. Revert Helm releases via Terraform, remove gems from Gemfile, remove initializer. Independent rollback per repo.

Decomposition Assessment

This ticket has 3 discrete changes across 3 repos:

1. Platform infra (Tempo + OTel Collector Helm releases + Grafana datasource) -- pal-e-platform
2. App instrumentation (gems + initializer) -- landscaping-assistant
3. Env var injection (OTLP endpoint) -- pal-e-deployments

These are sequentially dependent (infra must exist before app can send traces, env var must exist before app can find the collector). The three-thing limit is met exactly. An agent could execute this as one ticket with ordered PRs: platform first, then deployments, then app.

At 8 points this is on the large side but not decomposition-worthy -- the changes are tightly coupled and individually small.

Recommendation

1. (MUST FIX) Correct Terraform file targets. Replace terraform/tempo.tf, terraform/otel-collector.tf, and terraform/grafana.tf with paths that match the modular structure. Options: (a) Add resources to terraform/modules/monitoring/main.tf, or (b) Create a new terraform/modules/tracing/ module with its own main.tf/variables.tf/outputs.tf/versions.tf. Explicitly state which approach to use.
2. (MUST FIX) Update repo owner from forgejo_admin to ldraney. The canonical owner is ldraney. API calls and PR submissions should target ldraney/pal-e-platform and ldraney/pal-e-deployments.
3. (MUST FIX) Add Tempo Helm chart migration note. Add a constraint: "Use the Tempo chart from grafana-community/helm-charts (OCI: oci://ghcr.io/grafana-community/helm-charts/tempo), not the deprecated grafana/helm-charts repo."
4. (SHOULD FIX) Add application-level test expectation. Add at minimum: "Rails boots cleanly with OTel gems (RAILS_ENV=test bundle exec rails runner 'puts OpenTelemetry.tracer_provider')" to catch initializer errors before deployment.
5. (SHOULD FIX) Add landscaping-observability to Related section. This ticket lives on board-landscaping-observability. The Related section should reference the project it belongs to.
6. (SHOULD FIX) Add "do not touch" file targets. List config/initializers/yabeda.rb, config/routes.rb, and config/puma.rb as files agents should NOT modify, reinforcing the "must not break yabeda" constraint.

Verdict: NEEDS_REFINEMENT

Template Completeness

• [x] Type -- Feature
• [x] Lineage -- Child of #43, depends on #85 (Tempo)
• [x] Repo -- Two repos: forgejo_admin/pal-e-platform (Helm + Terraform), ldraney/landscaping-assistant (app instrumentation)
• [x] User Story -- Clear: platform operator wants continuous profiling with flame graphs linked to traces
• [x] Context -- Strong motivation, references current gap, explains value proposition
• [/] File Targets -- 4 files listed, but 2 have incorrect paths (see File Targets section)
• [x] Acceptance Criteria -- 4 testable conditions
• [/] Test Expectations -- 2 items, thin relative to constraints (see Acceptance Criteria section)
• [x] Constraints -- 3 items: MinIO backend, perf benchmark, Tempo dependency
• [x] Checklist -- Multi-repo PR checklist
• [x] Related -- References parent #43, prerequisite #85, project slug

Traceability

• [x] story:observability -- present on board item #1310
• [x] arch:platform, arch:rails-app -- present on board item #1310 (correctly reflects dual-repo scope)
• [x] Forgejo issue -- #87 exists and is open
• [x] parent:43 -- label present, matches Lineage in issue body
• [x] type:feature -- label present, matches issue Type header

File Targets

• [!] terraform/pyroscope.tf (pal-e-platform) -- WRONG PATH. The pal-e-platform repo organizes all monitoring Helm releases inside terraform/modules/monitoring/main.tf. Existing releases (kube-prometheus-stack, loki-stack, blackbox-exporter) are resource blocks in that file. A top-level terraform/pyroscope.tf would be outside the monitoring module scope -- the namespace, provider config, and variable references are module-scoped. An agent following this target would produce broken Terraform.
• [!] terraform/grafana.tf (pal-e-platform) -- WRONG PATH. The Grafana Loki datasource is a ConfigMap in terraform/modules/monitoring/main.tf. A top-level grafana.tf has no access to the module's resource references. Correct location: add a ConfigMap resource to terraform/modules/monitoring/main.tf or a new file within that module.
• [x] Gemfile (landscaping-assistant) -- Exists. No pyroscope gem present yet. Correct target.
• [x] config/initializers/pyroscope.rb (landscaping-assistant) -- Does not exist yet. Parent directory config/initializers/ exists. Correctly new.

Assessment: The two pal-e-platform file targets would misdirect an agent. The correct targets are resources within terraform/modules/monitoring/main.tf (or a new terraform/modules/monitoring/pyroscope.tf if the preference is to split by component). Also missing: terraform/modules/monitoring/variables.tf will likely need new variables (MinIO bucket config, Pyroscope version pin).

Repo Placement

This is correctly structured as a multi-repo ticket: platform infrastructure in pal-e-platform, app instrumentation in landscaping-assistant. The checklist includes separate PR checkboxes for each repo. However, it is filed as a single Forgejo issue in landscaping-assistant only. For a multi-repo change, the convention is either a tracking issue with child issues per repo, or a single issue that explicitly names both repos (which this does). Acceptable as-is given the scope is manageable, but worth noting that the pal-e-platform work is the larger portion.

Dependencies

• [/] #85 Tempo -- OPEN (not yet started, also in backlog). The ticket correctly notes trace-to-profile linking depends on Tempo but Pyroscope can deploy standalone. This is well-handled -- the hard dependency is only on AC #4 (trace-to-profile correlation), not the full deployment. However, AC #4 is not marked as conditional in the Acceptance Criteria. An agent would try to verify trace linking and fail if Tempo is not deployed.
• [x] MinIO -- Already deployed (used by existing Loki stack). No new dependency.
• [x] Grafana -- Already deployed (kube-prometheus-stack). No new dependency.

Acceptance Criteria

• AC1: Pyroscope deployed and healthy -- Testable. kubectl pod check + ready status.
• AC2: Grafana Explore returns profiles -- Testable. Navigate to Grafana, query Pyroscope datasource.
• AC3: Flame graphs visible for Rails endpoints -- Testable but vague. Which endpoints? After what traffic? Needs a concrete verification step (e.g., "hit /properties 10 times, flame graph shows ActionController and ActiveRecord frames").
• AC4: Profiles link to traces -- Testable only if Tempo (#85) is deployed. Not marked conditional. Should say "conditional on #85" or be split to a follow-up ticket.

Test Expectations gap: Constraints require "benchmark before/after Puma request throughput" but this is absent from Test Expectations. The constraint creates an implicit acceptance criterion with no verification method.

Blast Radius

• Files touched: 4 files across 2 repos (2 Terraform resources, 1 Gemfile entry, 1 initializer)
• Services affected: Monitoring namespace (new pod), landscaping-assistant (new gem in runtime)
• What could break: (1) Puma throughput degradation from profiling overhead -- mitigated by benchmark constraint. (2) MinIO storage growth from continuous profile data -- no retention policy mentioned in AC or constraints. (3) Monitoring namespace resource pressure from additional pod.
• Rollback: Straightforward. Remove Helm release, remove gem + initializer, deploy. No data migration involved.

Decomposition Assessment

• Discrete changes: 3 (Helm release + datasource, Gemfile addition, initializer config). Within the three-thing limit.
• Time estimate: Within the five-minute rule for an agent -- the pattern is well-established (copy loki-stack Helm release pattern, add gem, add initializer).
• Parallelization: The pal-e-platform PR and landscaping-assistant PR are independent and could be executed in parallel. The issue structure supports this (separate checklist items).

No decomposition needed. Scope is appropriate for a single ticket.

Technical Verification

• pyroscope gem: CONFIRMED. Exists on RubyGems as pyroscope v1.0.7 (released 2026-05-15). Actively maintained by Pyroscope team. Apache-2.0 license. 331K+ downloads. No deprecation notices.
• Helm chart: CONFIRMED. Chart name is pyroscope in the https://grafana.github.io/helm-charts repository. Matches the observability roadmap Technology Reference table (grafana/pyroscope).
• Observability roadmap: docs/observability-roadmap.md exists and confirms Pyroscope as Phase 3. Architecture diagram, rollout plan, and technology reference are all consistent with the issue scope.

Recommendation

1. Fix Terraform file targets. Change terraform/pyroscope.tf and terraform/grafana.tf to either:
   • terraform/modules/monitoring/main.tf -- add Helm release + datasource ConfigMap (matches existing loki-stack pattern), or
   • terraform/modules/monitoring/pyroscope.tf -- new file within the monitoring module (acceptable alternative, state explicitly)
2. Add Puma benchmark to Test Expectations. The Constraints section requires "benchmark before/after" but Test Expectations has no corresponding verification step. Add: "Benchmark: compare Puma requests/sec with and without pyroscope gem (must not degrade >X%)".
3. Mark AC #4 (trace-to-profile linking) as conditional on #85. Either add "(requires #85 Tempo)" to the criterion, or split it to a follow-up ticket that depends on both #85 and #87.
4. Tighten AC #3. "Flame graphs visible for Rails endpoints" is vague. Specify a concrete verification: e.g., "After generating traffic to /properties, Pyroscope shows CPU flame graph with Rails framework frames (ActionController, ActiveRecord)".

Verdict: NEEDS_REFINEMENT

Board item: board-landscaping-observability#1312
Forgejo issue: ldraney/landscaping-assistant #89 -- "Deploy Grafana Faro for frontend real user monitoring"
Reviewed: 2026-06-04 by QA agent

Template Completeness

• [x] Type -- Feature
• [x] Lineage -- Child of #43, Phase 5, depends on #85
• [x] Repo -- Two repos: pal-e-platform + landscaping-assistant
• [x] User Story -- Clear "who wants what and why"
• [x] Context -- Sufficient background for a fresh-context agent
• [ ] File Targets -- Present but underspecified (see findings)
• [x] Acceptance Criteria -- 4 testable conditions
• [x] Test Expectations -- 2 integration-level items
• [ ] Constraints -- Present but missing critical packaging constraint (see findings)
• [x] Checklist -- Covers both repos
• [x] Related -- References project and parent issue

Traceability

• [x] story:observability -- present on board item #1312
• [x] arch:platform, arch:rails-app -- present on board item #1312
• [x] Forgejo issue #89 -- exists and is open
• [x] parent:43 -- label present, lineage matches

File Targets

• [ ] "OTel Collector config (pal-e-platform)" -- No specific file path given. This is in a different repo (pal-e-platform) which is outside the scope of direct verification here, but the issue must name the exact file (e.g., terraform/otel-collector.tf or a values file).
• [ ] app/javascript/application.js -- File exists (verified). However, the issue says "or layout" which is ambiguous. The file currently contains a hand-rolled error reporter (reportClientError, window.onerror, unhandledrejection handlers at lines 5-25) that Faro would replace. The issue does not mention this existing code or whether it should be removed.
• [ ] Missing target: config/importmap.rb -- Would need modification to pin any Faro JS, but see packaging blocker below.

Assessment: Targets are NOT specific enough for an agent to act on without guessing.

Repo Placement

This is a multi-repo ticket spanning forgejo_admin/pal-e-platform (OTel Collector Faro receiver) and ldraney/landscaping-assistant (frontend SDK). The issue correctly identifies both repos and the checklist has separate PR checkboxes for each. This is acceptable as structured -- it acts as a tracking issue with work in two repos.

However, the two repos have independent scopes. The platform-side work (Faro receiver endpoint on OTel Collector) could be its own ticket. Consider whether splitting would improve parallelism.

Dependencies

• [ ] #85 (Tempo + OTel Collector) -- PENDING. Issue #85 is open, in backlog column on the board, with no labels indicating progress. The OTel Collector itself (deployed in #85) is a hard prerequisite -- Faro has nowhere to send telemetry without it. The Faro trace propagation AC ("Frontend spans propagate TraceID to backend") explicitly requires Tempo. This dependency is correctly documented in the issue but is unresolved.

Acceptance Criteria

• "Faro SDK loaded in production frontend" -- Testable. Can verify via browser devtools network tab or page source.
• "Core Web Vitals (LCP, FID, CLS) visible in Grafana" -- Testable but FID is deprecated. Google replaced FID with INP (Interaction to Next Paint) in March 2024. Faro SDK v1.4+ reports INP. Should say INP, not FID.
• "JS errors captured with stack traces in Loki" -- Testable. The test expectation ("Intentional JS error appears in Loki within 30s") maps to this.
• "Frontend spans propagate TraceID to backend (W3C trace context headers)" -- Testable but blocked by #85. Cannot verify without Tempo.

Blast Radius

• Files touched: 2-3 in landscaping-assistant (JS entrypoint, possibly importmap config, possibly layout), 1+ in pal-e-platform (OTel Collector config)
• What could break: If the SDK packaging is wrong, page loads could fail (JS errors blocking Turbo). If the Faro receiver is misconfigured, the OTel Collector could reject all telemetry. Existing window.onerror handler conflict with Faro's error capture is a risk.
• Rollback: Straightforward -- revert JS changes, Faro is additive observability that does not affect app functionality.

Decomposition Assessment

Three-thing limit: This ticket has 4+ discrete changes:

1. Configure Faro receiver on OTel Collector (pal-e-platform)
2. Install and initialize Faro SDK (landscaping-assistant)
3. Handle Turbo Drive navigation in Faro config
4. Remove or reconcile existing error reporter code

The platform-side work (#1) is independent and could be a separate ticket. However, the landscaping-assistant changes (#2-4) are tightly coupled and should stay together. Borderline -- acceptable as-is if the platform work is simple, but splitting would be cleaner.

Five-minute rule: The packaging question (see Finding #1 below) adds significant ambiguity that could cause an agent to spend considerable time researching solutions. With the packaging strategy specified, this is a reasonable-sized ticket.

Findings

Finding 1 (BLOCKING): Faro SDK packaging incompatible with importmap-rails

The issue says to install @grafana/faro-web-sdk and @grafana/faro-web-tracing in app/javascript/application.js. However, landscaping-assistant uses importmap-rails with no Node.js toolchain, no package.json, no bundler. The Faro SDK is an npm package with deep dependency trees (OpenTelemetry JS SDK internals) that cannot be pinned via importmap.

The issue MUST specify a packaging strategy:

1. jsbundling-rails migration -- Switch to esbuild/vite. Significant prerequisite that should be its own ticket.
2. Vendored pre-built bundle -- Build Faro offline, place in app/assets/builds/, pin via importmap. Avoids toolchain change but requires a manual rebuild process for updates.
3. CDN script tag -- Load Faro from unpkg.com or cdn.jsdelivr.net in the layout. Simplest but adds external runtime dependency.

Finding 2 (BLOCKING): Observability roadmap references SvelteKit -- this app is Rails

The observability roadmap (docs/observability-roadmap.md) Phase 5 section (line 191) states: "@grafana/faro-web-sdk in SvelteKit apps". The architecture diagram (line 14) labels the frontend as SVELTE["SvelteKit Frontend"]. But landscaping-assistant is a Rails 8.1 app with Hotwire/Turbo/Stimulus. The issue body correctly identifies "Rails frontend uses Hotwire/Turbo" in the Constraints section, but the source document it references is wrong.

Either fix docs/observability-roadmap.md to reference Rails/Hotwire, or add a note in the issue that the roadmap's SvelteKit reference is incorrect.

Finding 3 (BLOCKING): File Targets underspecified

Two problems:

• The pal-e-platform target has no file path -- just "OTel Collector config".
• The landscaping-assistant target says "application.js or layout" -- ambiguous. An agent must know which.
• Missing: no mention of removing the existing hand-rolled error reporter in app/javascript/application.js (lines 5-25) which conflicts with Faro's native error capture.

Finding 4 (Non-blocking): FID deprecated, should be INP

Acceptance criteria reference "Core Web Vitals (LCP, FID, CLS)". FID was replaced by INP as a Core Web Vital in March 2024. Update to "(LCP, INP, CLS)".

Finding 5 (Non-blocking): Test expectations lack unit-level coverage

Both test expectations are integration-level (requires Grafana + Loki). Consider adding a smoke test: "Faro SDK initializes without JS console errors on page load".

Finding 6 (Non-blocking): Bundle size constraint needs verification method

The constraint "Faro SDK bundle size must stay under 15KB gzipped" is good but the issue should specify how to verify it, especially since the packaging strategy affects measurement.

Recommendation

1. Specify the Faro SDK packaging strategy. Add a Constraints entry choosing between jsbundling-rails migration, vendored bundle, or CDN script tag. If jsbundling-rails, create a prerequisite ticket for the migration.
2. Fix file targets. Replace "OTel Collector config (pal-e-platform)" with the exact file path. Replace "application.js or layout" with a definitive choice. Add a note that the existing reportClientError / window.onerror / unhandledrejection code in app/javascript/application.js (lines 5-25) should be removed since Faro handles this natively.
3. Fix the observability roadmap. Update docs/observability-roadmap.md Phase 5 and the architecture diagram to reference Rails/Hotwire instead of SvelteKit.
4. Update FID to INP in acceptance criteria.
5. Add a smoke test expectation: "Faro SDK initializes without JS console errors on page load."

Verdict: BLOCK

Board item: board-landscaping-observability#1313 (backlog, 5 points)
Forgejo issue: ldraney/landscaping-assistant #90 -- "Deploy Pyrra for SLO tracking and Falco for runtime security"
Reviewer: QA Agent
Date: 2026-06-04

Template Completeness

Checked against template-issue-feature:

• [x] Type -- Feature
• [x] Lineage -- Child of #43, Phase 6 of observability roadmap
• [ ] Repo -- States forgejo_admin/pal-e-platform but issue is filed in ldraney/landscaping-assistant. Mismatch.
• [ ] User Story -- Combines two unrelated capabilities in one "I want" clause. "SLO tracking with error budgets" and "runtime container security monitoring" are distinct user needs for distinct personas (SRE vs Security).
• [x] Context -- Present. Acknowledges "Two independent capabilities bundled" but does not justify why.
• [ ] File Targets -- Three targets listed, all in pal-e-platform. SLO YAML location is ambiguous: "pal-e-platform or pal-e-deployments" is a question, not a spec.
• [x] Acceptance Criteria -- 5 criteria covering both components.
• [x] Test Expectations -- 3 items, all kubectl-based manual verification. No automated tests.
• [ ] Constraints -- "SLO targets need to be decided per-service before deployment" is an unresolved prerequisite, not a constraint. This is a blocker hiding as a constraint.
• [x] Checklist -- Present.
• [x] Related -- References project-pal-e-platform and parent #43.

Traceability

• [x] story:observability -- present on board item labels
• [x] arch:platform -- present on board item labels
• [x] Forgejo issue -- #90 exists and is open
• [x] parent:43 -- present on board item labels, matches Lineage

Traceability triangle is intact for the board item. However, the traceability to the platform plan is broken (see Decomposition Assessment below).

File Targets

• [ ] terraform/pyrra.tf (pal-e-platform) -- Cannot verify from landscaping-assistant repo. File would be new. Parent directory terraform/ exists in pal-e-platform per project page.
• [ ] terraform/falco.tf (pal-e-platform) -- Same. New file in pal-e-platform.
• [ ] SLO YAML definitions -- Location undecided ("pal-e-platform or pal-e-deployments"). Agent cannot act on this without a decision.

Assessment: Targets are in a different repo than where the issue is filed. An agent dispatched against landscaping-assistant would not find these files.

Repo Placement

PROBLEM: The issue is filed in ldraney/landscaping-assistant but all work targets are in forgejo_admin/pal-e-platform. This is a cross-repo mismatch. The issue body even says Repo: forgejo_admin/pal-e-platform. Either:

• Move the issue to pal-e-platform (where the Terraform files live), or
• Make #90 a tracking issue in landscaping-assistant with child issues in pal-e-platform (but this adds unnecessary indirection for what is fundamentally platform work)

Dependencies

Per plan-pal-e-platform, the two components have distinct dependency chains:

• [ ] SLO engine (Phase 16) depends on Phase 14 (synthetic monitoring) and Phase 15 (DORA re-baseline) -- status of both is unclear from available data
• [ ] Falco (Phase 20b) depends on Phase 10 (vulnerability scanning -- COMPLETED) and Phase 19 (Kyverno -- NOT STARTED)

The plan explicitly states "Tier 1.5 gates Tier 2." Falco is Tier 2. Phase 19 (Kyverno) is a prerequisite for Phase 20 and is NOT STARTED. The issue does not acknowledge or justify skipping this dependency.

Acceptance Criteria

Evaluating each AC:

1. "Pyrra deployed and generating PrometheusRules from SLO definitions" -- Testable via kubectl get prometheusrules. Specific enough.
2. "Grafana dashboard shows error budget remaining and burn rate" -- Testable but requires manual verification. No dashboard JSON path specified.
3. "At least one SLO defined (e.g., landscaping-assistant 99.5% availability)" -- Testable. But the constraint says "SLO targets need to be decided per-service before deployment" -- so this AC depends on an unresolved prerequisite.
4. "Falco DaemonSet running and detecting test anomaly" -- Testable via kubectl exec test. Specific enough.
5. "Falco alerts route through Alertmanager to Telegram" -- Testable. Requires Alertmanager config changes not listed in file targets.

AC quality is adequate individually, but conflated across two independent capabilities.

Blast Radius

• Repos touched: 1-2 (pal-e-platform, possibly pal-e-deployments for SLO YAMLs)
• Services affected: All (Falco DaemonSet runs on every node; SLO definitions reference all monitored services)
• Risk: Falco DaemonSet is medium-risk -- it monitors syscalls on every node. A misconfigured Falco can generate alert storms or consume node resources. SLO engine is low-risk -- namespace-scoped deployment.
• Rollback: Both are independent Helm releases, so rollback is straightforward per-component. But bundling them in one PR makes selective rollback harder.

Decomposition Assessment

Three-thing limit: VIOLATED. This ticket has at least 5 discrete changes:

1. Deploy Pyrra/Sloth Helm release
2. Write SLO YAML definitions
3. Create Grafana SLO dashboard
4. Deploy Falco Helm release (DaemonSet)
5. Configure Falco alerting through Alertmanager

Five-minute rule: VIOLATED. An agent would need substantial time to: resolve the Sloth/Pyrra decision, decide SLO YAML location, write SLO definitions for services, create Grafana dashboard JSON, deploy and test two independent Helm releases, and configure Alertmanager routing.

Critical conflict -- Sloth vs Pyrra:

• The platform plan Phase 16 (phase-pal-e-platform-16-slo-error-budgets) is titled "SLO Governance (Sloth)" and explicitly scopes deploying Sloth.
• The plan-pal-e-platform summary references "SLO governance/Sloth (16)".
• The validation pipeline diagram in project-pal-e-platform shows "Sloth -- SLO YAML to Recording Rules".
• This issue says "Pyrra uses Prometheus-native SLOs (not Sloth)" -- directly contradicting the plan.
• The docs/observability-roadmap.md in landscaping-assistant references Pyrra.

Two sources of truth disagree. An agent cannot execute this without a human decision.

Recommendation: Split into two issues aligned with existing plan phases:

• Issue A: SLO engine -- aligned with Phase 16. Filed in pal-e-platform. Resolve Sloth vs Pyrra first.
• Issue B: Falco runtime security -- aligned with Phase 20b. Filed in pal-e-platform. Acknowledge Kyverno (Phase 19) dependency.

Helm Chart Verification

• Pyrra: Official Helm chart repo is pyrra-dev/helm-charts (not pyrra-dev/pyrra which is the project repo). Chart name: pyrra. ArtifactHub listing exists.
• Falco: Official Helm chart repo is falcosecurity/charts (not falcosecurity/falco which is the project repo). Chart name: falco. Current version 9.0.0 confirmed.

The observability-roadmap.md references project repos, not chart repos. Terraform helm_release blocks will need the correct chart repository URLs.

Recommendation

1. Resolve Sloth vs Pyrra. Human decision required. Update the losing side's documentation (either Phase 16 + project-pal-e-platform validation diagram, or observability-roadmap.md + this issue).
2. Split into two issues aligned with existing plan phases: Phase 16 (SLO engine) and Phase 20b (Falco). File both in forgejo_admin/pal-e-platform.
3. Address dependency chains in each new issue. Phase 16 depends on 14+15. Phase 20b depends on 10+19. Either complete prerequisites or document why they can be skipped.
4. Decide SLO YAML location before the SLO issue moves forward. "pal-e-platform or pal-e-deployments" is not actionable.
5. Add Alertmanager config to file targets for the Falco issue -- routing Falco alerts requires Alertmanager config changes not currently listed.

Related

• plan-pal-e-platform -- authoritative platform hardening plan
• phase-pal-e-platform-16-slo-error-budgets -- Phase 16: SLO Governance (Sloth)
• phase-platform-20-security-deepening -- Phase 20: Security Deepening (Falco is subphase 20b)
• template-issue-feature -- issue template validated against
• template-review -- review template used for this note

Vision

Full Datadog-equivalent observability for the landscaping-assistant app and pal-e platform using the open-source Grafana ecosystem. Everything stores in MinIO, queries through Grafana, alerts through Alertmanager. The Datadog capability matrix is the benchmark — every row moves from NOT STARTED to COMPLETE. Detailed rollout in docs/observability-roadmap.md (landscaping-assistant repo).

User Stories

Key	Role	Summary	Success Metric
tracing	Platform Operator	Trace requests end-to-end across services	Click a slow request in Grafana, see the full trace with span-level latency
database-visibility	Platform Operator	Identify slow queries and database bottlenecks	Grafana dashboard shows top queries by time, calls, rows; slow queries in Loki
profiling	Platform Operator	See where CPU/memory time goes per endpoint	Flame graphs in Grafana linked to specific traces
cluster-viz	Platform Operator	Visual cluster overview without kubectl	Web UI showing namespaces, pods, resource usage, log streaming
frontend-monitoring	Platform Operator	See what users experience in the browser	Core Web Vitals, JS errors, frontend-to-backend trace correlation
slo-tracking	Platform Operator	Measure reliability against targets with error budgets	Grafana dashboard shows budget remaining and burn rate per service
runtime-security	Platform Operator	Detect anomalous container behavior	Falco alerts for unexpected shells, file access, outbound connections via Alertmanager

Architecture

Single deployment target: pal-e k8s cluster (single-node k3s). All observability components deploy via Terraform Helm releases in pal-e-platform. Service instrumentation ships via app-level SDK changes + kustomize overlay env vars in pal-e-deployments.

The stack is the Grafana LGTP ecosystem:

• Loki — logs (COMPLETE, 7d retention, MinIO backend)
• Grafana — dashboards, explore, alerting (COMPLETE, 26 dashboards)
• Tempo — traces (NOT STARTED, MinIO backend)
• Pyroscope — profiles (NOT STARTED, MinIO backend)

Supporting components: OpenTelemetry Collector (trace ingestion), Faro (frontend RUM), Headlamp (cluster UI), Pyrra (SLO engine), Falco (runtime security).

Key decisions:

• Tempo over Jaeger — native Grafana integration, MinIO backend matches Loki pattern
• Pyroscope over standalone pprof — trace-to-profile linking, Grafana-native
• OTel Collector as DaemonSet, not sidecar — simpler, matches Promtail pattern
• Tempo in monolithic mode — single-node cluster doesn't need microservices mode
• Pyrra over Sloth — better Grafana integration for SLO dashboards

Datadog Benchmark

The scorecard. Every Datadog capability mapped to our open-source equivalent.

Datadog Capability	Our Equivalent	Status	Issue
Infrastructure Monitoring	kube-prometheus-stack + node-exporter	COMPLETE	—
Log Management	Loki + Promtail + Grafana Explore	COMPLETE	—
Application Metrics	yabeda-rails + yabeda-prometheus	COMPLETE	#19 (done)
Dashboards	Grafana (26 dashboards + golden signals)	COMPLETE	#16 (done)
Alerting	Alertmanager → Telegram + Slack	PARTIAL	#17
CI Visibility	DORA exporter + Woodpecker API	COMPLETE	#20 (done)
Synthetic Monitoring	Blackbox exporter (13 probes)	COMPLETE	#21 (done)
Container Scanning	Harbor Trivy	COMPLETE	—
APM / Distributed Tracing	OpenTelemetry + Grafana Tempo	NOT STARTED	#85
Database Monitoring (query-level)	pg_stat_statements + Grafana dashboard	NOT STARTED	#86
Continuous Profiling	Grafana Pyroscope	NOT STARTED	#87
Infrastructure Visualization	Headlamp + Grafana NodeGraph	NOT STARTED	#88
RUM (Frontend)	Grafana Faro	NOT STARTED	#89
SLOs / Error Budgets	Pyrra	NOT STARTED	#90
Runtime Security	Falco	NOT STARTED	#90

Board

Primary kanban: board-landscaping-observability

9 items: 1 done (#83 roadmap doc), 8 in backlog. 40 total points. All issues filed in ldraney/landscaping-assistant repo, parent: #43.

Dependency chain: #85 (Tempo) is the keystone — #87 (Pyroscope) and #89 (Faro) depend on it for trace linking. #86 (pg_stat_statements) and #88 (Headlamp) are independent.

Status

8 of 15 Datadog capabilities are COMPLETE (pre-existing platform infrastructure). 6 are NOT STARTED (the new work). 1 is PARTIAL (#17 PrometheusRule alerts, needs issue refinement). The foundation — Prometheus, Loki, Grafana, Alertmanager — is solid. The gaps are tracing, profiling, database query analysis, frontend monitoring, infrastructure visualization, SLOs, and runtime security.

Milestones

2026-06-03: Observability roadmap doc merged (PR #84). Datadog gap analysis, mermaid target architecture diagram, 6-phase rollout plan. Six child issues created (#85-#90).

Repos

Repo	Platform	Role	Status
landscaping-assistant	Forgejo	App instrumentation (OTel SDK, Pyroscope gem, Faro JS)	Active
pal-e-platform	Forgejo	Helm releases (Tempo, Pyroscope, Headlamp, Pyrra, Falco)	Active
pal-e-deployments	Forgejo	Kustomize overlays (env vars, CNPG config)	Active