pal-e-docs

Plan: DORA Metrics Dashboard

plan-2026-03-01-dora-metrics-dashboard Plan

plan completed
Project

Plan: DORA Metrics Dashboard

Vision

DORA is the platform axiom. Every plan, every capability, every SOP exists to move one of four numbers. This plan makes those numbers visible — automatically, continuously, in Grafana — replacing manual measurement with a Prometheus exporter that polls Woodpecker and Forgejo APIs.

Projects & Repos Touched

Project/Repo Platform Role in this plan
<code>forgejo_admin/pal-e-dora-exporter</code> (new) Forgejo Python exporter service — polls APIs, exposes /metrics
<code>pal-e-platform</code> (terraform/) Forgejo Deploy exporter + ServiceMonitor + Grafana dashboard ConfigMap
<code>pal-e-docs</code> (knowledge) Forgejo Plan, issues, doc updates

Context

DORA baseline was manually measured on 2026-03-01 (see dora-framework). Confidence is Low-Medium because most metrics are estimates. The data exists in Woodpecker (pipeline events) and Forgejo (PR events) but nothing feeds it to Prometheus, so Grafana has nothing to show.
The existing plans (TF CI Phase 5, Observability Phase 4) bury DORA measurement at the end of long dependency chains. But app pipelines already exist — we can measure them now. This plan has zero dependencies on those plans.
What's already done:
  • [x] ldraney-woodpecker-sdk 0.1.0 published to Forgejo PyPI (httpx-based, 117 endpoints)
  • [x] ldraney-forgejo-sdk 0.1.0 published to Forgejo PyPI (httpx-based, 304 endpoints)
  • [x] Grafana sidecar configured: dashboards.enabled = true, searchNamespace = "ALL"
  • [x] Prometheus configured: serviceMonitorSelectorNilUsesHelmValues = false (scrapes ALL ServiceMonitors)
  • [x] Harbor container registry operational
  • [x] Woodpecker CI operational
  • [x] dora-framework note with manual baseline + band definitions

Previous Plan

None — standalone plan. Captures work that was buried in plan-2026-02-26-tf-ci-team-hardening Phase 5 and plan-2026-02-25-platform-observability Phase 4.

Depends On

None. All dependencies are already met (SDKs published, monitoring stack deployed).

Decisions Made

Decision Rationale
Deploy in <code>monitoring</code> namespace Alongside Prometheus/Grafana. This is a platform observability service, not an app.
Use <code>kubernetes_deployment</code> not Helm Too simple for a Helm chart. Single deployment, service, secret, ServiceMonitor, dashboard ConfigMap.
Use <code>kubernetes_manifest</code> for ServiceMonitor ServiceMonitor is a CRD — native TF kubernetes provider doesn't support CRDs natively.
Forgejo SDK uses basic auth (user/password) SDK uses <code>FORGEJO_USER</code> + <code>FORGEJO_PASSWORD</code> env vars, not token auth. Config must match.
Woodpecker SDK uses Bearer token SDK uses <code>WOODPECKER_URL</code> + <code>WOODPECKER_TOKEN</code> env vars.
SDK import names: <code>woodpecker_sdk</code>, <code>forgejo_sdk</code> Pip packages are <code>ldraney-woodpecker-sdk</code> and <code>ldraney-forgejo-sdk</code> but Python imports use underscores without prefix.
Poll interval: 60s DORA metrics are slow-moving (daily/weekly cadence). 60s is more than sufficient and keeps API load minimal.
FastAPI + prometheus_client FastAPI for health endpoints, prometheus_client for /metrics. Standard pattern.
Forgejo PyPI as pip index Both SDKs are published there. Dockerfile needs <code>--extra-index-url</code> pointing to Forgejo packages API.
Combined Phase 2+3 into single PR Both touch same TF files, faster for sprint deadline. Decision made 2026-03-02.
Reuse existing Forgejo admin creds No need for separate <code>forgejo_api_user</code>/<code>forgejo_api_password</code> vars — reuse <code>forgejo_admin_username</code> + <code>forgejo_admin_password</code>.

Architecture

Phases

Phase 1: Exporter Service — COMPLETE

Slug: phase-2026-03-01-1-dora-exporter-service
Goal: A Python service that exposes DORA Prometheus metrics for all Woodpecker-active repos.
Owner: Dev agent
Issue: issue-pal-e-dora-exporter-service (resolved)
Status: COMPLETE — merged 2026-03-01
Deliverables:
  • Forgejo repo: forgejo_admin/pal-e-dora-exporter
  • PR #1 merged (squash) — "Add DORA metrics exporter service" — 555 additions, 11 files
  • FastAPI app with /health and /metrics endpoints, lifespan-managed background tasks
  • Woodpecker collector: dora_deployments_total, dora_deployment_duration_seconds, dora_deployment_last_success_timestamp
  • Forgejo collector: dora_pr_merges_total, dora_pr_lead_time_seconds, dora_pr_review_count
  • Dockerfile with Forgejo PyPI extra-index-url
  • Woodpecker CI pipeline: build + push to Harbor (harbor.tail5b443a.ts.net/pal-e-dora-exporter/dora-exporter)
Review notes: Dev agent self-reviewed (review-fix loop). One fix: replaced deprecated asyncio.get_event_loop() with asyncio.get_running_loop(). Clean on second pass. No separate QA agent review — noted as process gap.

Phase 2+3: Deploy to Cluster + Grafana Dashboard — COMPLETE

Slug: phase-2026-03-01-2-dora-deploy-cluster + phase-2026-03-01-3-dora-grafana-dashboard
Goal: Exporter running in k8s, scraped by Prometheus, with DORA dashboard in Grafana.
Owner: Dev agent
Issue: issue-pal-e-platform-dora-deploy-dashboard (resolved) / Forgejo #9
Status: COMPLETE — PR #10 merged 2026-03-02 (squash). Pending tofu apply.
Deliverables:
  • PR #10 on pal-e-platform — "Deploy DORA exporter + Grafana dashboard (Phase 2+3)" — +665/-4, 4 files
  • terraform/main.tf: 5 new k8s resources (secret, deployment, service, ServiceMonitor, dashboard ConfigMap)
  • terraform/dashboards/dora-dashboard.json: 13 Grafana panels, 2 template variables (datasource + repo)
  • terraform/variables.tf: woodpecker_api_token (sensitive), dora_exporter_image (default Harbor)
  • Makefile: woodpecker_api_token added to TF_SECRET_VARS
  • Woodpecker API token encrypted and added to Salt pillar (salt/pillar/secrets/platform.sls)
Review notes: Dev agent self-reviewed (review-fix loop). One fix: MTTR overview panel PromQL query used non-existent status="failure" label on dora_deployment_last_success_timestamp. Clean on second pass.
Post-merge: Operator runs make tofu-plan then make tofu-apply to deploy.

Key Files

Phase File Repo Change
1 <code>src/main.py</code> pal-e-dora-exporter New — FastAPI app
1 <code>src/collectors/woodpecker.py</code> pal-e-dora-exporter New — Woodpecker metrics collector
1 <code>src/collectors/forgejo.py</code> pal-e-dora-exporter New — Forgejo metrics collector
1 <code>src/config.py</code> pal-e-dora-exporter New — env var config
1 <code>Dockerfile</code> pal-e-dora-exporter New — container build
1 <code>.woodpecker.yaml</code> pal-e-dora-exporter New — CI pipeline
2+3 <code>terraform/main.tf</code> pal-e-platform Add exporter deployment + service + ServiceMonitor + secret + dashboard ConfigMap
2+3 <code>terraform/variables.tf</code> pal-e-platform Add <code>woodpecker_api_token</code>, <code>dora_exporter_image</code>
2+3 <code>terraform/dashboards/dora-dashboard.json</code> pal-e-platform New — Grafana dashboard JSON
2+3 <code>Makefile</code> pal-e-platform Add <code>woodpecker_api_token</code> to TF_SECRET_VARS

Verification

  • [x] Phase 1: Exporter container builds in Woodpecker, pushes to Harbor. curl localhost:8000/metrics returns Prometheus metrics with real pipeline data.
  • [ ] Phase 2+3: kubectl get pods -n monitoring shows exporter running. Prometheus targets page shows exporter as UP. promql: dora_deployments_total returns data. Grafana dashboard loads at grafana.tail5b443a.ts.net with all four DORA metric panels showing real data.

Next Plan Seeds

  • Agent DORA metrics (PRs shipped/day, rework rate, plan-to-ship time) — requires pal-e-docs API integration
  • Alerting rules for DORA threshold violations (e.g., CFR exceeds 15%)
  • Historical DORA trend analysis and weekly reports
  • Update dora-framework to reference automated dashboard instead of manual measurement
  • dora-framework — the axiom. This plan makes it measurable.
  • plan-2026-02-26-tf-ci-team-hardening — Phase 5 had DORA baseline (now superseded by this plan)
  • plan-2026-02-25-platform-observability — Phase 4 had DORA dashboard (now superseded by this plan)
  • platform-maturity-matrix — DORA metrics validate maturity claims