Milestone: Woodpecker Postgres Migration + DORA Pipeline Complete
Milestone: Woodpecker Postgres Migration + DORA Pipeline Complete
Date: 2026-03-14
Session deliverables: 5 PRs merged, 3 issues closed, 2 phases completed, 1 hotfix shipped
What We Shipped
| PR | What | DORA Impact |
|---|---|---|
| <strong>#58</strong> | CI <code>-lock=false</code> + internal Forgejo URL | DF: eliminates state lock contention that blocked CI applies |
| <strong>#59</strong> | Woodpecker SQLite → Postgres (CNPG) | ALL: reliable pipeline API unlocks trustworthy DORA measurement |
| <strong>#61</strong> | CNPG backup verification CronJob | MTTR: automated DR confidence, daily backup freshness checks |
| <strong>#65</strong> | OAuth URL split (hotfix) | DF: unblocks Woodpecker UI access for repo management |
Plus: 28 repos re-activated, 6 global + 31 repo secrets re-created, agent PVC reset, kubeconfig CI fix, API token rotation — all via scripted automation.
Why This Matters
This session completed the DORA measurement data pipeline:
Before today, this pipeline existed but the upstream data source (Woodpecker SQLite API) was broken — log streaming failed, API responses were unreliable, debugging was archaeology. The DORA exporter was scraping garbage. Now it produces 726 metric lines across 28 repos from a reliable Postgres-backed API.
Observability Stack — Current State
| Capability | Technology | Status |
|---|---|---|
| Infrastructure metrics | kube-prometheus-stack (Prometheus + node-exporter) | 19 ServiceMonitors, 25 dashboards |
| Application metrics | prometheus-fastapi-instrumentator + ServiceMonitor | pal-e-docs golden signals dashboard live |
| Database metrics | CNPG PodMonitors (pal-e-postgres + woodpecker-db) | 3 PodMonitors, Postgres metrics in Prometheus |
| CI/CD metrics | DORA exporter → Prometheus | 726 metrics: deployments, PR merges, per-repo |
| Log aggregation | Loki + Promtail | All pod logs searchable in Grafana Explore |
| Dashboards | Grafana (ConfigMap sidecar) | 26 dashboards (25 infra + 1 DORA + 1 golden signals) |
| Alerting | Alertmanager → Telegram + Slack | 31 rule groups, custom platform-alerts, noise floor at 3 |
| Container scanning | Harbor Trivy | Vulnerability scanning on all pushed images |
| Backup verification | CronJob (MinIO WAL check) | Daily at 03:00 UTC, checks both CNPG clusters |
DORA Band Movement
| Dimension | Before (2026-03-01) | After (2026-03-14) | Key Change |
|---|---|---|---|
| Infra Pipeline DF | Medium (manual apply) | <strong>High</strong> (merge=deploy CI) | Automated <code>tofu apply</code> on merge to main |
| Infra Pipeline LT | Medium (hours-days) | <strong>High</strong> (minutes) | PR merge triggers immediate apply |
| Measurement Confidence | Low-Medium (estimates) | <strong>Medium-High</strong> (automated) | DORA exporter on reliable Postgres API |
Live URLs
- Grafana:
https://grafana.tail5b443a.ts.net— 26 dashboards including DORA + golden signals - Alertmanager:
https://alertmanager.tail5b443a.ts.net— Telegram + Slack routing - Woodpecker:
https://woodpecker.tail5b443a.ts.net— now on Postgres, logs visible - Harbor:
https://harbor.tail5b443a.ts.net— Trivy scanning enabled
What "Complete Datadog" Looks Like
We're building a sovereign observability platform. Here's where we stand against a full-featured observability suite:
| Datadog Capability | Our Equivalent | Status | Gap |
|---|---|---|---|
| Infrastructure Monitoring | kube-prometheus-stack + node-exporter | COMPLETE | — |
| APM (traces) | prometheus-fastapi-instrumentator | PARTIAL | No distributed tracing (OpenTelemetry + Tempo) |
| Log Management | Loki + Promtail + Grafana Explore | COMPLETE | — |
| Dashboards | Grafana (26 dashboards) | COMPLETE | Need SLO/error budget dashboards |
| Alerting | Alertmanager → Telegram + Slack | COMPLETE | No escalation/on-call rotation (one-man team) |
| CI Visibility | DORA exporter + Woodpecker API | COMPLETE | No flaky test detection |
| Synthetic Monitoring | — | NOT STARTED | Need uptime checks per Tailscale funnel |
| Database Monitoring | CNPG PodMonitors | COMPLETE | No query-level analysis (pg_stat_statements) |
| Container Security | Harbor Trivy | COMPLETE | No runtime security (Falco) |
| RUM (frontend) | — | NOT STARTED | SvelteKit Faro integration possible |
| SLOs / Error Budgets | — | NOT STARTED | Pyrra or Sloth for Prometheus-native SLOs |
| Cost Management | — | N/A | Self-hosted = zero cloud cost. Track resource allocation instead. |
Related
plan-pal-e-platform— Platform Hardening (8/13 phases complete)dora-framework— DORA axiom and baseline measurementsplatform-maturity-matrix— capability scorecardphase-observability-5-architecture— architecture review completed this session