pal-e-docs

Plan: Platform Hardening

plan-pal-e-platform Plan

plan deprecated
Project

Vision

Harden the pal-e platform from a working dev cluster into a production-grade, seven-pillar validated system. Three tiers — Foundation (observability + SLO governance), Hardening (policy-as-code + security deepening + progressive delivery), and Advanced Validation (load testing + chaos engineering) — prove platform reliability through structured, tiered validation. Every tool feeds the same Prometheus/Grafana layer. The platform that proves one human + AI agents can operate at enterprise grade.

Projects & Repos Touched

Project/Repo Platform Role
pal-e-platform Forgejo OpenTofu IaC + SaltStack — core platform infrastructure
pal-e-services Forgejo Service onboarding (ArgoCD, var.services for_each)
pal-e-deployments Forgejo Kustomize bases + overlays (ArgoCD syncs from here)

Context

The platform is stable and operational — k3s, Tailscale, Forgejo, Woodpecker, Harbor, MinIO, monitoring stack, CNPG Postgres all deployed. But it's operating at ~21% maturity (per platform-maturity-matrix). This plan is the DORA enablement engine: every phase directly improves one or more DORA metrics. Observability (Phases 1-5) reduces MTTR and Change Failure Rate. CI hardening (Phase 6) increases Deployment Frequency. Kustomize patterns (Phase 7) reduce Lead Time. Security and isolation (Phases 8-9) reduce blast radius. Stubs 10-13 cover the long tail — vulnerability scanning, dependency management, incident SOPs, backup verification. The through-line: you can't achieve DORA Elite without a platform that makes production smooth enough that teams push to it from the beginning.

Previous Plan

Consolidates plan-2026-02-25-platform-observability (active, 5 phases reparented) plus 4 deferred plans and 4 plan stubs that were never promoted. See completed plans table on project-pal-e-platform.

Depends On

Nothing — this is foundation work. Other projects depend on this.

Decisions Made

Decision Rationale
One plan per project 13 separate plans/stubs caused fragmentation. One living plan with phases that get worked in priority order.
Observability phases first Can't harden what you can't see. Alerting and dashboards are prerequisite for everything else.
Deferred plans become phases Content is preserved, just re-homed. Former inline phases become the phase's scope section.
Stubs become phases Stubs were proto-phases waiting for promotion. Now they're in the plan with a position.
DORA is the through-line Every phase maps to a DORA metric. Platform hardening isn't an ops checkbox — it's the engine that drives Deployment Frequency, Lead Time, Change Failure Rate, and MTTR across the entire agency. DORA = Observability (production health) + Kanban (value throughput).
Tier 1.5: Operational Excellence gates Tier 2 (2026-03-17) Discovered during session: 8 open TODOs (CI state locks, Harbor drift, onboarding gaps) are more dangerous than missing capabilities (Sloth, Kyverno). Deploying new tools on unreliable CI/CD compounds failures. Fix the foundation before extending it. TODOs stay as board items (per convention: too small for phase notes). Phase 17b is the only exception — it's architectural.

Phases

See child phase notes: list_notes(parent_slug="plan-pal-e-platform")
Summary: 27 phases (+ subphases) organized in a seven-pillar, three-tier platform validation framework. Phases 1-8, 10, 12-16(alert tuning), 17a, 24, 26 COMPLETED (19/27 main phases). Phase 9 DEFERRED. Phase 11 DEFERRED (absorbed into 20a). Phase 17b (Terraform State Governance) IN PROGRESS — 17b.1 CI Lock Recovery COMPLETED (PR #100), 17b.2 State Hygiene SOP COMPLETED, 17b.3 future. kube-router bug RESOLVED (NetworkPolicies re-enabled 2026-03-17).

Tier 1 — Foundation (Observability + SLO Governance): Phases 1-5, 14, 15, 16(alert) COMPLETED. Phase 16 (SLO/Sloth), Phase 17 (OTel/Tempo), Phase 18 (Ops Dashboard) NOT STARTED.
Tier 1.5 — Operational Excellence (Fix What We Have): Phase 17b IN PROGRESS (17b.1+17b.2 done). CI Pipeline FIXED (2026-03-21): PR #139 merged (OAuth override removed). PR #134 merged (alpine/git clone + SA token auth + internal MinIO + sysctl IPv6 disable + CoreDNS fix). Pipeline fully green: clone → validate → plan all pass on PR events. Root causes fixed: CoreDNS MagicDNS forwarder removed, Woodpecker trusted repo + clone plugin configured, kubeconfig moved to SA token auth. This tier gates Tier 2.
Tier 2 — Hardening (Policy + Security + Progressive Delivery): Phase 8 (Network) COMPLETED. Phase 19 (Kyverno), Phase 20 (Security), Phase 21 (Argo Rollouts) NOT STARTED.
Tier 3 — Advanced Validation (Load + Chaos): Phase 22 (k6), Phase 23 (LitmusChaos — capstone) NOT STARTED.
Tier 4 — MinIO Mobile Interface: Phase 24 (SDK) COMPLETED. Phase 25 (API) NOT STARTED (depends on 24). Phase 26 (Playground) COMPLETED. Phase 27 (SvelteKit) NOT STARTED (depends on 24+25+26).

Key Files

  • terraform/ — cluster-level IaC
  • salt/ — host-level configuration
  • Makefile — unified CLI

Verification

  • Platform maturity score increases from 21% baseline
  • Alerting fires on real incidents
  • CI catches TF errors before apply
  • New services deploy via Kustomize base inheritance

Next Plan Seeds

  • Multi-node cluster (Hetzner expansion)
  • GitOps for Salt (Salt states in separate repo, ArgoCD-like for host config)
  • project-pal-e-platform — project page
  • platform-maturity-matrix — capability scorecard
  • service-onboarding-sop — how services consume the platform