pal-e-docs

Project: pal-e-platform

project-pal-e-platform Project Page

active project-page mermaid
Project

pal-e-platform

Vision

The infrastructure pillar of a DORA Elite AI Enterprise. In the three-pillar model (platform=DevOps/SRE, docs=product, agency=process+enforcement), pal-e-platform proves the DORA numbers — Deployment Frequency and MTTR. A developer adds one entry to var.services, pushes code to Forgejo, and gets: a namespace, CI pipeline, container registry project, GitOps deployment, TLS ingress, monitoring, log aggregation, and alerting. The Terraform is the control plane. The platform is the product.
Three repos, three control planes, one system. pal-e-platform (Terraform + Salt) provisions the foundation: k3s cluster, Tailscale networking, Forgejo, Woodpecker CI, Harbor, MinIO, CNPG Postgres, Keycloak, and the full monitoring + validation stack. pal-e-services (Terraform) onboards services via ArgoCD and a for_each automation pattern. pal-e-deployments (Kustomize + ArgoCD) defines how applications deploy via GitOps overlays — the source ArgoCD reads for all 6 services. Three control planes manage three layers: Terraform manages what exists in the cluster (Helm releases, namespaces, RBAC). GitOps/ArgoCD manages how applications are delivered (kustomize overlays, image tags, auto-sync). SaltStack manages the host (k3s, nftables firewall, packages, GPG-encrypted pillar). Everything self-hosted. No external cloud dependencies except Tailscale for networking.
Operating thesis: This platform proves that one human architect + AI agent orchestration can build and operate infrastructure that traditionally requires a 50-person engineering organization. Three control planes: Terraform manages everything inside the cluster. GitOps manages application delivery. SaltStack manages everything on the host. A seven-pillar validation framework (observability, SLO governance, policy, security, progressive delivery, load testing, chaos engineering) proves it all works — not through architecture documents, but through measured, repeatable evidence. DORA is the proof.
DORA thesis: Platform hardening IS DORA enablement. Every phase in the hardening plan directly improves one or more DORA metrics — observability reduces MTTR and Change Failure Rate, CI hardening increases Deployment Frequency and reduces Lead Time, Kustomize patterns make deploys repeatable, network security and env isolation reduce blast radius. The virtuous cycle: platform maturity → developers trust production → they ship more often → DORA metrics improve → which validates the platform investment. DORA is two systems measured as one: Observability (SRE — production health) + Kanban (DevEx — value throughput via pal-e-docs boards). The platform provides the observability. Pal-e-docs provides the Kanban. DORA proves both work. This is what makes it an elite AI enterprise — not just that AI agents write the code, but that the system they operate within is measured, observable, and continuously improving.

User Stories

Who uses the platform, what they need, and how we measure success. pal-e-platform serves one primary role: the Superuser who deploys and operates infrastructure for all projects.
Role Story Success Metric story:X key
<strong>Superuser</strong> (Lucas) I can deploy infrastructure changes via <code>tofu plan/apply</code> and see them succeed in Woodpecker CI without manual intervention. Pipeline success rate >95%. Zero manual kubectl interventions for routine deploys. <code>story:superuser-deploy</code>
<strong>Superuser</strong> (Lucas) I can observe the health of all services via Grafana dashboards. When something breaks, I see it before users report it. MTTR <30min for infrastructure incidents. Alert-to-awareness <5min. <code>story:superuser-observe</code>
<strong>Superuser</strong> (Lucas) I can recover from failures using documented SOPs. Every failure mode has a runbook. All failure modes covered by recovery SOPs. Zero novel failure responses (every response follows an SOP). <code>story:superuser-recover</code>
<strong>Superuser</strong> (Lucas) I can onboard a new service to the platform (Forgejo repo, Woodpecker CI, k3s deployment, Tailscale funnel) following a documented procedure. Service onboarding follows <code>service-onboarding-sop</code>. New service deploys in <1 day. <code>story:superuser-onboard-service</code>
<strong>Superuser</strong> (Lucas) I can SSH into the platform from any device (phone, laptop, tablet) using any standard SSH client without browser-based approval gates. SSH from Termius/any client succeeds on first attempt. Zero browser redirects in the SSH flow. <code>story:superuser-remote-access</code>
<strong>Superuser</strong> (Lucas) I can log in once via Keycloak and access all platform services (Forgejo, Grafana, Harbor, MinIO) without re-authenticating. A single admin dashboard gives me click-through access to everything. Zero re-login prompts when navigating between services. All services accessible from one landing page on mobile. <code>story:superuser-sso</code>
<strong>Superuser</strong> (Lucas) I can see a consistent, mobile-friendly visual identity across all platform services. The platform feels like one product, not a collection of open-source tools. All service login pages share design tokens (font, palette). Mobile-friendly on phone. <code>story:superuser-unified-ui</code>
<strong>Superuser</strong> (Lucas) I can iterate on the pal-e-docs frontend with live hot-reload, protected behind SSO, so I can build the interface to manage my documentation system. File save to browser reflect in <2s. Keycloak-gated. API connectivity to pal-e-docs backend. <code>story:superuser-docs-frontend</code>
<strong>Superuser</strong> (Lucas) I can search platform knowledge semantically — natural language queries return relevant notes ranked by meaning, not just keyword matches. The embedding pipeline processes new content automatically. Semantic search returns relevant results for natural language queries. Embedding worker processes new blocks within 60s. Zero manual re-indexing. <code>story:semantic-search</code>

Architecture

Domain Model




          
  
Data Flow



          
  



          
  
Deployment



          
  



          
  
Validation Pipeline (Target State — Phases 16-23)



          
  



          
  
Plan



          
    
Active: plan-pal-e-platform — Platform Hardening



          
    
Harden from working dev cluster to production-grade, seven-pillar validated system. 23 phases across three tiers: Tier 1 Foundation — observability (1-5, 14-15), SLO governance/Sloth (16), distributed tracing/OTel (17), operations dashboard (18). Tier 2 Hardening — network security (8), policy-as-code/Kyverno (19), security deepening/Renovate+Falco+Cosign+ZAP (20a-d), progressive delivery/Argo Rollouts (21). Tier 3 Advanced Validation — load testing/k6 (22), chaos engineering/LitmusChaos (23, capstone). 16/23 main phases COMPLETED. Phase 17a (Woodpecker Secrets) in-progress. Every validation tool feeds Prometheus/Grafana — single pane of glass.



          
    
Completed plans:



          
  

    

        
          

              Plan
              Completed
              Summary
          

        
      
          

              <code>plan-2026-02-26-tf-modularize-postgres</code>
              2026-03-13
              SQLite to Postgres migration + CNPG operator deployment
          

          

              <code>plan-2026-02-25-platform-observability</code>
              2026-03-13
              5 phases reparented into plan-pal-e-platform
          

          

              <code>plan-2026-02-26-salt-host-management</code>
              2026-02-28
              SaltStack: host audit, bootstrap, codify 27 states, GPG pillar, nftables
          

          

              <code>plan-2026-02-24-minio-object-storage</code>
              2026-02-25
              MinIO standalone deployment
          

          

              <code>plan-2026-03-01-dora-metrics-dashboard</code>
              2026-03-02
              DORA framework + metrics foundation
          

      
    

  



          
  
Board



          
    
board-pal-e-platform — Pal E Platform Board. Continuous kanban. 26 items (1 plan, 22 phases, 3 issues). Columns: Backlog → In Progress → Done. Auto-syncs plan phases via sync_board. Forgejo issues auto-sync via sync-issues.



          
  
Status



          
  
    
      
  • Platform stable and operational — all core infrastructure deployed and running. Seven-pillar validation framework scoped (Phases 16-23).
    • 
      
  • k3s cluster with Tailscale funnels for ingress/TLS (no cert-manager, no Traefik)
    • 
      
  • Forgejo, Woodpecker CI (Postgres-backed via CNPG), Harbor, MinIO, Keycloak all operational
    • 
      
  • CNPG Postgres operator deployed — WAL archiving to MinIO, daily base backups, PITR verified
    • 
      
  • Monitoring stack: Prometheus (15d retention), Grafana (3 custom dashboards + kube-prometheus defaults), Loki (7d retention), Promtail, Alertmanager (Telegram), Blackbox Exporter (13 probe targets)
    • 
      
  • DORA measurement pipeline LIVE — exporter producing 726 metrics. Platform Overall: High-Elite. 262 PRs merged, 11.4/day, p50 lead time 10 min.
    • 
      
  • Ollama + NVIDIA device plugin deployed (GPU workloads, Qwen3-Embedding-4B)
    • 
      
  • Keycloak IdP LIVE — OIDC chain: Keycloak → basketball-api JWKS → westside-app Auth.js. 50 users, role-based access (admin/coach/player).
    • 
      
  • 6 services onboarded via pal-e-services: pal-e-docs, basketball-api, pal-e-app, westsidekingsandqueens, platform-validation, gcal-scheduler
    • 
      
  • Woodpecker CI automated — plan-on-PR + apply-on-merge for pal-e-platform. Merge = deploy.
    • 
      
  • Kustomize migration COMPLETE — all 6 services on centralized overlays in pal-e-deployments. ArgoCD reads from pal-e-deployments.
    • 
      
  • Network security COMPLETE (Phase 8) — three-layer defense: NetworkPolicies (15 namespaces), Tailscale ACLs (role-scoped), nftables host firewall (Salt-managed).
    • 
      
  • Alert tuning COMPLETE (Phase 16-alert) — 5 PRs across 4 repos. Alerts reduced from 19 to stale-only.
    • 
      
  • Resource usage: 12 cores / 125GB RAM / 1.8TB NVMe. Cluster uses ~11% CPU, ~9% RAM. Massive headroom for validation tooling.
    • 
      
  • Salt plan COMPLETE — host fully codified as 27 Salt states, GPG-encrypted pillar, nftables firewall applied.
    • 
      
  • Source of truth: Forgejo — migrated from GitHub 2026-02-27. GitHub is historical only.
    • 
      
  • 16 of 23 main phases completed + subphases — see plan-pal-e-platform. Three-tier framework: Tier 1 (SLO, OTel, Dashboard) → Tier 2 (Kyverno, Security 20a-d, Rollouts) → Tier 3 (k6 Load, LitmusChaos Capstone). Phase 17a (Woodpecker Secrets) in-progress.
    • 



          
  
Milestones



          
  

    

        
          

              Date
              Milestone
              Impact
          

        
      
          

              2026-03-14
              <a href="/notes/milestone-2026-03-14-woodpecker-postgres-dora-pipeline">Woodpecker Postgres Migration + DORA Pipeline Complete</a>
              5 PRs, 2 phases completed (5+13), DORA measurement pipeline reliable. Infra DF/LT moved from Medium→High. 726 metrics across 28 repos. <a href="https://grafana.tail5b443a.ts.net">Grafana</a> | <a href="https://alertmanager.tail5b443a.ts.net">Alertmanager</a> | <a href="https://woodpecker.tail5b443a.ts.net">Woodpecker</a>
          

          

              2026-03-14
              Platform Hardening: 8/13 phases complete
              Phases 1-6, 10, 13 COMPLETED. Observability stack: 26 Grafana dashboards, 31 alert rule groups, 19 ServiceMonitors, 3 PodMonitors. Alert noise floor: 23→3.
          

          

              2026-03-02
              Platform bootstrap complete
              k3s + Tailscale + Forgejo + Woodpecker + Harbor + MinIO + kube-prometheus-stack + Loki + CNPG + ArgoCD all deployed via OpenTofu. Salt codifies host.
          

      
    

  



          
  
Repos



          
  

    

        
          

              Repo
              Platform
              Role
              Status
          

        
      
          

              <a href="https://forgejo.tail5b443a.ts.net/forgejo_admin/pal-e-platform">pal-e-platform</a>
              Forgejo
              OpenTofu IaC + SaltStack for base platform
              active
          

          

              <a href="https://forgejo.tail5b443a.ts.net/forgejo_admin/pal-e-services">pal-e-services</a>
              Forgejo
              OpenTofu IaC for service onboarding
              active
          

          

              <a href="https://forgejo.tail5b443a.ts.net/forgejo_admin/deployments">pal-e-deployments</a>
              Forgejo
              Kustomize bases + per-service overlays
              active
          

          

              <a href="https://forgejo.tail5b443a.ts.net/forgejo_admin/minio-sdk">minio-sdk</a>
              Forgejo
              Pure Python S3 SDK with custom Signature V4 signing
              active
          

          

              <a href="https://forgejo.tail5b443a.ts.net/forgejo_admin/minio-playground">minio-playground</a>
              Forgejo
              Mobile-first vanilla HTML/CSS/JS file browser prototype
              active
          

          

              <a href="https://forgejo.tail5b443a.ts.net/forgejo_admin/gmail-sdk">gmail-sdk</a>
              Forgejo
              Gmail API SDK — OAuth auth, token lifecycle, email operations
              active
          

          

              <a href="https://forgejo.tail5b443a.ts.net/forgejo_admin/gmail-mcp">gmail-mcp</a>
              Forgejo
              MCP server for Gmail — wraps gmail-sdk for Claude Code
              active
          

      
    

  



          
  
Infrastructure



          
  

    

        
          

              Component
              Details
          

        
      
          

              Host
              Arch Linux · 12 cores · 125GB RAM · 1.8TB NVMe · NVIDIA GPU
          

          

              Cluster
              k3s single-node · Tailscale funnels for ingress/TLS
          

          

              Control Plane 1
              <strong>Terraform</strong> (OpenTofu) — pal-e-platform deploys Helm charts, pal-e-services onboards services
          

          

              Control Plane 2
              <strong>GitOps</strong> — ArgoCD reads kustomize overlays from pal-e-deployments. Image Updater polls Harbor tags.
          

          

              Control Plane 3
              <strong>SaltStack</strong> — 27 states, GPG-encrypted pillar, nftables firewall, k3s lifecycle
          

          

              CI
              Woodpecker CI (Postgres-backed via CNPG). Plan-on-PR, apply-on-merge for Terraform repos. Test+build+push for app repos.
          

          

              Container Registry
              Harbor — Trivy scanning, robot accounts per service, SBOM storage (future: Cosign signatures)
          

          

              Object Storage
              MinIO — CNPG WAL archives, Loki chunks, Tempo traces (future)
          

          

              Identity
              Keycloak — OIDC provider. Realms: westside-basketball, mcd-tracker. JWKS validation in app APIs.
          

          

              Monitoring
              Prometheus (15d) · Grafana (26 dashboards) · Loki (7d) · Promtail · Alertmanager → Telegram · Blackbox (13 probes) · DORA Exporter (726 metrics)
          

          

              Secrets
              Salt GPG pillar (21 secrets) + SOPS/Age in kustomize overlays (6 app secrets). Two paths per <code>sop-secrets-management</code>.
          

          

              GPU
              Ollama + NVIDIA device plugin — Qwen3-Embedding-4B for pal-e-docs semantic search
          

          

              Services onboarded
              6: pal-e-docs, basketball-api, pal-e-app, westsidekingsandqueens, platform-validation, gcal-scheduler
          

      
    

  



          
  
Inbox



          
    
Untriaged TODOs awaiting scoping into plan-pal-e-platform. See convention-todo-lifecycle.



          
    
All 7 platform TODOs are now parked under plan phases. No unparented items in the inbox.