pal-e-docs

Phase 23: Chaos Engineering (LitmusChaos) — Capstone

phase-platform-23-chaos-engineering Phase

phase
Project

Phase 23: Chaos Engineering (LitmusChaos) — Capstone

Goal: Prove platform resilience under failure through controlled chaos experiments. The capstone that validates everything built in Tiers 1-2 actually works when things break.
Owner: Dev agent
Repo: forgejo_admin/pal-e-platform
Depends on: Phase 16 (SLO — chaos without baselines is noise), Phase 22 (Load — need normal behavior before testing abnormal)
Tier: 3 — Advanced Validation (capstone). Zero exceptions in the research: chaos without a solid foundation creates noise, not confidence. When you get here, it's the proof the platform is mature.
DORA: MTTR — chaos experiments prove recovery time under failure conditions. A claimed 15-minute MTTR that's never been tested is hope, not data.

Scope

  • Deploy LitmusChaos via Helm — ChaosCenter UI + ChaosEngine CRDs + Prometheus exporter
  • Experiment library (graduated by blast radius):
  • Chaos → SLO correlation: During each experiment, track SLO burn rate. Does a killed pod violate the SLO? How fast does the error budget recover? This closes the observability loop back to Tier 1.
  • LitmusChaos Prometheus exporter → chaos experiment results in Grafana (pass/fail, duration, blast radius)
  • Blast radius controls: Start in non-production namespaces (platform-validation), graduate to production with time-bounded experiments
  • SOP: sop-chaos-gameday — scheduled chaos experiments with pre/post checklists, escalation procedures, blast radius limits

Deliverables

  • TBD — filled after completion
  • plan-pal-e-platform — parent plan
  • phase-pal-e-platform-16-slo-error-budgets — SLOs define what "healthy" means; chaos tests whether it holds
  • phase-platform-22-load-testing — load baselines define "normal"; chaos tests the abnormal
  • phase-pal-e-platform-18-operations-dashboard — the dashboard that shows chaos impact in real-time
  • sop-incident-response — chaos experiments should trigger and validate the incident response flow