pal-e-backup

Plan 1

Plan: Off-Site Platform Backup plan-pal-e-backup
Vision

If the PC dies, everything can be rebuilt from cloud backups. One unified off-site destination, one IaC-managed pipeline, zero manual steps. Every piece of irreplaceable platform state — databases, git repos, object storage, identity, secrets — backed up daily to external S3 with automated verification and alerting.

Projects & Repos Touched
- pal-e-backup — new repo, terraform + backup scripts + k8s CronJob manifests
- pal-e-platform — may need minor changes for RBAC, service accounts
Context

The platform runs on a single k3s node (Arch Linux, 12-core, 128GB RAM, 1.8TB NVMe, GTX 1070). All persistent state lives on local-path PVCs on that one disk. CNPG databases (pal-e-docs, woodpecker) have WAL archiving to MinIO, but MinIO is on the same disk. Terraform state backs up to MinIO daily. Everything else — Forgejo git repos, Keycloak identity, Harbor metadata, basketball-api DB, mcd-tracker DB — has zero backup. A single disk failure loses the entire platform.

Previous Plan

None. First plan for this project.

Depends On
- External S3 account (Backblaze B2 recommended, Hetzner Object Storage as alternative)
- k8s RBAC for CronJobs to access database pods and secrets
Decisions Made
- Daily pg_dump over CNPG migration — basketball-api and mcd-tracker stay as plain Postgres pods. pg_dump gives backup protection without the complexity of CNPG migration. CNPG migration is a separate initiative.
- Backblaze B2 recommended — $6/TB, S3-compatible, proven. Hetzner Object Storage is the alternative if staying in Hetzner ecosystem for future two-node architecture.
- Unified bucket structure — one bucket, directory-per-service. Not one bucket per service.
- Daily frequency for cloud backups — CNPG does continuous WAL locally. Cloud copy is disaster insurance, daily is sufficient.
Phases

Phase 1: Foundation — External S3 + Repo Setup (NOT STARTED)

Create the pal-e-backup Forgejo repo. Provision external S3 bucket with terraform. Store cloud credentials as k8s secrets. Verify S3 connectivity from within the cluster. Establish the bucket directory structure.

Deliverables: Repo exists. Terraform provisions S3 bucket. mc can write to external S3 from a k8s pod. Bucket structure: s3://pal-e-backups/{postgres,forgejo,keycloak,minio-mirror,k8s-secrets}/

Phase 2: Database Backups (NOT STARTED)

CronJobs that pg_dump each database to external S3 daily. Four databases: pal-e-docs (CNPG, postgres namespace), woodpecker (CNPG, woodpecker namespace), basketball-api (plain pod, basketball-api namespace), mcd-tracker (plain pod, mcd-tracker namespace). Each dump is gzipped, timestamped, uploaded to s3://pal-e-backups/postgres/{db-name}/YYYY-MM-DD.sql.gz. 30-day retention with automated pruning.

Deliverables: 4 CronJobs running daily. Recent dumps visible in S3. Old dumps auto-pruned at 30 days.

Phase 3: Forgejo Backup (NOT STARTED)

CronJob that runs gitea dump inside the Forgejo pod (or a sidecar with access to the Forgejo PVC). This captures git repositories, the SQLite database (issues, PRs, users, settings), configuration, and avatars in a single zip. Upload to s3://pal-e-backups/forgejo/gitea-dump-YYYY-MM-DD.zip. 30-day retention.

Deliverables: Daily Forgejo dump in S3. Full restore tested — git repos, issues, and user data recoverable.

Phase 4: MinIO Mirror (NOT STARTED)

CronJob that runs mc mirror from local MinIO to external S3. This replicates WAL archives (CNPG PITR capability), terraform state snapshots, and assets to the cloud. Destination: s3://pal-e-backups/minio-mirror/. Mirror is incremental — only new/changed objects transfer.

Deliverables: MinIO contents mirrored to external S3. CNPG PITR possible from cloud-stored WAL archives.

Phase 5: Identity & Secrets Backup (NOT STARTED)

Two CronJobs: (1) Keycloak realm export — uses Keycloak admin CLI to export all realms, users, clients, roles to JSON. Upload to s3://pal-e-backups/keycloak/. (2) k8s secrets export — dumps all secrets from critical namespaces (encrypted with age or GPG before upload) to s3://pal-e-backups/k8s-secrets/. 30-day retention for both.

Deliverables: Keycloak realm export in S3. Encrypted k8s secrets in S3. Identity can be rebuilt from exports.

Phase 6: Monitoring & Verification (NOT STARTED)

Verification CronJob that checks all backup types exist in S3 and are less than 25 hours old. Prometheus metrics for backup age per category. AlertManager rules that fire if any backup is stale. Fix existing cnpg-backup-verify CronJob (currently failing). Route alerts to existing Slack/Telegram pipeline.

Deliverables: Prometheus dashboard showing backup freshness. Alert fires within 1 hour if any daily backup is missing. Zero silent failures.

Phase 7: Disaster Recovery Test (NOT STARTED)

Full restore test. Provision a temporary environment (could be a Hetzner VPS or local VM). Restore every component from cloud backups only — no access to original data. Verify: pal-e-docs serves notes, Forgejo has repos, Keycloak authenticates users, basketball-api has player data. Document the restore procedure as an SOP. Time the full restore.

Deliverables: SOP: full platform restore from cloud backups. Tested end-to-end. Restore time documented.

Key Files

TBD — repo doesn't exist yet. Expected structure:
```
pal-e-backup/
├── terraform/
│   ├── main.tf          # S3 bucket + IAM provisioning
│   ├── cronjobs.tf      # All backup CronJob definitions
│   └── monitoring.tf    # Prometheus rules + alerting
├── scripts/
│   ├── pg-dump.sh       # Database backup script
│   ├── forgejo-dump.sh  # Forgejo backup script
│   ├── minio-mirror.sh  # MinIO mirror script
│   ├── keycloak-export.sh
│   └── verify-backups.sh
└── CLAUDE.md
```
Verification
- All 6 backup categories present in external S3 with daily freshness
- Prometheus alerts fire within 1 hour of a missed backup
- Full restore tested and documented
- Restore time under 2 hours for complete platform
Epilogue

Empty — plan has not started.

Related
- sop-postgres-restore — existing CNPG restore SOP (covers local MinIO only, needs cloud extension)
- plan-pal-e-platform — platform hardening (future: two-node architecture builds on backup destination)
- deployment-lessons — operational lessons including backup failures

Board 1

Pal-E Backup Board board-pal-e-backup

Pal-E Backup Board

Project Page 1

Project: pal-e-backup project-pal-e-backup

Vision

Off-site backup and disaster recovery for the entire Pal-E platform. If the PC dies, everything can be rebuilt from cloud backups — databases, git repos, identity, secrets, object storage. One unified pipeline, one cloud destination, managed as IaC.

User Stories

Role	Key	Story	Success Metric
Platform owner	`sleep-at-night`	I want to know that a disk failure won't destroy my platform	Full restore from cloud backups tested and documented
Platform owner	`backup-confidence`	I want to be alerted if any backup fails	Alert fires within 1 hour of a missed backup
Platform owner	`restore-speed`	I want to restore the full platform in under 2 hours	Restore time documented from DR test

Plan

Active: plan-pal-e-backup — Off-Site Platform Backup

7 phases: foundation (S3 + repo), database backups, Forgejo backup, MinIO mirror, identity/secrets, monitoring/verification, disaster recovery test.

Board

board-pal-e-backup — Pal-E Backup Board. Continuous kanban.

Status

Current backup coverage: pal-e-docs DB and woodpecker DB have CNPG WAL archiving to local MinIO. Terraform state backed up to local MinIO daily. Everything else has zero backup.
Off-site backup: None. All backups are on the same disk as the data they protect.
Backup verification: cnpg-backup-verify CronJob exists but is currently failing.

Milestones

None yet. First milestone will be defined when Phase 7 (DR test) completes — "Platform Protected."

Architecture

flowchart TD
    subgraph k3s["k3s Node (archbox)"]
        PG_DOCS["pal-e-docs DB
CNPG"]
        PG_WP["woodpecker DB
CNPG"]
        PG_BBALL["basketball-api DB
plain pod"]
        PG_MCD["mcd-tracker DB
plain pod"]
        FORGEJO["Forgejo
git repos + SQLite"]
        KEYCLOAK["Keycloak
H2 file DB"]
        MINIO["MinIO
WAL + TF state + assets"]
        K8S_SECRETS["k8s Secrets"]
    end

    subgraph cron["Backup CronJobs"]
        PGDUMP["pg_dump
daily"]
        GDUMP["gitea dump
daily"]
        MIRROR["mc mirror
daily"]
        KEXPORT["keycloak export
daily"]
        SDUMP["secrets export
daily (encrypted)"]
    end

    subgraph cloud["External S3 (Backblaze B2)"]
        S3["s3://pal-e-backups/"]
        S3_PG["postgres/"]
        S3_FG["forgejo/"]
        S3_MM["minio-mirror/"]
        S3_KC["keycloak/"]
        S3_SEC["k8s-secrets/"]
    end

    PG_DOCS --> PGDUMP
    PG_WP --> PGDUMP
    PG_BBALL --> PGDUMP
    PG_MCD --> PGDUMP
    FORGEJO --> GDUMP
    MINIO --> MIRROR
    KEYCLOAK --> KEXPORT
    K8S_SECRETS --> SDUMP

    PGDUMP --> S3_PG
    GDUMP --> S3_FG
    MIRROR --> S3_MM
    KEXPORT --> S3_KC
    SDUMP --> S3_SEC

    S3_PG --> S3
    S3_FG --> S3
    S3_MM --> S3
    S3_KC --> S3
    S3_SEC --> S3

Backup Flow. Five CronJobs run daily in the k3s cluster. Each targets a specific data category, compresses/encrypts as appropriate, and uploads to a single external S3 bucket organized by directory. A verification job checks freshness and alerts on failure.

Repos

Repo	Platform	Role	Status
pal-e-backup	Forgejo	Terraform + backup scripts + CronJob manifests	planned

Inbox

No untriaged items.

pal-e-backup

Notes

Vision

Projects & Repos Touched

Context

Previous Plan

Depends On

Decisions Made

Phases

Phase 1: Foundation — External S3 + Repo Setup (NOT STARTED)

Phase 2: Database Backups (NOT STARTED)

Phase 3: Forgejo Backup (NOT STARTED)

Phase 4: MinIO Mirror (NOT STARTED)

Phase 5: Identity & Secrets Backup (NOT STARTED)

Phase 6: Monitoring & Verification (NOT STARTED)

Phase 7: Disaster Recovery Test (NOT STARTED)

Key Files

Verification

Epilogue

Related

Pal-E Backup Board

pal-e-backup

Vision

User Stories

Plan

Board

Status

Milestones

Architecture

Repos

Inbox