pal-e-docs

Plan: Off-Site Platform Backup

plan-pal-e-backup Plan

plan deprecated
Project

Vision

If the PC dies, everything can be rebuilt from cloud backups. One unified off-site destination, one IaC-managed pipeline, zero manual steps. Every piece of irreplaceable platform state — databases, git repos, object storage, identity, secrets — backed up daily to external S3 with automated verification and alerting.

Projects & Repos Touched

  • pal-e-backup — new repo, terraform + backup scripts + k8s CronJob manifests
  • pal-e-platform — may need minor changes for RBAC, service accounts

Context

The platform runs on a single k3s node (Arch Linux, 12-core, 128GB RAM, 1.8TB NVMe, GTX 1070). All persistent state lives on local-path PVCs on that one disk. CNPG databases (pal-e-docs, woodpecker) have WAL archiving to MinIO, but MinIO is on the same disk. Terraform state backs up to MinIO daily. Everything else — Forgejo git repos, Keycloak identity, Harbor metadata, basketball-api DB, mcd-tracker DB — has zero backup. A single disk failure loses the entire platform.

Previous Plan

None. First plan for this project.

Depends On

  • External S3 account (Backblaze B2 recommended, Hetzner Object Storage as alternative)
  • k8s RBAC for CronJobs to access database pods and secrets

Decisions Made

  • Daily pg_dump over CNPG migration — basketball-api and mcd-tracker stay as plain Postgres pods. pg_dump gives backup protection without the complexity of CNPG migration. CNPG migration is a separate initiative.
  • Backblaze B2 recommended — $6/TB, S3-compatible, proven. Hetzner Object Storage is the alternative if staying in Hetzner ecosystem for future two-node architecture.
  • Unified bucket structure — one bucket, directory-per-service. Not one bucket per service.
  • Daily frequency for cloud backups — CNPG does continuous WAL locally. Cloud copy is disaster insurance, daily is sufficient.

Phases

Phase 1: Foundation — External S3 + Repo Setup (NOT STARTED)

Create the pal-e-backup Forgejo repo. Provision external S3 bucket with terraform. Store cloud credentials as k8s secrets. Verify S3 connectivity from within the cluster. Establish the bucket directory structure.
Deliverables: Repo exists. Terraform provisions S3 bucket. mc can write to external S3 from a k8s pod. Bucket structure: s3://pal-e-backups/{postgres,forgejo,keycloak,minio-mirror,k8s-secrets}/

Phase 2: Database Backups (NOT STARTED)

CronJobs that pg_dump each database to external S3 daily. Four databases: pal-e-docs (CNPG, postgres namespace), woodpecker (CNPG, woodpecker namespace), basketball-api (plain pod, basketball-api namespace), mcd-tracker (plain pod, mcd-tracker namespace). Each dump is gzipped, timestamped, uploaded to s3://pal-e-backups/postgres/{db-name}/YYYY-MM-DD.sql.gz. 30-day retention with automated pruning.
Deliverables: 4 CronJobs running daily. Recent dumps visible in S3. Old dumps auto-pruned at 30 days.

Phase 3: Forgejo Backup (NOT STARTED)

CronJob that runs gitea dump inside the Forgejo pod (or a sidecar with access to the Forgejo PVC). This captures git repositories, the SQLite database (issues, PRs, users, settings), configuration, and avatars in a single zip. Upload to s3://pal-e-backups/forgejo/gitea-dump-YYYY-MM-DD.zip. 30-day retention.
Deliverables: Daily Forgejo dump in S3. Full restore tested — git repos, issues, and user data recoverable.

Phase 4: MinIO Mirror (NOT STARTED)

CronJob that runs mc mirror from local MinIO to external S3. This replicates WAL archives (CNPG PITR capability), terraform state snapshots, and assets to the cloud. Destination: s3://pal-e-backups/minio-mirror/. Mirror is incremental — only new/changed objects transfer.
Deliverables: MinIO contents mirrored to external S3. CNPG PITR possible from cloud-stored WAL archives.

Phase 5: Identity & Secrets Backup (NOT STARTED)

Two CronJobs: (1) Keycloak realm export — uses Keycloak admin CLI to export all realms, users, clients, roles to JSON. Upload to s3://pal-e-backups/keycloak/. (2) k8s secrets export — dumps all secrets from critical namespaces (encrypted with age or GPG before upload) to s3://pal-e-backups/k8s-secrets/. 30-day retention for both.
Deliverables: Keycloak realm export in S3. Encrypted k8s secrets in S3. Identity can be rebuilt from exports.

Phase 6: Monitoring & Verification (NOT STARTED)

Verification CronJob that checks all backup types exist in S3 and are less than 25 hours old. Prometheus metrics for backup age per category. AlertManager rules that fire if any backup is stale. Fix existing cnpg-backup-verify CronJob (currently failing). Route alerts to existing Slack/Telegram pipeline.
Deliverables: Prometheus dashboard showing backup freshness. Alert fires within 1 hour if any daily backup is missing. Zero silent failures.

Phase 7: Disaster Recovery Test (NOT STARTED)

Full restore test. Provision a temporary environment (could be a Hetzner VPS or local VM). Restore every component from cloud backups only — no access to original data. Verify: pal-e-docs serves notes, Forgejo has repos, Keycloak authenticates users, basketball-api has player data. Document the restore procedure as an SOP. Time the full restore.
Deliverables: SOP: full platform restore from cloud backups. Tested end-to-end. Restore time documented.

Key Files

TBD — repo doesn't exist yet. Expected structure:

Verification

  • All 6 backup categories present in external S3 with daily freshness
  • Prometheus alerts fire within 1 hour of a missed backup
  • Full restore tested and documented
  • Restore time under 2 hours for complete platform

Epilogue

Empty — plan has not started.
  • sop-postgres-restore — existing CNPG restore SOP (covers local MinIO only, needs cloud extension)
  • plan-pal-e-platform — platform hardening (future: two-node architecture builds on backup destination)
  • deployment-lessons — operational lessons including backup failures