pal-e-docs

Plan: Salt Host Configuration Management

plan-2026-02-26-salt-host-management Plan

plan completed
Project

Plan: Salt Host Configuration Management

Vision

The internal developer platform for the pal-e AI agency. A developer adds one entry to var.services, pushes code to Forgejo, and gets: a namespace, CI pipeline, container registry project, GitOps deployment, TLS ingress, monitoring, log aggregation, and alerting. The Terraform is the control plane. The platform is the product.
This plan makes the host machine a managed, reproducible, continuously enforced surface — not a snowflake configured from memory. SaltStack manages everything outside the cluster: packages, firewall, kernel modules, systemd services, k3s lifecycle, and secrets. Salt is the root of the platform's secret trust chain and the foundation for multi-node scaling. Terraform manages what's inside the cluster. Salt manages what's under it.

Status: COMPLETE

Phases 1-3 delivered the core value: managed host, encrypted secrets, enforced firewall. Phase 4 (k3s lifecycle + DR) deferred — deliverables redistributed to plans where they have better prerequisites. See Decisions Made for rationale.

Projects & Repos Touched

Project/Repo Platform Role in this plan
pal-e-platform Forgejo (private) New <code>salt/</code> directory alongside <code>terraform/</code>. Makefile at repo root. Bootstrap script.
pal-e-docs (knowledge) Forgejo Host inventory note, firewall SOP, disaster recovery runbook, secret registry documentation

Preconditions

  • Repo is private (2026-02-27): forgejo_admin/pal-e-platform set to private on Forgejo. Required because Salt pillar will contain GPG-encrypted secrets, and infrastructure layout itself is sensitive. forgejo_admin/pal-e-services also private for the same reason.
  • ~/secrets/ inventory: Plaintext secrets in ~/secrets/pal-e-platform/secrets.env and ~/secrets/pal-e-services/secrets.env plus ~/secrets/pal-e-services/forgejo.env. These 3 files migrate to GPG-encrypted Salt pillar in Phase 2b. The rest of ~/secrets/ (30+ service directories) stays as-is — only pal-e-platform and pal-e-services secrets are in scope.

Context

The Arch box host is completely unmanaged. k3s was installed by hand. Packages are installed by hand. The NVIDIA runtime, kernel modules (xpad, uinput), systemd services, and firewall (currently policy ACCEPT — no rules) were all configured ad-hoc during various projects. If the disk dies, we're reconstructing from memory + Terraform (which only covers what's inside the cluster, not the host itself).
Secrets live as plaintext files in ~/secrets/ with no encryption, no audit trail, no rotation tracking. Kubernetes application secrets are created via manual kubectl create secret. This doesn't scale to multiple environments or multiple developers.
Salt becomes two things: the host configuration manager (packages, firewall, services, k3s) and the secret authority for the entire platform (GPG-encrypted pillar stores the age keys that unlock SOPS-encrypted application secrets in Kustomize overlays).
What's already done:
  • [x] k3s running on Arch box (12 cores, 125GB RAM, 1.8TB NVMe, GTX 1070)
  • [x] Current cluster uses ~1.4 cores (11%) and ~12GB RAM (9%) — massive headroom
  • [x] NVIDIA container runtime configured (runtime_class_name = "nvidia")
  • [x] Tailscale installed and operational
  • [x] Platform services all healthy (Forgejo, Woodpecker, Harbor, MinIO, monitoring)
  • [x] Security assessment completed — documented all host-level gaps
  • [x] Repos migrated to Forgejo and set to private (2026-02-27)
  • [x] Salt master + minion running on localhost (Phase 1 complete, PR #1 merged 2026-02-27)
  • [x] Host inventory documented (host-inventory-archbox)
  • [x] Makefile with unified CLI (salt-* and tofu-* targets)
  • [x] Host state codified as Salt states (Phase 2a complete, PR #2 merged 2026-02-27)
  • [x] Secrets migrated to GPG-encrypted pillar (Phase 2b complete, PR #4 merged 2026-02-27)
  • [x] Terraform reads secrets from Salt pillar via make tofu-secrets
  • [x] Age keypair generated and stored in encrypted pillar for SOPS
  • [x] Host firewall: nftables with default deny inbound (Phase 3 complete, PR #6 merged 2026-02-28)

Previous Plan

None — first host management plan. Triggered by architecture discussions (2026-02-26) that identified the host as a blind spot. See insight-devops-materializes-at-team-onboarding.

Depends On

None — this plan is foundational. Other plans depend on it.

Decisions Made

Decision Rationale
SaltStack specifically (not Ansible, not shell scripts) Persistent minion provides continuous enforcement, not one-shot configuration. Event bus enables reactive automation (future: Prometheus alert → Salt action). Master-minion topology scales to multi-node without architecture change. Ansible is push-only with no enforcement loop.
Master + minion on the same Arch box, master-minion topology from day one Single box today, but designed for multi-node. When a Hetzner node comes online, it registers a minion with the master over Tailscale. No architecture change needed. Starting with masterless Salt would mean re-architecting later.
Salt states + pillar live in pal-e-platform repo Host infrastructure and cluster infrastructure are the same concern at different layers. Same repo, different directories, different tools. <code>terraform/</code> for cluster, <code>salt/</code> for host.
Repo is private on Forgejo Salt pillar contains GPG-encrypted secrets. Even encrypted, the infrastructure layout (what's running, how it's configured, network topology) is sensitive. Private repo is the minimum security posture for infrastructure-as-code with embedded secrets.
Bootstrap via bash script + Makefile Salt can't manage its own initial installation (chicken-and-egg). A <code>bootstrap.sh</code> is the one manual step. After that, Salt manages itself. Makefile provides unified entry point for both Salt and Terraform operations.
nftables for firewall (not ufw/iptables) nftables is the modern Linux firewall (iptables successor). Salt has a mature nftables module. Rules defined as pillar data (what to allow) applied by states (enforcement). ufw is a frontend that adds indirection without value when Salt manages rules directly.
Salt pillar with GPG encryption as the secret authority Salt pillar is the root of the trust chain. GPG-encrypted pillar stores: age private keys (for SOPS), Terraform infrastructure secrets (replacing ~/secrets/), k3s tokens, rotation schedules, backup locations. One GPG key (physical backup) unlocks everything.
Phase 2 split into 2a (states) + 2b (secrets) Two different concerns with different risk profiles. 2a is agent-friendly code from host inventory. 2b requires hands-on GPG key generation, physical backup, and Terraform integration changes. Smaller PRs, cleaner blast radius.
GPG key identity: service identity Use <code>Salt Master (pal-e-platform) &lt;salt@pal-e.local&gt;</code>. Role-based, not person-based. Stays with the infrastructure if ownership changes. Keeps personal GPG key (future: git signing, email) separate from infrastructure encryption. Revoking one doesn't affect the other. Standard practice for automation keys.
Terraform integration: secrets.auto.tfvars rendered by Makefile Makefile target calls <code>salt-call pillar.get</code> to decrypt GPG pillar and render <code>terraform/secrets.auto.tfvars</code> (gitignored). The <code>.auto.tfvars</code> suffix is auto-loaded by Terraform — no <code>-var-file</code> flags. Non-sensitive config stays in <code>k3s.tfvars</code>. <code>make tofu-plan</code> depends on the render target. Decrypted file is a cache on disk (same security as ~/secrets/ today — ephemeral deletion adds friction for no security gain since the GPG key is on the same host).
~/secrets/ archived, not deleted After migration is verified, <code>~/secrets/</code> is archived (pushed to private GitHub repo as backup, then directory can be removed at operator's discretion). Never deleted without explicit operator approval. The directory becomes redundant once encrypted pillar is the source of truth.
Dev cluster and environment isolation moved to dedicated plan The former Phase 4 (dev k3s cluster) has been absorbed into <code>plan-2026-02-27-environment-isolation-secret-boundaries</code>. Environment isolation is a cross-cutting concern that deserves its own phased progression (single key → per-env keys → OS isolation → physical isolation). This plan stays focused on host management and the prod trust chain. Decided 2026-02-27.
Only pal-e-platform + pal-e-services secrets in scope <code>~/secrets/</code> contains 30+ service directories beyond pal-e. Only the 3 files for pal-e-platform and pal-e-services migrate to Salt pillar. Everything else stays in <code>~/secrets/</code> as-is. Scope is platform infrastructure secrets, not all application secrets.
GPG algorithm: RSA 4096 Chosen over ed25519 for maximum compatibility with Salt's GPG renderer (python-gnupg 0.5.2). Infrastructure encryption key — speed doesn't matter, reliability does. Decided 2026-02-27.
Phase 2b Steps 8-9 deferred Physical GPG backup (Step 8) and ~/secrets/ archive (Step 9) are manual operator tasks. Code work is complete and merged. These steps remain documented in the plan and can be done at operator's discretion. Does not block Phase 3. Decided 2026-02-27.
Phase 4 deferred — deliverables redistributed Phase 4 had three deliverables: k3s version pinning, DR runbook, and physical GPG backup. Gap analysis (2026-02-28) revealed: (1) k3s version pinning is a guard rail for upgrades, but without a dev cluster for canary testing it's premature — redistributed to environment isolation plan Phase 1. (2) DR runbook would document incomplete recovery since there are no off-host backups — if the NVMe dies, MinIO data (including Litestream backups), TF state (k3s etcd), Forgejo repos, and Harbor images are all lost. Writing the runbook after off-host backups exist (TF CI plan Phase 1: state backups) produces a much more useful document. (3) Physical GPG backup is a standalone manual task tracked as <code>todo-gpg-physical-backup</code>. Closing this plan with Phases 1-3 as the delivered scope. Decided 2026-02-28.

Phases

Phase 1: Host Audit + Salt Bootstrap + Makefile — COMPLETE

Slug: phase-2026-02-26-1-salt-bootstrap
Goal: Full host inventory documented. Salt master + minion running. Makefile provides unified CLI for all platform operations. Salt is operational but not yet enforcing.
Issue: issue-pal-e-platform-salt-bootstrap — resolved
PR: #1 merged (squash, 2026-02-27)
Results: salt-call test.ping True. salt-onedir 3007.13. Services active (disabled). Idempotent re-run verified. make tofu-plan unbroken.

Phase 2a: Codify Host State as Salt States — COMPLETE

Slug: phase-2026-02-27-2a-codify-host-state
Goal: Host is fully described as Salt states. salt-call state.apply test=True shows zero changes (reality matches code). No secrets involved — pure state codification.
Owner: Agent (worktree, pal-e-platform repo — PR with Salt states)
Issue: resolved
PR: #2 merged (squash, 2026-02-27)
Results: Salt states for packages, kernel, services, users, nvidia, k3s, ssh. top.sls assigns all to archbox.

Phase 2b: GPG Encryption + Secret Migration — COMPLETE

Slug: phase-2026-02-27-2b-gpg-secret-migration
Goal: Platform secrets (3 files from ~/secrets/) migrated to GPG-encrypted pillar. Age keypair generated and stored. Secret registry with rotation tracking. Terraform reads secrets from Salt-rendered secrets.auto.tfvars.
Owner: Main session (key generation, physical backup, Terraform integration)
Issue: issue-pal-e-platform-salt-phase-2b-gpg-secrets — resolved
PR: #4 merged (squash, 2026-02-27)
Results:
  • 22 secrets across 4 encrypted pillar files (platform, services, forgejo, sops)
  • Secret registry with rotation tracking and metadata
  • make tofu-secrets renders secrets.auto.tfvars from encrypted pillar
  • k3s.tfvars eliminated — replaced by secrets.auto.tfvars + variable defaults
  • Age keypair generated (public in plaintext, private GPG-encrypted)
  • make tofu-plan = "No changes" with zero warnings
  • Sudo pre-check, Python value escaping, stderr visibility — all hardened during review
Deferred (manual, tracked separately):
  • Step 8: Physical backup of GPG private key. Tracked as todo-gpg-physical-backup.
  • Step 9: Archive ~/secrets/ pal-e files. Private GitHub backup repo (ldraney/secrets) retained indefinitely. Archive at operator's discretion.

Phase 3: Firewall States (nftables) — COMPLETE

Slug: phase-2026-02-26-3-nftables-firewall
Goal: Host has a continuously enforced, code-managed firewall. Default deny inbound. Rules defined as pillar data, applied by Salt states.
Owner: Agent (worktree, pal-e-platform repo — PR with firewall states)
Issue: issue-pal-e-platform-salt-phase-3-nftables — resolved
PR: #6 merged (squash, 2026-02-28)
Results:
  • Pillar-driven firewall: rules defined as structured data in salt/pillar/firewall.sls
  • Jinja template renders valid nftables ruleset from pillar
  • Default deny inbound, accept outbound (Tailscale safe), drop forward
  • Allowed: tailscale0, lo, flannel 10.42.0.0/16, k8s services 10.43.0.0/16, LAN SSH (22/tcp from 10.0.0.0/24)
  • Conntrack (established/related) in both input and forward chains
  • ICMPv6 neighbor discovery rules included (IPv6 safe)
  • Forward chain covers both saddr and daddr for k3s CIDRs
  • Salt enforces: manual nftables changes reverted on next highstate
  • QA reviewed: triple-safe for Tailscale (outbound accept + tailscale0 allow + conntrack)
Operator action required: Apply manually with revert timer. Code is merged but firewall is not yet active on the host. Recommended sequence:
  • git pull forgejo main
  • salt-call state.apply firewall test=True — dry run
  • sudo nft -f /etc/nftables.conf && sleep 120 && sudo nft flush ruleset — apply with 2-min revert timer
  • Verify: Tailscale, k3s, LAN SSH all working
  • If good: salt-call state.apply firewall — permanent apply with service enabled

Phase 4: k3s Lifecycle + Disaster Recovery — DEFERRED

Slug: phase-2026-02-26-4-lifecycle-dr
Status: Deferred (2026-02-28). Deliverables redistributed to plans with better prerequisites.
Gap analysis (2026-02-28):
  • No off-host backups. If the NVMe dies, the following are lost: MinIO data (Litestream backups for pal-e-docs), Terraform state (k3s etcd), Forgejo repos (GitHub mirrors exist for some), Harbor container images. A DR runbook written today would document an incomplete recovery path.
  • No dev cluster for canary. k3s version pinning without a dev cluster to test upgrades first is a guard rail for a road that doesn't exist yet.
  • Current k3s state is verify-only. salt/states/k3s/init.sls checks binary exists + service running. No version awareness. Current version: v1.34.4+k3s1.
Redistribution:
Deliverable Redistributed to Rationale
k3s version pinning (pillar + state) <code>plan-2026-02-27-environment-isolation-secret-boundaries</code> Phase 1 Version pinning becomes meaningful when a dev cluster exists for canary upgrades.
DR runbook (<code>sop-disaster-recovery</code>) <code>plan-2026-02-26-tf-ci-team-hardening</code> (after Phase 1: state backups) Write the runbook after off-host backups exist so it documents a real recovery path, not a gap list.
Physical GPG backup Standalone TODO: <code>todo-gpg-physical-backup</code> Manual operator task. No code dependency. Can be done anytime.

Key Files

Phase File Repo Change
1 ✓ <code>Makefile</code> pal-e-platform Created — unified CLI for Salt + Terraform
1 ✓ <code>salt/bootstrap.sh</code> pal-e-platform Created — one-time Salt installation script
1 ✓ <code>salt/master.conf</code>, <code>salt/minion.conf</code> pal-e-platform Created — Salt configuration
2a ✓ <code>salt/states/**/*.sls</code> pal-e-platform Created — all host state definitions
2b ✓ <code>salt/pillar/secrets/*.sls</code> pal-e-platform Created — GPG-encrypted secrets (replaces ~/secrets/ pal-e files)
2b ✓ <code>salt/pillar/secrets_registry.sls</code> pal-e-platform Created — secret metadata registry with rotation tracking
2b ✓ <code>terraform/secrets.auto.tfvars</code> pal-e-platform Created (gitignored) — rendered from encrypted pillar by Makefile
3 ✓ <code>salt/pillar/firewall.sls</code> pal-e-platform Created — firewall rule definitions as structured pillar data
3 ✓ <code>salt/states/firewall/init.sls</code> pal-e-platform Created — install nftables, render config, enable service
3 ✓ <code>salt/states/firewall/nftables.conf.j2</code> pal-e-platform Created — Jinja template for nftables ruleset

Verification

  • [x] Phase 1: salt-call test.ping returns True. Host inventory documented. make salt-test and make tofu-plan both work.
  • [x] Phase 2a: salt-call state.apply test=True shows 0 changes (host matches states). No secrets involved. PR #2 merged.
  • [x] Phase 2b: GPG keypair generated. GPG renderer configured. 22 secrets migrated to encrypted pillar across 4 files. make tofu-plan = "No changes". Age keypair in pillar. Secret registry complete. PR #4 merged. Steps 8-9 (physical backup, archive) deferred.
  • [x] Phase 3: Pillar-driven nftables firewall. Default deny inbound, accept outbound. Tailscale triple-safe. QA reviewed, PR #6 merged. Operator must apply manually with revert timer.
  • [n/a] Phase 4: Deferred. Deliverables redistributed. See decision table above.

Next Plan Seeds

  • Salt CI pipeline — Woodpecker validates Salt state syntax on PR (salt-call state.show_sls), runs test=True on merge. Integrates with the TF CI pipeline plan.
  • Automated secret rotation — Salt state generates new passwords on schedule, SOPS-encrypts, commits to deployments repo. Coordinated with application restart.
  • Salt monitoring — highstate success/failure as Prometheus metrics. Alert on drift (state.apply shows unexpected changes).
  • Pod Security Standards — enforce restricted security contexts. Orthogonal to network policies but related to host hardening.
  • plan-2026-02-27-environment-isolation-secret-boundaries — absorbed the former Phase 4 (dev cluster) and now also receives k3s version pinning from this plan's deferred Phase 4. That plan depends on this plan's Phase 2b (GPG trust chain, COMPLETE) and Phase 3 (firewall, COMPLETE).
  • plan-2026-02-26-kustomize-service-bases — Phase 2 (SOPS) depends on this plan's Phase 2b (age keypair in pillar, COMPLETE).
  • plan-2026-02-26-network-security-hardening — Phase 2 (host firewall) depends on this plan's Phase 3 (nftables states, COMPLETE).
  • plan-2026-02-26-tf-ci-team-hardening — Salt CI is a future extension of TF CI. DR runbook redistributed to after that plan's Phase 1 (state backups).
  • plan-2026-02-26-tf-modularize-postgres — repo structure now includes salt/ alongside terraform/.
  • plan-2026-02-25-platform-observability — Salt monitoring is a natural extension. Alerting (Phase 3) can detect Salt drift.
  • insight-devops-materializes-at-team-onboarding — the context note explaining why host management surfaced now.
  • tf-architecture-assessment-2026-02-26 — identified host as a blind spot.
  • todo-gpg-physical-backup — standalone manual task extracted from deferred Phase 4.