pal-e-docs

Phase: Environment Isolation & Secret Boundaries

phase-pal-e-platform-env-isolation Phase

deferred phase
Project

Plan: Environment Isolation & Secret Boundaries

DEFERRED (2026-03-15). Thesis: delayed until we know we really need it or have budget for a remote cloud provider VPS. Rationale from planning discussion:

1. Narrow use case: The primary value is a terraform playground for platform experiments (tofu plan/apply without risking prod). For app development (pal-e-docs, westside-app, basketball-api), the existing CI/CD pipeline (branch → PR → merge → ArgoCD auto-deploy) handles dev→prod cleanly — a dev cluster doesn't improve this flow, it adds a hop.

2. Complexity tax: Two clusters on one box means double the secrets, double the CNPG clusters, double the monitoring, split 64GB RAM. The maintenance burden outweighs the benefit for a solo operator.

3. Prod is stable: Phase 16 (Alert Tuning) proved the platform is maturing — 19 alerts reduced to clean. The risk profile has shifted from 'bad config breaks things' to 'external factors break things.'

4. Wrong problem: A dev cluster on the same box doesn't address the real vulnerability — single node. Power outage, disk failure, or kernel panic takes down both clusters. If capital becomes available, a Hetzner VPS ($14/mo) is better spent on prod redundancy than dev isolation.

Re-activate when: (a) A tofu apply breaks prod badly enough to justify the overhead, (b) budget allows a second node for true physical isolation (Phase 4), or (c) a second operator joins and needs a safe sandbox.

Vision

The internal developer platform for the pal-e AI agency. A developer adds one entry to var.services, pushes code to Forgejo, and gets: a namespace, CI pipeline, container registry project, GitOps deployment, TLS ingress, monitoring, log aggregation, and alerting. The Terraform is the control plane. The platform is the product.
This plan builds the environment isolation story — dev and prod as genuinely separate trust boundaries, not just different namespaces on the same cluster. Each environment gets its own cluster, its own secret encryption key, and progressively stronger isolation: from logical (pillar targeting) to OS-level (user accounts) to physical (separate hosts). A dev who has full access to the dev environment cannot accidentally or maliciously touch prod secrets or workloads.

Projects & Repos Touched

Project/Repo Platform Role in this plan
pal-e-platform Forgejo (private) Salt states for dev k3s, pillar targeting per environment, GPG key management, user account isolation
pal-e-services Forgejo (private) ArgoCD multi-cluster targeting, per-environment SOPS age keys
deployments Forgejo Multi-cluster overlays (prod/ and dev/ per service), per-env SOPS encryption
pal-e-docs (knowledge) Forgejo Environment architecture docs, secret boundary diagrams, developer onboarding for dev cluster

Context

The platform today runs a single k3s cluster. Dev and prod workloads coexist as separate namespaces on the same cluster (e.g., basketball-api and basketball-api-dev). There is no secret isolation — a compromised dev pod on the same cluster could potentially reach prod secrets. There is no environment-level access control — a developer with kubeconfig access sees everything.
The Salt Host Management plan (Phase 2b, in progress) establishes the GPG trust chain with a single key — Option A from the 2026-02-27 architecture discussion. This is the correct starting point: get the encryption infrastructure working before layering on environment isolation. This plan picks up where that leaves off.
The progression is deliberate: each phase adds a stronger isolation boundary while remaining independently deployable. Phase 1 gives you a real dev cluster. Phase 2 adds cryptographic separation. Phase 3 adds OS-level separation. Phase 4 adds physical separation. You can stop at any phase and have a working, improved setup.
What's already done:
  • [x] Single k3s cluster running all workloads (prod + dev namespaces)
  • [x] Salt master + minion operational (Salt plan complete: Phases 1-3)
  • [x] GPG trust chain bootstrapped (Salt plan Phase 2b — COMPLETE)
  • [x] Host firewall active (Salt plan Phase 3 — COMPLETE, pending operator apply)
  • [ ] No dev cluster — dev workloads share prod cluster
  • [ ] No secret isolation between environments
  • [ ] No per-environment SOPS age keys
  • [ ] No multi-cluster ArgoCD targeting
  • [ ] Massive resource headroom available (89% RAM free, 89% CPU free)

Previous Plan

plan-2026-02-26-salt-host-management (COMPLETE) — Phase 2b establishes the single-key GPG trust chain. Phase 3 establishes the host firewall. Phase 4 (k3s lifecycle) was deferred, with k3s version pinning redistributed to this plan's Phase 1.

Depends On

  • plan-2026-02-26-salt-host-management Phase 2b — COMPLETE. GPG trust chain operational.
  • plan-2026-02-26-salt-host-management Phase 3 — COMPLETE. Firewall includes dev cluster CIDRs capability.
All dependencies resolved. This plan is unblocked.

Decisions Made

Decision Rationale
Options A → B → D → C as progressive phases Each phase adds stronger isolation while being independently deployable. You can stop at any phase and have a working setup. Matches the natural progression from solo dev to team to multi-node. Discussed 2026-02-27.
Dev cluster on same box first (not Hetzner) Massive resource headroom (89% free). Avoids capital spend until proven needed. Same Salt master manages both clusters. When Hetzner comes online, the dev cluster migrates — Salt minion re-points, no architecture change.
One Salt master, pillar targeting for environment isolation Salt pillar top.sls naturally targets different data to different minions. Prod minion gets prod secrets, dev minion gets dev secrets. Same master, different views. Simpler than running two masters on one box. Second master is a Phase 4 consideration (separate host).
Separate GPG keys per environment (Phase 2) Even though the same Salt master holds both keys, the encryption is separate. Compromising the dev GPG key cannot decrypt prod pillar. The keys are logically isolated even if physically co-located. True physical isolation comes in Phase 3 (user accounts) and Phase 4 (separate hosts).
Absorbed Salt plan Phase 4 and Kustomize "multi-cluster overlays" seed Environment isolation is a cross-cutting concern that was scattered across multiple plans as footnotes and seeds. Consolidating into one plan gives it proper phasing, dependencies, and ownership. The Salt plan stays focused on host management; this plan owns the environment story.
k3s version pinning absorbed from Salt plan Phase 4 Salt plan Phase 4 was deferred (2026-02-28). k3s version pinning is most useful when a dev cluster exists for canary upgrades. Redistributed to this plan's Phase 1 as a natural fit. See <code>plan-2026-02-26-salt-host-management</code> decision table.

Phases

Phase 1: Dev k3s Cluster + Pillar Targeting + k3s Version Pinning

Slug: phase-2026-02-27-1-dev-cluster-pillar-targeting
Goal: Second k3s instance running on the Arch box for dev workloads. Salt pillar targets different (non-secret) config to each minion. k3s version pinned in pillar for both clusters. ArgoCD on prod cluster sees both clusters. Developers get dev-only kubeconfig.
Owner: Agent (worktree, pal-e-platform repo — Salt states + Terraform for ArgoCD cluster registration)
  • Write Salt state for dev k3s: 
    salt/
  states/
    k3s/
      dev.sls              # second k3s instance: different port, CIDR, data dir
    Configuration:
  • k3s version pinning (redistributed from Salt plan Phase 4):
  • Update firewall pillar to allow dev k3s CIDRs
  • Set up pillar targeting in top.sls: 
    base:
  'archbox':
    - config.prod       # prod cluster config (non-secret)
  'archbox-dev':
    - config.dev        # dev cluster config (non-secret)
  '*':
    - secrets.platform  # shared platform secrets (single GPG key, Option A still)
  • Register dev cluster in ArgoCD (prod cluster):
  • Create dev-specific kubeconfig for developers (read-write on dev cluster, no access to prod)
  • Verify: kubectl --kubeconfig dev.kubeconfig get nodes works. ArgoCD sees both clusters. A test deployment lands on dev cluster. Dev kubeconfig has no access to prod. k3s version matches pillar for both clusters.
Issue: create when phase becomes active

Phase 2: Per-Environment GPG Keys + SOPS Age Keys

Slug: phase-2026-02-27-2-per-env-gpg-sops
Goal: Prod and dev pillar secrets encrypted to different GPG keys. Each environment gets its own age keypair for SOPS. Compromising the dev key cannot decrypt prod secrets.
Owner: Main session (key generation, pillar re-encryption, ArgoCD secret deployment)
  • Generate second GPG keypair: Salt Dev (pal-e-platform) <salt-dev@pal-e.local>
  • Split pillar secrets by environment: 
    salt/
  pillar/
    secrets/
      prod/
        platform.sls    # GPG-encrypted to salt@pal-e.local (prod key)
        sops.sls        # GPG-encrypted: prod age private key
      dev/
        platform.sls    # GPG-encrypted to salt-dev@pal-e.local (dev key)
        sops.sls        # GPG-encrypted: dev age private key
  • Update pillar top.sls targeting: 
    base:
  'archbox':
    - secrets.prod.platform
    - secrets.prod.sops
  'archbox-dev':
    - secrets.dev.platform
    - secrets.dev.sops
  • Generate separate age keypairs per environment:
  • Update deployments repo .sops.yaml with per-directory encryption rules: 
    creation_rules:
  - path_regex: overlays/.*/prod/.*
    age: AGE_PROD_PUBLIC_KEY
  - path_regex: overlays/.*/dev/.*
    age: AGE_DEV_PUBLIC_KEY
  • Dev GPG key can be shared more freely (dev onboarding). Prod GPG key stays locked down.
  • Physical backup of both GPG keys (separate backup locations documented in secret registry)
  • Verify: salt-call --id=archbox pillar.get secrets returns prod secrets. salt-call --id=archbox-dev pillar.get secrets returns dev secrets. Neither can see the other's. SOPS encryption in deployments repo uses correct key per environment.
Issue: create when phase becomes active

Phase 3: OS-Level Isolation (Separate User Accounts)

Slug: phase-2026-02-27-3-os-user-isolation
Goal: Dev cluster operations run under a separate OS user. File permissions prevent the dev user from reading prod GPG keys or prod kubeconfig. Root compromise still exposes both (accepted risk on single host).
Owner: Main session (user creation, permission audit) + Agent (Salt states for user management)
  • Create salt-dev OS user via Salt state:
  • Move dev GPG private key from root/ldraney's keyring to salt-dev's keyring
  • Configure dev k3s to run as salt-dev user (or at minimum, dev Salt operations use salt-dev's GPG keyring)
  • File permission audit:
  • Salt master configuration: evaluate whether the master process needs both GPG keys (it does for serving pillar to both minions) vs having the master delegate decryption per-minion
  • Document the user account model and permission boundaries
  • Verify: sudo -u salt-dev gpg --list-secret-keys shows only dev key. sudo -u salt-dev cat /root/.gnupg/... fails with permission denied.
Note: This phase has a subtlety — the Salt master process needs access to both GPG keys to serve pillar to both minions. The user isolation protects against a compromised dev application or dev user session, not against the Salt master itself. True Salt-level isolation requires Phase 4 (separate master on separate host).
Issue: create when phase becomes active

Phase 4: Physical Isolation (Hetzner Node)

Slug: phase-2026-02-27-4-hetzner-physical-isolation
Goal: Dev cluster runs on a separate Hetzner VPS. Salt minion registers over Tailscale. Physical compromise of the dev host cannot access prod secrets. Dev GPG key lives only on the Hetzner node.
Owner: Main session (Hetzner provisioning, Tailscale enrollment) + Agent (Salt states for remote minion)
  • Provision Hetzner VPS (evaluate cost: CPX21 ~$7/mo for 3 vCPU, 4GB RAM, or CPX31 ~$14/mo for 4 vCPU, 8GB RAM)
  • Install Tailscale on Hetzner node, join tailnet
  • Update Salt master to listen on Tailscale IP (currently 127.0.0.1):
  • Bootstrap Salt minion on Hetzner node:
  • Install dev GPG private key on Hetzner node (transferred securely, then removed from Arch box)
  • Evaluate: does the Salt master still need the dev GPG key? If pillar decryption happens on the master, yes. If we switch to minion-side decryption, the master can serve encrypted blobs and the minion decrypts locally. Research Salt's GPG pillar architecture for this.
  • Install k3s on Hetzner node via Salt highstate
  • Migrate dev workloads from Arch box dev k3s to Hetzner k3s
  • Decommission dev k3s on Arch box, remove salt-dev user
  • Update ArgoCD cluster registration to point at Hetzner dev cluster
  • Update firewall pillar: remove dev CIDRs from Arch box nftables (dev traffic now goes over Tailscale to Hetzner)
  • Verify: dev workloads run on Hetzner. Prod workloads unaffected. Physical access to Hetzner node cannot decrypt prod pillar. salt '*' test.ping shows both minions healthy over Tailscale.
Capital required: ~$7-14/month for Hetzner VPS. Defer until budget allows.
Issue: create when phase becomes active

Key Files

Phase File Repo Change
1 <code>salt/states/k3s/dev.sls</code> pal-e-platform Create — dev k3s instance state
1 <code>salt/states/k3s/init.sls</code> pal-e-platform Update — add version pinning from pillar
1 <code>salt/pillar/top.sls</code> pal-e-platform Update — pillar targeting per minion
1 <code>salt/pillar/config/</code> pal-e-platform Create — per-environment non-secret config (includes k3s version)
1 <code>terraform/main.tf</code> pal-e-platform Update — ArgoCD cluster registration for dev
2 <code>salt/pillar/secrets/prod/</code>, <code>salt/pillar/secrets/dev/</code> pal-e-platform Create — per-environment encrypted pillar
2 <code>.sops.yaml</code> deployments Update — per-directory age key rules
3 <code>salt/states/users/salt-dev.sls</code> pal-e-platform Create — dev user account state
4 <code>salt/master.conf</code> pal-e-platform Update — listen on Tailscale IP
4 <code>salt/pillar/firewall.sls</code> pal-e-platform Update — remove dev CIDRs from Arch box

Verification

  • [ ] Phase 1: Dev k3s running (kubectl --kubeconfig dev.kubeconfig get nodes). ArgoCD sees both clusters. Test deployment lands on dev cluster. Dev kubeconfig has no access to prod. Pillar targeting delivers different config per minion. k3s version matches pillar for both clusters.
  • [ ] Phase 2: Prod and dev pillar encrypted to different GPG keys. salt-call --id=archbox pillar.get and salt-call --id=archbox-dev pillar.get return different secrets. SOPS encryption in deployments repo uses correct age key per environment directory.
  • [ ] Phase 3: salt-dev user exists. Cannot read prod GPG key or prod kubeconfig. Dev operations run under salt-dev context.
  • [ ] Phase 4: Dev cluster on Hetzner. Salt minion healthy over Tailscale. Dev GPG key removed from Arch box. Physical access to Hetzner cannot decrypt prod pillar.

Next Plan Seeds

  • Staging environment — third environment between dev and prod. Same pattern: own cluster, own GPG key, own age key. Canary deployments promote dev → staging → prod.
  • Second Salt master (full HA) — if Hetzner node should be fully independent (not dependent on Arch box master over Tailscale). Each host runs its own master+minion. Pillar synced via git pull. Eliminates single-master SPOF.
  • Automated dev provisioning — Salt reactor watches for new Tailscale nodes, auto-accepts keys, runs highstate. New developer machine joins tailnet and gets a dev environment automatically.
  • Secret rotation per environment — rotate dev secrets more aggressively (weekly) than prod (monthly). Dev GPG key rotation doesn't require prod re-encryption.
  • Client/tenant isolation — if pal-e onboards multiple basketball programs or clients, each may need its own secret boundary. Evaluate per-tenant SOPS keys or per-tenant namespaces with separate age keys.
  • plan-2026-02-26-salt-host-management (COMPLETE) — Phase 2b (GPG trust chain) and Phase 3 (firewall) were dependencies, both resolved. k3s version pinning redistributed from that plan's deferred Phase 4 to this plan's Phase 1.
  • plan-2026-02-26-kustomize-service-bases — Phase 3 (env-aware var.services) aligns with this plan's multi-cluster story. "Multi-cluster overlays" seed is absorbed into this plan's Phase 1-2.
  • plan-2026-02-26-network-security-hardening — Phases 3-4 (NetworkPolicies) benefit from dev cluster for testing. Dev cluster is a safe place to test default-deny policies before applying to prod.
  • plan-2026-02-26-tf-ci-team-hardening — Phase 5 (developer onboarding) needs dev cluster kubeconfig. CI pipeline should target dev cluster for plan-on-PR.
  • plan-2026-02-25-platform-observability — Dev cluster needs its own monitoring stack or a multi-cluster Prometheus federation.