pal-e-docs
Project not found.

SOP: Secrets Management

sop-secrets-management Sop

sop active security
Project

Overview

Six secrets mechanisms exist across the platform, with two delivery paths: host-side (Salt pillar → tfvars → manual tofu apply) and CI-side (Woodpecker secrets → TF_VAR_* env → automated tofu apply). Both paths must stay in sync for CI-enabled repos. This SOP documents each mechanism, when to use it, and how they connect.

Secrets Layers

Layer Mechanism Where Encryption When to Use
1. Salt Pillar GPG-encrypted YAML (<code>#!yaml|gpg</code>) <code>salt/pillar/secrets/*.sls</code> in pal-e-platform GPG key <code>81A03D1CF874DC90</code> Platform infrastructure secrets (Terraform tfvars values)
2. ~/secrets/ Plaintext <code>.env</code> files <code>~/secrets/{service}/</code> on archbox None (host-level access control only) Local CLI usage (MCP servers, curl, manual operations)
3. Kubernetes Secrets Manual <code>kubectl create secret</code> Per-namespace in k3s cluster At-rest (k3s default encryption) App runtime secrets (DB creds, API keys, session keys)
4. SOPS + Age Age encryption, SOPS decryption at deploy Age keypair in Salt pillar, deployed to ArgoCD namespace Age key <code>age15ct78fr4scv4vxzj3k6q76wshywzlu0mdc64a624e264dst7zfaq6tjzjr</code> Secrets that need to live in Git repos (committed encrypted, decrypted at deploy)
5. Terraform tfvars Plaintext in <code>*.tfvars</code> <code>terraform/k3s.tfvars</code> in pal-e-platform and pal-e-services None (gitignored) Values consumed by <code>tofu apply</code>
6. Woodpecker CI Secrets Encrypted key-value store Woodpecker UI or MCP tools Encrypted at rest by Woodpecker CI pipeline steps that need credentials (git push, registry auth, deploy tokens)

CI Secrets (Woodpecker)

Woodpecker has two levels of secrets:
  • Global secrets — available to all repos. Use for platform-wide credentials.
  • Repo secrets — scoped to a single repo. Use for repo-specific credentials.

Current Global Secrets

Name Events Source (Layer 2) Purpose
<code>forgejo_user</code> push, pull_request <code>~/secrets/pal-e-services/forgejo.env</code> Git operations in CI
<code>forgejo_password</code> push, pull_request <code>~/secrets/pal-e-services/forgejo.env</code> Git operations in CI
<code>forgejo_url</code> push, pull_request <code>~/secrets/pal-e-services/forgejo.env</code> Internal Forgejo URL
<code>forgejo_publish_user</code> all <code>~/secrets/pal-e-services/forgejo.env</code> Package registry publish
<code>forgejo_publish_token</code> all <code>~/secrets/pal-e-services/forgejo.env</code> Package registry publish
<code>forgejo_pypi_url</code> push <code>~/secrets/pal-e-services/forgejo.env</code> PyPI registry URL

Current Repo Secrets

Repo Name Events Purpose
pal-e-platform <code>kubeconfig_content</code> push, pull_request k8s backend + provider auth for tofu plan/apply
pal-e-platform <code>forgejo_token</code> push, pull_request Post tofu plan as PR comment + cross-pillar review issue creation
pal-e-platform <code>tf_var_tailscale_oauth_client_id</code> push, pull_request Tailscale operator OAuth
pal-e-platform <code>tf_var_tailscale_oauth_client_secret</code> push, pull_request Tailscale operator OAuth
pal-e-platform <code>tf_var_grafana_admin_password</code> push, pull_request Grafana admin credential
pal-e-platform <code>tf_var_forgejo_admin_password</code> push, pull_request Forgejo admin credential
pal-e-platform <code>tf_var_woodpecker_forgejo_client</code> push, pull_request Woodpecker OAuth app client ID
pal-e-platform <code>tf_var_woodpecker_forgejo_secret</code> push, pull_request Woodpecker OAuth app secret
pal-e-platform <code>tf_var_harbor_admin_password</code> push, pull_request Harbor admin credential
pal-e-platform <code>tf_var_harbor_secret_key</code> push, pull_request Harbor encryption key
pal-e-platform <code>tf_var_minio_root_password</code> push, pull_request MinIO root credential
pal-e-platform <code>tf_var_woodpecker_api_token</code> push, pull_request Woodpecker API token (for repo activation)
pal-e-platform <code>tf_var_keycloak_admin_password</code> push, pull_request Keycloak admin credential
pal-e-platform <code>tf_var_paledocs_db_password</code> push, pull_request pal-e-docs CNPG database password
pal-e-platform <code>tf_var_slack_webhook_url</code> push, pull_request Alert notifications (Slack)
pal-e-platform <code>tf_var_telegram_bot_token</code> push, pull_request Alert notifications (Telegram)
pal-e-platform <code>tf_var_telegram_chat_id</code> push, pull_request Alert notifications (Telegram)
pal-e-services <code>kubeconfig_content</code> all k8s backend + provider auth for tofu plan/apply
pal-e-services <code>forgejo_token</code> all Post tofu plan as PR comment + cross-pillar review issue creation
pal-e-services <code>tfvars_content</code> all Base64-encoded <code>k3s.tfvars</code> — decoded in CI for tofu plan/apply. <strong>Must be re-encoded and updated whenever k3s.tfvars changes.</strong> See "Updating tfvars_content" procedure below.
pal-e-docs <code>harbor_username</code> all Harbor registry push (Kaniko)
pal-e-docs <code>harbor_password</code> all Harbor registry push (Kaniko)
pal-e-docs <code>forgejo_token</code> push CI commit-back (auto-update deployment image tag)

Adding a CI Secret

  • Check if the value exists in ~/secrets/pal-e-services/ (Layer 2)
  • If not, generate or obtain it and add to the appropriate .env file
  • Add to Woodpecker: mcp__woodpecker__create_repo_secret or create_global_secret
  • Reference in .woodpecker.yaml as from_secret: secret_name
  • Update this SOP with the new secret in the table above

Updating tfvars_content (pal-e-services)

The tfvars_content Woodpecker secret for pal-e-services contains the entire k3s.tfvars file, base64-encoded. Every time you edit ~/secrets/pal-e-services/k3s.tfvars, you must update this secret or CI plan/apply will use stale values.
  • Edit ~/secrets/pal-e-services/k3s.tfvars with the new values
  • Encode: base64 -w0 ~/secrets/pal-e-services/k3s.tfvars
  • Update: mcp__woodpecker__update_repo_secret(repo_full_name="ldraney/pal-e-services", name="tfvars_content", value=<base64 output>)
  • Verify by pushing a branch — CI plan step should decode the secret and produce a valid plan
Why base64? Woodpecker secrets are key-value strings. k3s.tfvars is multi-line HCL with nested blocks — base64 encoding preserves the exact content without escaping issues. The CI pipeline decodes it with echo "$TFVARS_CONTENT" | base64 -d > /tmp/k3s.tfvars.

Keeping Salt and CI in Sync

For CI-enabled repos (currently: pal-e-platform, pal-e-services), the same secret values live in two places: the local tfvars file (for break-glass manual apply) and the Woodpecker encrypted store (for CI plan/apply). These must stay in sync.
pal-e-platform: Individual secrets are stored as separate TF_VAR_* Woodpecker secrets. When adding or rotating a secret, update both the Salt pillar entry AND the Woodpecker repo secret.
pal-e-services: All secrets are bundled into a single tfvars_content secret (base64-encoded k3s.tfvars). When any value in k3s.tfvars changes, re-encode and update the secret. See "Updating tfvars_content" above.
If these diverge, CI apply will use different values than a manual break-glass apply, causing state drift.

Flow: How Secrets Get to Services




          
  
Current Service Secrets



          
  

    

        
          

              Service
              k8s Secrets
              How Created
          

        
      
          

              pal-e-docs
              <code>pal-e-docs-secrets</code> (seed email, password, session key), <code>litestream-creds</code> (MinIO S3), <code>harbor-creds</code> (image pull)
              Manual kubectl + Terraform (harbor-creds)
          

          

              westside-basketball
              <code>rails-env</code> (POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_HOST, SECRET_KEY_BASE), <code>harbor-creds</code> (image pull)
              Terraform (both)
          

          

              Platform (monitoring)
              <code>dora-exporter</code> (Woodpecker + Forgejo creds)
              Terraform
          

      
    

  



          
  
Registry



          
    
Full metadata for every secret (origin, rotation schedule, provider) lives in salt/pillar/secrets_registry.sls. Check there before creating new secrets.



          
  
Procedures



          
    
Decision gate — choose the right path before starting: Does Terraform create a k8s Secret for this value? If YES → follow "Adding a new platform secret" (full 10-step pipeline). If NO → follow "Adding a new app secret (SOPS path)." Both paths share common Steps 1–4 (obtain value → plaintext backup → GPG-encrypt in Salt pillar → registry metadata). Steps 1–4 are the canonical backup layer and are always required regardless of path. Steps 5–10 are only for secrets that Terraform consumes. Adding variables.tf entries or Makefile references for secrets Terraform doesn't use creates dead code and future drift.



          
  
Adding a new platform secret



          
  
    
      
  • Generate or obtain the value
    • 
      
  • Store plaintext in ~/secrets/{service}/{name}.env
    • 
      
  • Encrypt with GPG and add to salt/pillar/secrets/platform.sls
    • 
      
  • Add metadata entry to salt/pillar/secrets_registry.sls
    • 
      
  • Add variable to terraform/variables.tf + reference in terraform/main.tf
    • 
      
  • Add value to terraform/k3s.tfvars (for break-glass manual apply)
    • 
      
  • Add Woodpecker repo secret: mcp__woodpecker__create_repo_secret(repo_id=29, name='tf_var_{name}', value=..., events=['push','pull_request'])
    • 
      
  • Commit to branch, open PR — CI runs tofu plan and posts output as PR comment
    • 
      
  • Merge to main — CI runs tofu apply -auto-approve (merge = deploy per convention-apply-before-merge)
    • 
      
  • Update this SOP with the new secret in both the Repo Secrets table and the registry
    • 



          
  
Adding a new app secret (SOPS path — preferred for new services)



          
  
    
      
  • Common Steps 1–4 (canonical backup layer — always do these first):
    1a. Generate or obtain the value
    1b. Store plaintext in ~/secrets/{service}/{name}.env
    1c. Encrypt with GPG and add to salt/pillar/secrets/platform.sls
    1d. Add metadata entry to salt/pillar/secrets_registry.sls
    • 
      
  • Create a SOPS-encrypted secret YAML in the app repo or kustomize overlay
    • 
      
  • Encrypt with Age public key: sops --encrypt --age age15ct78fr4scv4vxzj3k6q76wshywzlu0mdc64a624e264dst7zfaq6tjzjr
    • 
      
  • Commit encrypted file to repo
    • 
      
  • ArgoCD decrypts at deploy time using Age private key (deployed by Salt to ArgoCD namespace)
    • 
      
  • Commit pillar + registry changes to pal-e-platform (branch → PR → merge)
    • 



          
  
Adding a new app secret (manual path — legacy)



          
  
    
      
  • Store plaintext in ~/secrets/{service}/
    • 
      
  • kubectl create secret generic {name} --from-env-file=~/secrets/{service}/{file}.env -n {namespace}
    • 
      
  • Reference in deployment manifest
    • 



          
  
Open Issues



          
  
    
      
  • tfvars contain plaintext passwords — k3s.tfvars in pal-e-services has passwords in cleartext. These files are gitignored but not encrypted at rest.
    • 
      
  • Manual k8s secrets are not reproducible — if the cluster is rebuilt, manual secrets must be recreated by hand. Migrate to SOPS path.
    • 
      
  • No rotation automation — registry tracks rotation_days but nothing enforces it.
    • 



          
  


          
  
    
      
  • salt/pillar/secrets_registry.sls — full secret metadata registry
    • 
      
  • salt/pillar/secrets/sops.sls — Age keypair for SOPS
    • 
      
  • convention-apply-before-merge — merge = deploy for CI-enabled repos (manual apply is break-glass only)
    • 
      
  • sop-platform-tf-changes — full Terraform change lifecycle including CI plan/apply
    • 
      
  • convention-cross-pillar-triggers — this SOP was updated as a cross-pillar review triggered by Platform Phase 6.3/6.4
    • 
      
  • .woodpecker.yaml in pal-e-platform — CI pipeline with 17 from_secret references
    • 
      
  • .woodpecker/terraform.yaml in pal-e-services — CI pipeline with 3 from_secret references
    •