pal-e-docs

SOP: Network Security

sop-network-security Sop

sop active
Project

SOP: Network Security

Three-layer defense-in-depth for the pal-e platform. Each layer operates independently — a failure in one doesn't compromise the others. All three were deployed during Phase 8 of plan-pal-e-platform (2026-03-15).

Architecture Overview

Layer Scope Tool Managed By Rollback
1. NetworkPolicy Pod-to-pod (k8s) Kustomize manifests ArgoCD (pal-e-deployments) <code>kubectl delete networkpolicy -n {ns} {name}</code>
2. Tailscale ACL Device-to-device (tailnet) Terraform <code>tailscale_acl</code> CI apply-on-merge (pal-e-platform) Tailscale admin console → ACL history
3. Host Firewall Bare-metal inbound nftables via Salt <code>salt-call state.apply firewall</code> <code>sudo nft flush ruleset</code>

Layer 1: NetworkPolicy (Pod-to-Pod)

Current State

15 of 16 namespaces have default-deny NetworkPolicies. argocd is deferred (Helm-managed, complex internal communication).
Convention: base policy in pal-e-deployments/bases/standard/network-policy.yaml. Platform namespaces in pal-e-platform/terraform/network-policies.tf.

How to Add a Policy for a New Service

  • Service namespace gets default-deny via kustomize base (automatic for services using bases/standard/)
  • If the service needs custom ingress (e.g. specific port from specific namespace), add an overlay in pal-e-deployments/overlays/{service}/prod/network-policy.yaml
  • ArgoCD syncs automatically on merge

How to Modify an Existing Policy

  • Edit the relevant file (base or overlay)
  • PR to pal-e-deployments → QA → merge
  • ArgoCD syncs within 3 minutes
  • Verify: kubectl get networkpolicy -n {ns}

Emergency Rollback

Layer 2: Tailscale ACL (Device-to-Device)

Current State

4 role-scoped grants replace the original *:*:* (PR #79):
  • autogroup:admin → *:* (full access)
  • tag:k8s → tag:k8s on * (inter-node)
  • tag:k8s → autogroup:admin on * (callbacks)
  • group:developers → tag:k8s on 443 (future stub, empty group)
SSH, nodeAttrs (funnel capability), and tagOwners are separately scoped.

How to Modify the ACL

  • Edit terraform/main.tf → tailscale_acl.this resource (grants block)
  • tofu plan -lock=false -var-file=k3s.tfvars to preview
  • PR to pal-e-platform → QA → merge
  • CI apply-on-merge deploys automatically

How to Onboard a Developer

  • Add their Tailscale identity to group:developers in the ACL
  • They get access to tag:k8s on port 443 only (Forgejo + Woodpecker web UIs)
  • For broader access, create a new group or add to autogroup:admin

Emergency Rollback

  • Go to Tailscale admin console → Access Controls
  • Click "History" to see previous ACL versions
  • Revert to a previous version with one click
  • Note: Terraform will show drift on next plan — re-apply from the reverted state or update main.tf to match

Layer 3: Host Firewall (nftables via Salt)

Current State

INPUT policy DROP. Rules loaded from /etc/nftables.conf (Salt-managed). Boot ordering: nftables starts after tailscaled via systemd drop-in (PR #81).
What's allowed inbound:
  • tailscale0 — all traffic (Tailscale overlay)
  • lo — all traffic (localhost)
  • 10.42.0.0/16 — flannel pod CIDR
  • 10.43.0.0/16 — k8s service CIDR
  • 10.0.0.0/24 tcp/22 — SSH from LAN
  • ICMP echo-request
  • Everything else: DROP

How to Check Current Rules

How to Add a Firewall Rule

  • Edit salt/pillar/firewall.sls — add to port_rules, allowed_cidrs, or allowed_interfaces
  • PR to pal-e-platform → QA → merge
  • Apply: sudo salt-call state.apply firewall
  • Rules reload automatically when config changes (Salt cmd.wait watches the config file)

How to Apply with Revert Timer (for risky changes)

Emergency Rollback

Boot Ordering

nftables must start after tailscaled (the tailscale0 interface must exist). Systemd drop-in at /etc/systemd/system/nftables.service.d/after-tailscale.conf ensures this. Deployed via Salt (PR #81).

Diagnosis: "Traffic is Blocked — Which Layer?"

Decision Tree

  • Is the source inside the k8s cluster (pod-to-pod)?
  • Is the source a Tailscale device accessing a tailnet service?
  • Is the source on the LAN accessing the host directly?

Quick Checks

End-to-End Verification Checklist

Run after any change to network security layers:
  • [ ] kubectl get nodes — k8s API accessible
  • [ ] Blackbox probes: kubectl exec -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -- wget -qO- 'http://localhost:9090/api/v1/query?query=probe_success' — all UP
  • [ ] Admin can SSH to host
  • [ ] Admin can access Grafana, Forgejo, ArgoCD via Tailscale
  • [ ] Funneled services reachable from internet (check any https://*.tail5b443a.ts.net)
  • [ ] sudo nft list chain inet filter input — policy drop
  • [ ] LAN device CANNOT reach port 6443 (k8s API) — curl -k https://10.0.0.217:6443 should timeout

Operational Lessons (Phase 8, 2026-03-15)

  • nftables boot ordering race: nft validates interface names at load time. If tailscale0 doesn't exist, service fails silently (enabled but dead). Fix: systemd drop-in After=tailscaled.service.
  • nftables is Type=oneshot: "inactive (dead)" after loading is NORMAL. Rules live in kernel memory, not a daemon. Don't use service.running in Salt — use service.enabled + cmd.wait reload.
  • Tailscale ACL revert is instant: Admin console has full history. Safest layer to experiment with.
  • NetworkPolicy is highest-value: Contains blast radius within the cluster. Most services only need ingress from monitoring (Prometheus scrape) + their own namespace.
  • Revert timer pattern works: nft -f ... && sleep 300 && nft flush ruleset gives 5 minutes to verify before committing. Used successfully during 8c deployment.

Key Files

File Repo Layer
<code>terraform/network-policies.tf</code> pal-e-platform 1 (platform namespaces)
<code>bases/standard/network-policy.yaml</code> pal-e-deployments 1 (service namespaces)
<code>terraform/main.tf</code> (tailscale_acl) pal-e-platform 2
<code>salt/pillar/firewall.sls</code> pal-e-platform 3 (rule definitions)
<code>salt/states/firewall/init.sls</code> pal-e-platform 3 (state + boot ordering)
<code>/etc/nftables.conf</code> Salt-rendered 3 (live config)
  • plan-pal-e-platform — Phase 8 (Network Security Hardening)
  • doc-network-traffic-map — traffic flows between namespaces
  • sop-incident-response — escalation when network issues cause incidents
  • sop-platform-tf-changes — Terraform change workflow (ACL changes)