pal-e-docs

Phase 8: Network Security Hardening

phase-pal-e-platform-network-security Phase

phase
Project

Phase 8: Network Security Hardening

Vision

Close the platform's biggest remaining security gap: zero network boundaries. Today, any pod can talk to any pod, the host firewall is wide open (INPUT ACCEPT), and the Tailscale ACL grants *:*:*. A compromised or misconfigured pod can reach Forgejo, Harbor, MinIO, and the k8s API. This phase establishes defense-in-depth with three independently deployable layers: pod-level (NetworkPolicy), host-level (nftables), and overlay-level (Tailscale ACL).

Lineage

plan-pal-e-platform → Phase 8

Context

What changed since original scoping (2026-02-27):
  • Platform is now on Forgejo (not GitHub)
  • Kustomize migration COMPLETE (Phase 7) — all 6 services on pal-e-deployments overlays. Adding NetworkPolicy to bases/standard/ now propagates to every service automatically.
  • Salt host management plan COMPLETED — nftables states exist but NOT APPLIED (host INPUT policy is still ACCEPT). Bug bug-nftables-service-running-oneshot was fixed but rules were never applied with revert timer.
  • SOPS CMP sidecar DEPLOYED — secrets are encrypted in Git, decrypted at deploy time.
  • 6 namespaces with services, 7+ platform namespaces — all flat network.
Current security posture (verified 2026-03-15):
  • kubectl get networkpolicies -A → No resources found
  • iptables -S → INPUT ACCEPT (no inbound filtering)
  • Tailscale ACL → *:*:* (assumed, verify in Phase 8b)
  • k8s API (6443) → reachable from LAN

Depends On

  • Phase 7 (Kustomize) — COMPLETED. NetworkPolicy in bases/standard/ now propagates to all 6 services.
  • plan-2026-02-26-salt-host-management — COMPLETED. nftables states exist, just need to be applied.

Decisions Made

Decision Rationale
Use k3s built-in kube-router (no Calico/Cilium swap) kube-router is running (<code>KUBE-ROUTER-INPUT</code> iptables chain exists). Supports L3/L4 NetworkPolicy. Cilium L7 is overkill for current scale.
Default-deny per namespace, explicit allow Zero-trust. Each namespace starts deny-all, we whitelist pod-to-pod traffic.
Service namespace policies in kustomize base, platform policies in Terraform Service default-deny inherits automatically via <code>bases/standard/</code>. Platform policies are infrastructure (Terraform).
Each subphase produces or updates an SOP Security without SOPs is how you lock yourself out at 2am. Every change must have a documented rollback procedure.
3 subphases, independently deployable Each layer (pod, host, tailnet) can be deployed, tested, and rolled back without affecting the others.

Phases

Phase 8a: NetworkPolicy — Pod-Level Isolation (highest value)

Goal: Every namespace has default-deny ingress. Platform namespaces have explicit allow rules. Service namespaces inherit default-deny from kustomize base with allow for funnel, Prometheus, and Promtail.
Owner: Agent (worktree, pal-e-platform + pal-e-deployments repos)
Deliverables:
  • Security assessment doc (COMPLETED) — doc-network-traffic-map published. 25 namespaces, ~70 pods, 19 funnels, 12 cross-namespace flows documented.
  • Default-deny NetworkPolicy in kustomize base (COMPLETED) — PR #7 merged (pal-e-deployments). bases/standard/networkpolicy.yaml added. All 6 overlays patched. QA caught cross-namespace blocker (pal-e-app→pal-e-docs, westside-app→basketball-api) — fixed with overlay-specific ingress rules. Issue #6 closed.
  • Platform namespace NetworkPolicies in Terraform (COMPLETED) — PR #77 merged (pal-e-platform). terraform/network-policies.tf added. 9 policies covering monitoring, forgejo, woodpecker, harbor, minio, keycloak, postgres, ollama, cnpg-system. QA caught 5 missing ingress rules (monitoring→forgejo/woodpecker/minio/cnpg-system, woodpecker→harbor, cnpg-system→woodpecker, tofu-state→minio) — all fixed. Issue #76 closed. ArgoCD namespace deferred (Helm-managed, no kubernetes_namespace_v1).
  • SOP — DEFERRED to Phase 8d. Consolidated into single sop-network-security covering all three layers (NetworkPolicy + Tailscale ACL + Host Firewall).
Verification:
  • [x] kubectl get networkpolicies -A shows policies in 5/6 service namespaces (pal-e-app pending ArgoCD sync cycle)
  • [x] App pod in service namespace cannot reach pal-e-docs from unauthorized namespace (playground, basketball-api both BLOCKED)
  • [x] App pod CAN receive traffic from its funnel proxy (Blackbox probes all UP)
  • [x] Prometheus CAN scrape all ServiceMonitor targets (metrics verified)
  • [x] Promtail CAN collect logs from all namespaces (hostPath, not affected by NetworkPolicy)
  • [x] ArgoCD CAN sync all apps (6/6 Synced + Healthy)
  • [x] All Blackbox Exporter probes still passing (13/13 UP)
  • [ ] Platform namespace policies deployed (deliverable 3)
  • [ ] SOP published and tested (Phase 8d)

Phase 8b: Tailscale ACL Tightening (COMPLETED — PR #79 merged)

Goal: Replace *:*:* ACL with scoped grants. Admin gets full access, k8s node gets funnel access, future developers get limited access.
Owner: Agent (worktree, pal-e-platform repo)
Deliverables:
  • Audit current ACL — DONE. Single *:*:* grant documented.
  • Design scoped ACL — DONE. 4 role-scoped grants: autogroup:admin → *:*, tag:k8s → tag:k8s (inter-node), tag:k8s → autogroup:admin (callbacks), group:developers → tag:k8s:443 (future stub).
  • Apply via Terraform — DONE. PR #79 merged. CI apply-on-merge deploys. Issue #78 closed.
  • SOP: sop-tailscale-acl — DEFERRED to Phase 8d. Consolidated into sop-network-security.
Verification:
  • [x] tailscale_acl no longer grants *:*:*
  • [x] Admin device can still SSH, access ArgoCD, Grafana, Forgejo — VERIFIED (pipeline #49 success, 13/13 probes UP)
  • [x] Funneled services still reachable from internet — VERIFIED (13/13 Blackbox probes UP post-apply)
  • [ ] SOP published — DEFERRED to Phase 8d

Phase 8c: Host Firewall Verification (COMPLETED — PR #81 + manual apply)

Goal: nftables rules applied. Host has default-deny inbound. k8s API (6443) only reachable from localhost + Tailscale. Salt continuously enforces.
Owner: Main session (manual apply with revert timer — too risky for agent)
Deliverables:
  • Investigate nftables gap — DONE. Root cause: boot ordering race. nftables.service started before tailscaled.service created tailscale0. Host ran INPUT ACCEPT for 10 days (March 4-14).
  • Boot ordering fix — DONE. Salt deploys systemd drop-in After=tailscaled.service (PR #81, Issue #80).
  • Apply with revert timer — DONE. sudo nft -f /etc/nftables.conf loaded. 5-minute revert timer set. All services verified. Timer killed.
  • Verify — DONE. k8s API OK. 13/13 Blackbox probes UP. nft INPUT policy DROP. systemctl restart nftables persists rules.
  • Make permanent — DONE. systemctl restart nftables exited status=0. Service enabled. Drop-in After=tailscaled.service confirmed in systemd After chain.
  • SOP: sop-host-firewall — DEFERRED to Phase 8d. Consolidated into sop-network-security.
Verification:
  • [x] nft list ruleset shows rules (not empty)
  • [x] iptables -S no longer shows -P INPUT ACCEPT — nft policy DROP active
  • [x] k3s still healthy — node Ready, 13/13 probes UP
  • [ ] k3s still healthy after 24 hours — PENDING (check 2026-03-15 ~midnight)
  • [ ] Manual rule change reverted by next Salt highstate — PENDING
  • [ ] SOP published — DEFERRED to Phase 8d

Phase 8d: Network Security SOP (COMPLETED — sop-network-security published)

Goal: One comprehensive SOP covering all three network security layers. Written after 8a-8c are complete — when we've actually operated all three layers and know the real gotchas.
Owner: Main session (Betty Sue)
Deliverables:
  • sop-network-security — consolidated SOP. Sections:
  • End-to-end verification — run the full verification checklist from 8a + 8b + 8c after all layers are in place. Document results.
  • Update doc-network-traffic-map — refresh with any new flows discovered during 8b/8c implementation.
Verification:
  • [x] SOP published as sop-network-security in pal-e-docs
  • [ ] SOP tested: intentionally break each layer + rollback using SOP procedures — PENDING (24-hour bake)
  • [ ] Diagnosis flowchart works for a real blocked-traffic scenario — PENDING (next incident)
  • [x] doc-network-traffic-map — no new flows discovered during 8b/8c (all traffic paths already documented)

Key Files

Phase File Repo Change
8a <code>bases/standard/networkpolicy.yaml</code> pal-e-deployments Default-deny + allow funnel/Prometheus/Promtail
8a <code>terraform/network-policies.tf</code> pal-e-platform Platform namespace NetworkPolicies
8b <code>terraform/main.tf</code> (tailscale_acl) pal-e-platform Scoped ACL grants
8c <code>salt/states/nftables/</code> pal-e-platform Verify + apply existing states

SOPs Produced

  • sop-network-security — consolidated SOP (Phase 8d). Covers all three layers: NetworkPolicy (k8s pod-to-pod), Tailscale ACL (tailnet device-to-device), Host Firewall (nftables). Includes diagnosis flowchart, emergency rollback procedures, and common gotchas.
  • doc-network-traffic-map — reference doc (Phase 8a, COMPLETED). Living document of all legitimate traffic flows. SOP references this for "what should be allowed."

Risk Assessment

Subphase Risk Mitigation
8a (NetworkPolicy) Medium — wrong policy can break pod-to-pod traffic Deploy one namespace at a time. Test connectivity before and after. NetworkPolicies are additive (delete to rollback).
8b (Tailscale ACL) Low — Tailscale admin console has ACL history for instant revert Test from admin device before tightening further.
8c (Host firewall) HIGH — wrong rules can lock out SSH, break k8s networking Apply with revert timer. Manual only (no agent). Test from multiple access paths before making permanent.

Verification

  • [ ] 8a: Every namespace has NetworkPolicy. App pods isolated from platform. All services healthy.
  • [ ] 8b: Tailscale ACL scoped. Admin full access. Funnels work. SOP published.
  • [ ] 8c: nftables active. Host hardened. Salt enforces continuously. SOP published.
  • plan-pal-e-platform — parent plan
  • phase-pal-e-platform-kustomize — Phase 7 (COMPLETED). Enables 8a: base NetworkPolicy propagates to all services.
  • plan-2026-02-26-salt-host-management — COMPLETED. nftables states exist but NOT APPLIED. Phase 8c investigates.
  • bug-nftables-service-running-oneshot — fixed but rules may never have been applied.
  • phase-pal-e-platform-env-isolation — Phase 9. Benefits from network boundaries established here.
  • sop-incident-response — rollback procedures reference these SOPs.