pal-e-docs

BUG: kube-router ipset population broken — NetworkPolicies block all traffic

bug-kube-router-ipset-empty Doc

bug resolved
Project

BUG: kube-router ipset population broken — NetworkPolicies block all traffic

Problem

NetworkPolicies deployed in Phase 8 (2026-03-15, PR #77) create iptables chains (KUBE-NWPLCY-*) and ipsets (KUBE-SRC-*/KUBE-DST-*), but ipsets are never populated with pod IPs. All allow rules match against empty sets, so everything falls through to DROP. 340 iptables rules exist, zero ipset members. Re-creating policies does NOT fix it — new ipsets are also empty.
Error from Woodpecker kaniko build:

Root Cause

k3s v1.34.4 embeds kube-router for NetworkPolicy enforcement. The kube-router component that watches pod events and populates ipsets with pod IPs is not functioning. The iptables structure is correct but the data (pod IPs) is missing. Health probes pass because kubelet traffic comes from the host network and bypasses NetworkPolicy.

Fix

  • Immediate (done): Deleted all NetworkPolicies across all namespaces per sop-network-security emergency rollback. Platform still protected by Tailscale ACLs (Layer 2) + nftables (Layer 3).
  • Proper fix needed: Investigate k3s kube-router state. Options: k3s patch upgrade, switch to Cilium/Calico CNI, or find kube-router restart mechanism within k3s.
  • ArgoCD: App-level policies (pal-e-deployments) will re-apply on sync — need to either fix root cause first or remove policies from kustomize bases temporarily.

Impact

  • All CI pipelines failed for 44h (Woodpecker kaniko → Harbor blocked)
  • Cross-namespace pod communication broken cluster-wide
  • Prometheus scraping of services likely degraded
  • Layer 1 (NetworkPolicy) of three-layer defense is offline until fix

Acceptance Criteria

  • NetworkPolicies re-applied with populated ipsets (verify: sudo ipset list KUBE-SRC-* | grep "^10\.")
  • Woodpecker CI pipeline passes end-to-end (clone → test → build-push → smoke-test)
  • Pod-to-pod traffic works across allowed namespaces with policies active
  • All three security layers operational simultaneously
  • project-pal-e-platform — affected project
  • sop-network-security — three-layer architecture, emergency rollback used
  • plan-pal-e-platform Phase 8 — where NetworkPolicies were deployed
  • pal-e-platform PR #77 — terraform that created the policies