pal-e-docs

Incident: pal-e-streamlit public funnel exposed PII (2026-04-10)

incident-2026-04-10-pal-e-streamlit-public-funnel Doc

security incident postmortem
Project

Incident: pal-e-streamlit public funnel exposed PII

Date: 2026-04-10
Severity: P1 (data exposure — PII for minors' families)
Status: Remediated
Owner: Lucas / Ava (main session)
Exposure window: ~4 hours (ingress creationTimestamp 2026-04-10T17:14:43Z until discovery and mitigation ~21:00Z same day)

Summary

The Streamlit operator dashboard deployed as pal-e-streamlit was served at a public Tailscale funnel (pal-e-streamlit.tail5b443a.ts.net) with zero authentication in the application. For approximately 4 hours, any visitor from the public internet could load the dashboard and see player names, parent emails and phone numbers, monthly fee amounts, jersey sizes and numbers, contract status, and custom notes — PII belonging to minors' families. Discovered by Ava during an architecture conversation about the westside-streamlit project; mitigated within the same session by removing the tailscale.com/funnel annotation from the live ingress, followed by a source-of-truth PR (#109) to prevent drift-back on future applies.

Timeline (UTC)

  • 17:14:43 — Ingress pal-e-streamlit-funnel created in pal-e-streamlit namespace with tailscale.com/funnel: "true" annotation. Deployment bypassed GitOps — no ArgoCD Application, no committed overlay.
  • ~18:00 — Pod pal-e-streamlit-7fc6bb9d66-xzn9g Ready. Streamlit app serving publicly.
  • ~21:00 (approximate, same session) — Ava discovered the exposure while auditing the deployment during a westside-streamlit architecture conversation. Key findings: funnel annotation, no app-level auth (grep -il -e keycloak -e oauth -e auth ~/pal-e-streamlit/*.py returned empty), no OAuth sidecar.
  • ~21:02 — Immediate mitigation applied: kubectl -n pal-e-streamlit annotate ingress pal-e-streamlit-funnel tailscale.com/funnel-. Verified empty. Pod remained 1/1 Running. Tailnet access preserved.
  • ~21:10 — Incident issue #108 filed in pal-e-deployments with full bug template.
  • ~21:15 — Dev agent spawned, PR #109 opened committing overlays/pal-e-streamlit/dev/ingress.yaml (new file, no funnel annotation) to align source-of-truth.
  • ~21:20 — Fresh QA review agent per pr-review-loop: APPROVE, zero blockers, zero nits.
  • ~21:25 — PR #109 merged to main.

Root cause

This was a five-layer failure. Any single layer holding would have prevented the incident. All five failed simultaneously:
  • No in-app authentication. The Streamlit app.py contained raw psycopg → pandas → widget code with no auth guard, no login check, no role gate.
  • Public funnel annotation. The ingress was configured with tailscale.com/funnel: "true" instead of the tailnet-private default. This single annotation is the difference between "trusted operators only" and "whole internet."
  • Overlay never committed to git. The kustomize files lived only on archbox disk. No PR, no code review, no diff anyone else could have seen before the ingress hit the cluster.
  • No ArgoCD Application. Even if the overlay had been committed, no ArgoCD app pointed at it. The service bypassed the entire GitOps pipeline.
  • Manual kubectl apply as the deployment mechanism. Direct apply with no review, no template check, no peer eye on the manifests.
The deep root cause is that the service was bootstrapped without any gate. Each layer that should have caught it was implicitly opt-in, and none were opted into.

Remediation

  • Live cluster: kubectl annotate removed the funnel annotation. ~5 seconds, zero disruption, fully reversible.
  • Source of truth: PR #109 committed overlays/pal-e-streamlit/dev/ingress.yaml without the funnel annotation, merged to main.
  • Verification: Post-merge, live ingress annotation empty, pod 1/1 Running, tailnet access confirmed working, public URL confirmed blocked at the Tailscale edge.

Lessons learned

  • Funnel annotations require auth verification before merge. No ingress with tailscale.com/funnel: "true" should ever land without a documented auth proof (Keycloak OIDC in-app, oauth2-proxy sidecar, or equivalent). Worth a pre-commit or pre-apply hook that greps for the annotation and demands evidence of an auth layer.
  • Every new service in a prod-adjacent namespace must be GitOps-managed from day one. Manual kubectl apply for new services is a footgun — the commit-to-git requirement is the review gate that catches things like rogue funnel annotations.
  • Auth decisions must be made at service bootstrap. "We'll add auth later" is how data leaks. Authentication is a gate that must exist before any traffic hits the service, not a retrofit.
  • Fast-iteration dev patterns must be explicitly distinguished from prod patterns. The hostPath bind mount pattern is powerful for iteration but conflates "my laptop edits are now serving production." A deployment using that pattern must be clearly scoped to tailnet-private and never exposed on a public funnel.
  • "Accidental misname" is a symptom of fast bootstrap without a spec. pal-e-streamlit vs westside-streamlit confusion contributed by obscuring that this was a westside tool with westside data — if the project identity had been locked in pal-e-docs first, the namespace name would have forced a scope review.

Action items

  • [x] Remove funnel annotation from live cluster
  • [x] Commit source-of-truth ingress.yaml (#109 merged)
  • [x] File incident issue #108
  • [x] Write this postmortem
  • [ ] Complete #106 — commit the remaining 4 overlay files (deployment, service, namespace, kustomization)
  • [ ] Create ArgoCD Application for the overlay (after #106)
  • [ ] Add Keycloak OIDC with admin-role gate on westside-basketball realm (same-realm SSO with westside-app)
  • [ ] Rename pal-e-streamlit → westside-streamlit (namespace + hostname + local dir + overlay path)
  • [ ] Rotate and remove hardcoded PGURL fallback in ~/pal-e-streamlit/app.py:8
  • [ ] Delete empty westside-ops namespace (orphan)
  • [ ] Create project-westside-streamlit pal-e-docs project-page and board-westside-streamlit (scope review would have caught this)
  • [ ] Hook proposal: pre-commit or pre-apply check that flags tailscale.com/funnel: "true" without a documented auth layer
  • [ ] Hook proposal: block kubectl apply of new ingress/service resources in prod-adjacent namespaces without an ArgoCD Application pointing at the same overlay
  • [ ] Check Tailscale funnel access logs for the 4-hour exposure window (source IPs, request counts, UA strings) — exposure assessment