pal-e-docs

Review: westsidekingsandqueens.com TLS handshake fails

review-1614-2026-06-25 Review

review needs-refinement
Project

Verdict: NEEDS_REFINEMENT

Template Completeness

  • [x] Type -- Bug
  • [x] Lineage -- Related to pal-e-platform#463
  • [x] Repo -- ldraney/pal-e-platform
  • [x] What Broke -- TLS handshake fails, clear error output
  • [x] Repro Steps -- curl commands provided
  • [x] Expected Behavior -- stated clearly
  • [x] Environment -- DNS, ingress, and funnel hostname documented
  • [x] Acceptance Criteria -- 3 criteria, testable
  • [x] Related -- project and original issue referenced
All required bug template sections are present.

Traceability

  • [x] story:WS-S9 label -- "As an admin, I want to track payment status per player so that I know who owes what" (found in project-westside-basketball user-stories section under Admin)
  • [x] story note verified -- found in project-westside-basketball user-stories section
  • [ ] arch:tailscale label -- arch note MISSING -- [SCOPE] No arch-tailscale note exists in pal-e-docs. Search returned zero results.
  • [x] Forgejo issue -- ldraney/pal-e-platform#464, state: open
story:WS-S9 label accuracy: WS-S9 is about payment tracking. This bug is about the public domain's TLS cert for the whole westside site. The story label is a stretch -- this is infrastructure/networking, not payment tracking. A more accurate story would be WS-S1 (superadmin deploy via IaC) or no story (foundational infra bug). Not blocking, but worth noting.

File Targets

The issue does not specify explicit file targets (it is a bug report with symptoms). However, investigation reveals the relevant files:
  • [x] salt/pillar/caddy.sls -- verified: PR #463 added the westside site entry with proxy_target: westside-basketball.tail5b443a.ts.net. This entry exists at HEAD.
  • [x] salt/states/caddy/Caddyfile.j2 -- verified: template generates site blocks from pillar data, pattern is correct.
  • [x] salt/states/caddy/init.sls -- verified: manages Caddyfile rendering and Caddy service reload.
  • [ ] pal-e-deployments/overlays/westside-ror/prod/ -- ISSUE: NO ingress resource exists in the prod overlay. The k3s.tfvars.example comment says "kustomize overlay manages TWO Tailscale Funnel ingresses" but the prod overlay only has kustomization.yaml and deployment-patch.yaml -- zero ingress files. The dev overlay at overlays/westside-ror/dev/ingress.yaml has one funnel ingress, but prod does not.
  • [ ] terraform/modules/networking/main.tf -- verified: this file manages all Tailscale funnels for platform services (grafana, forgejo, woodpecker, etc.) but has NO westside funnel. The westside-ror service has funnel = false in the services map, delegating to kustomize.

Repo Placement

ISSUE: The Forgejo issue is filed on ldraney/pal-e-platform, and the Caddy pillar fix is indeed there. However, the root cause spans TWO repos:
  • ldraney/pal-e-platform -- Caddy pillar + Salt states (proxy config is correct)
  • ldraney/pal-e-deployments -- Missing Tailscale funnel ingress in the prod kustomize overlay
The Caddy proxy targets westside-basketball.tail5b443a.ts.net:443 as the upstream, but no Tailscale funnel ingress serves that hostname. The fix likely needs a prod ingress resource in pal-e-deployments (following the dev overlay's pattern) AND possibly verifying the Caddy highstate was applied on the edge server. A single pal-e-platform issue may be sufficient if the scope documents both repos, but currently it does not mention pal-e-deployments at all.

Dependencies

  • Board item #1613 (pal-e-services#148) -- Harbor pull creds bug. Also sprint:5, also type:bug. This bug prevents new pods from deploying (ImagePullBackOff). Even if the TLS/funnel fix lands, the westside-ror pod may be running stale code until #1613 is resolved. These two bugs are co-dependent blockers for sprint 4/5 auth changes going live.
  • PR #463 (merged) -- Added the Caddy pillar entry. The pillar is correct but may not have been applied via Salt highstate on the edge server yet.
  • Salt highstate -- Not a board item, but the Caddy config won't take effect until salt '*' state.highstate runs on the edge-proxy node.

Acceptance Criteria

The 3 criteria are clear and testable via curl commands. However:
  • AC #1 ("loads with valid TLS cert") is verifiable: curl -sv https://westsidekingsandqueens.com 2>&1 | grep "SSL certificate verify ok"
  • AC #2 ("response matches tailnet URL content") is verifiable: diff the response bodies
  • AC #3 ("no regression in tailnet URL") is verifiable: curl https://westside-ror.tail5b443a.ts.net
  • Missing AC: www redirect should be tested: curl -sI https://www.westsidekingsandqueens.com should return 301 to apex domain (Caddy pillar has www_redirect: true)

Blast Radius

Two other domains use the same Caddy edge-proxy pattern:
  • palinks.app -- proxy_target: palinks.tail5b443a.ts.net
  • landscaping-assistant.app -- proxy_target: landscaping-assistant.tail5b443a.ts.net
Both of these have funnel = true in the services map (Terraform manages their funnel ingress automatically). The westside-ror case is unique because funnel = false delegates to kustomize, and the kustomize overlay is incomplete. No blast radius beyond westside.

Decomposition Assessment

2 file targets across 2 repos (Caddy pillar verification in pal-e-platform, ingress addition in pal-e-deployments). 3 acceptance criteria. Estimated agent work under 5 minutes. No decomposition needed.

Root Cause Analysis (Reviewer Addition)

The issue body describes the symptom correctly but the root cause analysis is misleading. It says "Tailscale funnel not serving public domain cert" -- but the architecture does NOT use Tailscale funnel for public domain cert provisioning. The architecture is:
  • DNS A record points westsidekingsandqueens.com to Hetzner edge-proxy (178.156.129.142)
  • Caddy on edge-proxy terminates TLS via Let's Encrypt ACME
  • Caddy reverse-proxies to westside-basketball.tail5b443a.ts.net:443 via Tailscale mesh
  • That hostname requires a Tailscale funnel ingress in the k8s cluster
The failure chain: Caddy cannot reach the upstream because no Tailscale funnel ingress exists for hostname westside-basketball in the westside-ror namespace. The westsidekingsandqueens-funnel ingress mentioned in the issue appears to be manually created and serves the wrong hostname (westsidekingsandqueens.tail5b443a.ts.net instead of westside-basketball.tail5b443a.ts.net).

Recommendation

  • [BODY] Update issue body to clarify root cause: the fix is adding a Tailscale funnel ingress to pal-e-deployments/overlays/westside-ror/prod/ (following the dev overlay pattern at overlays/westside-ror/dev/ingress.yaml), and verifying Salt highstate was applied on the edge-proxy for the Caddy config.
  • [BODY] Add file targets section listing: (1) pal-e-deployments/overlays/westside-ror/prod/ingress.yaml (new file, funnel ingress), (2) pal-e-deployments/overlays/westside-ror/prod/kustomization.yaml (add ingress to resources), (3) verify Salt highstate on edge-proxy.
  • [BODY] Add missing acceptance criterion: https://www.westsidekingsandqueens.com should 301 redirect to apex domain.
  • [BODY] Document the pal-e-deployments repo involvement -- currently only pal-e-platform is mentioned.
  • [SCOPE] Create architecture note arch-tailscale for the Tailscale networking component (funnels, ACLs, subnet router, edge proxy pattern).
  • [LABEL] Consider changing story label from story:WS-S9 to story:WS-S1 (superadmin IaC/deploy) -- this is infra, not payment tracking.