pal-e-docs

Review: Stripe webhooks unreachable — basketball-api needs Tailscale Funnel

review-836-2026-04-04 Doc

review needs-refinement
Project

Verdict: NEEDS_REFINEMENT

Template Completeness

  • [x] Type — Bug
  • [x] Lineage — references #340
  • [ ] Repo — listed but INCORRECT (see Repo Placement below)
  • [x] What Broke — detailed impact description with dollar amounts
  • [x] Repro Steps — clear 4-step reproduction
  • [x] Expected Behavior — clear
  • [x] Environment — detailed with webhook ID, namespace, funnel status
  • [x] Acceptance Criteria — 4 criteria listed (but AC1 is already satisfied, AC4 is a data fix not infra)
  • [x] Related — references project page and related issues
  • [ ] File Targets — listed but WRONG (see File Targets below)

Traceability

  • [x] story:WS-S18 label — "As a parent, I want to receive a branded email with jersey ordering link so that I can order without bringing cash" — verified in project-westside-basketball user-stories section
  • [x] story note verified — found in project-westside-basketball user-stories section under Parent stories
  • [x] arch:basketball-api label — present on board item
  • [ ] arch note MISSING — no arch-basketball-api note found in pal-e-docs. [SCOPE] Create architecture note arch-basketball-api for component basketball-api.
  • [x] Forgejo issue — forgejo_admin/basketball-api#343, open

File Targets

  • [ ] terraform/modules/networking/main.tf (pal-e-platform) — ISSUE: This file contains platform-level funnels (Grafana, Forgejo, Woodpecker, Harbor, etc.). Service-level funnels are managed in pal-e-services/terraform/services.tf via the var.services map with a funnel boolean. basketball-api already has funnel = true in pal-e-services/terraform/k3s.tfvars (line 182). The funnel ingress already exists in the cluster: kubectl get ingress -n basketball-api shows basketball-api-funnel with address basketball-api.tail5b443a.ts.net on port 443, created 40 days ago. The tailscale.com/funnel: "true" annotation is present.
  • [x] basketball-api/src/basketball_api/routes/webhooks.py — verified: @router.post("/stripe") endpoint exists at line 263. Handler validates stripe-signature header and constructs events correctly.

Repo Placement

ISSUE: The ticket says the fix is in ldraney/pal-e-platform (infrastructure) but the Forgejo issue is filed on forgejo_admin/basketball-api. The actual funnel configuration lives in pal-e-services/terraform/services.tf (line 177), not pal-e-platform. Moreover, the funnel is already configured and deployed — the k8s ingress exists with the correct annotation and has an assigned hostname for 40 days. The issue's core assumption ("basketball-api needs Tailscale Funnel") is factually incorrect — the funnel exists. The real bug requires investigation into WHY the existing funnel isn't serving public traffic (Tailscale operator behavior, ACL permissions, or a Funnel vs Ingress distinction). The Forgejo issue should be filed on the repo where the fix will land — likely pal-e-platform or pal-e-services, not basketball-api.

Dependencies

  • [x] Board item #823 (in_progress) — forgejo_admin/basketball-api#340 "Jersey checkout load failed" — this is the original symptom that led to #343's discovery. Related but independent.
  • [x] Board item #719 (qa) — "Stripe webhook not syncing payment to order status" — directly related. If the funnel isn't serving public traffic, this QA item cannot pass either.
  • [x] Board item #720 (backlog) — "Legacy jersey webhook handler missing payment_status guard" — downstream bug that matters once webhooks are reachable.
  • [ ] Tailscale operator version (1.94.2 per pal-e-platform networking module) — may need investigation for Funnel behavior. Status: unknown.

Acceptance Criteria

  • AC1 "Tailscale Funnel configured for basketball-api.tail5b443a.ts.net" — already satisfied. The ingress exists, annotation is set, hostname is assigned. This AC needs rewriting to reflect the actual problem: "Public internet traffic reaches basketball-api.tail5b443a.ts.net (verified via curl from off-tailnet host)."
  • AC2 "Stripe webhook test event delivered successfully to /webhooks/stripe" — testable via Stripe dashboard "Send test webhook." Good AC.
  • AC3 "New jersey payment triggers checkout.session.completed with pending_webhooks=0" — testable but requires a real payment or Stripe CLI. Good AC.
  • AC4 "Verify no Yussuf Duro (id=116) stuck in pending/opt_out state gets fixed by replay" — this is a data remediation task, not an infrastructure fix. Should be split into a separate ticket.

Blast Radius

If the funnel truly isn't routing public internet traffic, ALL external webhook providers (not just Stripe) would be affected. However, no other services currently receive external webhooks through basketball-api. The westsidekingsandqueens funnel works (frontend is publicly accessible), but it has funnel = false in tfvars — meaning its funnel is managed by kustomize overlay, not Terraform. This is a different codepath and may explain why basketball-api's Terraform-managed funnel behaves differently. Rollback is low-risk since the current state is already broken.

Decomposition Assessment

Apply the three-thing limit and five-minute rule:
  • The ticket has 2 discrete changes (infra investigation/fix + data replay for Yussuf Duro). The data replay (AC4) should be a separate ticket.
  • The infra investigation may exceed 5 minutes if the root cause is non-obvious (Tailscale operator logs, ACL debugging, funnel vs ingress semantics). However, this is a single-focus investigation — once root cause is found, the fix is likely a 1-file change.
  • No independent subtasks to parallelize — this is sequential (investigate, then fix).
  • No decomposition needed once AC4 is split out. The remaining scope (investigate + fix funnel reachability) fits a single agent pass.

Recommendation

  • [BODY] Rewrite "What Broke" section: the funnel IS configured (ingress exists with tailscale.com/funnel: "true" annotation and assigned hostname basketball-api.tail5b443a.ts.net). The real problem is that despite the funnel ingress existing for 40 days, Stripe cannot reach the endpoint from the public internet. Investigation should focus on Tailscale Funnel operator behavior, whether the operator is actually advertising the route publicly, and whether the nodeAttrs funnel permission in pal-e-platform networking ACL applies correctly to the operator-managed proxy.
  • [BODY] Fix file targets: remove terraform/modules/networking/main.tf (pal-e-platform). Add pal-e-services/terraform/services.tf (line 177, service_funnel resource) and pal-e-services/terraform/k3s.tfvars (line 178, basketball-api service config with funnel=true). Add Tailscale operator logs as an investigation target.
  • [BODY] Fix repo: the Forgejo issue is filed on basketball-api but the fix will land in pal-e-platform or pal-e-services. Consider moving the issue or filing a new one on the correct repo.
  • [BODY] Rewrite AC1: replace "Tailscale Funnel configured" (already true) with "Public internet traffic reaches basketball-api.tail5b443a.ts.net (verified via curl from off-tailnet host)".
  • [BODY] Split AC4 (Yussuf Duro id=116 data fix) into a separate ticket — it is a data remediation task, not an infrastructure fix.
  • [SCOPE] Create architecture note arch-basketball-api for component basketball-api.