pal-e-docs

Validation: In-app registration with Keycloak phone field and Lead sync

validation-59-2026-07-17 Doc

validation fail
Project

Verdict: FAIL

Re-validated 2026-07-17. Previous validation also FAIL; root cause has shifted from messaging test failures (now fixed by PR #76) to minitest/mock infrastructure issue and secret placeholder values.

Ticket

ldraney/intelligentstaffingsystems#59 — In-app registration: post-Keycloak flow, phone validation, LeadsController reconciliation. Merged via PR #72 (commit dfe3b8d). Board item #1831 on board-iss.

Environment

Production cluster, namespace intelligentstaffingsystems. URL: https://intelligentstaffingsystems.ai. ArgoCD application: intelligentstaffingsystems (Synced / Degraded). Keycloak realm: iss at https://keycloak.tail5b443a.ts.net/realms/iss.
Two pods in namespace:
  • intelligentstaffingsystems-6d6896dd95-ls9k4 — Running, 1/1, 0 restarts, 11h. Serving traffic on old image.
  • intelligentstaffingsystems-8474c67bc8-nxphp — Init:CrashLoopBackOff, 0/1, 13 restarts, 42m. New ReplicaSet failing to start.

Tiers Executed

Tier 1 (local tests) — blocked, CI test step broken. Tier 3 (production) — executed, image not updated.

Checks

# Criterion How Verified Result Evidence
1 Woodpecker pipeline green for merge commit mcp__woodpecker__get_pipeline_status (pipeline #90, latest push to main) FAIL Pipeline #90: clone OK, database OK, bundle-install OK, lint OK, security OK, <strong>test FAIL</strong> (exit 1), build-and-push SKIPPED. Error: <code>cannot load such file -- minitest/mock (LoadError)</code> from <code>test/test_helper.rb:4</code>. Ruby 3.4 moved minitest/mock out of stdlib into bundled_gems. All pipelines since #19 have failed; no new image has been pushed to Harbor.
2 New image tag propagated to pods <code>kubectl get pods -n intelligentstaffingsystems -o jsonpath='{.items[*].spec.containers[*].image}'</code> FAIL Both pods running image <code>harbor.tail5b443a.ts.net/intelligentstaffingsystems/app:93b9a79b8a88cbb07a7f8fda1b6d85803580cbbb</code>. This corresponds to commit <code>93b9a79</code> ("fix: align pipeline push repo with Argo CD Image Updater annotation"), which is <strong>many commits behind</strong> PR #72's merge at <code>dfe3b8d</code>. No new image was ever built because build-and-push is always skipped.
3 Pod running and ready, restarts = 0 <code>kubectl get pods -n intelligentstaffingsystems</code> + <code>kubectl describe pod</code> FAIL Old pod (ReplicaSet <code>6d6896dd95</code>): Running, 0 restarts — healthy but on pre-PR-#72 code. New pod (ReplicaSet <code>8474c67bc8</code>): <code>Init:CrashLoopBackOff</code>, 13 restarts. Init container <code>migrate</code> fails: <code>ActiveRecord::DatabaseConnectionError: There is an issue connecting with your hostname: PLACEHOLDER_POSTGRES_HOST</code>. The Kubernetes secret <code>intelligentstaffingsystems-secrets</code> contains placeholder values.
4 Smoke test: health endpoint returns 200 <code>curl -s -o /dev/null -w "%{http_code}" https://intelligentstaffingsystems.ai/up</code> PASS (stale) Both <code>/</code> and <code>/up</code> return HTTP 200. However, this is the old deployment serving on pre-PR-#72 code. The registration feature is NOT present.
5 Post-Keycloak registration flow (SessionsController 3-path resolve_lead with phone) Cannot verify — code not deployed FAIL PR #72 added <code>extract_phone</code>, <code>sync_phone</code>, and 3-path <code>resolve_lead</code> (by sub, by email claim, walk-in) to SessionsController. Code is in the repo at commit <code>dfe3b8d</code> but not in the running container image (<code>93b9a79</code>).
6 Phone validation (Lead::PHONE_FORMAT) on Lead model Cannot verify — code not deployed FAIL <code>PHONE_FORMAT = /\A\+?[\d\s\-\(\)\.]{7,20}\z/</code> regex and format validation exist in source but not in running image.
7 Keycloak register.ftl theme with phone field Keycloak theme deployment is separate ticket (platform#541, board #1865) N/A The <code>register.ftl</code> and <code>register.css</code> files are in the repo. Their deployment to Keycloak is tracked by platform#541 (done) and svc#184 (#1864, in validation). This ticket covers only Rails-side wiring.
8 Backfill migration (phone placeholders for existing leads) Cannot verify — migration never ran in production FAIL Migration <code>20260717100000_backfill_lead_phone_placeholders.rb</code> exists in source but the image containing it was never deployed. The new pod's migrate init container crashes before reaching this migration (PLACEHOLDER_POSTGRES_HOST).

Regression Check

The existing site at intelligentstaffingsystems.ai returns HTTP 200 at / and /up. No regression from the merge — the old deployment continues serving. However, ArgoCD health is Degraded due to the new pod's CrashLoopBackOff, indicating a deployment configuration problem independent of the code change.

Root Cause

Two independent blockers prevent PR #72's code from reaching production:
  • CI test runner broken (minitest/mock LoadError): Ruby 3.4's bundled_gems change moved minitest/mock out of the standard library. test/test_helper.rb:4 does require 'minitest/mock' which fails. The original messaging test failures (from the first validation) were fixed by PR #76, but this new infrastructure issue now blocks all pipelines. The test step exits 1, causing build-and-push to be skipped. No new image has been pushed since commit 93b9a79.
  • Kubernetes secret has placeholder values: The new ReplicaSet's init container migrate fails because POSTGRES_HOST in secret intelligentstaffingsystems-secrets is set to PLACEHOLDER_POSTGRES_HOST. Even if a new image were built, it could not deploy until the secret is corrected.

Discovered Issues

  • BLOCKER: minitest/mock LoadError in CI. All Woodpecker pipelines fail at the test step. require 'minitest/mock' is no longer available in Ruby 3.4 without an explicit gem dependency. Fix: add gem "minitest" (with version that includes mock) to Gemfile, or adjust test_helper to not require minitest/mock if it's not needed. This blocks ALL ISS deployments.
  • BLOCKER: PLACEHOLDER_POSTGRES_HOST in prod secrets. The Kubernetes secret intelligentstaffingsystems-secrets has a placeholder for POSTGRES_HOST. Any new pod (even on the same image) will fail to start. The old running pod predates this secret change.
  • ArgoCD Degraded health. Application shows Synced/Degraded. The CrashLoopBackOff pod from the new ReplicaSet is the cause.