pal-e-docs

Validation: Add PSA-restricted securityContext to westside-admin overlay (#137)

validation-137-2026-05-01 Doc

validation pass
Project

Verdict: PASS

This ticket's scope is fully delivered. The PSA-restricted securityContext patch is rendered correctly in cluster, the new ReplicaSet is admitted with zero PSA violations, and the wrong-container-name failure mode (caught in the third PR review) is not present. M1 (first admin login) is not reachable yet, but the remaining blocker is a separate, pre-existing infra gap (placeholder Harbor pull credentials) — it is not a regression from this change. See Discovered Issues for the new ticket needed.

Ticket

forgejo_admin/pal-e-deployments#137 — Add PSA-restricted securityContext to overlays/westside-admin/prod/deployment-patch.yaml so the deployment is admitted under pod-security.kubernetes.io/enforce: restricted.
Board item: 1118 on board-westside-admin (column qa).
Merged PR: pal-e-deployments#141 — squash-merged 2026-05-01, merge commit 2e10af964e473d342366ac1a76454613bb04dfc4.

Environment

Cluster: archbox (k3s, prod).
Namespace: westside-admin.
ArgoCD app: westside-admin (sync revision 2e10af96, status Synced, health Progressing).
Image: harbor.tail5b443a.ts.net/westside-admin/app:803e9439806cb644cc824fa8fbd87cc5d0f567e2.
External funnel: https://westside-admin.tail5b443a.ts.net (Tailscale).
Active ReplicaSet (post-merge): westside-admin-8786bffd6.
Validation run: 2026-05-01.

Checks

# Criterion How Verified Result Evidence
1 ArgoCD app has synced to a revision newer than the pre-merge head <code>576eacac</code> <code>kubectl get application westside-admin -n argocd -o jsonpath='{.status.sync.revision}'</code> PASS Returned <code>2e10af964e473d342366ac1a76454613bb04dfc4</code>, the merge commit of PR #141. <code>operationState.message: successfully synced (all tasks run)</code> at <code>2026-05-01T14:36:05Z</code>.
2 Rendered Deployment in cluster carries all 4 PSA-required fields at the pod level (<code>runAsNonRoot</code>, <code>runAsUser</code>, <code>runAsGroup</code>, <code>fsGroup</code>) plus <code>seccompProfile.type=RuntimeDefault</code> <code>kubectl get deployment westside-admin -n westside-admin -o json | jq '.spec.template.spec.securityContext'</code> PASS <pre>{ "fsGroup": 1000, "runAsGroup": 1000, "runAsNonRoot": true, "runAsUser": 1000, "seccompProfile": { "type": "RuntimeDefault" } }</pre>
3 Container-level securityContext sets all 4 PSA-restricted fields (<code>allowPrivilegeEscalation=false</code>, <code>capabilities.drop=[ALL]</code>, <code>runAsNonRoot=true</code>, <code>seccompProfile.type=RuntimeDefault</code>) <code>kubectl get deployment ... | jq '.spec.template.spec.containers[0].securityContext'</code> PASS <pre>{ "allowPrivilegeEscalation": false, "capabilities": { "drop": ["ALL"] }, "runAsNonRoot": true, "seccompProfile": { "type": "RuntimeDefault" } }</pre>
4 Container count is exactly 1, named <code>westside-admin</code> (the wrong-name failure mode caught in PR review #3 — patch referenced <code>app</code> while base used <code>westside-admin</code>) <code>kubectl get deployment westside-admin -n westside-admin -o jsonpath='{.spec.template.spec.containers[*].name}'</code> PASS Returned <code>westside-admin</code> (single container). The patch's strategic merge correctly targeted the named container in the base — securityContext is on the right container, not on a phantom <code>app</code> sidecar.
5 Post-merge ReplicaSet has zero <code>FailedCreate</code> events from PSA admission <code>kubectl describe rs westside-admin-8786bffd6 -n westside-admin</code> PASS Events on the new RS contain only <code>SuccessfulCreate</code>. All historical <code>FailedCreate</code> events from PSA (on stale RS <code>566d4cc96d</code>, <code>bcb595854</code>, <code>6fb5db9c6</code>) predate the merge — the new RS is admitted cleanly. PSA is no longer the failure mode.
6 Deployment reaches <code>1/1 ready</code>, <code>Available=True</code>, <code>/health</code> returns 200 <code>kubectl get deployment</code>; <code>curl https://westside-admin.tail5b443a.ts.net/health</code> FAIL (out-of-scope) Deployment is <code>0/1</code>, funnel returns <code>HTTP 502</code>. Pod is stuck in <code>Waiting</code> with reason <code>illegal base64 data at input byte 11</code> — kubelet can't decode the <code>auth</code> field in the <code>harbor-creds</code> dockerconfigjson because it's the literal placeholder <code>PLACEHOLDER_REPLACE_AFTER_PAL_E_SERVICES_HARBOR_PROJECT_PROVISIONED</code> (byte 11 = <code>_</code>, not a valid base64 char). This blocker is <strong>pre-existing</strong> and is not caused by PR #141 — it would block the pod regardless of the PSA fix. Tracked as new discovered issue (see below). <strong>Not counted against this ticket's verdict</strong> — PR #137 was scoped to the PSA admission gate, which is now passed.

Why Verdict is PASS Despite Pod Not Ready

The acceptance criteria of issue #137 are: "the deployment manifest carries the four PSA-required securityContext fields and the namespace's restricted enforcement no longer blocks pod creation." Both are met:
  • The four required fields are present at both pod and container level (checks 2 and 3).
  • The new ReplicaSet's pods are admitted by PodSecurity (check 5 — zero FailedCreate from PSA).
Pod liveness, image pull, and runtime startup are downstream of admission. The pod-startup failure is governed by a separate prerequisite (Harbor robot credentials) that pal-e-services#64 was supposed to wire up but didn't replace the placeholder values in cluster. PR #141 cannot make a pod start that has unusable image-pull creds — and it doesn't claim to. The PSA gate is closed; the next gate (Harbor auth) is open and is now visible because PSA stopped masking it.

Regression Check

  • ArgoCD sync: app remains Synced after PR; operationState.message: successfully synced (all tasks run). No drift introduced.
  • Other pal-e-deployments apps: PR was scoped to overlays/westside-admin/prod/deployment-patch.yaml only. No other app is affected.
  • kustomize build: ArgoCD's server-side render of the overlay produced a valid Deployment spec with the patched fields landing on the correct container — confirms strategic merge by container name worked.
  • Pre-existing PSA failures elsewhere: spike #139 (westside-streamlit) and audit #140 (system-wide) remain open and are unaffected.

Discovered Issues

(NEW — needs Forgejo issue + board item per convention-discovered-scope)
D-1: harbor-creds placeholder injection — westside-admin namespace's harbor-creds Secret still contains the literal placeholder string from pal-e-deployments bootstrap.
  • Symptom: kubelet rejects pod creation with illegal base64 data at input byte 11 when attempting to construct the docker auth header for image pull.
  • Root cause: The Secret's .dockerconfigjson contains "auth":"PLACEHOLDER_REPLACE_AFTER_PAL_E_SERVICES_HARBOR_PROJECT_PROVISIONED". The auth field of a dockerconfigjson must be base64-encoded username:password. The placeholder fails base64 validation at byte 11 (_ is not in the base64 alphabet).
  • Bootstrap intent: pal-e-services#64 (closed 2026-04-26) onboarded westside-admin into terraform/k3s.tfvars but did not replace the placeholder Secret in the running namespace. The terraform onboarding provisions the Harbor robot account and (per existing services pattern) generates the dockerconfigjson — but the rendered Secret was never applied into the westside-admin namespace.
  • Verification of placeholder location: 
    kubectl get secret harbor-creds -n westside-admin -o jsonpath="{.data\.dockerconfigjson}" | base64 -d
# {"auths":{"harbor.tail5b443a.ts.net":{"auth":"PLACEHOLDER_REPLACE_AFTER_PAL_E_SERVICES_HARBOR_PROJECT_PROVISIONED","username":"robot$westside-admin+westside-admin-pull","password":"PLACEHOLDER"}}}
  • Impact: blocks M1. Until the real auth token is injected, no pod can pull the image, regardless of any other deployment fix. This is now the sole remaining blocker for first admin login.
  • Suggested remediation path: per the existing services pattern (pal-e-app, basketball-api), Harbor robot creds for new namespaces are bootstrapped via the pal-e-services terraform module + a one-shot kubectl apply of the rendered Secret. Likely fix is one of: (a) re-run pal-e-services apply with output verification, (b) add a missing kubectl apply step that pal-e-services#64 was supposed to include, or (c) document a manual kubectl create secret docker-registry harbor-creds step for service onboarding. Lucas/Ava to scope.
  • Recommended labels: type:bug, arch:harbor, blocks:deploy-chain, blocks:M1, story:admin-row-crud.
  • Related: pal-e-services#64 (closed but did not deliver complete bootstrap), spike #139 / #140 (PSA — different domain), board item #1088/#1098 (already in qa, but they validated terraform apply not secret injection).

Implications for Bootstrap Arc / M1

Both items #1117 (validate-step fix) and #1118 (PSA fix) had their merged code reach cluster: image 803e9439… was successfully built/pushed by westside-admin pipeline #3, ArgoCD picked up #141's manifest, the new RS was admitted, and the pod was scheduled. All deploy-chain work this validation was scoped to has succeeded. But M1 (first admin login achievable) is still blocked by the harbor-creds placeholder — a third, previously-hidden blocker that PSA was masking. The bootstrap arc is not complete: it has one more step. Recommend creating the discovered issue (D-1) and routing it through the standard backlog → review → todo pipeline before declaring M1 unblocked.

Recommendation

  • Move board item #1118 (this ticket) to done.
  • Move board item #1117 (validate-step fix in westside-admin#12) to done — its image successfully reached cluster, proving the validate-step fix worked end-to-end.
  • Create a new Forgejo issue + board item in backlog for D-1 (harbor-creds placeholder injection). Do not reopen pal-e-services#64 — fresh ticket with clean repro and fix scope.
  • Hold off on declaring M1 reached until D-1 is shipped, pod becomes 1/1 Running, /health returns 200, and Lucas verifies first admin login at https://westside-admin.tail5b443a.ts.net.