Validation: westside-ai-assistant#30 — Nemo qwen infra

validation-30-2026-04-03 Doc

validation partial

Project

Ticket

Forgejo issue: westside-ai-assistant#30 (Nemo qwen infra)
Board item: #748
PR: pal-e-deployments#82 — merged to main as commit 21a33a2
What shipped: Replaced Anthropic model env with Ollama for Nemo AI assistant.

Environment

Production cluster (k3s on archbox). Namespace: westside-ai-assistant. ArgoCD app: westside-ai-assistant.

Checks

#	Criterion	How to Verify	Result	Evidence
1	ArgoCD synced at correct revision	kubectl get application -n argocd	PASS	ArgoCD app <code>westside-ai-assistant</code>: Synced at revision <code>21a33a2f89bd1cc4f768daf49159768fc26dc81d</code>. Matches PR #82 merge commit.
2	ArgoCD health	kubectl get application -n argocd	PARTIAL	Health status: <code>Progressing</code>. Not yet Healthy.
3	Old pod running	kubectl get pods -n westside-ai-assistant	PASS	Pod <code>westside-ai-assistant-9946bd988-lww8s</code> Running 1/1 Ready (old ReplicaSet, started 2026-03-31).
4	New pod rollout	kubectl get pods -n westside-ai-assistant	FAIL	New pod <code>westside-ai-assistant-dd855f577-c94mp</code> stuck in <code>ImagePullBackOff</code>. Error: 401 Unauthorized pulling <code>harbor.tail5b443a.ts.net/westside-ai-assistant/api:latest</code>.
5	Deployment status	kubectl get deployment	PARTIAL	Deployment shows 1/2 ready replicas. Rollout incomplete.

Verdict

PARTIAL — ArgoCD synced at the correct revision, but the new pod cannot pull the image due to a 401 Unauthorized error from Harbor. The old pod is still running and serving traffic. The rollout is blocked by a Harbor authentication issue for the westside-ai-assistant/api:latest image.

Discovered Issues

BLOCKER: Harbor image pull auth failure for westside-ai-assistant namespace. The new ReplicaSet pod gets 401 Unauthorized when pulling from Harbor. Likely missing or expired imagePullSecret in the westside-ai-assistant namespace. Needs a Forgejo issue to fix Harbor pull credentials.