Review: Restore Ollama Helm release to ops module (embedding pipeline broken)

Verdict: NEEDS_REFINEMENT

Template Completeness

• [x] Type — Feature
• [x] Lineage — standalone, references removal commit 7872dac
• [x] Repo — ldraney/pal-e-platform (terraform/modules/ops/main.tf)
• [x] User Story — platform operator, Ollama restored, embedding worker functional
• [x] Context — thorough history of deploy, removal, and current state
• [x] File Targets — 3 modify targets + 3 don't-touch targets
• [x] Feature Flag — none (correct for infra work)
• [x] Acceptance Criteria — 7 items
• [x] Test Expectations — 4 items with run command
• [x] Constraints — 6 items, specific and actionable
• [x] Checklist — present
• [x] Related — references project, commits, related issue

Traceability

• [x] story:superuser-query label — "I can query the knowledge base by meaning (semantic search)"
• [x] story note verified — found in project-pal-e-docs user-stories section, row 1
• [x] arch:infra label — infrastructure component
• [ ] arch note MISSING — [SCOPE] No arch-infra note exists in pal-e-docs. Create architecture note documenting the infra layer (ops module, NVIDIA plugin, Ollama, embedding worker metrics).
• [x] Forgejo issue — ldraney/pal-e-services#114, open
• [ ] Repo mismatch — [SCOPE] Issue filed on pal-e-services but all work targets pal-e-platform. Should be filed on pal-e-platform or documented why it lives on pal-e-services.

File Targets

• [x] terraform/modules/ops/main.tf — verified: file exists (213 lines). Ollama namespace + helm_release were surgically removed between nvidia_device_plugin (line 6-26) and embedding_worker_metrics (line 28-53). Git history at 7872dac~1 confirms the exact config to restore (namespace + helm_release, ~70 lines).
• [x] terraform/modules/ops/variables.tf — verified: file exists. No Ollama variables existed before removal (confirmed via git show 7872dac~1). No changes needed here.
• [ ] terraform/modules/ops/outputs.tf — ISSUE: [BODY] File exists but is EMPTY. The pre-removal version had output "ollama_namespace" that should be restored. Issue body says to check "if outputs existed" but doesn't explicitly list this as a target. The output existed and should be mentioned as a definite restore target.

• [x] terraform/main.tf — confirmed: module "ops" call at line 123, moved{} blocks exist for non-Ollama resources. No Ollama references remain. Correct to leave untouched.
• [x] NVIDIA / embedding-worker-metrics resources — confirmed in current main.tf, should not be modified.

Repo Placement

Dependencies

• #1434 (Scale embedding worker to 0) — alternative stopgap. Becomes unnecessary if #1432 succeeds. Not a blocker.
• #1433 (pg_stat_statements + Grafana) — independent, no dependency.
• NVIDIA device plugin — already deployed and managed by ops module. Issue correctly notes depends_on relationship. Not a blocker.
• Embedding worker — downstream consumer. Already deployed but failing due to missing Ollama. Will self-heal once Ollama is reachable. Confirmed: pal_e_docs/config.py hardcodes ollama.ollama.svc.cluster.local:11434.

Acceptance Criteria

• [x] AC 1-2: namespace + helm_release restored — verifiable via tofu plan output
• [x] AC 3: add-only plan — verifiable via tofu plan
• [x] AC 4: tofu apply succeeds — verifiable (requires cluster access)
• [x] AC 5: pods running — verifiable via kubectl
• [x] AC 6: embedding worker connectivity — verifiable via logs
• [x] AC 7: embedding backfill — verifiable but vague. No specific command given. Agent may need guidance on how to trigger backfill (the embedding_worker.py has a run_backfill function invoked via CLI arg).

Blast Radius

• pal-e-docs embedding worker — direct consumer. Config hardcodes the Ollama service URL. Will start working immediately once Ollama is reachable. No code changes needed in pal-e-docs.
• pal-e-docs semantic search — degraded to keyword-only while embeddings are missing. Will gradually improve as backfill processes blocks.
• No other services reference Ollama. The westside-ai-assistant that originally used it was decommissioned. Grep across pal-e-deployments found zero Ollama references.
• GPU contention — Ollama will claim the GPU. If other GPU workloads are added later, contention may occur. Issue acknowledges this in the Decision section.

Decomposition Assessment

• 2-3 file targets in 1 repo — under threshold
• 7 AC but work is essentially a single git-restore operation — under 5 minutes
• Single agent pass is appropriate

Recommendations

• [SCOPE] Repo placement: Issue is on pal-e-services but work targets pal-e-platform. Either move the issue to pal-e-platform or add a note explaining cross-repo placement. Agent spawn must clone pal-e-platform.
• [SCOPE] Create architecture note arch-infra documenting the ops module components (NVIDIA plugin, Ollama, embedding worker metrics, TF state backup).
• [BODY] outputs.tf: Add explicit mention that outputs.tf needs the ollama_namespace output restored (confirmed it existed at 7872dac~1). Currently the issue says "if outputs existed" — they did.
• [BODY] AC 7 (backfill): Specify the exact backfill command. The embedding worker supports --backfill mode per embedding_worker.py.
• [LABEL] Board item type mismatch: Board label is type:bug but issue ### Type is Feature. Update board label to type:feature.
• [LABEL] Board title mismatch: Board says "Redeploy Ollama" but Forgejo title is "Restore Ollama Helm release to ops module". Align titles.