Review: Restore Ollama -- unblock pal-e-docs semantic search & embedding worker

Verdict: NEEDS_REFINEMENT

Template Completeness

• [x] Type -- Bug
• [x] Lineage -- story: semantic-search | arch: infra
• [x] Repo -- pal-e-platform
• [x] What Broke -- clear description of Ollama removal impact
• [x] Repro Steps -- concrete 3-step repro
• [x] Expected Behavior -- clear
• [x] Acceptance Criteria -- 5 criteria, all verifiable
• [x] Environment -- k3s cluster, ops module
• [x] Related -- PR #380, commit 3233620, cross-references to other tickets

Traceability

• [ ] story:semantic-search label -- MISSING from project-pal-e-platform user stories table. The project page has stories for superuser-deploy, superuser-observe, superuser-recover, etc., but no semantic-search story. [SCOPE] Create user story entry on project-pal-e-platform user-stories section.
• [ ] arch:infra label -- too generic. No arch-infra note exists in pal-e-docs. The actual architecture component is the ops module (NVIDIA + Ollama + embedding worker + tf-state-backup). [LABEL] Change arch:infra to arch:ops or a more specific label, and [SCOPE] create the backing architecture note.
• [x] Forgejo issue -- ldraney/pal-e-platform#460, open

File Targets

• [x] terraform/modules/ops/main.tf -- verified: Ollama namespace + Helm release were removed here (81 lines deleted). Currently contains NVIDIA device plugin and embedding worker metrics service. Restore target confirmed.
• [x] terraform/modules/ops/outputs.tf -- verified: ollama_namespace output was removed. File is now empty. Output needed for network policy reference.
• [x] terraform/network-policies.tf -- verified: netpol_ollama resource was removed (22 lines). Allowed ingress from pal-e-app and westside-ai-assistant namespaces to ollama on port 11434.
• [x] terraform/main.tf -- verified: two moved blocks for ollama namespace and helm release were removed. Comment on line 121 still references Ollama ("NVIDIA, Ollama, embedding worker, tf-state-backup"). Moved blocks are one-time migration aids and do NOT need to be restored.
• [x] salt/states/services/init.sls -- verified: ollama service.running block was removed. However, issue AC does not mention Salt restoration. The AC says "Ollama namespace re-created in ops module" and "Helm release restored" which are terraform-only. Salt service state may or may not be needed depending on whether Ollama runs as a k8s pod (Helm) or a host service.

Repo Placement

Dependencies

• Phase 6: Vector Search (pgvector) -- currently in_progress on board-pal-e-platform. This Ollama restore is a prerequisite for that phase to function.
• Board item #1543 (pal-e-platform#456) -- in todo column, type:chore, arch:network-policy. May affect the network policy restoration approach.
• pal-e-services#114 -- referenced as "misplaced ticket, same issue" -- potential duplicate coordination needed.
• pal-e-deployments#202 -- referenced as "embedding worker stopgap" -- downstream dependency.
• claude-custom#274 -- tracking ticket in different project.
• NVIDIA device plugin -- already deployed (AC confirms "No changes to NVIDIA device plugin"), no dependency conflict.

Acceptance Criteria

• "Ollama namespace re-created in ops module" -- verifiable via terraform plan/apply output
• "Helm release restored (otwld chart v1.49.0, qwen3-embedding:4b model, GPU-accelerated, hostPath volume)" -- verifiable against commit diff
• "Network policy allowing pal-e-docs namespace to reach ollama on port 11434" -- NOTE: the original netpol allowed pal-e-app and westside-ai-assistant, not "pal-e-docs namespace". The issue AC says "pal-e-docs namespace" which may be a rename of pal-e-app, or may need verification. [BODY] Verify correct namespace name for network policy ingress rule (original was pal-e-app, not pal-e-docs).
• "kubectl get pods -n ollama shows running pod" -- verifiable post-apply
• "No changes to NVIDIA device plugin (already deployed)" -- verifiable via plan output showing no changes to that resource

Blast Radius

• The original network policy also allowed westside-ai-assistant to reach Ollama. The issue AC only mentions pal-e-docs. If westside-ai-assistant still needs Ollama access, the netpol should include it.
• The embedding worker metrics service (already in ops/main.tf) references pal-e-app namespace. This confirms the embedding worker runs in pal-e-app, meaning the netpol should allow pal-e-app (not pal-e-docs) unless namespace was renamed.
• Salt service state for ollama was also removed. If Ollama runs as both a host systemd service AND a k8s Helm release, the salt state may need restoration too. The AC is silent on this.

Decomposition Assessment

Recommendations

• [SCOPE] Create user story "semantic-search" on project-pal-e-platform user-stories section, or map to an existing story (e.g., superuser-observe or superuser-deploy).
• [LABEL] Change arch:infra to a more specific label (e.g., arch:ops or arch:ollama) and [SCOPE] create the backing architecture note.
• [BODY] Verify correct namespace in AC #3: original netpol allowed pal-e-app (not pal-e-docs). Confirm whether pal-e-app was renamed to pal-e-docs or if the AC has the wrong namespace.
• [BODY] Clarify whether Salt service state (salt/states/services/init.sls) needs restoration in addition to terraform resources.
• [BODY] Clarify whether westside-ai-assistant should still be allowed in the Ollama network policy (it was in the original).