Review: Scale down k8s Ollama to free GPU (r2)

Verdict: APPROVED

Template Completeness

• [x] Type -- Feature
• [x] Lineage -- Standalone, part of Phase 3 pivot
• [x] Repo -- ldraney/pal-e-platform
• [x] User Story -- well-formed
• [x] Context -- includes Terraform rationale (kubectl would be reverted by tofu apply)
• [x] File Targets -- specific: terraform/modules/ops/main.tf (helm_release.ollama). Fixed from previous review.
• [x] Feature Flag -- "none"
• [x] Acceptance Criteria -- 5 items, all Terraform-based. Fixed from previous review.
• [x] Test Expectations -- 3 items
• [x] Constraints -- 3 items (reversibility, preserve PVCs, no side effects)
• [x] Checklist -- now includes "Tests pass". Fixed from previous review.
• [x] Related -- project and downstream issue referenced

Traceability

• [x] story:gpu-freedom label -- GPU Freedom story
• [x] story note verified -- story-pal-e-world-gpu-freedom exists in pal-e-docs (user-story type, active status)
• [ ] story note on project page -- NOT listed in project-pal-e-world user-stories table. [SCOPE] Add gpu-freedom entry to project-pal-e-world user-stories section.
• [x] arch:k3s label -- k3s infrastructure
• [ ] arch note MISSING -- no arch-k3s note found in pal-e-docs. [SCOPE] Create architecture note arch-k3s for k3s component, or relabel to arch:ollama (consistent with review-1552-2026-06-22-r2).
• [x] Forgejo issue -- ldraney/pal-e-platform#486, open

File Targets

• [x] terraform/modules/ops/main.tf -- VERIFIED. File exists in pal-e-platform repo. helm_release.ollama resource found at line 47. Contains GPU config (nvidia runtime, gpu.enabled=true, gpu.number=1), model pull (qwen3-embedding:4b), hostPath volume, and depends_on nvidia_device_plugin. Exact match to issue description.
• [x] Exclusions correctly scoped -- pal-e-docs code (separate Groq migration issue pal-e-api #283) and Salt states (host Ollama already disabled).

Repo Placement

Dependencies

• Board item #1653 ("Groq API migration for pal-e-docs embeddings", story:gpu-freedom, 5 pts) depends on THIS ticket completing first. Documented in issue Context and Related sections. Sequencing correct.
• No items in in_progress that block this work.
• No undocumented blockers detected.

Acceptance Criteria

• [x] AC1: "Terraform Helm values set Ollama replicas to 0 (or release disabled via count conditional)" -- verifiable by code review of helm_release.ollama
• [x] AC2: "tofu plan shows the expected change (scale down, not destroy)" -- verifiable by running tofu plan
• [x] AC3: "After tofu apply, nvidia-smi shows no Ollama processes" -- verifiable manually
• [x] AC4: "pal-e-docs semantic search degrades to keyword-only without crashing" -- verifiable manually
• [x] AC5: "Change is reversible -- can set replicas back to 1 for re-embedding" -- verifiable by code review

Blast Radius

• pal-e-docs semantic search -- acknowledged in issue. Degrades to keyword-only. Covered by AC4.
• Embedding worker (pal-e-api/k8s/embedding-worker.yaml) -- runs with replicas=1, connects to http://ollama.ollama.svc.cluster.local:11434. Will get ConnectError when Ollama scales down. Worker has error handling (marks blocks as errored, continues polling). Will generate embedding_errors_total Prometheus metrics. Acceptable temporary state before Groq migration (#1653). Not a blocker.
• NVIDIA device plugin -- separate Helm release in same module, not affected.
• Salt states -- ollama-service: service.dead already in place for host systemd. No conflict.
• Other ops module resources -- terraform/modules/ops/main.tf also contains TF state backup CronJob and NVIDIA device plugin. Change is scoped to helm_release.ollama only. Constraints section correctly warns "Must not affect other resources."

Decomposition Assessment

Recommendation

• [SCOPE] Add gpu-freedom story entry to project-pal-e-world user-stories table (story note exists but is not registered on the project page).
• [SCOPE] Create architecture note arch-k3s for k3s component. Or relabel board item to arch:ollama if preferred (consistent with prior review review-1552-2026-06-22-r2).