pal-e-docs

Review: Upgrade Nemo model from Haiku 4.5 to Sonnet 4.6

review-730-2026-03-30 Doc

review needs-refinement
Project

Verdict: NEEDS_REFINEMENT

Template Completeness

  • [x] Type -- Feature
  • [x] Lineage -- Standalone
  • [x] Repo -- forgejo_admin/westside-ai-assistant
  • [x] User Story -- "As Marcus (admin), I want Nemo to understand complex questions..."
  • [x] Context -- explains Haiku-to-Sonnet motivation and SQL generation need
  • [x] File Targets -- listed (but incorrect, see below)
  • [x] Acceptance Criteria -- 3 criteria listed
  • [x] Test Expectations -- manual testing described
  • [x] Constraints -- kustomize persistence + cost monitoring noted
  • [x] Checklist -- standard PR checklist
  • [x] Related -- project-westside-basketball + sibling issue #27

Traceability

  • [ ] story:WS-S5 label -- PRESENT but WRONG mapping. WS-S5 (Superadmin) = "basketball-api database on CNPG shared cluster." WS-S5 (Admin) = "view registration stats on dashboard." Neither relates to AI model upgrades. Needs a new user story or correct mapping.
  • [ ] story note MISMATCH -- [SCOPE] Create a user story for AI assistant model/quality improvements on project-westside-basketball, or map to an existing story that covers Nemo capabilities.
  • [x] arch:westside-ai-assistant label -- present on board item
  • [ ] arch note MISSING -- [SCOPE] Create architecture note arch-westside-ai-assistant for the AI assistant component.
  • [x] Forgejo issue -- forgejo_admin/westside-ai-assistant#28, open

File Targets

  • [ ] Issue claims current model is claude-haiku-4-5-20251001 -- WRONG. Actual value in pal-e-deployments/overlays/westside-ai-assistant/prod/deployment-patch.yaml line 16 is claude-sonnet-4-20250514 (already Sonnet 4, not Haiku 4.5).
  • [ ] Issue does not specify the actual file path. The env var lives in pal-e-deployments/overlays/westside-ai-assistant/prod/deployment-patch.yaml, not in the westside-ai-assistant repo.
  • [ ] Target value claude-sonnet-4-6-20250514 needs verification as a valid Anthropic model identifier. The current deployed model ID format is claude-sonnet-4-20250514.

Repo Placement

MISMATCH. Issue is filed on forgejo_admin/westside-ai-assistant, but the only file to change is pal-e-deployments/overlays/westside-ai-assistant/prod/deployment-patch.yaml, which lives in forgejo_admin/pal-e-deployments. The Forgejo issue should either be re-filed on pal-e-deployments, or the issue body must explicitly state the cross-repo target with the full path.

Dependencies

  • [x] Board item #729 (Add read-only SQL tool, issue #27) -- in backlog, same labels. Issue #28 Context says Sonnet upgrade benefits the db_query tool. Complementary but independent; no hard blocker. However, the primary motivation ("Sonnet is better at SQL generation") is weakened if the SQL tool doesn't ship.
  • [x] No items in in_progress block this work.

Acceptance Criteria

  • [x] "ANTHROPIC_MODEL env var set to claude-sonnet-4-6-20250514" -- verifiable via kubectl get deploy -o yaml. Testable.
  • [ ] "Nemo responds with higher quality answers" -- subjective, not programmatically verifiable. Acceptable for a GroupMe bot but should reference baseline comparison (Sonnet 4 vs 4.6, not Haiku vs Sonnet).
  • [x] "Latency under 5s for typical questions" -- measurable manually. Acceptable.
AC assumes upgrading FROM Haiku. Since the model is already Sonnet 4, the quality delta from Sonnet 4 to Sonnet 4.6 will be smaller than described. AC should reflect the actual before/after.

Blast Radius

Low. ANTHROPIC_MODEL is only used in the westside-ai-assistant deployment (confirmed via grep across pal-e-deployments and pal-e-services). No downstream consumers affected. Cost impact from Sonnet 4 to Sonnet 4.6 is the main concern (already noted in Constraints). Rollback is trivial -- revert the env var value.

Decomposition Assessment

No decomposition needed. Single env var change in one file. Well under the five-minute rule. 1 file target, 3 acceptance criteria, 0 code changes. This is a minimal-scope configuration ticket.

Recommendation

  • [BODY] Fix current model reference: claude-haiku-4-5-20251001 is wrong. Actual current value is claude-sonnet-4-20250514. Update Context and File Targets sections to reflect the real before/after (Sonnet 4 to Sonnet 4.6, not Haiku to Sonnet).
  • [BODY] Add explicit file path: pal-e-deployments/overlays/westside-ai-assistant/prod/deployment-patch.yaml line 16.
  • [BODY] Fix repo placement: either re-file on forgejo_admin/pal-e-deployments or add a cross-repo note in File Targets stating the change is in pal-e-deployments, not westside-ai-assistant.
  • [LABEL] Fix story:WS-S5 label -- WS-S5 does not match this work. Either create a new user story for AI assistant quality/model management, or remove the label and mark as foundational work.
  • [SCOPE] Create architecture note arch-westside-ai-assistant for the AI assistant component.
  • [SCOPE] Create or assign a user story that covers Nemo model upgrades / AI quality improvements on project-westside-basketball.