pal-e-docs

Review: Evaluate gpt-oss-120b on real agentic tasks via Goose + Groq

review-1742-2026-07-03 Review

review needs-refinement
Project

Verdict: NEEDS_REFINEMENT

Template Completeness

  • [x] Type — Spike
  • [x] Lineage — "Standalone — follows from Groq pivot"
  • [x] Repo — ldraney/personal-assistant
  • [x] Question — well-structured with 7 sub-questions covering file edits, MCP tool use, multi-step workflows, code review, reasoning tasks, coverage percentage, and model comparison
  • [x] Deliverables — docs/model-evaluation.md + follow-up tickets
  • [x] Time-box — 1 session (2-3 hours)
  • [x] Related — project reference and dependency on #6
All required spike template sections present. Extra ### Note section documents model name change from Kimi K2 to gpt-oss-120b (acceptable context).

Traceability

  • [x] story:groq-pivot label — present on board item
  • [ ] story note MISSING — [SCOPE] The user-stories section of project-personal-assistant has four generic stories (developer, operator, business owner) but none are identified as "groq-pivot". The Groq pivot represents a strategic shift from local Ollama inference to Groq cloud API. Create a user story entry for the groq-pivot story on project-personal-assistant user-stories section.
  • [x] arch:personal-assistant label — present on board item
  • [ ] arch note MISSING — [SCOPE] No arch-personal-assistant note found in pal-e-docs. The project page (project-personal-assistant) has an architecture section with Mermaid diagram, three-layer breakdown, and cost model, which may suffice. Create architecture note arch-personal-assistant or confirm the project page architecture section covers this role.
  • [x] Forgejo issue — ldraney/personal-assistant#23, open

File Targets

  • [x] docs/model-evaluation.md — deliverable to be created (does not exist yet, expected for spike). The docs/ directory exists with 6 existing files (anatomy.md, architecture.md, deployment-pipeline.md, mcp-integration.md, model-serving.md, prompt-engineering.md). Target path is valid.

Repo Placement

OK. Issue filed on ldraney/personal-assistant, work targets the same repo. Single-repo scope. No mismatch.

Dependencies

  • [ ] [LABEL] Issue body states "Should run after #6 (system prompts affect evaluation quality)" but board item #1742 has no depends:6 label. Issue #6 ("Adapt core system prompts for Goose") is still open and in backlog (board item #1488). Add depends:6 label to track this dependency.
  • Board item #1488 (issue #6) itself depends on #3 and #5 (both done), so its own blockers are cleared.
  • Other groq-pivot story items in backlog: #1745 (issue #26, sprint:2), #1746 (issue #27, sprint:3), #1743 (issue #24, sprint:3). No circular dependencies detected.

Acceptance Criteria

Deliverables are verifiable by an agent:
  • docs/model-evaluation.md — agent can verify file exists and contains pass/partial/fail results, overall score, and recommendation
  • Follow-up tickets — agent can verify new Forgejo issues created and linked
The question sub-bullets (simple file edits, MCP tool use, multi-step workflows, code review, reasoning tasks) provide clear test categories. The "compare with qwen/qwen3-32b and llama-3.3-70b-versatile if time permits" is appropriately scoped as optional.

Blast Radius

Low. This is a spike (investigation only, no code changes to production systems). The evaluation results will inform model selection for the broader personal-assistant project. No downstream consumers affected. No existing code patterns at risk.

Decomposition Assessment

  • File targets: 1 (docs/model-evaluation.md)
  • Repos: 1
  • Deliverables: 2
  • Time-box: 1 session (2-3 hours, inherently interactive)
No decomposition needed. Single-file output, time-boxed, single repo. Well within the 5-minute rule for agent work (the spike itself is manual/interactive evaluation, not automatable).

Recommendation

  • [LABEL] Add depends:6 to board item #1742 labels — issue body documents dependency on #6 but board item does not track it
  • [SCOPE] Create "groq-pivot" user story entry in project-personal-assistant user-stories section describing the strategic shift to Groq cloud API
  • [SCOPE] Create architecture note arch-personal-assistant or explicitly confirm the project page architecture section fulfills that role