pal-e-docs

Review: Embedding pipeline -- Solid Queue + Ollama + neighbor gem

review-1562-2026-06-22 Review

review needs-refinement
Project

Verdict: NEEDS_REFINEMENT

Re-review (rev 2, 2026-06-22). Prior SCOPE items for story note and arch note are now resolved. Issue body has been expanded. Decomposition still required.

Template Completeness

  • [x] Type -- Feature
  • [x] Lineage -- depends on paldocs#49, part of consolidation
  • [x] Repo -- ldraney/paldocs
  • [x] User Story -- well-formed "As the platform / I need / So that"
  • [x] Context -- thorough: describes current Python pipeline, Rails replacement architecture, coexistence strategy with race-avoidance plan
  • [x] File Targets -- 7 files to create/modify, 2 "do not touch" entries, all with descriptions
  • [x] Feature Flag -- "none" with justification (replaces existing functionality with coexistence strategy)
  • [x] Acceptance Criteria -- 7 ACs, all in testable "When/Then" format
  • [x] Test Expectations -- 5 unit/integration tests plus run command
  • [x] Constraints -- Ollama endpoint, model (qwen3-embedding:4b), dimensions (2560), block granularity, mermaid skip, queue depth (~5600 blocks), Solid Queue
  • [x] Checklist -- 8 items
  • [x] Related -- project, story, arch note, upstream/parallel/downstream issues all listed
All required sections for a Feature template are present and well-populated.

Traceability

  • [x] story:consolidation label -- present on board item
  • [x] story note verified -- story-paldocs-consolidation exists in pal-e-docs (note ID 2092), titled "Story: Consolidation -- absorb pal-e-docs into Rails monolith". Entry found in project-paldocs user-stories section.
  • [x] arch:rails-jobs label -- present on board item
  • [x] arch note verified -- arch-rails-jobs exists in pal-e-docs (note ID 2095), titled "Architecture: Rails Jobs -- Solid Queue embedding pipeline". Contains sequence diagram matching the issue's described flow.
  • [x] Forgejo issue -- ldraney/paldocs#52, state: open
Traceability triangle is complete. Prior review's [SCOPE] items for missing story and arch notes have been resolved.

File Targets

  • [x] app/models/block.rb -- verified: file exists (5 lines, belongs_to :note). Ready for after_save callback + has_neighbors :embedding declaration.
  • [x] app/jobs/embed_block_job.rb -- does not exist yet (will be created). app/jobs/ directory exists with application_job.rb. Solid Queue is configured (Gemfile, puma.rb, queue.yml).
  • [x] app/services/ollama_client.rb -- does not exist yet (will be created). NOTE: app/services/ directory does NOT exist yet -- agent must create it.
  • [x] app/services/hybrid_search.rb -- does not exist yet (will be created). Same directory note.
  • [ ] app/controllers/api/search_controller.rb -- ISSUE: app/controllers/api/ directory does not exist. Issue body says "replaces stub from #51" but paldocs#51 (API endpoints) is still open and not started -- no API namespace, no base API controller, no API routes exist. This target either depends on #51 completing first or the agent must also create the entire API namespace and routing.
  • [x] config/initializers/ollama.rb -- does not exist yet (will be created). config/initializers/ directory exists.
  • [x] Gemfile -- verified: exists. Neighbor gem is NOT present yet. Issue correctly notes "(may already be added in paldocs#49)". Since #49 is in_progress, this ticket should own adding neighbor if #49 doesn't.

Repo Placement

OK. Issue filed on ldraney/paldocs, all file targets are in paldocs. Single-repo scope. No mismatch.

Dependencies

  • BLOCKER: paldocs#49 (schema ownership transfer) -- board item #1559, currently in_progress. This ticket explicitly depends on #49 for: structure.sql with the vector column, pgvector extension, and potentially the neighbor gem. Until #49 merges, there is no Rails-owned schema with the embedding column. Dependency is documented in the issue.
  • Soft dependency: paldocs#51 (API endpoints) -- open, not started. The search controller file target references "replaces stub from #51", implying #51 creates the API namespace and base controller first. If #51 is not done, this ticket must also create the api/ controller namespace and routing infrastructure.
  • Parallel (no conflict): paldocs#50 (sprint schema, wave:1), paldocs#56 (sprint UI)
  • Downstream: paldocs#54 (new MCP server, wave:3) depends on this ticket's search endpoints
Dependencies are well-documented in the issue's Lineage and Related sections.

Acceptance Criteria

All 7 ACs are in testable "When/Then" format:
  • AC1 (job enqueued on block save) -- unit testable via ActiveJob test adapter
  • AC2 (Ollama call + vector write) -- testable with mocked HTTP, Ollama URL confirmed matching existing Python worker (http://ollama.ollama.svc.cluster.local:11434/api/embed)
  • AC3 (mermaid skip) -- unit testable, clear criterion
  • AC4 (semantic search endpoint) -- integration testable
  • AC5 (hybrid search with RRF) -- integration testable, well-specified
  • AC6 (retry with exponential backoff, 3 attempts, dead letter) -- testable with connection stubbing, specific retry count given
  • AC7 (result equivalence with pal-e-docs) -- Vague. "Results are equivalent" lacks a concrete threshold or test dataset. However, since the same model (qwen3-embedding:4b) and dimensions (2560) are used, functional equivalence is reasonable to assume. Suggest rewording to "hybrid search returns fused keyword+semantic results for the same queries that pal-e-docs semantic_search serves."

Blast Radius

  • Coexistence with Python embedding worker is explicitly addressed in the issue body. Strategy is sound: deploy Rails job first, verify, then scale Python worker to 0. PostgreSQL LISTEN/NOTIFY trigger stays (no conflict with ActiveRecord callbacks).
  • Ollama URL confirmed matching existing Python worker: http://ollama.ollama.svc.cluster.local:11434 (verified in pal-e-docs/src/pal_e_docs/config.py).
  • Both systems write to the same blocks.embedding column -- last-writer-wins is documented and accepted.
  • MCP server (pal-e-docs) currently serves semantic_search. During transition, search results could diverge if both systems are live, but this is inherent to the migration and acceptable.
  • No other services share the embedding column -- blast radius is contained to paldocs + pal-e-docs.

Decomposition Assessment

NEEDS DECOMPOSITION -- route to skill-decompose-ticket
  • File targets: 7 files (5 new, 2 modified) in 1 repo
  • Acceptance criteria: 7 ACs -- exceeds the 5 AC threshold
  • Test expectations: 5 distinct test items
  • Estimated agent work: Well over 5 minutes. Must create: Ollama HTTP client with error handling, embedding job with retry/dead-letter logic, model callbacks with mermaid skip, RRF hybrid search algorithm, search controller with routing, initializer with configuration.
  • Natural decomposition into 2 sub-tickets:

Recommendation

  • [BODY] Fix search controller reference: "replaces stub from #51" is misleading since #51 is not started and no API namespace exists. Clarify that this ticket (or its sub-ticket) must create the app/controllers/api/ directory and routing if #51 is not done first.
  • [BODY] Clarify AC7 (result equivalence): "results are equivalent" is not automatable. Suggest rewording to "hybrid search returns fused keyword+semantic results for known test content."
  • [BODY] Note that app/services/ directory does not exist and must be created by the agent.
  • [DECOMPOSE] 7 ACs across 7 file targets, 3 distinct subsystems (Ollama client, embedding pipeline, search/RRF), estimated well over 5 min agent work. Route to skill-decompose-ticket. Natural split: (1) embedding pipeline -- Ollama client + job + model callback + initializer, (2) search endpoints -- hybrid search service + controller + routing.