Review: Embedding pipeline -- Solid Queue + Ollama + neighbor gem (Round 2)

Verdict: APPROVED

Template Completeness

• [x] Type -- Feature
• [x] Lineage -- depends on paldocs#49, search split to #61
• [x] Repo -- ldraney/paldocs
• [x] User Story -- well-formed "As the platform / I need / So that"
• [x] Context -- thorough: current Python pipeline, Rails replacement architecture, coexistence strategy with race-avoidance plan
• [x] File Targets -- 5 files (3 new, 2 modified), plus "do not touch" entries with rationale
• [x] Feature Flag -- "none" with justification (replaces existing functionality with coexistence strategy)
• [x] Acceptance Criteria -- 4 ACs, all in testable "When/Then" format (down from 7 pre-split)
• [x] Test Expectations -- 3 unit tests plus run command
• [x] Constraints -- Ollama endpoint, model (qwen3-embedding:4b), dimensions (2560), block granularity, mermaid skip, queue depth (~5600 blocks), Solid Queue
• [x] Checklist -- 7 items
• [x] Related -- project, story, arch note, upstream/downstream issues

Traceability

• [x] story:consolidation label -- present on board item
• [x] story note verified -- story-paldocs-consolidation exists in pal-e-docs, entry found in project-paldocs user-stories section
• [x] arch:rails-jobs label -- present on board item
• [x] arch note verified -- arch-rails-jobs exists in pal-e-docs (note ID 2095), titled "Architecture: Rails Jobs -- Solid Queue embedding pipeline". Contains sequence diagram matching the issue's described flow.
• [x] Forgejo issue -- ldraney/paldocs#52, state: open

File Targets

• [x] app/models/block.rb -- verified: file exists (5 lines, belongs_to :note). Ready for after_save callback + has_neighbors :embedding declaration.
• [x] app/jobs/embed_block_job.rb -- does not exist yet (will be created). app/jobs/ directory exists with application_job.rb. Solid Queue is configured (Gemfile line 26, puma.rb, queue.yml).
• [x] app/services/ollama_client.rb -- does not exist yet (will be created). NOTE: app/services/ directory does NOT exist -- agent must create it. Issue body now notes this.
• [x] config/initializers/ollama.rb -- does not exist yet (will be created). config/initializers/ directory exists.
• [x] Gemfile -- verified: exists. Neighbor gem is NOT present yet. Issue correctly notes "(may already be added in paldocs#49)". Since #49 is in_progress, this ticket should own adding neighbor if #49 doesn't.

Repo Placement

Dependencies

• BLOCKER: paldocs#49 (schema ownership transfer) -- board item #1559, currently in_progress. This ticket explicitly depends on #49 for structure.sql with the vector column, pgvector extension, and potentially the neighbor gem. Until #49 merges, there is no Rails-owned schema with the embedding column. Dependency is documented in the issue.
• Downstream: paldocs#61 (hybrid search) -- board item #1570, backlog. Depends on this ticket for embeddings to exist. Correctly split out.
• No other blocking dependencies. Parallel items (sprint schema #50, sprint UI #56) do not conflict.

Acceptance Criteria

• AC1 (job enqueued on block save) -- unit testable via ActiveJob test adapter
• AC2 (Ollama call + vector write) -- testable with mocked HTTP. Ollama URL confirmed matching existing Python worker.
• AC3 (mermaid skip) -- unit testable, clear criterion with embedding_status='skipped'
• AC4 (retry with exponential backoff, 3 attempts, dead letter) -- testable with connection stubbing, specific retry count given

Blast Radius

• Coexistence with Python embedding worker is explicitly addressed. Strategy is sound: deploy Rails job first, verify, then scale Python worker to 0.
• Both systems write to the same blocks.embedding column -- last-writer-wins is documented and accepted.
• No other services share the embedding column -- blast radius is contained.
• No search endpoints in this ticket -- blast radius further reduced by the split.

Decomposition Assessment

• File targets: 5 files (3 new, 2 modified) in 1 repo
• Acceptance criteria: 4 ACs -- within the 5 AC threshold
• Test expectations: 3 unit tests
• Estimated agent work: Under 5 minutes. Create Ollama HTTP client, embedding job with retry logic, model callback with mermaid skip, initializer. Straightforward Rails patterns.

Recommendation

• Search controller and hybrid search moved to #61 -- resolves [BODY] and [DECOMPOSE] items
• Story note and arch note exist -- resolves [SCOPE] items from round 1
• Issue body is clean and focused on embedding pipeline only