Review: Embedding pipeline -- Solid Queue + Ollama + neighbor gem (retroactive)

Verdict: APPROVED

Template Completeness

• [x] Type -- Feature
• [x] Lineage -- depends on paldocs#49 (merged), search split to paldocs#61
• [x] Repo -- ldraney/paldocs
• [x] User Story -- "As the platform / I need Rails to generate vector embeddings / So that the Python embedding worker can be deprecated"
• [x] Context -- thorough: describes current Python pipeline, Rails replacement architecture, and coexistence strategy with race-avoidance plan
• [x] File Targets -- 5 files (3 new, 2 modified) with "do not touch" entries and rationale. Notes app/services/ may need to be created.
• [x] Feature Flag -- "none" with justification (replaces existing functionality)
• [x] Acceptance Criteria -- 4 ACs in testable When/Then format
• [x] Test Expectations -- 3 unit tests plus run command (bundle exec rspec)
• [x] Constraints -- Ollama endpoint, model (qwen3-embedding:4b), dimensions (2560), block granularity, mermaid skip, queue depth (~5600), Solid Queue
• [x] Checklist -- 7 items
• [x] Related -- project, story, arch note, upstream/downstream issues

Traceability

• [x] story:consolidation label -- present on board item
• [x] story note verified -- story-paldocs-consolidation exists (note ID 2092, user-story type, project paldocs). Listed in project-paldocs user-stories section.
• [x] arch:rails-jobs label -- present on board item
• [x] arch note verified -- arch-rails-jobs exists (note ID 2095, architecture type, project paldocs). Contains sequence diagram matching the described flow.
• [x] Forgejo issue -- ldraney/paldocs#52, state: open

File Targets

• [x] app/models/block.rb -- verified on main (5 lines, belongs_to :note). PR branch adds has_neighbors + after_save callback as specified.
• [x] app/jobs/embed_block_job.rb -- PR creates this. app/jobs/ exists on main with application_job.rb. Implementation includes mermaid skip, text extraction, retry logic.
• [x] app/services/ollama_client.rb -- PR creates this. app/services/ did not exist; PR creates the directory. HTTP client with typed errors (Error, TimeoutError).
• [x] config/initializers/ollama.rb -- PR creates this. ENV-based config for URL, model, dimensions, timeout, input prefix.
• [x] Gemfile -- PR adds neighbor gem (line 18) and webmock for test HTTP stubbing (line 48). solid_queue already present (line 26).

Repo Placement

Dependencies

• paldocs#49 (schema ownership transfer) -- board item #1559, now in done column (state: closed). Blocker resolved. structure.sql on main has the embedding column (vector(2560)), pgvector extension, and HNSW index.
• Downstream: paldocs#61 (hybrid search) -- board item #1570, backlog. Depends on embeddings from this ticket. Correctly separated.
• No blocking dependencies remain. Parallel sprint:1 items (#50 sprint schema, #67 docs) do not conflict.

Acceptance Criteria

• AC1 (job enqueued on block save) -- verified in PR: spec/models/block_spec.rb has 4 tests including enqueue on create, enqueue on content update, skip on position-only change
• AC2 (Ollama call + vector write) -- verified in PR: spec/jobs/embed_block_job_spec.rb has 8 tests covering the full job flow
• AC3 (mermaid skip with embedding_status='skipped') -- verified in PR: job checks block_type == "mermaid" and also skips blank content
• AC4 (retry with exponential backoff, 3 attempts) -- verified in PR: retry_on with polynomially_longer, attempts: 3, for both TimeoutError and Error

Blast Radius

• No existing embedding code on main branch -- this is net-new Rails functionality replacing external Python worker.
• Coexistence strategy is documented: both systems write to blocks.embedding, last-writer-wins during transition.
• update_columns used in job to bypass callbacks -- avoids re-enqueue loop. Sound pattern.
• No search endpoints included -- blast radius contained by the split to #61.

Decomposition Assessment

• File targets: 5 files (3 new, 2 modified) in 1 repo -- under threshold
• Acceptance criteria: 4 ACs -- under 5 AC threshold
• Estimated agent work: Under 5 minutes. PR is already implemented with 483 insertions across standard Rails patterns.

Recommendation