pal-e-docs

Architecture: Rails Jobs — Solid Queue embedding pipeline

arch-rails-jobs Architecture

architecture active
Project

Rails Jobs: paldocs

Diagram

Components

Component Purpose Notes
EmbedBlockJob Primary embedding job Triggered by after_save on Block model
Solid Queue Job backend Rails 8 default, replaces Sidekiq/Redis
Ollama HTTP API Embedding model host qwen3-embedding:4b, 2560 dimensions
neighbor gem Vector column writes Writes to blocks.embedding column

Key Decisions

  • Solid Queue over Sidekiq — eliminates Redis dependency, aligns with Rails 8 defaults
  • after_save callback — every block save triggers re-embedding, keeping vectors fresh
  • Mermaid block skipping — blocks containing only mermaid diagrams are not embedded (no semantic value)
  • Retry with exponential backoff — handles Ollama timeouts gracefully
  • project-paldocs — parent project page
  • arch-rails-db — schema where vectors are stored
  • arch-infra — k8s deployment where jobs run
  • story-paldocs-consolidation — driving user story