pal-e-docs

Decision: Phase 6 Vector Search Architecture

decision-phase6-vector-search-architecture Doc

active concept
Project

Context

Phase 6 adds semantic/vector search to pal-e-docs using pgvector. Before scoping the work, we researched embedding models, evaluated our hardware constraints, and made architectural decisions about what to embed and how.

Hardware Profile (archbox — single-node k3s)

Component Spec Availability
CPU Intel i7-8700K, 6c/12t, 4.8GHz boost 11% utilized by cluster
RAM 128GB DDR4 ~116GB available (8% used)
GPU GTX 1070, 8GB VRAM, CUDA 13.0, compute 6.1 Effectively idle (54MB/8192MB, 0% compute). NVIDIA device plugin deployed.

Embedding Model Research (March 2026)

Models Evaluated

Model Params Dims MTEB Rank Key Feature Fits GPU?
Qwen3-Embedding-0.6B 600M 32-1024 configurable Competitive with 7B models Instruction-aware, 100+ languages Yes (1.2GB)
<strong>Qwen3-Embedding-4B</strong> 4B 32-1024 configurable Near-8B quality Sweet spot for GTX 1070 <strong>Yes (fits 8GB VRAM)</strong>
Qwen3-Embedding-8B 8B 32-1024 configurable #1 open-source on MTEB (70.58) Max quality No (16GB, CPU only)
nomic-embed-text 137M 768 fixed Beats ada-002 Proven, small Yes
mxbai-embed-large 335M 1024 fixed Beats text-embedding-3-large Strong quality Yes

Why Not Qwen3.5?

Qwen3.5 (Feb 2026) is a general LLM family only — chat/base models (9B, 35B-A3B MoE). No Qwen3.5 embedding model exists as of March 2026. Qwen3-Embedding (June 2025) remains the latest and best open-source embedding family. Only Google's proprietary Gemini-Embedding beats it on MTEB overall.

Why Not External APIs?

The self-hosted RAG vision (concept-phase5-self-hosted-rag) explicitly targets zero external dependencies — no Pinecone, no OpenAI embedding API, no data leaving the cluster. All inference runs on-cluster.

Decisions Made

# Decision Choice Rationale
1 Embedding model <strong>Qwen3-Embedding-4B via Ollama</strong> Fits entirely in GTX 1070's 8GB VRAM for GPU-accelerated inference. Near-8B quality. Instruction-aware for domain tuning. Configurable dimensions (32-1024). Available on Ollama (<code>qwen3-embedding:4b</code>).
2 Dimensions <strong>768</strong> Good balance of quality vs storage. Can re-embed at 1024 later if needed. Configurable at inference time — no model change required.
3 What to embed <strong>Per-block (not per-note)</strong> Section-level semantic search. "Find the section about credentials" returns the specific block, not the whole note. Aligns with Phase 7's block content model. More embeddings but dramatically better retrieval precision.
4 Embedding pipeline <strong>Async via PostgreSQL LISTEN/NOTIFY</strong> Writes don't block on embedding generation. Block create/update fires a Postgres trigger → <code>NOTIFY embedding_queue</code> → worker picks up immediately. Zero new infrastructure (no Redis/Celery). Event-driven, not polling. Sub-30-second staleness SLA.
5 Dependency <strong>Phase 7 (blocks) before Phase 6 (vectors)</strong> Per-block embedding requires blocks to exist. Building blocks first avoids throwaway per-note embeddings and a migration. "Do it right, not fast."
6 Ollama deployment <strong>Platform service</strong> Own namespace + deployment, like CNPG. Any app on the platform can generate embeddings. Reusable capability.
7 Worker architecture <strong>Separate Kubernetes Deployment</strong> Independent from API pod. Owns the <code>nvidia.com/gpu: 1</code> resource request. Independent failure domain — API restarts don't kill embedding jobs, API pod doesn't get a GPU it doesn't need. Clean GPU isolation.
8 Instruction prefixes <strong>Asymmetric query/document</strong> Qwen3-Embedding is instruction-aware. Document side: <code>"Represent this platform knowledge base section for retrieval: {block_text}"</code>. Query side: <code>"Find the platform documentation about: {user_query}"</code>. Different prefixes for indexing vs querying improves retrieval quality.
9 Block type filtering <strong>Embed semantic content, skip rendering artifacts</strong> Embed: paragraph, list, heading, table (flattened to text), code. Skip: mermaid, raw HTML, empty/structural blocks. Headings embedded with parent context for hierarchy.

Dependency Chain Change

Original plan had Phase 6 and Phase 7 as independent (both depend only on Phase 5). With per-block embedding, Phase 6 now depends on Phase 7. Phase 7 is the critical path.

Open Questions — RESOLVED (2026-03-08)

All five open questions from original scoping have been resolved:
Question Resolution Decision #
Queue mechanism PostgreSQL LISTEN/NOTIFY + trigger. No new infra. #4
Worker architecture Separate k8s Deployment with GPU resource request. #7
Staleness window Event-driven, sub-30-second. LISTEN/NOTIFY is near-instant. #4
Instruction prefix Asymmetric: different prefixes for document indexing vs query embedding. #8
Block type filtering Embed semantic content (paragraph, list, heading, table, code). Skip mermaid, raw HTML. #9
  • phase-postgres-6-vector-search — the phase this decision supports
  • phase-postgres-7-block-content — prerequisite phase (blocks must exist before per-block embedding)
  • concept-phase5-self-hosted-rag — the RAG architecture vision
  • concept-phase5-database-side-intelligence — the database-side intelligence pattern
  • benchmark-phase5-knowledge-baseline — baseline measurements