pal-e-docs

Phase 6: Vector Search (pgvector)

phase-postgres-6-vector-search Phase

Project
Goal: Add semantic/vector search to pal-e-docs using pgvector. Enable AI agents to find conceptually related notes, not just keyword matches.
Owner: Dev agent
Repo: pal-e-platform, pal-e-docs, pal-e-docs-sdk, pal-e-docs-mcp
Depends on: Phase 5 (full-text search) — COMPLETED, Phase 7 (block content model) — COMPLETED, Phase 7f (clean data) — COMPLETED

Why

Full-text search finds keyword matches. Vector search finds conceptual matches. Example: searching "how do we handle credentials" should return the secrets management SOP even if it never uses the word "credentials."

Key Decisions (see decision-phase6-vector-search-architecture)

Decision Choice
Embedding model <strong>Qwen3-Embedding-4B</strong> via Ollama (GPU-accelerated on GTX 1070)
Dimensions <strong>768</strong> (configurable, can increase later)
What to embed <strong>Per-block</strong> (paragraph, list, heading, table, code. Skip mermaid.)
Pipeline <strong>Async via PostgreSQL LISTEN/NOTIFY</strong> (no Redis/Celery, event-driven, sub-30s)
Ollama <strong>Platform service</strong> (own namespace, like CNPG)
Worker <strong>Separate k8s Deployment</strong> (owns GPU resource request, independent failure domain)
Instruction prefixes <strong>Asymmetric</strong> — different prefixes for document indexing vs query embedding

Sub-Phase Status

# Sub-Phase Repo Status
6a Deploy Ollama as platform service pal-e-platform COMPLETED
6b pgvector extension + schema migration pal-e-docs COMPLETED
6b-1 Fix extension ownership (migration-only) pal-e-docs COMPLETED (PR #140, Issue #126)
6c Async embedding pipeline + backfill pal-e-docs COMPLETED (PR #130, Issue #129)
6c-1 Enforce Closes #N + strengthen post-merge reminder claude-custom COMPLETED (PR #74, Issue #73)
6c-2 QA nits — dead code, N+1 query, dep hygiene, k8s hardening pal-e-docs COMPLETED (PR #136, Issue #135)
6d Semantic search API + SDK + MCP tool pal-e-docs, sdk, mcp COMPLETED (API #138, SDK #20, MCP #30)
6d-1 SDK integration tests pal-e-docs-sdk COMPLETED (PR #23, Issue #22)
6e Hybrid ranking (tsvector + vector) pal-e-docs COMPLETED (PR #141, Issue #139)

Sub-Phase Details

6a: Deploy Ollama as platform service

Repo: pal-e-platform (Terraform + Helm)
Status: COMPLETED — PR #25 (initial deploy), PR #27 (runtimeClassName fix). Issue #24, #26.
  • Create ollama namespace
  • Deploy Ollama via Helm chart with GPU resource request (nvidia.com/gpu: 1)
  • Pull qwen3-embedding:4b model on startup
  • ClusterIP service on port 11434 for internal access
  • Verify GPU acceleration is active (ollama ps shows GPU layers)
Verification (2026-03-08): Node reports nvidia.com/gpu: 1 capacity. Ollama pod Running. Model qwen3-embedding:4b (4.0B, Q4_K_M) fully loaded in VRAM (3.5GB). Embedding API returns 768-dim vectors.

6b: pgvector extension + schema migration

Repo: pal-e-docs
Status: COMPLETED — PR #122. Issue #121.
  • Enable pgvector extension via CREATE EXTENSION IF NOT EXISTS vector
  • Alembic migration l2g3h4i5j6k7: add embedding vector(768) column to blocks table
  • Add embedding_status varchar(20) column (default 'pending')
  • Create HNSW index on embedding column (vector_cosine_ops)
  • Postgres trigger: on block INSERT or UPDATE of content or block_type, sets embedding_status = 'pending' and NOTIFY embedding_queue. Skips mermaid blocks (sets 'skipped').
  • Added pgvector>=0.3 dependency

6b-1: Fix extension ownership (platform-provides pattern)

Repo: pal-e-docs (migration fix), deployments (CNPG CRD)
Status: NOT STARTED — Issue #126.
Problem: The 6b migration includes CREATE EXTENSION IF NOT EXISTS vector, which requires superuser. The Alembic migration runs as the paledocs app user (not superuser), causing CrashLoopBackOff on deploy. Fixed manually via kubectl exec as postgres superuser, but this breaks fresh deployments.
Fix: Follow the platform-provides/app-consumes pattern:
  • Remove CREATE EXTENSION from Alembic migration, replace with existence check + informative error
  • Add extension provisioning to CNPG Cluster CRD in deployments repo (bootstrap.initdb.postInitSQL)
  • Remove DROP EXTENSION from downgrade

6c: Async embedding pipeline + backfill

Repo: pal-e-docs (new module + k8s Deployment). Same Docker image, different entrypoint.
  • Embedding worker process: src/pal_e_docs/embedding_worker.py — standalone Python process (not FastAPI). LISTEN embedding_queue as primary trigger, periodic poll fallback (every 60s) for missed notifications during restarts.
  • Block text extraction: block_type-aware content JSON → plain text. Paragraph: strip HTML. List: join items. Heading: "{note_title} > {heading_text}" (parent context join). Table: flatten headers + rows. Code: raw text. Mermaid: already skipped by trigger.
  • Ollama integration: POST http://ollama.ollama.svc.cluster.local:11434/api/embed with model qwen3-embedding:4b. Document prefix: "Represent this platform knowledge base section for retrieval: {block_text}". Store 768-dim vector in blocks.embedding.
  • State machine: embedding_status transitions: pending → processing → completed | error. The processing state prevents duplicate work on pod restart. Existing skipped state (mermaid) unchanged.
  • Reliability: retry with exponential backoff on Ollama transient errors. Batch processing (10 blocks/cycle live, higher for backfill). Graceful SIGTERM handling — finish current batch, don't leave blocks in processing state.
  • Observability: Prometheus metrics — embedding_total, embedding_errors_total, embedding_duration_seconds, embedding_queue_depth. Health endpoint (/healthz) for k8s liveness/readiness probes. Structured logging.
  • k8s Deployment: k8s/embedding-worker.yaml — same image as API pod, entrypoint python -m pal_e_docs.embedding_worker. No GPU resource request (worker calls Ollama over HTTP). Minimal resources (10m CPU, 64Mi request, 256Mi limit). Add to kustomization.yaml.
  • Config: add ollama_url to Settings (PALDOCS_OLLAMA_URL, default: in-cluster service URL).
  • Dependencies: add httpx to main deps (Ollama HTTP client). Use raw psycopg2 connection for LISTEN (SQLAlchemy doesn't expose it).
  • Backfill: --backfill flag — one-time run to embed all ~5K pending blocks. Rate-limited batches, progress logging. Can run as kubectl exec into the worker pod.

6d: Semantic search API + SDK + MCP tool

Repos: pal-e-docs (API), pal-e-docs-sdk (client), pal-e-docs-mcp (tool)
  • GET /notes/semantic-search?q=...&limit=10 — query prefix applied, cosine similarity search
  • Returns: matching blocks with note context, similarity score, block content snippet
  • SDK: client.semantic_search(query, limit)
  • MCP: semantic_search tool wrapping SDK

6e: Hybrid ranking (tsvector + vector)

Repo: pal-e-docs
  • Combine full-text search score (tsvector ts_rank) with vector similarity (cosine distance)
  • Weighted ranking: configurable alpha between keyword relevance and semantic similarity
  • GET /notes/search?q=...&mode=hybrid — unified search endpoint
  • MCP: update search_notes tool to support mode parameter

Dependency Chain

6a and 6b can run in parallel (different repos). 6b-1 doesn't block 6c (extension already installed). 6c depends on both 6a and 6b. 6d depends on 6c. 6e depends on 6d.
  • decision-phase6-vector-search-architecture — full decision record with model research and hardware analysis
  • phase-postgres-7-block-content — prerequisite phase (COMPLETED)
  • phase-postgres-7f-doc-cleanup-sop — prerequisite phase (COMPLETED, clean data)
  • concept-phase5-self-hosted-rag — RAG architecture vision
  • concept-phase5-database-side-intelligence — database-side intelligence pattern
  • benchmark-phase5-knowledge-baseline — baseline measurements