pal-e-docs

Phase: Activate Semantic Search Pipeline

phase-pal-e-docs-activate-semantic-search Phase

phase
Project
Goal: Activate the dormant embedding pipeline and semantic search, giving every agent contextual awareness across the full 260+ note knowledge base.
Owner: Dev agent
Repo: forgejo_admin/pal-e-docs
Depends on: None (all infrastructure already built — pgvector, Ollama, embedding worker code, search API, SDK, MCP tools)

Scope

The entire semantic search stack was built during Act 2 but left dormant (replicas: 0). This phase activates it:
  • Scale embedding worker — change k8s/embedding-worker.yaml replicas from 0 to 1
  • Verify image currency — ensure the manifest image tag matches a build that contains embedding_worker.py. Update if stale.
  • Verify connectivity — worker must reach Ollama at http://ollama.ollama.svc.cluster.local:11434 and Postgres via PALDOCS_DATABASE_URL
  • Initial backfill — 5,643 blocks with embedding_status='pending' need embedding. Worker has --backfill mode or will process via LISTEN/NOTIFY loop.
  • Verify search modes — confirm /search?mode=semantic and /search?mode=hybrid return results via API
  • Verify MCP tool — confirm semantic_search() MCP tool returns ranked results

Progress (2026-03-14)

  • PR #155 merged — replicas 0→1, image tag updated. Embedding worker running.
  • PR #161 merged — Fixed SQLAlchemy text() parameter binding bug: :query_vec::vector → CAST(:query_vec AS vector). Semantic search SQL now works. Issue #160 closed.
  • SDK v0.3.0 published — to Forgejo PyPI. MCP server venv updated. MCP process needs restart to load new SDK.
  • Backfill in progress — 266/5,921 blocks completed (~4.5%). Batch size patched to 50, poll interval to 10s for ~10x speedup. ETA ~1 hour.
  • Remaining — verify search endpoints return results after backfill progresses further; verify MCP tool after session restart.

Key Context

  • Embedding worker: src/pal_e_docs/embedding_worker.py (610 lines, production-ready)
  • K8s manifest: k8s/embedding-worker.yaml (replicas: 1, same image as API with different entrypoint)
  • Search service: src/pal_e_docs/services/search.py (RRF fusion — keyword + semantic)
  • Model: qwen3-embedding:4b (2560-dim vectors, 3.5GB VRAM, loaded in Ollama)
  • DB state: ~5,639 pending, 16 skipped, ~266 completed (actively backfilling)
  • ArgoCD reads from pal-e-docs/k8s/ — push to main triggers deploy

Deliverables

  • Embedding worker running in production (1 replica) ✓
  • All embeddable blocks have embedding_status='completed' — in progress
  • Hybrid search returns meaningful results via API and MCP tool — SQL fix merged, pending verification
  • plan-pal-e-docs — parent plan
  • phase-pal-e-docs-5a-embedding-dimension-fix — sub-phase (completed)
  • bug-mcp-silent-load-failure — semantic search could help agents find recovery SOPs
  • plan-2026-03-09-template-rendering — sibling capability (template rendering)