pal-e-docs

Phase F12: Semantic Search Recovery

phase-pal-e-docs-f12-semantic-search-recovery Phase

active phase
Project

Phase F12: Semantic Search Recovery

Goal: Restore semantic search to working state and add alerting so Ollama/embedding failures are detected within 10 minutes instead of silently rotting.
Owner: Dev agent
Repo: forgejo_admin/pal-e-docs, forgejo_admin/pal-e-platform, forgejo_admin/pal-e-deployments
Depends on: None (Ollama and pgvector already deployed from Act 2 Phase 6a)

Scope

The semantic_search MCP tool returns 503. Root cause diagnosed (2026-03-15): Ollama pod is healthy (running 6 days, 0 restarts) but the qwen3-embedding:4b model is NOT loaded. The 10Gi PVC was recreated 6 days ago and only chat models (qwen3.5:4b, qwen3:4b) were pulled. Embedding worker has 152 errors, 0 successful embeddings. embedding_queue_depth reads 0 because failed blocks are marked error, not pending — this is important for alerting strategy.
F12-1: Fix Ollama model persistence. Swap PVC for hostPath volume mount (e.g. /var/lib/ollama) in the Ollama Terraform deployment. This ensures models survive any k8s lifecycle event (pod restart, deployment recreate, terraform apply). Then pull qwen3-embedding:4b (~3GB). Repos: pal-e-platform (Terraform), pal-e-deployments (if kustomize overlay). Hardware: GTX 1070 (8GB VRAM), model fits comfortably alongside chat models.
F12-2: Backfill stale embeddings. Query SELECT embedding_status, count(*) FROM blocks GROUP BY embedding_status to assess damage. Reset error blocks to pending. Run python -m pal_e_docs.embedding_worker --backfill. Verify semantic_search returns results.
F12-3: Add alerting. The worker exposes Prometheus metrics on :8001/metrics. IMPORTANT: embedding_queue_depth is NOT sufficient — it reads 0 during failures because blocks get marked error after 3 retries. The correct alerts are: rate(embedding_errors_total[5m]) > 0 → warning (active failures), and embedding_total == 0 for > 10 minutes while embedding_errors_total is increasing → critical (complete embedding failure). Add Prometheus scrape config for the worker. Route to existing Slack/Telegram pipeline (Phase 16 infra).

Deliverables

  • semantic_search MCP tool returns results (not 503)
  • All blocks have embedding_status = completed or skipped (zero pending/error)
  • Prometheus alert fires within 10 minutes if embeddings stop processing
  • plan-pal-e-docs — parent plan
  • plan-pal-e-platform — alerting infrastructure (Phase 16)
  • phase-pal-e-docs-design-overhaul — sibling phase (F11, in progress)