pal-e-docs

Phase 6d: Semantic Search API + SDK + MCP Tool

phase-postgres-6d-semantic-search Phase

Project
Goal: Add a semantic search endpoint so agents can find related knowledge by meaning (cosine similarity over pgvector embeddings), then expose it through the SDK and MCP tool.
Owner: Dev agent
Repo: forgejo_admin/pal-e-docs (API), forgejo_admin/pal-e-docs-sdk (SDK), forgejo_admin/pal-e-docs-mcp (MCP)
Depends on: phase-postgres-6c (async embedding pipeline — embeddings must exist in blocks table)

Why

Phase 6a deployed Ollama with GPU. Phase 6b added pgvector schema. Phase 6c built the embedding pipeline and backfilled 5K+ blocks. But there's no way to query those embeddings yet. This phase closes the loop — agents can ask a natural language question and get ranked, relevant blocks back without reading every document.

Scope

Three deliverables across three repos, executed sequentially (each depends on the previous):
# Deliverable Repo Forgejo Issue Status
1 API endpoint: <code>GET /notes/semantic-search</code> pal-e-docs <a href="https://forgejo.tail5b443a.ts.net/forgejo_admin/pal-e-docs/issues/137">#137</a> COMPLETED (PR #138)
2 SDK method: <code>client.semantic_search(query, limit)</code> pal-e-docs-sdk <a href="https://forgejo.tail5b443a.ts.net/forgejo_admin/pal-e-docs-sdk/issues/18">#18</a> COMPLETED (PR #20)
3 MCP tool: <code>semantic_search</code> pal-e-docs-mcp <a href="https://forgejo.tail5b443a.ts.net/forgejo_admin/pal-e-docs-mcp/issues/29">#29</a> COMPLETED (PR #30)

Architecture

Query flow: Agent calls MCP tool → SDK calls API → API embeds query via Ollama → cosine similarity search on blocks.embedding → return ranked blocks with note context.
The API endpoint mirrors GET /notes/search (tsvector) but operates at block granularity. Key differences:
  • Queries the blocks table (not notes), joining to notes for context
  • Uses pgvector <=> cosine distance operator (HNSW index: ix_blocks_embedding)
  • Requires an Ollama call to embed the query text before searching
  • Returns block-level results: anchor_id, block_type, content snippet, similarity score

Deliverables

  • (filled after completion)
  • phase-postgres-6-vector-search — parent phase
  • plan-2026-02-26-tf-modularize-postgres — parent plan
  • decision-phase6-vector-search-architecture — embedding model research + architectural decisions
  • phase-postgres-6c — embedding pipeline (prerequisite)