pal-e-docs

Phase 7c: Backfill Migration (HTML → Blocks)

phase-postgres-7c-backfill-migration Phase

Project
Goal: Convert all 256 existing notes from monolithic html_content to typed blocks. Populate compiled_pages with compiled HTML + TOC. One-time backfill.
Owner: Dev agent
Repo: pal-e-docs
Depends on: Phase 7a (tables exist), Phase 7b (parser + compiler)
Parent phase: Phase 7 (Block-Structured Content Model)
Forgejo Issue: #100
PR: #101 (MERGED)
Status: IN PROGRESS — Deliverable 1 (migration script) complete. Deliverables 2-3 (run against production + validation) pending.

Why This Phase

The parser and compiler are proven by unit tests in 7b. Now we run them against the full corpus and populate the blocks table. This is a one-time data migration — the most critical step in Phase 7.

Deliverables

  • Migration script: For each note, run the parser on html_content, store resulting blocks in the blocks table with correct positions and anchor IDs. COMPLETE — PR #101 merged.
  • Compiled pages: For each note, compile blocks back to HTML, generate TOC JSON, compute content hash, store in compiled_pages. PENDING — requires running script against production.
  • Validation report: Compare compiled_pages.html vs original html_content for all 256 notes. Flag any semantic differences. PENDING — requires running script against production.
  • Nest orphaned docs: After hierarchy relaxation (7a), set parent_note_id for the 6 identified orphans.

Migration Strategy

  • Run as a one-time Alembic data migration or standalone script
  • html_content stays populated (backward compat — existing MCP tools still read it)
  • Blocks are additive — they coexist with html_content, not replace it
  • Use kubectl exec + kubectl cp pattern (port-forward is unreliable on k3s)

Acceptance Criteria

  • All 256 notes have blocks in the blocks table
  • All 256 notes have a compiled_pages entry
  • Round-trip validation: compiled HTML is semantically equivalent to original for 95%+ of notes
  • Remaining notes (if any) have documented differences with manual review
  • 6 orphaned docs nested under their logical parents
  • Existing API / MCP behavior unchanged

Expected Block Counts

Based on baseline data (avg 7.7 headings/note, plus paragraphs/tables/lists between them):
  • Estimated ~15-25 blocks per note average
  • Total estimated blocks: ~4,000-6,500
  • This is well within Postgres performance for indexed queries
  • phase-postgres-7a-schema-hierarchy — tables must exist
  • phase-postgres-7b-parser-compiler — parser + compiler must be proven
  • benchmark-phase7-block-baseline — corpus metrics