pal-e-docs

Phase 7b: HTML-to-Blocks Parser + Blocks-to-HTML Compiler

phase-postgres-7b-parser-compiler Phase

Project
Goal: Build the two core library functions: parse HTML into typed blocks, and compile blocks back into deterministic HTML with stable anchor IDs.
Owner: Dev agent
Repo: pal-e-docs
Depends on: Phase 7a (schema must exist for block types)
Parent phase: Phase 7 (Block-Structured Content Model)

Completion Summary

Forgejo Issue: #96
PR: #97 (merged)
Bug fix PR: #99 (merged) — fixed mermaid <br/> stripping and heading inline <code> loss. Forgejo issue #98.
Deliverables: Parser (6 block types: heading, paragraph, table, code, list, mermaid), compiler (deterministic HTML + stable anchor IDs), 121 tests (105 original + 16 regression).
QA: Both PRs approved. PR #97 had 5 non-blocking nits; PR #99 fixed 2 of them.
Level 2 validation: Real-data round-trip test against 268 live notes. 99.3% semantic match (2 bugs fixed by PR #99). Remaining diffs are by design: anchor ID additions (112 notes) and table whitespace normalization (~129 notes).

Why This Phase

The parser and compiler are the heart of the block system. They must be correct, deterministic, and handle all existing content. This phase is pure library code — no DB writes, no API changes, no behavior change. Fully testable in isolation.

Deliverables

1. HTML-to-Blocks Parser

Takes html_content (string) and returns a list of typed block dicts.
Must handle all content found in the 256 existing notes (see benchmark-phase7-block-baseline):
Block Type HTML Pattern Content JSONB Notes in Corpus
<code>heading</code> <code>&lt;h2&gt;</code>, <code>&lt;h3&gt;</code>, <code>&lt;h4&gt;</code> <code>{"level": 2, "text": "..."}</code> 229 notes (89%)
<code>paragraph</code> <code>&lt;p&gt;...&lt;/p&gt;</code> <code>{"html": "..."}</code> ~256 notes
<code>table</code> <code>&lt;table&gt;...&lt;/table&gt;</code> <code>{"headers": [...], "rows": [...]}</code> 118 notes (46%)
<code>code</code> <code>&lt;pre&gt;&lt;code&gt;...&lt;/code&gt;&lt;/pre&gt;</code> <code>{"language": "...", "content": "..."}</code> 72 notes (28%)
<code>list</code> <code>&lt;ul&gt;</code>, <code>&lt;ol&gt;</code> <code>{"ordered": false, "items": [...]}</code> 239 notes (93%)
<code>mermaid</code> <code>&lt;pre class="mermaid"&gt;</code> <code>{"definition": "..."}</code> 37 notes (14%)
<code>callout</code> TBD — currently no callout pattern in corpus <code>{"type": "info", "content": "..."}</code> 0 (future use)

2. Blocks-to-HTML Compiler

Takes a list of blocks and produces deterministic HTML with stable anchor IDs.
  • Deterministic: Same blocks → same HTML every time. No random IDs.
  • Stable anchor IDs: Each heading block generates an anchor from its text (slugified). Example: <h3 id="acceptance-criteria">Acceptance Criteria</h3>
  • Round-trip safe: compile(parse(html)) ≈ html (semantically equivalent, may normalize whitespace/formatting)

Edge Cases to Handle

  • Paragraphs between headings (group with preceding heading or as standalone block?)
  • Nested lists (items containing sub-lists)
  • Tables inside other elements
  • Inline code vs code blocks
  • Empty content / notes with no headings (27 notes)
  • HTML entities and special characters
  • The nh3 sanitizer may have altered some HTML patterns

Acceptance Criteria

  • Parser handles all 7 block types
  • Round-trip test: parse then compile 20 representative notes, verify semantic equivalence
  • Mermaid blocks preserved exactly (whitespace-sensitive)
  • Table structure preserved (headers vs body rows)
  • Anchor IDs are deterministic and unique within a note
  • Unit tests for each block type + edge cases

Design Decision: Grouping Strategy

Open question: how do we group content under headings?
  • Option A: Flat blocks. Every HTML element is its own block. A heading is one block, the paragraph after it is another. Simple, but no "section" concept.
  • Option B: Section blocks. A heading + everything until the next heading at the same or higher level = one "section" block. More useful for get_block("acceptance-criteria") returning the heading + its content.
Recommendation: Option A (flat) with section grouping at query time. Store flat blocks but provide a "get section" query that returns a heading block + all blocks until the next heading. This keeps storage simple and grouping flexible.
  • phase-postgres-7a-schema-hierarchy — prerequisite (block types defined in schema)
  • benchmark-phase7-block-baseline — content type distribution in the corpus
  • html-style-guide — HTML patterns used in existing notes