Phase 8g: Deploy Pipeline Smoke Tests

phase-postgres-8g-smoke-tests Phase

Project

Goal: Add a post-deploy smoke test step to the pal-e-docs Woodpecker pipeline that automatically verifies the live service is healthy after every deploy to main.

Owner: Dev agent

Repo: forgejo_admin/pal-e-docs-sdk (smoke module), forgejo_admin/pal-e-docs (pipeline step)

Depends on: 8f (SDK published on Forgejo PyPI, all endpoints covered)

Scope

Two deliverables across two repos:

Smoke test module in pal-e-docs-sdk — src/pal_e_docs_sdk/smoke_test.py. Runnable as python -m pal_e_docs_sdk.smoke_test. Hits 5 key endpoints, asserts minimum counts, exits 0/1/2. Includes retry loop for ArgoCD sync delay.
Pipeline step in pal-e-docs — new smoke-test step in .woodpecker.yaml after update-deployment-tag. Installs SDK from Forgejo PyPI, runs smoke module against internal service URL.

This completes Phase 8 (SDK + MCP Rewrite + Integration Tests). After 8g, the full stack has: typed SDK → integration tests → MCP tools → post-deploy smoke tests.

Why

Today there is zero automated verification that a deploy actually works. The Phase 5 deployment outage (incident-phase5-deployment-outage-2026-03-06) was discovered by manually curling endpoints. The pipeline that builds should be the pipeline that verifies — don't call it done until you've proven it works.

Design Decisions

Decision	Rationale
Smoke module lives in SDK repo	Uses SDK methods directly. Published with the package. Runnable as <code>python -m pal_e_docs_sdk.smoke_test</code>.
Pipeline step lives in pal-e-docs repo	That's where <code>.woodpecker.yaml</code> is. The step installs SDK and runs the module.
Internal service URL	<code>http://pal-e-docs.pal-e-docs.svc.cluster.local:8000</code>. No Tailscale/TLS dependency. Pipeline runs in-cluster.
Retry loop in smoke module	ArgoCD sync is async (30s-90s after tag update). Module waits 30s initial, then retries up to 5 times at 15s intervals. Total window: ~105s.
Health verification, not version check	Verify the service is up and returning correct data. No <code>/version</code> endpoint needed. Simpler, sufficient.
In-pipeline verification (Option A)	Enterprise pattern. The pipeline that builds is the pipeline that verifies. Tight feedback loop. No detection delay.

Smoke Test Endpoints

Check	SDK Call	Assertion	What It Proves
API up + DB connected	<code>list_notes()</code>	count > 200	FastAPI serving, SQLAlchemy connected, notes table populated
Full-text search	<code>search_notes("postgres")</code>	results > 0	tsvector index working, search endpoint functional
Block content	<code>get_note_toc("plan-2026-02-26-tf-modularize-postgres")</code>	headings >= 5	Blocks table populated, TOC endpoint functional
Sprint tables	<code>list_sprints()</code>	count >= 1	Sprint schema intact, endpoint functional
Tags	<code>list_tags()</code>	count > 5	Tags table, lightweight health check

Smoke Module Design

Pipeline Step

Execution Order

Two repos means two PRs in sequence:

PR on pal-e-docs-sdk: Add smoke_test.py + __main__.py wiring. Bump version to 0.2.0. Merge → publishes to PyPI.
PR on pal-e-docs: Add smoke-test step to .woodpecker.yaml. Merge → next deploy triggers smoke test.

Risks

ArgoCD sync delay: If sync takes longer than 105s window, smoke test exits 2. Can tune INITIAL_WAIT and MAX_RETRIES.
Forgejo PyPI in-cluster access: Pipeline already uses forgejo-http.forgejo.svc.cluster.local for git clone. PyPI should work the same way.
SDK version mismatch: Smoke test installs latest SDK. If SDK has breaking change, smoke fails even though API is fine. Low risk — we control both.

Deliverables

To be filled after completion

phase-postgres-8-mcp-optimization — parent phase
phase-postgres-8f-mcp-rewrite — SDK must be published (done)
phase-postgres-8e-integration-tests — smoke test is a subset of integration tests, tuned for post-deploy
incident-phase5-deployment-outage-2026-03-06 — the incident that motivates this