SOP: Post-Merge Validation

SOP: Post-Merge Validation

Purpose

Steps

• Identify the repo type. The validation procedure depends on what kind of repo was changed. See Validation by Repo Type below for the specific checklist per type.
• Check dependency order. If the merged change spans multiple repos (e.g., API + frontend, or platform + services), validate in dependency order: platform first, then services, then APIs, then frontends. Never validate a downstream consumer before its upstream dependency is confirmed healthy.
• Execute the repo-type checklist. Follow the appropriate checklist from the section below. Collect evidence at each step.
• Record validation evidence. Every validation must produce at least one piece of concrete proof — a pipeline URL, kubectl output, curl response, or screenshot. See Validation Evidence below.
• Create the validation note. Use the /validate-ticket skill or manually follow template-validation to create a validation note recording the ticket, environment, checks, verdict, and any discovered issues.
• Move board item to done. After all checks pass and the validation note shows PASS, move the board item from validation to done using update_board_item. The DORA Lead Time clock stops here.
• Run /update-docs. Trigger the post-merge documentation chain walk. Validation must happen before docs update — never update docs for unvalidated changes.

Validation by Repo Type

Terraform repos (pal-e-platform, pal-e-services)

• Run tofu plan -lock=false against the target workspace. The plan must show no changes (clean state). If it shows drift, the merge did not apply correctly.
• If tofu apply is required (state was not applied during merge), run tofu apply -lock=false and confirm it succeeds with zero errors.
• Verify the specific resource changed: check Kubernetes objects (kubectl get), Helm releases (helm list), or provider state as appropriate.

API repos (pal-e-api, basketball-api, minio-api, mcd-tracker-api)

• Confirm Woodpecker pipeline is green for the merge commit. Record the pipeline URL.
• Verify the new image tag propagated: kubectl get pods -n {namespace} -o jsonpath='{.items[*].spec.containers[*].image}' should show the expected tag.
• Confirm pod is running and ready: kubectl get pods -n {namespace} — status Running, restarts = 0.
• Smoke test the affected endpoint: curl -s -o /dev/null -w "%{http_code}" https://{service-url}/health returns 200.

Frontend repos (westside-app, pal-e-app, mcd-tracker-app, pal-e-docs)

• Confirm Woodpecker pipeline is green. Record the pipeline URL.
• Confirm deployment is live: check pod image tag or static asset hash matches the merge commit.
• Route-level smoke test (required). Check the project page for a ### Routes section listing critical routes. Navigate every listed route using Playwright (mcp__playwright__browser_navigate) or curl, and confirm each returns HTTP 200. A root-URL-only check is not sufficient — broken sub-routes (e.g., /admin returning 500) are invisible to root-only validation. If the project page has no Routes section, at minimum test / plus any routes touched by the PR.
• Visual check: load the affected page in a browser or via screenshot. Confirm the change is visible and nothing is broken.

claude-custom (hooks and agent config)

• Restart the Claude Code session to pick up new hooks.
• Verify hooks load without errors: check session startup output.
• Run the relevant test suite or trigger the hook manually to confirm behavior.

pal-e-docs notes (content changes via MCP)

• Read the updated note via get_note(slug=...) or get_section(slug=..., anchor_id=...). Confirm content renders correctly.
• Verify internal links: any code slug references should resolve to existing notes.
• Check the note appears in the correct project, has correct tags, and follows template structure.

Kustomize repos (pal-e-deployments)

• Confirm ArgoCD has synced the application: kubectl get application -n argocd {app-name} -o jsonpath='{.status.sync.status}' shows Synced.
• Confirm the application is healthy: kubectl get application -n argocd {app-name} -o jsonpath='{.status.health.status}' shows Healthy.
• Verify the target pods are running the new image and are not crash-looping.

Validation Evidence

Lessons Learned

• Merged does not equal applied. Terraform state can drift from the merged code. A PR merged in pal-e-platform means nothing until tofu apply runs successfully. Always verify with tofu plan -lock=false.
• Pipeline green does not equal deployed. The CI pipeline can build and push a new image, but the image tag may not propagate to the running pod. ArgoCD sync delays, kustomize tag mismatches, and registry pull errors can all silently prevent deployment.
• ArgoCD sync does not equal healthy. An application can show Synced while pods are in CrashLoopBackOff. Always check both sync status and health status. A running pod with restart count > 0 is a red flag.
• Formatting PRs still need CI verification. Even "cosmetic" changes — whitespace, comments, README updates — can break CI. YAML indentation, trailing commas, and encoding issues are silent killers. Every merge gets validated, no exceptions.
• Cross-repo changes need dependency-order validation. If pal-e-platform changes a Helm value that pal-e-services consumes, validate platform first. Validating downstream before upstream gives false confidence. The dependency chain is: platform → services → deployments → APIs → frontends.
• Validation must happen before docs update. Running /update-docs on an unvalidated merge propagates false state into pal-e-docs. The docs say "done" but production says otherwise. Validate first, document second.
• Use -lock=false with tofu plan. Without this flag, the plan acquires a state lock that blocks Woodpecker CI pipelines. Always pass -lock=false for validation checks.
• CNPG changes need pg_isready verification. Database operator changes (CloudNativePG) can appear healthy at the pod level while the database is actually in recovery or failover. Run pg_isready -h {cluster-rw-service} to confirm the database is accepting connections.
• Keycloak changes need login flow verification. Theme changes, realm config, and client updates can break the login flow even when the Keycloak pod is healthy. Always test an actual login — load the login page, submit credentials, confirm redirect.
• Network policy changes need connectivity verification. A new NetworkPolicy can silently break inter-service communication. After any network policy merge, verify the affected services can still reach each other: kubectl exec into a pod and curl the target service.
• Root URL 200 does not equal all routes healthy. A frontend app can return 200 at / while sub-routes like /admin return 500 due to missing env vars or config. PlayMe2K shipped a broken admin page because validation only checked the root URL. Always test every critical route listed on the project page, not just /. Use Playwright or per-route curl. (2026-04-05: PlayMe2K /admin 500 — ADMIN_SECRET in dev overlay only, never prod.)

Rules

• Every merge gets validated. No exceptions. Formatting PRs, documentation PRs, and one-line fixes all go through validation. The validation column is not optional.
• Validation before docs. Never run /update-docs until the change is confirmed live and healthy in production.
• Evidence is required. A board item cannot move from validation to done without at least one piece of concrete proof (pipeline URL, kubectl output, screenshot, or curl response).
• Dependency order is mandatory. Cross-repo changes validate upstream before downstream. No shortcuts.
• Validation is not QA. QA reviews the code (left side of the board). Validation confirms the deployment (right side). These are separate gates with separate concerns.
• Stuck in validation is a signal. If an item sits in validation for more than one session, investigate. Common causes: forgot to apply terraform, ArgoCD out of sync, image tag mismatch.
• Never move directly from needs_approval to done. The validation column exists precisely because merged-does-not-equal-deployed. Skipping it defeats the purpose.
• Route-level checks are mandatory for frontends. A root-URL-only curl is not a valid frontend validation. Test every route listed on the project page.

DORA Integration

• Validation Latency — time from merge to validation PASS. This is the gap between needs_approval and done. Shorter is better.
• Deployment Frequency — only incremented when items reach done. Unvalidated merges do not count as deployments.
• Change Failure Rate — validation failures (items that bounce back from validation to in_progress) are CFR signals. They indicate the merge introduced a regression that passed QA but failed in production.
• Mean Time to Recovery — for incident-triggered items, validation confirms the fix is live. MTTR clock stops at validation PASS, not at merge.

Related

• sop-board-workflow — defines the full board lifecycle including the validation column and two-gate model
• convention-validation-checkpoints — the convention that mandates validation gates
• template-validation — template for validation evidence notes
• skill-validate-ticket — the /validate-ticket agent skill that automates validation note creation and checks
• pr-lifecycle — the 7-stage PR flow that feeds into validation
• hook-catalog — enforcement hooks including post-merge reminders and validation-gate enforcement