pal-e-docs

Convention: Validation Pipeline

convention-validation-pipeline Convention

active convention
Project

Convention: Validation Pipeline

Three-tier validation model ensuring merged work actually works before it reaches users. Each tier catches a different class of failure. No ticket moves to done without passing all applicable tiers.

Purpose

Validation is the right-side gate of the kanban board, mirroring the left-side review gate at todo. The pipeline answers one question: does the merged change work in a real environment? Unit tests answer "does the code compile." Validation answers "does the system behave."

Three Tiers

Tier Name Environment What It Tests Speed
1 <strong>Dev</strong> Local / k8s dev namespace Integration tests against local code with volume mounts. Confirms the change works beyond unit tests in a real service context. Seconds
2 <strong>Staging</strong> Containerized, production-like Full deploy smoke tests. Harbor images, ArgoCD sync, same pipeline as prod but targeting a staging namespace. Confirms the built artifact deploys and runs. Minutes
3 <strong>Prod</strong> Production Post-deploy health checks. Validates the deployed service is healthy and the specific acceptance criteria from the ticket are met. Minutes

When Each Tier Runs

Tier Trigger Prerequisite Gate
Dev Before QA review (during <code>in_progress</code>) <code>git pull origin main</code> must have latest merged changes Dev PASS required before PR submission
Staging After merge, before prod deploy Woodpecker CI: build → push → deploy to staging → smoke test Staging PASS required before ArgoCD prod sync
Prod After prod deploy (ticket enters <code>validation</code> column) Service deployed and healthy in production Prod PASS required before ticket moves to <code>done</code>

Tier Details

Tier 1: Dev

  • Mechanism: Volume mount to local directory (Vite-on-host or k8s dev overlay)
  • Integration tests run against local code with real database, real API calls
  • Pull-before-dev requirement: Main branch must have the latest PR changes pulled before running dev tests. This is enforced by the pull-before-dev SOP and SessionStart hook.
  • Overlay location: pal-e-deployments/overlays/{service}/dev/
  • Fast feedback loop: seconds, not minutes. Developer gets immediate signal.

Tier 2: Staging

  • Mechanism: Full containerized deployment (Harbor images, ArgoCD sync to staging namespace)
  • Provisioned by: pal-e-services terraform (staging namespace) + pal-e-platform terraform (staging infra)
  • Salt managed like prod — same configuration management, different target
  • Woodpecker CI pipeline: build → push → deploy → smoke test
  • Same pipeline as prod, different target namespace
  • Overlay location: pal-e-deployments/overlays/{service}/staging/

Tier 3: Prod

  • Mechanism: Existing ArgoCD deployment
  • Post-deploy health checks against live production endpoints
  • Validation note created via /validate-ticket skill per template-validation
  • Overlay location: pal-e-deployments/overlays/{service}/prod/ (existing)

Directory Naming Convention

All overlays follow the kustomize convention (see convention-kustomize-overlay):
Each tier overlay follows the same structure: kustomization.yaml with base ref and patches, deployment-patch.yaml for strategic merge patches, and optional service-specific resources.

Pull-Before-Dev Requirement

Before running dev-tier tests, the local main branch must have the latest changes pulled. This prevents testing stale code and getting false positives.
  • Enforced by: SessionStart hook in claude-custom + sop-pull-before-dev
  • Why: If main is behind, dev tests pass against old code. The PR's changes are never actually tested.
  • Command: git checkout main && git pull origin main before any dev-tier validation

Board Integration

The validation column sits between needs_approval (merge) and done in the kanban board (see sop-board-workflow):
  • Entry: Betty Sue moves ticket to validation after merge
  • Exit: Validation PASS moves ticket to done. Validation FAIL creates a follow-up issue for the regression.
  • Enforcement: No ticket reaches done without a validation note with PASS verdict
  • Right-side gate: Mirrors the left-side review gate at todo (via /review-ticket)

DORA Integration

Validation is a first-class DORA measurement point:
  • Validation Latency: Time from merge to validation PASS. This is the validation column's DORA signal in the board workflow. Shorter = faster confidence in deployments.
  • Lead Time for Changes: Clock stops at validation PASS, not at merge. A merge without validation is not a completed change.
  • Change Failure Rate: FAIL verdicts signal regressions introduced by the merge. Each FAIL increments CFR.
  • Mean Time to Recovery: Time from merge to validation PASS = recovery verification latency. For bug-fix tickets, this measures how quickly we confirm the fix works.
  • Deployment Frequency: Only validated items (PASS) count as deployments. Unvalidated merges are not deployments.

Failure Classes by Tier

Tier Catches Example
Dev Logic errors, integration bugs, API contract breaks New endpoint returns wrong shape; DB migration breaks existing query
Staging Build failures, deployment config errors, env var misses Dockerfile copies wrong path; kustomize patch references missing field
Prod Environment-specific issues, DNS, TLS, external dependencies Tailscale funnel not configured; external API key expired

Architecture Labels

Components involved in the validation pipeline carry these architecture labels on board items:
  • arch:validation-pipeline — the pipeline itself
  • arch:kustomize — overlay structure
  • arch:ci-pipeline — Woodpecker CI steps
  • arch:argocd — deployment sync
  • arch:harbor — container registry
  • template-validation — validation note template (structure for PASS/FAIL notes)
  • skill-validate-ticket — the skill that orchestrates tier 1 → tier 2 → tier 3 checks
  • sop-board-workflow — kanban board column semantics including the validation column
  • convention-kustomize-overlay — overlay directory structure that dev/staging/prod tiers follow
  • board-validation-pipeline — decomposition board for building this pipeline
  • sop-frontend-dev-overlay — existing dev overlay SOP for frontend services