Service Onboarding SOP

How a New Service Joins the Platform

• Create Forgejo issue on the new service repo using template-issue-feature. MUST include: Dockerfile EXPOSE port, Harbor project name (matching service key), Keycloak realm + client ID (if auth needed). The issue is the spec for the scaffold agent.
• Add to var.services in k3s.tfvars — defines repo, image, port, funnel, and optionally target_revision, source_repo, source_path. This goes through a PR on pal-e-services — create issue, branch, PR, review. No manual git push.
• Create kustomize overlay in pal-e-deployments — create overlays/{service-name}/prod/ with kustomization.yaml and deployment-patch.yaml following the standard base. This goes through a PR on pal-e-deployments — create issue, branch, PR, review. Set source_repo and source_path in var.services to point ArgoCD at the overlay. See Convention: Kustomize Overlay for Deployments.
  
  ⚠️ Secrets warning: Kustomize overlays committed to git must NOT contain real secrets. A kubectl apply -k will overwrite any manually-created secrets with placeholder values from the overlay. Application secrets (DB passwords, API keys, OAuth tokens) must be created via kubectl create secret generic in the target namespace BEFORE the first kustomize apply or ArgoCD sync. The overlay should reference secrets by name (e.g. in envFrom) but must not define their data. If the overlay includes a Secret manifest, use stringData placeholders and apply secrets manually first so ArgoCD does not clobber them.
• Update NetworkPolicy for dependent services — if the new service needs access to MinIO, Postgres, Keycloak, or any other shared platform service, add the new namespace to the relevant NetworkPolicy allowlist in pal-e-platform/terraform/network-policies.tf. This goes through a PR on pal-e-platform — create issue, branch, PR, review. Without this, the new service gets connection refused when calling shared services. Check each dependency: MinIO (minio namespace policy), Postgres (postgres namespace policy), Keycloak (keycloak namespace policy). Run tofu plan -lock=false to verify the netpol diff before apply.
• Provision databases (if the service needs PostgreSQL) — add an entry to service_databases in k3s.tfvars. The map key becomes the PostgreSQL role name. Rails 8 apps typically need 4 databases: primary, cache, queue, cable. Example:

  westside_basketball = {
    password  = "generated-password"
    databases = ["basketball", "basketball_cache", "basketball_queue", "basketball_cable"]
  }

  This goes through a PR on pal-e-services. After merge, update the tfvars_content Woodpecker secret (base64 -w0 ~/secrets/pal-e-services/k3s.tfvars). The terraform creates the PostgreSQL role and databases via databases.tf.
  
  Rails apps also need a rails-env Kubernetes secret with POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_HOST, and SECRET_KEY_BASE. Add a kubernetes_secret_v1 resource in services.tf referencing the terraform-managed role and password. The app's database.yml production config must use ENV.fetch/ENV[] to read these values — never hardcode credentials.
  
  ⚠️ Database ownership: If databases were created manually before terraform import, table ownership may not match the terraform-managed role. Verify with SELECT tableowner FROM pg_tables and fix with REASSIGN OWNED BY if needed.
• Configure public domain routing (if the service has a public domain like example.com) — the full chain is:

  DNS (GoDaddy) → edge-proxy Caddy (178.156.129.142) → Tailscale funnel → k8s Service → Pod

  Each link must be configured:
• tofu plan/apply — creates ArgoCD Application, namespace, Harbor project, robot accounts, image pull secret, and Image Updater annotations. Always use -lock=false to avoid blocking CI. Requires Lucas approval before apply.
• Scaffold repo on Forgejo — FastAPI app with Dockerfile, .woodpecker.yaml, k8s manifests. Dispatch a dev agent with the Forgejo issue from step 1.
  
  CI image registry convention: The .woodpecker.yaml pipeline MUST use the internal Harbor URL (harbor-core.harbor.svc.cluster.local) for image push, NOT the external URL (harbor.tail5b443a.ts.net). The external URL routes through Tailscale DERP relay which is unreliable from inside the cluster and causes intermittent CI failures. A HARBOR_INTERNAL env var may be added to the Woodpecker agent config as the canonical source for this URL. All docker push and docker tag commands in the pipeline must target the internal endpoint.
  
  Ruff linter configuration (required for all Python repos): The scaffold MUST include a pyproject.toml with [tool.ruff] config (line-length = 88, select = ["E", "F", "I", "W"]) and a .pre-commit-config.yaml with the ruff pre-commit hook (both ruff lint and ruff-format). The CI pipeline MUST include ruff check . and ruff format --check . as gating steps. See Convention: Python Ruff Standard for the full specification.
  
  Dockerfile CMD (Rails apps): Use CMD ["./bin/rails", "server"] to run Puma directly on port 3000. Do NOT use ./bin/thrust unless the thruster gem is in the Gemfile and bin/thrust binstub exists. The Dockerfile EXPOSE port must match the k8s service port.
  
  database.yml (Rails apps): Production config must use env vars (ENV["POSTGRES_PASSWORD"], not ENV.fetch(...) { raise }). A hard raise crashes CI because Rails evaluates all environment configs through ERB even in test mode. Use ENV.fetch("VAR", "default") for non-secret values and ENV["VAR"] for secrets.
• Activate Woodpecker — GAP: no activate_repo tool exists in woodpecker-mcp. Currently requires manual UI activation (Woodpecker UI → Add repository). Tracked: woodpecker-sdk #6. When the MCP tool ships, this step becomes automated.
• Add Harbor secrets to Woodpecker — use mcp__woodpecker__create_repo_secret to add harbor_username and harbor_password (from tofu output). Do NOT use the UI — the MCP tool handles this.
• Push to trigger pipeline — merge the scaffold PR to main. Woodpecker builds, pushes to Harbor. Use mcp__woodpecker__list_pipelines to verify the pipeline succeeds.
• ArgoCD syncs — Image Updater picks up the new image, writes newTag to the overlay kustomization.yaml, ArgoCD deploys. Verify pod is running before marking complete.

Pre-Deploy Validation Checklist

Reference

var.services Fields

• forgejo_repo (string) — e.g., "forgejo_admin/platform-validation"
• image_repo (string) — e.g., "platform-validation/validator"
• port (number) — container port
• funnel (bool) — whether to create a Tailscale funnel ingress. Set false when the kustomize overlay manages its own ingress with a custom hostname (e.g., public domains that don't match the service key)
• target_revision (string, optional) — git branch ArgoCD watches, defaults to "main"
• source_repo (string, optional) — Forgejo repo for kustomize overlays, e.g., "forgejo_admin/pal-e-deployments". When set, terraform adds write-back-target: kustomization annotation and points ArgoCD at this repo instead of the service repo.
• source_path (string, optional) — path within source_repo to the overlay, e.g., "overlays/pal-e-docs/prod". Defaults to "k8s" if omitted.