TF: Environment Strategy (Dev/Prod)

tf-environment-strategy Doc

architecture terraform roadmap

Project

Terraform Environment Strategy

Current State: Single Production Cluster

Everything runs in one k3s cluster on archbox. There is no dev environment. The closest we have is the basketball-api-dev service key pattern in pal-e-services, which creates a separate namespace but shares the same cluster, monitoring, and networking.

Industry Best Practice

The textbook answer is "separate clusters per environment" with identical Terraform, different tfvars:

Or: Terragrunt/Terramate for DRY environment configs.

Our Reality: Why Full Separation Is Overkill (For Now)

One physical machine. A second k3s cluster on the same box gives isolation but not redundancy.
Cost. Running 2x of every Helm chart doubles memory usage on a single machine (~8GB current, ~16GB with duplication).
Complexity tax. Two clusters = two kubeconfigs, two Tailscale setups, two state files per repo, double the maintenance.

Our Approach: Namespace-Level Dev/Prod

We already do this implicitly with basketball-api vs basketball-api-dev. Formalize it:

Service-level environments (pal-e-services)

The var.services map already supports this. Add a convention:

Each dev variant gets its own: namespace, Harbor project, ArgoCD app, funnel URL. Same underlying repo, different branch.

Platform-level environments

Platform components (Forgejo, Harbor, Prometheus) do NOT get dev variants. They are shared infrastructure. Dev services run alongside prod services in the same cluster, hitting the same Harbor, same Forgejo, same monitoring.

When To Add a Real Dev Cluster

When any of these become true:

A second physical machine is available (or cloud VPS for dev)
A team member needs to test platform changes without affecting prod
Compliance requires environment isolation
We need to test k3s upgrades before applying to prod

At that point, the modularized Terraform (see tf-modularization-roadmap) makes this a tfvars-only change.

Terraform Workspace Alternative

Workspaces are NOT recommended for environment separation. They share the same backend config and can lead to accidental cross-environment applies. The tfvars-per-environment pattern with separate state files is safer.

Progressive Strategy

Now: Namespace-level dev/prod via service key convention in var.services
Next: Add environment field to var.services type, use it for labeling, resource limits, replica counts
Later: Second cluster for platform dev (testing Helm upgrades, Terraform changes before prod)