pal-e-docs

Terraform Architecture Assessment (2026-02-26)

tf-architecture-assessment-2026-02-26 Doc

architecture active terraform assessment
Project

Terraform Architecture Assessment (2026-02-26)

Executive Summary

Two repos, two state files, one cluster. The split is correct and the for_each service onboarding pattern in pal-e-services is genuinely excellent. But everything is in monolithic main.tf files with no modules, no CI pipeline, no environments, no rollback mechanism, and shared state with no locking strategy for parallel developers. We have a working platform that a solo developer built fast — now we need to harden it for a team.

What We Have Today

pal-e-platform (828 lines, 1 file)

Metric Value
Resources ~25 (6 namespaces, 7 helm releases, 5 funnels, 1 ACL, 1 configmap, 2 buckets, 1 IAM user, 1 IAM policy, 1 IAM attachment)
Providers 4 (kubernetes, helm, tailscale, minio)
Files 5 (<code>main.tf</code>, <code>variables.tf</code>, <code>outputs.tf</code>, <code>providers.tf</code>, <code>versions.tf</code>)
Modules 0
State backend Kubernetes secret (<code>tofu-state</code> namespace)
Environments 1 (production only)
CI/CD None — <code>tofu apply</code> from laptop

pal-e-services (413 lines, 2 files)

Metric Value
Resources ~8 static + 7 per service x N services (currently 4 services = ~36 total)
Providers 4 (kubernetes, helm, harbor, argocd)
Files 6 (<code>main.tf</code>, <code>services.tf</code>, <code>variables.tf</code>, <code>outputs.tf</code>, <code>providers.tf</code>, <code>versions.tf</code>)
Modules 0
State backend Kubernetes secret (<code>tofu-state</code> namespace)
Environments Pseudo-env via service key suffix (<code>basketball-api-dev</code>)
CI/CD None — <code>tofu apply</code> from laptop

What's Good (Keep These)

  • Two-repo separation — Platform infra (slow-changing) vs service onboarding (frequent) is textbook. Different change frequencies, different blast radii.
  • for_each service onboarding — The var.services map-driven pattern in services.tf is genuinely elegant. One tfvars entry provisions 7 resources. This is better than most enterprise setups.
  • Kubernetes state backend — No external dependency (S3, Terraform Cloud). State lives in the cluster it manages. Self-contained.
  • set_sensitive pattern — Secrets passed via tfvars, never in state as plaintext values. The type = "string" workaround is well-documented.
  • Explicit resource limits — Every helm release has resource requests and limits. This is rare and excellent.
  • depends_on clarity — Cross-resource dependencies are explicit and well-commented.
  • Operational comments — The codebase is full of "WHY" comments (e.g., ArgoCD provider two-phase apply, Harbor robot secret behavior). These are invaluable.

What Needs Work (Priority Order)

  • No rollback mechanism — Grafana and pal-e-docs both crashed in 48 hours. Only recovery is manual tofu apply from laptop. See tf-rollback-strategy.
  • No CI pipeline for Terraform — No tofu plan on PR, no tofu apply on merge. See tf-pipeline-design.
  • Monolithic main.tf — 828 lines in one file. No modules. See tf-modularization-roadmap.
  • No environments — Single production cluster. Dev changes go straight to prod. See tf-environment-strategy.
  • No state locking strategy for teams — Kubernetes backend supports locking, but no documented workflow for parallel developers. See tf-team-readiness.
  • Secrets in tfvars on disk — k3s.tfvars contains plaintext passwords. Gitignored but not encrypted. No vault integration.
  • No DORA metrics — No measurement of deployment frequency, lead time, change failure rate, MTTR. See tf-pipeline-design.
  • PostgreSQL not yet managed — Harbor runs its own internal PG. pal-e-docs is moving to PG. No shared PG operator or managed instance. See tf-postgres-strategy.
  • tf-current-filetree — Annotated file tree of both repos
  • tf-modularization-roadmap — Module extraction plan and target file tree
  • tf-pipeline-design — CI/CD pipeline for Terraform + DORA metrics
  • tf-environment-strategy — Dev/prod environment separation
  • tf-rollback-strategy — Rollback mechanisms and disaster recovery
  • tf-team-readiness — What's needed before a second developer runs tofu apply
  • tf-postgres-strategy — PostgreSQL onboarding and shared database strategy
  • tf-best-practices-comparison — Industry best practices vs our approach, with rationale for deviations

The Vision

We are building the internal developer platform — the Datadog/Heroku equivalent for our own services. A developer adds a service entry to tfvars, pushes to Forgejo, and gets: a namespace, CI pipeline, container registry project, GitOps deployment, TLS ingress, monitoring dashboards, log aggregation, and alerting. The Terraform is the control plane for all of this. It needs to be as reliable as the services it deploys.