pal-e-docs

Phase: Observability — Architecture Review

phase-observability-5-architecture Phase

Project

Phase 5: Architecture Review

Goal: Platform architecture is fully documented and interview-ready.
Status: COMPLETED (2026-03-14)

Infrastructure Decision Log

Decision Chosen Over Why
Ingress Tailscale Funnels Traefik / Nginx / cert-manager Zero-config TLS via Tailscale. No cert renewal, no ingress controller overhead. Each service gets a <code>.ts.net</code> hostname automatically. Trade-off: public access requires funnel annotation (opt-in security).
Container Registry Harbor (self-hosted) Docker Hub / GHCR Sovereign supply chain. Trivy vulnerability scanning built-in. Robot accounts for per-repo RBAC. No rate limits, no external dependency. Phase 10 enabled Trivy scanning.
CI/CD Woodpecker CI GitHub Actions / Drone / Jenkins Forgejo-native OAuth. Lightweight (128Mi server). K8s backend spawns ephemeral pipeline pods. YAML pipeline syntax. Open source, self-hosted. Known limitation: SQLite log streaming bug (#4409) — Postgres migration in progress (PR #59).
GitOps ArgoCD + Image Updater Flux / manual kubectl App-of-apps pattern. Image Updater watches Harbor for newest tags, writes <code>.argocd-source</code> annotations — no human in the deploy loop. Self-heal reverts manual kubectl changes. Trade-off: CRD-heavy (~500 CRDs).
Postgres CNPG (CloudNativePG) Standalone Postgres / PGO / Zalando Kubernetes-native operator. Automated WAL archiving to MinIO (barman). Declarative Cluster CR. PodMonitor integration with Prometheus. Single-binary operator, minimal footprint (256Mi). pgvector support for embeddings (pal-e-docs semantic search).
Observability Stack kube-prometheus-stack + Loki Datadog / New Relic / custom Full DORA metrics pipeline. Prometheus scrapes ServiceMonitors + PodMonitors. Grafana dashboards via ConfigMap sidecar. Loki for log aggregation. Alertmanager routes to Telegram + Slack. All self-hosted, zero SaaS cost.
IaC OpenTofu Terraform / Pulumi / Ansible BSL-free Terraform fork. Identical HCL syntax. Helm provider deploys charts. Kubernetes provider manages raw manifests. State stored in k8s secrets (tofu-state namespace). Daily backup to MinIO.
Object Storage MinIO (self-hosted) AWS S3 / Backblaze B2 S3-compatible API. Three buckets: postgres-wal (CNPG backups), tf-state-backups (TF state), harbor (registry blobs via redirect). IAM users with scoped policies. Sovereign — no cloud dependency.
Host Management SaltStack Ansible / manual SSH Declarative pillar-based config. GPG-encrypted secrets in git. Two control planes: Salt for host (packages, services, GPU drivers), Terraform for cluster (namespaces, Helm, CRDs). Salt handles what k8s can't: NVIDIA drivers, NVMe mounts, system packages.
Secrets Management Salt GPG + SOPS/Age Vault / Sealed Secrets / External Secrets Salt pillar GPG encrypts TF vars at rest — <code>make tofu-secrets</code> renders to <code>secrets.auto.tfvars</code>. SOPS/Age for app-level k8s secrets — ArgoCD CMP sidecar decrypts at sync time. No external service dependency. 15 secrets in Salt pipeline, Age pubkey for SOPS.
Auth Keycloak Auth0 / Firebase Auth / custom JWT Self-hosted OIDC IdP. Realm-based multi-tenancy. JWKS endpoint for API validation. Auth.js integration for SvelteKit apps. Start-dev mode with H2 file persistence (appropriate for &lt;100 users). Upgrade path to Postgres when needed.
GPU/AI NVIDIA Device Plugin + Ollama vLLM / standalone GPU passthrough nvidia-device-plugin DaemonSet exposes GPU to k8s scheduler. Ollama Helm chart for local LLM inference (embedding models for semantic search). RuntimeClass nvidia. pgvector stores embeddings in Postgres.

Production Patterns

GitOps Pipeline

Immutable Images

Every build produces a unique image tag (CI_COMMIT_SHA). No :latest in production. Harbor stores all versions. ArgoCD Image Updater uses newest-build strategy to always deploy the most recent SHA.

Infrastructure as Code

Two control planes, zero manual kubectl:
  • OpenTofu — 13 Helm releases, 19 Tailscale funnels, CNPG clusters, MinIO buckets, IAM policies, CronJobs, Secrets, ConfigMaps. CI runs tofu plan on PR, tofu apply on merge.
  • SaltStack — NVIDIA drivers, NVMe RAID, system packages, GPG key management. Pillar-encrypted secrets. salt '*' state.apply from control node.

Secret Management Pipeline

Observability Architecture

Metrics Pipeline

Logs Pipeline

Alerting Pipeline

Resource Utilization Snapshot (2026-03-14)

Metric Value Assessment
Node archbox (single node k3s) Production-grade hardware
CPU 16% (1964m / 12 cores) Healthy headroom
Memory 19% (25.5Gi / 128Gi) Significant headroom
Namespaces 29 Well-organized isolation
Running Pods 69 Stable
Helm Releases 13 All deployed status
Tailscale Funnels 19 Each service externally accessible
CNPG Clusters 1 (pal-e-postgres) + 1 pending (woodpecker-db) Postgres migration in progress
Woodpecker Repos 28 activated All repos with CI
CI Secrets 6 global + ~30 repo-level Salt pipeline manages TF vars

Incident Log

Woodpecker TLS Clone Failure (2026-03-14)

Symptom: Pipeline clones failed with TLS EOF errors. Root cause: Woodpecker used external Forgejo URL (https://forgejo.tail5b443a.ts.net) which traverses Tailscale funnel TLS from within the cluster. Fix: Changed WOODPECKER_FORGEJO_URL to internal service URL (http://forgejo-http.forgejo.svc.cluster.local:80). PR #56. Lesson: In-cluster traffic should never exit to external URLs.

Woodpecker SQLite Log Streaming Bug (ongoing)

Symptom: All pipeline logs empty in UI/API. queue.Done: cannot ack workflow + stream: not found in server logs. Root cause: Upstream bug #4409 — K8s backend log streaming fails with SQLite. Logs ARE stored but can't be retrieved. Fix: Postgres migration (PR #59). Workaround: Direct SQLite query on PVC.

State Lock Contention (2026-03-14)

Symptom: CI tofu plan locks state, blocking local tofu apply or concurrent CI runs. Fix: Added -lock=false to plan step (PR #58). Lesson: Read-only operations should never acquire write locks.

Grafana 502 Gateway Error (2026-03-08)

Symptom: Grafana dashboard 502 after kube-prometheus-stack upgrade. Root cause: HPA scaled Grafana replicas, but local-path PVC only supports ReadWriteOnce — second replica couldn't mount. Fix: Disabled HPA, single replica. Lesson: local-path storage doesn't support multi-replica stateful workloads.

DORA Exporter OOM (2026-03-14)

Symptom: dora-exporter pod OOMKilled. Root cause: Memory limit too low for Python process with Forgejo API calls. Fix: Increased to 256Mi limit (PR #54). Lesson: Python processes need more memory headroom than Go equivalents.

Telegram chat_id Type Mismatch (2026-03-14)

Symptom: Alertmanager Telegram notifications failing. Root cause: Helm chart expects chat_id as string, but numeric value was being passed. Fix: Wrapped in quotes (PR #54). Lesson: Always check Helm value types against chart schema.

Known Non-Running Workloads (2026-03-14)

  • 3x MCP remote pods in ImagePullBackOff (gmail, linkedin, notion) — pal-e-services scope, images never pushed to Harbor
  • basketball-api-dev — missing secret, app scope
  • palworld-client — Unknown state, game streaming experiment (not production)