pal-e-docs

Architecture: Kubernetes Deployment

arch-k8s-deploy Architecture

architecture active k8s prediction-assistant
Project

Kubernetes Deployment: prediction-assistant

How prediction-assistant is deployed to the k3s cluster. Kustomize overlays define environment-specific configuration, Woodpecker CI builds and pushes images to Harbor, and ArgoCD syncs manifests to the cluster. Tailscale funnel + Traefik ingress handle HTTPS routing.

Diagram




          
  
Directory Structure



          
  



          
  
Components



          
  

    

        
          

              Component
              Purpose
              Notes
          

        
      
          

              Base Kustomization
              Shared resources for all environments
              Sets namespace <code>prediction-assistant</code>, common labels (<code>app.kubernetes.io/name</code>, <code>managed-by</code>)
          

          

              Web Deployment
              Rails application server
              Image from Harbor, init container runs <code>db:prepare</code> before server start, liveness/readiness on <code>/up</code>
          

          

              Worker Deployment
              Solid Queue background job processor
              Same image, runs <code>bin/rails solid_queue:start</code>, liveness via <code>pgrep -f solid_queue</code>
          

          

              Service
              ClusterIP routing to web pods
              Port 80 → targetPort 3000
          

          

              ConfigMap
              Non-secret environment variables
              RAILS_ENV=production, RAILS_LOG_TO_STDOUT=true, RAILS_MAX_THREADS=3, PORT=3000
          

          

              Secrets (external)
              Sensitive credentials
              <code>prediction-assistant-secrets</code>: DATABASE_URL, SECRET_KEY_BASE, KALSHI_API_KEY_ID, KALSHI_PRIVATE_KEY (API keys optional)
          

      
    

  



          
  
Image Build & Push



          
    
Woodpecker CI (.woodpecker.yaml) handles the build pipeline:



          
  
    
      
  • Test step: Runs bin/rails test against a Postgres 16 sidecar service.
    • 
      
  • Build-push step (main branch only): Uses woodpeckerci/plugin-docker-buildx to build a multi-stage Docker image and push to Harbor.
    • 



          
    
Tags pushed: ${CI_COMMIT_SHA} and latest. Registry: harbor.tail5b443a.ts.net/ldraney/prediction-assistant.



          
    
The Dockerfile uses a two-stage build: build stage (ruby:3.4.9-slim) installs gems, precompiles bootsnap + assets; runtime stage copies artifacts into a minimal image running as non-root user rails (UID 1000).



          
  
Kustomize Overlays



          
  

    

        
          

              Setting
              Dev
              Prod
          

        
      
          

              Web replicas
              1
              2
          

          

              Worker replicas
              1
              1
          

          

              Web CPU request/limit
              50m / 250m
              200m / 1 core
          

          

              Web memory request/limit
              128Mi / 256Mi
              512Mi / 1Gi
          

          

              Worker CPU request/limit
              50m / 250m
              200m / 1 core
          

          

              Worker memory request/limit
              128Mi / 256Mi
              512Mi / 1Gi
          

          

              RAILS_LOG_LEVEL
              debug
              info
          

          

              Ingress / TLS
              none
              Traefik + Let's Encrypt
          

      
    

  



          
  
ArgoCD Sync Pattern



          
    
ArgoCD watches the repo's k8s/overlays/prod path (registered via the argocd-app Terraform module in pal-e-services). On each merge to main:



          
  
    
      
  • Woodpecker CI builds and pushes a new image tagged with the commit SHA + latest.
    • 
      
  • ArgoCD detects manifest changes and syncs the Kustomize overlay.
    • 
      
  • Kubernetes pulls the updated image from Harbor using the robot-account pull secret provisioned by the harbor-robot Terraform module.
    • 



          
  
Ingress & TLS



          
    
Production ingress chain:



          
  
    
      
  • Tailscale Funnel — Public HTTPS entrypoint provisioned by the tailscale-funnel Terraform module. Maps external traffic into the Tailscale network.
    • 
      
  • Traefik Ingress Controller — Routes prediction-assistant.com to the prediction-assistant-web ClusterIP service.
    • 
      
  • cert-manager ClusterIssuer — letsencrypt-prod issues TLS certificates via HTTP-01 challenge through Traefik.
    • 
      
  • Redirect Middleware — Traefik middleware forces HTTP → HTTPS with a permanent redirect.
    • 



          
    
DNS (managed in pal-e-platform via the godaddy-dns Terraform module) points prediction-assistant.com to the Tailscale funnel address.



          
  
Monitoring



          
    
The k8s/monitoring/ directory defines observability resources:



          
  
    
      
  • ServiceMonitor: Prometheus scrapes /metrics every 30s from pods in the prediction-assistant namespace.
    • 
      
  • PrometheusRule: Golden-signal alerts — HTTP error rate > 5%, p95 latency > 2s, pod not ready for 5m, Solid Queue failures or backlog > 100 jobs.
    • 
      
  • Grafana dashboard: ConfigMap-provisioned dashboard for at-a-glance service health.
    • 



          
  
Key Decisions



          
  
    
      
  • Kustomize over Helm: The app's configuration is simple enough that Kustomize overlays (base + patches) keep things readable without Helm templating complexity.
    • 
      
  • In-repo manifests: k8s manifests live alongside application code in the k8s/ directory. Changes to infrastructure and code ship together in the same PR.
    • 
      
  • Init container for migrations: The web deployment runs db:prepare in an init container, ensuring schema is ready before the server starts. Only the web pod runs migrations; the worker relies on web readiness.
    • 
      
  • Separate web and worker deployments: Allows independent scaling and resource allocation. The worker runs Solid Queue for background jobs (market scanning, order execution).
    • 
      
  • Harbor robot accounts: Pull secrets are provisioned by Terraform (pal-e-services), not manually created. Scoped to the prediction-assistant Harbor project.
    • 
      
  • Tailscale funnel over NodePort/LoadBalancer: No cloud load balancer needed on bare-metal k3s. Tailscale funnel provides a stable public entrypoint with automatic TLS.
    • 
      
  • Non-root container: Runtime image runs as UID 1000 (rails user) for security best practice.
    • 
      
  • Docker buildx over kaniko: Woodpecker CI uses plugin-docker-buildx for multi-stage builds. Tags both :SHA and :latest.
    •