pal-e-docs

Observability: prediction-assistant

arch-observability Architecture

architecture
Project

Observability: kalshi-assistant

Monitoring, alerting, and dashboards for kalshi-assistant. All observability resources are configured in pal-e-platform under terraform/modules/monitoring/main.tf, following the patterns established by existing services.

Diagram




          
  
Components



          
  

    

        
          

              Component
              Purpose
              Notes
          

        
      
          

              ServiceMonitor
              Tells Prometheus to scrape kalshi-assistant's /metrics endpoint
              Deployed via Kustomize prod overlay; matches service labels
          

          

              PrometheusRules
              Alert rules for golden signals (latency, traffic, errors, saturation)
              Defined in pal-e-platform monitoring module; follows existing service patterns
          

          

              AlertManager
              Routes and deduplicates alerts
              Shared cluster-wide; per-service routing via labels
          

          

              blackbox-exporter
              External HTTP probes for uptime monitoring
              Probes the public Tailscale funnel URL; alerts on non-2xx responses
          

          

              Grafana dashboards
              Golden signals dashboard for kalshi-assistant
              Request rate, error rate, latency percentiles, resource utilization
          

          

              monitoring/main.tf
              Terraform that provisions all monitoring resources
              Adding kalshi-assistant means adding a service block following the existing pattern
          

      
    

  



          
  
Key Decisions



          
  
    
      
  • Golden signals framework: Every service dashboard tracks the four golden signals (latency, traffic, errors, saturation). This standardization means any team member can read any service dashboard without learning a new layout.
    • 
      
  • Monitoring as code in pal-e-platform: Alert rules, blackbox targets, and dashboards are Terraform resources, not manual Grafana/Prometheus config. Changes are reviewed in PRs and applied deterministically.
    • 
      
  • Blackbox probes for external perspective: In addition to internal metrics scraping, blackbox-exporter probes the public URL. This catches issues that internal metrics miss (DNS, Tailscale funnel, TLS).
    • 
      
  • Follow existing service patterns: kalshi-assistant's monitoring block mirrors the structure used by other services in main.tf. Copy-paste-adapt, not reinvent.
    • 
      
  • ServiceMonitor in Kustomize, rules in Terraform: The scrape target (ServiceMonitor) lives with the deployment manifests. The alert rules and dashboards live in the platform repo. This split matches ownership: app team owns what to expose, platform owns how to alert on it.
    •