Bug: 19 alerts from broken/undeployed services (noise floor)
Bug: 19 alerts from broken/undeployed services
Problem
19 of 22 active Alertmanager alerts come from services that are either broken or never fully deployed. 12 from MCP remotes (gmail, linkedin, notion), 4 from westside-app, 3 from basketball-api-dev. Alert noise makes real problems invisible.
Root Cause
Services were onboarded via
var.services (creating namespaces, ServiceMonitors, etc.) but their deployments either have no working image or were never completed. ArgoCD created the deployments from git manifests, but the pods can't start. Prometheus scrapes them, kube-state-metrics reports them unhealthy, and Alertmanager fires alerts nobody receives.Fix
Scale broken deployments to 0 replicas:
kubectl scale deployment <name> -n <ns> --replicas=0. Or delete the deployments entirely if the services aren't needed. For westside-app, this resolves when the app is properly deployed (Issue #6 on westside-app repo).Impact
No service impact (these services are already broken). But the 19 noisy alerts create a 86% false-positive rate that makes the monitoring stack useless for incident detection. Any real alert would be buried.
Acceptance Criteria
- Active alerts reduced to 2 or fewer (Watchdog + any real issues)
- NodeClockNotSynchronising resolved separately
- No KubePodNotReady or KubeDeploymentRolloutStuck alerts from intentionally-offline services
Related
audit-observability-baseline-2026-03-13— discovered during baseline auditphase-observability-3-alerting— parent phase