pal-e-docs

Phase 16: Alert Tuning & Resource Right-Sizing

phase-platform-16-alert-tuning Phase

completed phase
Project
Goal: Eliminate all false-positive alerts so Telegram alerting channel is signal-only — currently 19 alerts firing on a healthy cluster.
Owner: Dev agent (code changes across 4 repos), Betty Sue (coordination)
Repo: pal-e-platform, pal-e-services, pal-e-deployments, pal-e-docs
Depends on: None

Scope

Five sub-phases covering the full alert cleanup:

16a: Fix Alertmanager Slack URL (pal-e-platform)

COMPLETED — PR #83 merged (pal-e-platform). Slack receiver removed from alertmanager config, variable removed from variables.tf, .woodpecker.yaml, and Makefile. Telegram remains sole receiver. QA nits: orphan Woodpecker secret tf_var_slack_webhook_url (cleanup later).

16b: Bump memory limits — OOMKilled fixes

  • COMPLETED — pal-e-docs: 128Mi → 256Mi (app repo PR #183 + production overlay pal-e-deployments PR #11, Issues #182 + #10 closed)
  • COMPLETED — argocd-image-updater: 128Mi → 256Mi (pal-e-services PR #13 merged, Issue #12 closed)
  • COMPLETED — argocd-application-controller: 512Mi → 1Gi (pal-e-services PR #13 merged, Issue #12 closed)

16c: Remove ServiceMonitors from non-metrics apps (pal-e-deployments)

COMPLETED — PR #9 merged (pal-e-deployments, Issue #8 closed). ServiceMonitors removed from 4 non-metrics overlays (pal-e-app, westside-app, gcal-scheduler, platform-validation). Kept for pal-e-docs and basketball-api via new opt-in bases/servicemonitor/ base. QA nit: dead files bases/standard/hpa.yaml + servicemonitor.yaml remain on disk.

16d: Remove HPAs — single-replica noise (pal-e-deployments)

COMPLETED — PR #9 merged (pal-e-deployments, Issue #8 closed). HPAs removed from all 6 overlays. 94 lines net removed. bases/standard/ slimmed to deployment + service + networkpolicy only.

16e: Investigate backup job + postgres TargetDown

PARTIAL — Failed cnpg-backup-verify-29559420 job deleted (KubeJobFailed alert cleared). Postgres pod restarted but metrics exporter still not listening on port 9187 — this is a CNPG configuration issue, not a stale process. The instance manager (CNPG 1.28.1) doesn't start the embedded metrics exporter despite enablePodMonitor: true. Deeper investigation deferred to TODO. TargetDown alert for postgres will persist until either the exporter is fixed or the PodMonitor is disabled.

Deliverables

  • TBD — filled after completion
  • plan-pal-e-platform — parent plan (Platform Hardening)