pal-e-docs

Review: CronJob stale failures causing persistent KubeJobFailed alerts

review-387-2026-03-26 Review

review ready
Project

Verdict: READY

Template Completeness

  • [x] Lineage — standalone, discovered during AlertManager triage 2026-03-26
  • [x] Repo — forgejo_admin/pal-e-platform
  • [ ] User Story — missing formal format, but "What Broke" section covers the bug context adequately
  • [x] Context — covered in "What Broke" and scope correction comment
  • [x] File Targets — added in scope correction: terraform/main.tf lines ~2288-2381 and ~2387-2514
  • [x] Acceptance Criteria — present and updated in scope correction (4 criteria, all verifiable)
  • [ ] Test Expectations — missing section, but AC includes tofu plan verification which is sufficient
  • [ ] Constraints — missing section (minor for a 2-line value change)
  • [ ] Checklist — missing section (minor)
  • [x] Related — project, user story, and arch component documented

File Targets

  • [x] terraform/main.tf:2288 — verified: kubernetes_cron_job_v1.tf_state_backup resource exists, failed_jobs_history_limit = 3 at line 2297
  • [x] terraform/main.tf:2387 — verified: kubernetes_cron_job_v1.cnpg_backup_verify resource exists, failed_jobs_history_limit = 3 at line 2396
  • [x] No other CronJobs in pal-e-platform — confirmed only 2 kubernetes_cron_job_v1 resources in main.tf
  • [x] daily-reboot confirmed in palworld-server repo (Helm chart values), correctly descoped

Repo Placement

OK. Issue filed on forgejo_admin/pal-e-platform, both target CronJobs are kubernetes_cron_job_v1 resources in terraform/main.tf of that repo. The daily-reboot CronJob lives in palworld-server (Helm-managed) and was correctly dropped from scope in the correction comment.

Dependencies

No active dependencies. Related completed item: board #224 / Forgejo #123 (tofu-state backup CronJob failures) — that fixed the backup job logic itself. This ticket cleans up the stale alert noise left behind. No blockers in in_progress or next_up.

Acceptance Criteria

All 4 criteria are agent-verifiable:
  • failedJobsHistoryLimit lowered to 2 — verifiable via tofu plan -lock=false showing exactly 2 in-place updates
  • 6 stale failed jobs deleted — verifiable via kubectl get jobs -A --field-selector=status.successful=0
  • KubeJobFailed alerts clear — verifiable via AlertManager API query
  • tofu plan shows only the 2 history limit changes — verifiable
Note: The manual kubectl delete job step is included in the scope correction with exact job names. Agent can execute this.

Blast Radius

Minimal. Only 2 integer value changes (3 → 2) on CronJob specs. No failed_jobs_history_limit found in pal-e-services. The KubeJobFailed alert is a built-in kube-prometheus-stack rule (kube_job_failed > 0) — not custom, so no alert rule modifications needed. The daily-reboot CronJob in palworld already has limit=1, no action needed there.

Recommendation

No action needed. Scope is solid after the correction comment. The ticket is agent-ready as a 2-line Terraform value change plus a manual cleanup step. Missing template sections (User Story, Test Expectations, Constraints, Checklist) are acceptable given the minimal scope.