pal-e-docs

Review: tofu-state backup CronJob failures (2 alerts)

review-224-2026-03-22 Review

review needs-refinement
Project

Verdict: NEEDS_REFINEMENT

Template Completeness

  • [x] Lineage
  • [x] Repo
  • [ ] User Story -- missing (bug template uses "What Broke" instead, which is present)
  • [x] Context (via "What Broke")
  • [ ] File Targets -- missing. The CronJob is defined at terraform/main.tf lines 2222-2308. This should be listed.
  • [x] Acceptance Criteria
  • [ ] Test Expectations -- missing explicit test commands
  • [ ] Constraints -- missing
  • [ ] Checklist -- missing
  • [x] Related

File Targets

  • [x] CronJob verified: kubernetes_cron_job_v1.tf_state_backup at terraform/main.tf:2222 -- confirmed exists
  • [x] Job history verified via kubectl: pattern matches ticket description (alternating Complete/Failed)
  • [x] MinIO bucket/IAM verified: minio_s3_bucket.tf_state_backups, minio_iam_user.tf_backup at main.tf:2123-2160
  • [x] Network policy verified: MinIO netpol at network-policies.tf:112 allows tofu-state namespace ingress
  • [x] Namespace verified: tofu-state namespace exists with correct label

Repo Placement

OK. Filed on pal-e-platform which owns the CronJob Terraform resource.

Dependencies

  • Board item #188 (Issue #109: Platform cleanup -- 15 alerts) is in_progress and this ticket is a child alert from that umbrella. Documented in Related section.
  • No blockers identified. MinIO netpol already allows tofu-state traffic.

Acceptance Criteria

Criteria are reasonable but incomplete:
  • "Next 3 consecutive backup jobs complete" -- verifiable but requires waiting 3 days (daily schedule at 02:00 UTC)
  • "Both KubeJobFailed alerts clear" -- verifiable via Alertmanager
  • "Root cause identified and fixed" -- appropriate for a bug
Missing: how to verify root cause. Failed pod logs are already cleaned up (confirmed: kubectl logs job/tf-state-backup-29568000 returns nothing). The ticket should note this observability gap and suggest adding failedJobsHistoryLimit: 3 to preserve failed pods for debugging (currently set to 3 but pods still get cleaned).

Blast Radius

  • The CronJob downloads mc and kubectl from external URLs on every run. This is a reliability risk -- external CDN outages cause job failures.
  • The backoff_limit is 2, but the ticket says "fails after 1 attempt." The BackoffLimitExceeded after 2 retries with restartPolicy: OnFailure means the container crashed/failed 2 times within the same pod, not that it ran 2 separate pods.
  • Memory limit is 128Mi. Installing apk packages + downloading mc + kubectl + running kubectl commands may exceed this, especially during the apk add phase. This is a likely root cause candidate.
  • No other CronJobs in the cluster use this same pattern, so blast radius is isolated.

Recommendation

Three refinements needed before this is READY:
  • Add File Targets section -- list terraform/main.tf:2222-2308 (CronJob resource) and terraform/network-policies.tf:112 (MinIO netpol tofu-state rule)
  • Add debugging strategy -- failed pod logs are unavailable. Ticket should specify: (a) increase memory limit as first hypothesis, (b) add an init container or pre-built image with mc+kubectl to eliminate external download failures, (c) add explicit error logging before tool downloads
  • Add Test Expectations -- kubectl get jobs -n tofu-state --sort-by=.metadata.creationTimestamp | tail -5 to verify consecutive completions