Review: Bug: platform-validation OOMKilled at 64Mi + stale alert rule

review-388-2026-03-26-v3 Review

review needs-refinement

Verdict: NEEDS_REFINEMENT

Template Completeness

  • [x] Type
  • [x] Lineage
  • [x] Repo (in body, but incomplete -- only lists pal-e-platform)
  • [ ] User Story -- absent from body and all comments
  • [x] Context / What Broke
  • [ ] File Targets -- absent from body; present in v3 correction comment only
  • [x] Acceptance Criteria (body has original; v3 comment has corrected version)
  • [ ] Test Expectations -- absent from body; present in v3 correction comment only
  • [ ] Constraints -- absent (tofu plan -lock=false, ArgoCD sync ordering, two separate PRs)
  • [ ] Checklist -- absent
  • [x] Related
  • [x] Environment
  • [x] Repro Steps

File Targets

  • [x] terraform/main.tf lines 263-274 -- VERIFIED: line 264 alert = "OOMKilled", line 265 expr with kube_pod_container_status_last_terminated_reason, line 266 for = "0m". v3 correction is accurate.
  • [x] overlays/platform-validation/prod/deployment-patch.yaml lines 22-27 (pal-e-deployments) -- VERIFIED: line 26 limits:, line 27 memory: 64Mi. Correct.

Repo Placement

The v3 correction correctly identifies two repos:
  • pal-e-deployments: memory limit fix (kustomize overlay)
  • pal-e-platform: alert rule fix (terraform PrometheusRule)
Issue is filed on pal-e-platform, which is reasonable since the alert rule is the more complex fix. However, the issue body still says only pal-e-platform. An agent reading the body alone would miss the pal-e-deployments work entirely.

Dependencies

  • No board-level blockers. Item #388 is in todo.
  • PR ordering dependency (v3 documents this correctly): pal-e-deployments memory bump must merge first so the rollout clears stale OOM history before the alert rule change takes effect.
  • Related item #387 (CronJob stale failures, issue #170) has a similar alert staleness pattern but is independent.

Acceptance Criteria

The v3 consolidated scope has concrete, verifiable acceptance criteria. Two issues:
  • Test command wrong: v3 says kustomize build overlays/platform-validation/prod/ but standalone kustomize is not installed. Must be kubectl kustomize overlays/platform-validation/prod/.
  • Alert clearance mechanism accurate: v3 correctly explains the rollout from memory bump creates a new pod with clean OOM history, which is what actually clears the stale alert. The for: 15m change is defense-in-depth for future OOMKills.

Blast Radius

  • PodRestartStorm rule (line 252-262): Also uses for = "0m". Same pattern -- fires immediately and may persist on historical data. Not in scope but same class of issue.
  • Other 64Mi services: 10+ services in pal-e-deployments use 64Mi request limits, but only platform-validation has 64Mi as both request AND limit. Static-site services (playground, svelte-playground) are also at 64Mi limits but likely fine for static content.
  • No downstream consumers affected: platform-validation is a standalone health-check service.

Recommendation

One blocking issue remains before this is agent-ready:
  • BLOCKING: Update the issue body to consolidate the v3 correction scope. The body currently says the repo is pal-e-platform only, has no File Targets section, lists invalid PromQL in acceptance criteria, and references wrong line numbers. An agent reading the body would produce incorrect work. Either edit the body to include the v3 consolidated scope, or at minimum add a bold note at the top of the body pointing to the v3 correction comment.
Two minor issues (non-blocking but should be fixed):
  • Test command: Change kustomize build to kubectl kustomize in the v3 correction.
  • Missing template sections: User Story, Constraints, Checklist are absent. These are optional for a bug ticket with clear scope, but Constraints in particular (tofu plan -lock=false, two PRs, ArgoCD sync ordering) would help the dev agent.

v3 Correction Assessment

The v3 correction comment (comment #6) addresses all issues raised by previous reviews:
  • [x] Line numbers corrected: 263-274 (verified against codebase)
  • [x] PR ordering documented: pal-e-deployments first, pal-e-platform second
  • [x] Alert staleness explained: rollout clears metric, for:15m is defense-in-depth
  • [ ] Issue body still stale: corrections live only in comments
  • [ ] Test command uses nonexistent kustomize standalone binary