Review: Bug: platform-validation OOMKilled at 64Mi + stale alert rule
Verdict: NEEDS_REFINEMENT
Template Completeness
- [x] Type
- [x] Lineage
- [x] Repo (in body, but incomplete -- only lists pal-e-platform)
- [ ] User Story -- absent from body and all comments
- [x] Context / What Broke
- [ ] File Targets -- absent from body; present in v3 correction comment only
- [x] Acceptance Criteria (body has original; v3 comment has corrected version)
- [ ] Test Expectations -- absent from body; present in v3 correction comment only
- [ ] Constraints -- absent (tofu plan -lock=false, ArgoCD sync ordering, two separate PRs)
- [ ] Checklist -- absent
- [x] Related
- [x] Environment
- [x] Repro Steps
File Targets
- [x]
terraform/main.tflines 263-274 -- VERIFIED: line 264alert = "OOMKilled", line 265 expr withkube_pod_container_status_last_terminated_reason, line 266for = "0m". v3 correction is accurate. - [x]
overlays/platform-validation/prod/deployment-patch.yamllines 22-27 (pal-e-deployments) -- VERIFIED: line 26limits:, line 27memory: 64Mi. Correct.
Repo Placement
The v3 correction correctly identifies two repos:
- pal-e-deployments: memory limit fix (kustomize overlay)
- pal-e-platform: alert rule fix (terraform PrometheusRule)
Issue is filed on pal-e-platform, which is reasonable since the alert rule is the more complex fix. However, the issue body still says only
pal-e-platform. An agent reading the body alone would miss the pal-e-deployments work entirely.Dependencies
- No board-level blockers. Item #388 is in
todo. - PR ordering dependency (v3 documents this correctly): pal-e-deployments memory bump must merge first so the rollout clears stale OOM history before the alert rule change takes effect.
- Related item #387 (CronJob stale failures, issue #170) has a similar alert staleness pattern but is independent.
Acceptance Criteria
The v3 consolidated scope has concrete, verifiable acceptance criteria. Two issues:
- Test command wrong: v3 says
kustomize build overlays/platform-validation/prod/but standalonekustomizeis not installed. Must bekubectl kustomize overlays/platform-validation/prod/. - Alert clearance mechanism accurate: v3 correctly explains the rollout from memory bump creates a new pod with clean OOM history, which is what actually clears the stale alert. The
for: 15mchange is defense-in-depth for future OOMKills.
Blast Radius
- PodRestartStorm rule (line 252-262): Also uses
for = "0m". Same pattern -- fires immediately and may persist on historical data. Not in scope but same class of issue. - Other 64Mi services: 10+ services in pal-e-deployments use 64Mi request limits, but only platform-validation has 64Mi as both request AND limit. Static-site services (playground, svelte-playground) are also at 64Mi limits but likely fine for static content.
- No downstream consumers affected: platform-validation is a standalone health-check service.
Recommendation
One blocking issue remains before this is agent-ready:
- BLOCKING: Update the issue body to consolidate the v3 correction scope. The body currently says the repo is
pal-e-platformonly, has no File Targets section, lists invalid PromQL in acceptance criteria, and references wrong line numbers. An agent reading the body would produce incorrect work. Either edit the body to include the v3 consolidated scope, or at minimum add a bold note at the top of the body pointing to the v3 correction comment.
Two minor issues (non-blocking but should be fixed):
- Test command: Change
kustomize buildtokubectl kustomizein the v3 correction. - Missing template sections: User Story, Constraints, Checklist are absent. These are optional for a bug ticket with clear scope, but Constraints in particular (tofu plan -lock=false, two PRs, ArgoCD sync ordering) would help the dev agent.
v3 Correction Assessment
The v3 correction comment (comment #6) addresses all issues raised by previous reviews:
- [x] Line numbers corrected: 263-274 (verified against codebase)
- [x] PR ordering documented: pal-e-deployments first, pal-e-platform second
- [x] Alert staleness explained: rollout clears metric, for:15m is defense-in-depth
- [ ] Issue body still stale: corrections live only in comments
- [ ] Test command uses nonexistent
kustomizestandalone binary