pal-e-docs

Review: ArgoCD CMP sidecar fails to render kustomize+SOPS overlays

review-525-2026-03-29 Doc

review needs-refinement
Project

Verdict: NEEDS_REFINEMENT

Template Completeness

  • [x] Type -- Feature
  • [x] Lineage -- board-pal-e-platform, story and arch references
  • [x] Repo -- listed as pal-e-platform and pal-e-services (but see Repo Placement)
  • [x] User Story -- clear "superuser needs CMP sidecar to render overlays"
  • [x] Context -- detailed error output, affected apps listed
  • [x] File Targets -- present but inaccurate (see below)
  • [x] Acceptance Criteria -- 3 testable conditions
  • [x] Test Expectations -- pod status and tofu plan verification
  • [x] Constraints -- age key path, no disruption to existing apps
  • [x] Checklist -- 4 discrete steps
  • [x] Related -- blocking items documented
All template sections are present. No structural gaps.

Traceability

  • [x] story:superuser-deploy -- present on board item
  • [x] arch:argocd -- present on board item
  • [x] arch:k8s-deploy -- present on board item
  • [x] Forgejo issue -- forgejo_admin/pal-e-platform#225, open
Traceability triangle is complete.

File Targets

  • [ ] terraform/modules/ci/ -- ISSUE: This directory in pal-e-platform contains Woodpecker CI configuration only. No ArgoCD content exists here. The ArgoCD Helm release (including the CMP sidecar definition) lives in pal-e-services/terraform/main.tf lines 35-191.
  • [ ] "ArgoCD repo-server pod spec (sidecar resources, volume mounts)" -- ISSUE: Vague descriptor, not an actionable file path. The actual targets are:
File targets do not match reality. An agent dispatched with these targets would search the wrong directory in the wrong repo.

Repo Placement

MISMATCH. The Forgejo issue is filed on forgejo_admin/pal-e-platform, but the ArgoCD Helm release with the CMP sidecar config lives entirely in pal-e-services/terraform/main.tf. The issue body says "pal-e-platform (ArgoCD Helm config), pal-e-services (ArgoCD app definitions)" -- this is inverted. ArgoCD Helm config (including CMP sidecar) is in pal-e-services, not pal-e-platform. pal-e-platform has zero ArgoCD Helm resources. The fix is scoped to pal-e-services only.

Dependencies

  • [x] Board item #521 "Apply 5+ pending terraform changes (ArgoCD migrations)" (column: todo) -- this item is BLOCKED by #525. The CMP sidecar must work before ArgoCD app migrations can apply. Documented in issue Related section.
  • [ ] Board item #515 "Validate: pal-e-deployments (k8s API unreachable)" (column: backlog) -- related k8s-deploy work, same arch labels. Not documented as a dependency.
  • [ ] Board item #191 "ArgoCD repo-server memory bump" (column: done) -- prior OOMKill fix for repo-server. Relevant if root cause is memory again. Not documented.
Primary dependency (#521) is documented. Note that #521 has label depends:#224 but should likely also reference #225.

Acceptance Criteria

  • [x] "tofu apply for pal-e-services succeeds for all ArgoCD apps" -- testable via tofu plan -lock=false showing 0 changes post-apply
  • [x] "ArgoCD dashboard shows all apps Synced" -- testable via ArgoCD UI or argocd app list
  • [x] "CMP sidecar logs show successful manifest generation" -- testable via kubectl logs -n argocd -l app.kubernetes.io/component=repo-server -c cmp-sops
All three criteria are verifiable. Specific kubectl and tofu commands should be added to the issue for agent clarity, but the criteria themselves are adequate.

Blast Radius

CMP sidecar config is isolated to pal-e-services/terraform/main.tf. All 10 services using pal-e-deployments overlays with .enc.yaml files are affected (pal-e-docs, gcal-scheduler, basketball-api, westside, mcd-tracker, etc.). The discover.fileName = "*.enc.yaml" triggers the CMP plugin for any overlay with encrypted secrets. Fixing the sidecar will impact all ArgoCD-managed apps simultaneously -- this is the desired outcome.
The init container downloads sops and age from GitHub on every pod restart. If GitHub is unreachable, init fails and CMP sidecar has no tools. This is a known fragility but out of scope for this ticket.
Rollback is straightforward: revert Helm values change via tofu apply.

Decomposition Assessment

File count: 1 file (pal-e-services/terraform/main.tf). AC count: 3. Discrete changes: likely 1-2 (resource limits or config fix). Estimated agent time: under 5 minutes for investigation + fix. No decomposition needed. This is a single-file investigation and fix that fits well within the three-thing limit and five-minute rule.

Recommendation

  • [BODY] Fix file targets: replace terraform/modules/ci/ with pal-e-services/terraform/main.tf (lines 35-191, specifically cmp-sops container at lines 114-167, init container at lines 99-113, plugin config at lines 51-76).
  • [BODY] Fix repo description: ArgoCD Helm config is in pal-e-services, not pal-e-platform. Correct the Repo section to read "pal-e-services (ArgoCD Helm config + CMP sidecar), pal-e-deployments (kustomize overlays consumed by ArgoCD)".
  • [SCOPE] Consider moving the Forgejo issue from pal-e-platform to pal-e-services, since all code changes will be in that repo. Alternatively, keep it on pal-e-platform as a cross-cutting platform concern but ensure the agent prompt specifies the pal-e-services working directory.
  • [LABEL] Consider changing type:feature to type:bug -- CMP sidecar fails to render (broken behavior), not new functionality.
  • [BODY] Add explicit investigation commands to Checklist: kubectl describe pod -n argocd -l app.kubernetes.io/component=repo-server and kubectl logs -n argocd -l app.kubernetes.io/component=repo-server -c cmp-sops and kubectl logs -n argocd -l app.kubernetes.io/component=repo-server -c install-sops.