pal-e-docs

Review: Woodpecker server log noise: orphaned queue.Done / stream errors

review-631-2026-03-28 Doc

review needs-refinement
Project

Verdict: NEEDS_REFINEMENT

Template Completeness

  • [x] Type — Bug
  • [x] Lineage — traced to #184 (Harbor connectivity timeout resolution)
  • [x] Repo — forgejo_admin/pal-e-platform
  • [ ] User Story — no explicit "As a..." statement. Fix Path describes the what, not the who/why.
  • [x] Context — What Broke section provides error messages and frequency (~5/hr)
  • [ ] File Targets — none specified. Fix is operational (server restart / DB vacuum), not a code change.
  • [x] Acceptance Criteria — 2 criteria present
  • [ ] Test Expectations — no specific verification commands beyond the repro grep. No post-fix assertion.
  • [x] Constraints — "Do NOT restart during active pipeline runs" + verify no stuck workflows
  • [ ] Checklist — no discrete execution steps. Fix Path is vague ("restart or DB vacuum").
  • [x] Related — parent incident #184 linked

Traceability

  • [x] story:superuser-deploy — present on board item #631
  • [x] arch:ci-pipeline — present on board item #631
  • [x] Forgejo issue — forgejo_admin/pal-e-platform#241, open

File Targets

No file targets specified. The issue describes an operational fix (server restart or DB vacuum), not a code change. Verified that Woodpecker is deployed via Helm at terraform/modules/ci/main.tf (line 182), chart version 3.5.1, backed by CNPG Postgres cluster woodpecker-db in the woodpecker namespace. No terraform or Helm config changes are implied by the fix path.
Assessment: an agent cannot act on this ticket without knowing whether the fix is kubectl commands, SQL queries, or a Helm values change. Targets are insufficiently specific.

Repo Placement

OK — issue filed on forgejo_admin/pal-e-platform, which owns the Woodpecker Helm deployment. No cross-repo concerns. Single namespace (woodpecker), single service.

Dependencies

  • [x] #184 (Harbor connectivity timeout) — satisfied, in done (board item #411)
  • [x] No items in in_progress or next_up block this ticket
  • [x] No active pipelines constraint — runtime dependency, not a board dependency
No unresolved dependencies blocking execution.

Acceptance Criteria

Two criteria from the issue:
  • "Woodpecker server logs clean of orphaned queue/stream errors" — testable via kubectl logs -n woodpecker deploy/woodpecker-server --tail=200 | grep -c "no rows", but needs a wait period after fix to confirm errors stopped (not just that the log buffer was flushed by restart).
  • "Pipelines still succeed after fix" — testable by triggering a pipeline, but no specific pipeline or repo is identified for the smoke test.
Issues: AC #1 needs a time window (e.g., "zero errors for 1 hour post-fix"). AC #2 needs a specific test pipeline reference. Both are verifiable in principle but underspecified for agent execution.

Blast Radius

  • Files/repos touched: 0 code files. Operational fix only.
  • What could break: Woodpecker server restart kills in-flight pipelines if constraint is violated. DB vacuum could remove legitimate records if query is wrong.
  • Rollback: server restart is self-healing (pod restarts automatically). DB changes are harder to roll back — needs a backup-first step.
Overall blast radius is low. Errors are cosmetic — pipelines succeed despite log noise. Risk is contained to the woodpecker namespace.

Decomposition Assessment

Apply the three-thing limit and five-minute rule:
  • Discrete changes: 1 (operational fix — restart or vacuum). Does NOT exceed 3.
  • Estimated agent time: under 5 minutes if the procedure is specified. Currently unspecified.
  • Independent subtasks: none. This is a single atomic operation.
No decomposition needed. The ticket is right-sized for a single agent pass once the fix path is clarified.

Recommendation

  • [BODY] Replace the vague "Fix Path" section with a concrete Checklist of execution steps. Specify: (a) diagnostic query to run against woodpecker-db to confirm orphaned records exist, (b) the exact fix operation (restart vs. SQL cleanup vs. both), (c) post-fix verification command with a time window.
  • [BODY] Add File Targets or explicitly mark as "Operational — no file changes" so agents know this is a kubectl/SQL task, not a code PR.
  • [BODY] Add Test Expectations: specify which pipeline to trigger for the smoke test and the expected log output after fix.
  • [SCOPE] Needs human decision: is this a one-time manual cleanup (not agent-dispatchable in the normal sense) or should a Helm values / config change prevent recurrence? If one-time, consider executing manually and closing rather than routing through the agent pipeline.