TODO: Token Metrics -- Correlate Token Usage with DORA and Sprints
TODO: Token Metrics -- Correlate Token Usage with DORA and Sprints
Empirical Calibration (2026-03-10 basketball-api session)
Key insight: Tokens are to AI agents what hours are to human developers — the unit of cost and effort. Sprint points traditionally map to time (1=half day, 3=2-3 days, 5=full week). For an AI workforce, they should map to tokens. First empirical data from a real session:
| Agent Task | Tokens | Tool Uses | Wall Time |
|---|---|---|---|
| Dev: single endpoint (Phase 2a) | 71K | 62 | 5 min |
| Dev: fix 2 QA blockers (Phase 3a) | 93K | 59 | 6 min |
| Dev: rebase + conflict resolve | 53K | 44 | 4 min |
| QA: full PR review | 38-73K | 19-54 | 2-5 min |
| QA: re-review after fix | 48K | 20 | 1.5 min |
| Full issue cycle (dev + QA + fix + re-QA) | ~150-250K | — | ~15-20 min |
| Points | Human Time | Token Equivalent | What It Looks Like |
|---|---|---|---|
| 1 | Half day | ~100K | Single fix, one agent cycle, no QA surprises |
| 2 | Full day | ~200K | One feature PR, dev + QA + fix cycle |
| 3 | 2-3 days | ~400K | Feature with rebase/conflict, 2 QA rounds |
| 5 | Full week | ~800K | Multiple PRs, plan updates, coordination |
| 8 | Full sprint | ~1.5M | Multi-phase, multiple agents in parallel |
| 13 | Multi-sprint | 3M+ | Cross-repo, architectural changes |
Next step: Instrument agent spawns with issue number tagging and token capture. Simplest path is a Claude Code hook that parses agent output files (they already report
total_tokens, tool_uses, duration_ms) and pushes to pal-e-docs as sprint item metadata. No k8s infrastructure needed — just a post-session hook. Calibrate the fibonacci mapping over several sprints with real data.The Idea
Measure token usage per issue, per sprint, per repo -- and correlate it with DORA metrics. Break token spend down by activity type: planning, development, review (QA), documentation. This gives us a cost-efficiency dimension that nobody else has because nobody else runs an AI agent workforce.
Why This Matters
Token usage alone is not directly valuable -- spending more tokens does not mean more value. But token usage correlated with outcomes is breakthrough:
- What is the token cost per successful deployment?
- Does more planning tokens reduce rework tokens? (proving that planning prevents waste)
- What is the token efficiency curve per repo as it matures?
- Which SOPs and claude config patterns reduce token waste?
- Sprint planning: look at token spend breakdown and predict capacity
Activity Categories
- Planning -- Betty Sue sessions: plan creation, architecture decisions, sprint planning
- Development -- Dev agent sessions: code writing, branch creation, PR submission
- Review -- QA agent sessions: PR review, review-fix loops
- Documentation -- Main session doc updates, pal-e-docs note creation/updates
Data Sources
- Claude API usage logs (if available) or Claude Code session metadata
- Could instrument hooks to log token counts per agent spawn
- Correlate with Forgejo issue URLs and sprint items
Vision: Grafana Board
One day: a Grafana dashboard showing per-repo DORA metrics alongside token metrics per repo. Sprint burndowns measured in tokens. Token cost per DORA band improvement. This is the economic proof that one human + AI agents operates at enterprise velocity at a fraction of the cost.
Related
dora-framework-- the metrics this would extendplan-2026-03-01-pal-e-sprints-- sprints provide the container for measuring token spend per iterationplan-2026-03-01-dora-metrics-dashboard-- the Grafana dashboard this would feed into