Wattage Turns AI Agent Token Waste into a CI-Testable Cost Signal

Introduction

AI agent cost problems rarely come from model pricing alone. They often hide in repeated prompt context, ineffective retrieval loops, redundant tool calls, oversized model choices, or reasoning tokens spent on steps that do not need them. Wattage positions itself as a “Kill-A-Watt meter” for agents: point it at a trace, and it shows where tokens are being burned, what the waste costs, and what can be fixed.

The project is not another agent framework. It is a profiling and regression-control layer for teams that already run agents or are starting to instrument them. Wattage ingests OTLP JSON trace exports based on OpenTelemetry GenAI semantic conventions, runs fully offline, and requires no API key or configuration file for the basic report flow.

Key points

• Trace-based cost profiling: wattage report trace.json breaks token usage into categories such as input, output, cache reads, cache creation, and reasoning.
• Real-dollar pricing: Wattage uses a vendored, dated pricing snapshot to price calls. If a model cannot be priced, the tool does not invent a number; in CI, that condition fails loudly.
• Eight waste detectors: The current detector set covers repeated stable prompt prefixes, weak cache redemption, excessive verbosity, redundant tool calls, non-converging loops, retrieval thrash, model mismatch, and reasoning overspend.
• CI regression gate: wattage ci can block changes when token efficiency drops, cost rises beyond a chosen threshold, or critical findings appear. It can also produce PR comments, SARIF, and JUnit XML.
• Score and badge workflow: wattage score and wattage badge expose a 0–100 Token Efficiency grade for README badges and automated review.

A notable part of the project is its convergence engine. The nonconvergence detector is designed to catch loops that look active but do not make real progress, including retries with fresh timestamps, oscillation between strategies, or stalls where each call is technically unique. In the project’s hand-reviewed set of 10 labeled synthetic loops, Wattage reports Precision, Recall, and F1 of 1.00. A SHA-256 exact-match baseline also reaches Precision of 1.00, but its Recall is 0.14 and F1 is 0.25, illustrating why exact duplicate detection is too narrow for many agent failure modes.

The repository also includes a captured agent trace example. In that trace, the prefix_churn simulation shows that enabling prompt caching on a stable prefix would reduce cost from $0.000199 to $0.000110, a 44.7% reduction. The dollar amount is tiny because the trace is a three-turn demo, but the same mechanism is relevant when repeated across production traffic.

Why it matters

As agents move into coding workflows, internal automation, support systems, and retrieval-heavy tasks, token spend becomes an engineering regression surface. A prompt change or orchestration tweak can make an agent more expensive even if the functional tests still pass. Wattage tries to make that cost visible before the change lands.

The project also treats quality as a first-class concern. Findings are tagged with a quality risk tier, and optimizations that may change output quality, such as model downgrades or reduced reasoning, only count toward the score when backed by a quality map. That is an important guardrail: cheaper is not automatically better.

Overall, Wattage is most useful for teams with agent traces or teams preparing to adopt OpenTelemetry instrumentation. It does not replace task-level evaluation, but it does turn common token waste patterns into repeatable, reviewable, and automatable signals for agent engineering.

Source: Hacker News