Repository navigation
Track token usage and cost of every LLM call - #136
Merged
Merged
Conversation
Subscription mode (claude-code / codex) already received cost_usd from caw, but each call overwrote backend.last_usage and only the incremental updater read it, so a full generate never reported what it cost. - UsageLedger (backend.py) records one entry per CLI process / agent run, with phase, kind, target, model, seconds, tokens and cost. It is thread-safe for caw's MCP worker threads. Nested caw sub-module agents record themselves, so totals add up without double counting. - Both backends route usage through LLMBackend._record_usage, which still sets last_usage for the updater. - Phases: clustering, modules, parent_modules, overview, update. - metadata.json gets a "usage" block (totals, by_phase, unpriced_models, calls) and generation_info.provider. - `codewiki generate` prints tokens and the estimated cost, notes that subscription cost is API-equivalent, and flags models with tokens but no price instead of showing $0.
caw main now prices codex runs on the model they actually run (including gpt-6.1-sol and newer OpenAI models), which the cost report relies on.
CI installs requirements.txt only, so importing caw failed collection. Use a stand-in usage object and a stub backend; the CawBackend test is skipped when caw is missing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In subscription mode (
claude-code/codex), caw already returnscost_usdfor every run, but each call overwrotebackend.last_usageand only the incremental updater read it. A fullcodewiki generatenever reported its tokens or cost. On top of that, Codex runs were almost always priced at $0 by caw.Changes
UsageLedger(backend.py) records one entry per CLI process / agent run: phase, kind, target, model, seconds, tokens and cost. It is thread-safe for caw's MCP worker threads. Nested caw sub-module agents are separate processes that record themselves, so totals add up without double counting.LLMBackend._record_usage, which still setslast_usagefor the updater.clustering,modules,parent_modules,overview,update.metadata.jsongets ausageblock (totals, by_phase, unpriced_models, calls) andgeneration_info.provider.caw pricing sethint for Codex.main, which includes Price codex runs on the default model and newer OpenAI models anhnh2002/caw#2: Codex is priced on the model it actually runs (-mor~/.codex/config.toml), newer OpenAI models fall back to the opencode table, andgpt-6.1-solis priced.guides/providers.md.Known gap
In API-key mode, pydantic-ai sub-module agents' usage isn't counted. Counting it would mean passing
usage=ctx.usage, which also makes the parent and its sub-agents share one request limit.Testing
tests/test_usage_ledger.py(9 tests): summing, phases, unpriced detection, thread safety, CawBackend recording, metadata block, CLI output.test_clustering_skip) also fails onmain.claude-code(Haiku) on a 2-file repo: the CLI printed 125,446 tokens and $0.08, andmetadata.jsonrecorded the same.CawBackendwith nomain_model: resolvedgpt-6.1-soland priced it.