Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 4 additions & 12 deletions .claude/harness-candidates.md
Original file line number Diff line number Diff line change
Expand Up @@ -545,18 +545,10 @@ divergences, so the deferred-work record is one place. Measurements in
as tool execution. Pinned by `unbounded_cases` in
`tests/_fixtures/timing_union_cases.json`, which both suites replay.

- [ ] **Pre-existing, surfaced by this work's final review: `TokenUsage._adopt_legacy_input_tokens`
double-counts the cache buckets.** The validator copies a legacy record's
full-prompt `input_tokens` straight into `uncached_input_tokens`, and the
computed `input_tokens` then adds `cache_creation` + `cache_read` again. A
legacy record with `input_tokens=1000`, `cache_creation=200`, `cache_read=150`
reloads as 1350 prompt tokens and bills 1000 at the uncached rate instead of
650. Affects every report, budget check and detached regrade over a
pre-split run that used prompt caching. NOT touched by the timing work
(token accounting was explicitly out of its scope) and not a guardrail
candidate — a real bug needing its own change, with a decision about
whether legacy records can be distinguished from current ones at all.
Caught in: timing-capture final review (gpt-5.6-sol).
- [x] ~~**Pre-existing, surfaced by this work's final review: `TokenUsage._adopt_legacy_input_tokens`
double-counts the cache buckets.**~~ **DONE.** The validator is deleted. Every run
uploaded in the last 30 days stores `uncached_input_tokens`, so no pre-split record
is read. Caught in: timing-capture final review (gpt-5.6-sol).

- [x] ~~No golden-corpus assertion of the four-bucket identity.~~ **DONE.** The
fixture clocks were unified (`_rebase_notifications` / `_rebase_lines` shift
Expand Down
19 changes: 9 additions & 10 deletions .claude/notes/lint-rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -714,16 +714,15 @@ None else self.open_start_ms)` collapse to one instant whenever `open_end_ms` is

An identity invariant, which is why it is its own id rather than a second arm of
CE058/CE059, both of which are about timing. Antigravity omitted the kwarg, so the field
defaulted to `None` on every message it ever recorded, and the evalboard — which groups
assistant emissions by `message_id` and falls back to a `SAME_EMISSION_GAP_MS` wall-clock
gap when either side lacks one — collapsed a whole turn's generations into ONE timeline
row as soon as the harness's generation windows became contiguous (the gap is then exactly
0 ms, always). Nothing failed: the consumer SUMS the group, so the totals and the
reconciliation invariant stayed right, and the golden snapshots had ratified the `null` on
the day they were written — a snapshot is regenerated from whatever the code currently
does, so it catches a later change and never an initial omission. The damage was not
confined to the timeline, which is why "only granularity is lost" was the wrong way to
describe it: a grouped emission is one API call to the evalboard's thinking-cost
defaulted to `None` on every message it ever recorded, and the evalboard — which then
grouped assistant emissions by `message_id`, else by a 100 ms wall-clock gap — collapsed a
whole turn's generations into ONE timeline row as soon as the harness's generation windows
became contiguous (the gap is then exactly 0 ms, always). Nothing failed: the consumer
SUMS the group, so the totals and the reconciliation invariant stayed right, and the
golden snapshots had ratified the `null` on the day they were written — a snapshot is
regenerated from whatever the code currently does, so it catches a later change and never
an initial omission. The damage was not confined to the timeline, which is why "only
granularity is lost" was the wrong way to describe it: a grouped emission is one API call to the evalboard's thinking-cost
simulator, whose prompt-cache cascade is quadratic in that count, so a single-shot
Antigravity run had every cascade coefficient pinned at zero; the `Messages` count and the
10 s slow-generation bar were per-turn too. Deriving the constructor set from each
Expand Down
3 changes: 1 addition & 2 deletions .claude/notes/persistence.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,8 +167,7 @@ render still sees it; the JSON dump excludes it.

The sibling is YAML rather than JSON because the transcript carries multi-line text, which
YAML's literal block scalar renders as readable paragraphs instead of one line full of `\n`
escapes. Its consumers are humans. The reader accepts `.json` too, so previously-spilled runs
keep rendering, and a row with no path at all is a no-op, so old inline records keep working.
escapes. Its consumers are humans. A row with no path at all is a no-op.

FILE ORDER IS LOAD-BEARING. Each filename is keyed off the criterion's position in its result
list, and the reader resolves the stored path, so each list must retain its order through
Expand Down
6 changes: 2 additions & 4 deletions docs/REPORT_SCHEMA.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,8 +191,7 @@ no longer `str | null`, so consumers must not string-handle it unconditionally.

### CriterionResult

A discriminated union on `result_kind` (`basic` / `judge` / `classification`); legacy
files without `result_kind` are inferred from `criterion_type`. Base fields
A discriminated union on `result_kind` (`basic` / `judge` / `classification`). Base fields
(`result_kind="basic"`):

`criterion_type`, `description`, `score` (0.0–1.0), `details`, `error`,
Expand Down Expand Up @@ -333,8 +332,7 @@ Serialized fields: `uncached_input_tokens`, `output_tokens`,
`cache_creation_input_tokens`, `cache_read_input_tokens`, `total_cost_usd`, plus a
computed **`input_tokens`** (= sum of the three input buckets). Note: `total_tokens`
is a plain property and is **not** serialized. Cost bills `uncached_input_tokens`,
not `input_tokens`. Legacy files that only have `input_tokens` are adopted as
`uncached_input_tokens` on read.
not `input_tokens`.

## FinalStatus

Expand Down
38 changes: 16 additions & 22 deletions docs/agents/HARNESS_PARITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -408,28 +408,22 @@ generic tool items now carry a duration where they previously carried none, so
`avg_command_time_ms` and `total_command_time_ms` for a Codex run describe every
tool call rather than shell commands alone.

**`message_id` is what splits the timeline.** The evalboard groups assistant
emissions by `message_id`, and falls back to a wall-clock gap threshold
(`SAME_EMISSION_GAP_MS`, 100 ms, in `evalboard/lib/runs.ts`) when either side
lacks one. Antigravity's `Step` stream carries no message id, so the harness
synthesizes one — and it must, because this harness's generation windows are
*contiguous* by construction: each opens exactly where the previous one closed,
so the gap between two of them is always 0 ms and the fallback would fold a
whole turn's generations into a single row. CE060 makes the kwarg mandatory in
`src/coder_eval/agents/` for that reason.

The collapse is a *display* defect, not an accounting one — the consumer SUMS a
group's token buckets and durations, so every total, percentage and cost is
identical either way, as is the reconciliation residual. But it is not
cosmetic, and three displayed figures do move when a turn stops collapsing:
the thinking-cost simulator's per-call cache cascade (`calls` in
`evalboard/lib/thinkingSim.ts` is the number of grouped emissions, and the
cascade is quadratic in it — on a single-shot run it was pinned at one call,
so every coefficient was zero), the `Messages` count and timeline heading, and
the "slow generation" count, whose 10 s bar was being applied to a whole turn's
summed generation time. All three move toward the figure they were always
meant to report, so the fix corrects them rather than breaking them — but a
trend compared across this change is not comparing like with like.
**`message_id` is what splits the timeline.** The evalboard joins assistant
emissions into one row only when they share a `message_id`
(`evalboard/lib/runs.ts`); an emission without one is never joined, so each of
its content blocks renders as its own row. Antigravity's `Step` stream carries
no message id, so the harness synthesizes one. CE060 makes the kwarg mandatory
in `src/coder_eval/agents/` for that reason.

A missing `message_id` is a *display* defect, not an accounting one — the
consumer SUMS a group's token buckets and durations, so every total, percentage
and cost is identical however the blocks are grouped, as is the reconciliation
residual. But it is not cosmetic: each split block counts as its own emission,
which skews three displayed figures — the thinking-cost simulator's per-call
cache cascade (`calls` in `evalboard/lib/thinkingSim.ts` is the number of
grouped emissions, and the cascade is quadratic in it), the `Messages` count
and timeline heading, and the "slow generation" count, whose 10 s bar applies
per emission.

The two synthetic schemes read differently on purpose: Codex deliberately REPEATS one
id across the sub-messages of a single generation — that is exactly the "the
Expand Down
9 changes: 4 additions & 5 deletions evalboard/app/_components/__tests__/harness-badge.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,14 @@ import { HarnessBadge, harnessShortLabel } from "../harness-badge";
describe("harnessShortLabel", () => {
test("every known harness has a human label", () => {
// A missing entry falls through to the raw id, which would put
// "delegate-sdk" in a legend next to "Codex" and "Antigravity".
// "delegate" in a legend next to "Codex" and "Antigravity".
for (const h of KNOWN_HARNESSES) {
expect(harnessShortLabel(h)).not.toBe(h);
}
});

test("the UiPath harness reads as Delegate", () => {
// The run data's id stays `delegate-sdk` (it's the registered
// `agent.type`); only the label people read is the short one.
test("the UiPath harness reads as Delegate under both of its ids", () => {
expect(harnessShortLabel("delegate")).toBe("Delegate");
expect(harnessShortLabel("delegate-sdk")).toBe("Delegate");
});

Expand All @@ -31,7 +30,7 @@ describe("harnessShortLabel", () => {

describe("HarnessBadge", () => {
test("names the vendor in the alt text, not just the product", () => {
render(<HarnessBadge harness="delegate-sdk" />);
render(<HarnessBadge harness="delegate" />);
expect(screen.getByAltText("Delegate · UiPath")).toBeInTheDocument();
});

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,11 @@ describe("HarnessSelector", () => {
});

test("a deep-linked harness that has aged out still shows as a segment", () => {
// Otherwise `?h=delegate-sdk` after a quiet fortnight renders a control
// Otherwise `?h=delegate` after a quiet fortnight renders a control
// with nothing selected, which reads as an unscoped page.
render(
<HarnessSelector
current="delegate-sdk"
current="delegate"
harnesses={["claude-code", "codex"]}
includeAll
/>,
Expand Down
21 changes: 5 additions & 16 deletions evalboard/app/_components/harness-badge.tsx
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import Image from "next/image";
import { normalizeHarness } from "@/lib/harness";

// Canonical harness constants live in the leaf lib/harness module (shared by
// the server data layer); re-exported here so existing badge importers are
Expand All @@ -23,20 +24,8 @@ const HARNESS_LOGO: Record<string, { src: string; label: string; short: string }
label: "Antigravity · Google Gemini",
short: "Antigravity",
},
// The Delegate harness is UiPath's own (the coder_eval_uipath plugin), so
// its vendor mark is the UiPath logo already served for the site header
// rather than a per-harness file under /harness. The key stays `delegate-sdk`
// because that is the `agent.type` the plugin registers and therefore what
// run.json carries; only the label people read is shortened.
"delegate-sdk": {
src: "/uipath.png",
label: "Delegate · UiPath",
short: "Delegate",
},
// coder_eval's own built-in Delegate agent (`agent.type: delegate`) — a
// distinct `AgentKind` from the coder_eval_uipath plugin's `delegate-sdk`
// above, so it needs its own key or it falls through to raw gray text
// (same fallback treatment pi gets below, matching that precedent).
// UiPath's own harness has no file under /harness; it uses the UiPath logo
// already served for the site header.
delegate: {
src: "/uipath.png",
label: "Delegate · UiPath",
Expand All @@ -47,7 +36,7 @@ const HARNESS_LOGO: Record<string, { src: string; label: string; short: string }
// Short human label for a harness id ("Claude Code"), for selectors and prose.
// Unknown ids fall through to the raw id.
export function harnessShortLabel(harness: string): string {
return HARNESS_LOGO[harness]?.short ?? harness;
return HARNESS_LOGO[normalizeHarness(harness)]?.short ?? harness;
}

export function HarnessBadge({
Expand All @@ -59,7 +48,7 @@ export function HarnessBadge({
harness?: string | null;
size?: number;
}) {
const key = harness ?? "claude-code";
const key = normalizeHarness(harness);
const logo = HARNESS_LOGO[key];
// Unknown harness: show the raw id rather than a misleading logo.
if (!logo) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,7 @@ describe("MessageTimelineSection — sub-agent grouping", () => {
const parent = makeMessage({
index: 1,
text: "PARENT_THREAD",
blockTypes: ["tool_use"],
blockTypes: ["text", "tool_use"],
toolUses: [agentTool("T1")],
});
const child = makeMessage({
Expand All @@ -321,31 +321,6 @@ describe("MessageTimelineSection — sub-agent grouping", () => {
expect(screen.getByText(/\[1, 2\]/)).toBeInTheDocument();
});

test("legacy run without branch info renders every message at top level", () => {
// No parent_tool_use_id recorded → both messages are top-level rows, and
// a message with an Agent tool call has no children to nest.
const a = makeMessage({
index: 1,
text: "ROW_A",
blockTypes: ["tool_use"],
toolUses: [agentTool("T1")],
parentToolUseId: undefined,
});
const b = makeMessage({
index: 2,
text: "ROW_B",
parentToolUseId: undefined,
});
const { container } = render(
<MessageTimelineSection messages={[a, b]} />,
);
// Two top-level rows in the table body.
const topRows = container.querySelectorAll("ol > li");
expect(topRows).toHaveLength(2);
expect(screen.getByText("ROW_A")).toBeInTheDocument();
expect(screen.getByText("ROW_B")).toBeInTheDocument();
});

test("shows each sub-agent call's token buckets on its per-call CallTokensRow", () => {
const parent = makeMessage({
index: 1,
Expand Down
57 changes: 20 additions & 37 deletions evalboard/app/runs/[id]/[...task]/_sections.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -485,8 +485,6 @@ export function MessageTimelineSection({
// flat stream tagged with the spawning tool_use_id (parent_tool_use_id).
// We key children by that id and render only the main thread at top level;
// each sub-agent's invocations nest, expandable, under its Agent call.
// Legacy runs (parentToolUseId undefined) have no string parents → the map
// stays empty and every message renders flat, exactly as before.
const childrenByParent = new Map<string, MessageEvent[]>();
for (const m of messages) {
const p = m.parentToolUseId;
Expand All @@ -496,9 +494,8 @@ export function MessageTimelineSection({
childrenByParent.set(p, arr);
}
}
// == null matches both null (main thread) and undefined (legacy, no branch
// info) → both render at the top level. Sub-agent emissions render nested
// inside the message that spawned them (see MessageRow).
// Sub-agent emissions render nested inside the message that spawned them
// (see MessageRow).
const topLevelMessages = messages.filter((m) => m.parentToolUseId == null);
// Count real generations only — the synthetic reconciliation row is a meta
// entry, not a message.
Expand Down Expand Up @@ -978,9 +975,8 @@ export function CostExplorerSection({
});
};

// No model → the run can't be projected (unpriced model, or it predates
// per-call branch capture so the cache cascade can't be modeled as a tree).
// The timeline still renders; the simulator + inline deltas are hidden.
// No model → the run can't be projected (unpriced model). The timeline
// still renders; the simulator + inline deltas are hidden.
const model = buildThinkingModel(messages, tokens, recordedCostUsd);
const leversActive =
model != null &&
Expand Down Expand Up @@ -1452,36 +1448,23 @@ function MessageBody({
};
// Render blocks in their actual emission order (content_blocks sequence),
// interleaving text and tool calls instead of forcing thinking→tools→text.
// `blockTypes` preserves order for both agents; thinking/text are aggregated
// per message, so each renders once at its first occurrence while tool_use
// blocks consume `toolUses` in order. Falls back to the legacy grouping when
// a (legacy) run recorded no blockTypes.
// thinking/text are aggregated per message, so each renders once at its
// first occurrence while tool_use blocks consume `toolUses` in order.
const ordered: ReactNode[] = [];
if (m.blockTypes.length === 0) {
if (thinkingRow) ordered.push(thinkingRow);
m.toolUses.forEach((t, i) => ordered.push(renderTool(t, i)));
if (textRow) ordered.push(textRow);
} else {
let toolIdx = 0;
let thinkingDone = false;
let textDone = false;
for (const bt of m.blockTypes) {
if (bt === "thinking") {
if (!thinkingDone && thinkingRow) ordered.push(thinkingRow);
thinkingDone = true;
} else if (bt === "text") {
if (!textDone && textRow) ordered.push(textRow);
textDone = true;
} else if (bt === "tool_use") {
const t = m.toolUses[toolIdx++];
if (t) ordered.push(renderTool(t, toolIdx - 1));
}
let toolIdx = 0;
let thinkingDone = false;
let textDone = false;
for (const bt of m.blockTypes) {
if (bt === "thinking") {
if (!thinkingDone && thinkingRow) ordered.push(thinkingRow);
thinkingDone = true;
} else if (bt === "text") {
if (!textDone && textRow) ordered.push(textRow);
textDone = true;
} else if (bt === "tool_use") {
const t = m.toolUses[toolIdx++];
if (t) ordered.push(renderTool(t, toolIdx - 1));
}
// Defensive: if blockTypes didn't list thinking/text but the aggregated
// field is present, still render it (thinking first, text last — the
// legacy positions) so content is never dropped.
if (!thinkingDone && thinkingRow) ordered.unshift(thinkingRow);
if (!textDone && textRow) ordered.push(textRow);
}
return <>{ordered}</>;
}
Expand Down Expand Up @@ -1620,7 +1603,7 @@ function MessageRow({
unit: Unit;
// Maps a sub-agent-spawn tool_use_id → the emissions that ran inside that
// sub-agent. Lets a message render its sub-agents' rows nested between the
// spawning Agent call and that call's result. Empty for legacy runs.
// spawning Agent call and that call's result. Empty when no sub-agent ran.
childrenByParent: Map<string, MessageEvent[]>;
// Per-Agent-call sub-agent token breakdown (input/output/cache-create/
// cache-read), keyed by tool_use_id. Shown on the Agent call's result row.
Expand Down
5 changes: 2 additions & 3 deletions evalboard/app/runs/[id]/[...task]/thinking-simulator.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -108,9 +108,8 @@ export function ThinkingSimulator({
)}
{noTools && (
<div className="text-xs text-gray-500 bg-gray-50 border border-gray-200 rounded p-2">
This run recorded no tool-result tokens (no tool calls, or a
run predating per-tool result sizing), so the tool-output
lever has nothing to scale.
This run recorded no tool-result tokens (no tool calls), so
the tool-output lever has nothing to scale.
</div>
)}

Expand Down
Loading
Loading