Skip to content

Publish suite rewards as Braintrust scores, including incomplete arms - #5

Merged
rgarcia merged 3 commits into
mainfrom
hypeship/braintrust-scores
Oct 13, 2026
Merged

rgarcia merged 3 commits into
mainfrom
hypeship/braintrust-scores

Conversation

@rgarcia

@rgarcia rgarcia commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Braintrust experiments showed only a list of traces. The summary bar, score distributions, and the comparison view's improvements/regressions were all empty, because the publisher put every reward in metadata. Braintrust's summarize API returned "scores": {} for the full ClawBench run.

Each trial's suite rewards in [0, 1] are now Braintrust scores. For ClawBench that means reward_lenient, reward_strict, intercepted, kernel_mcp_valid, and reward. Infrastructure failures get no scores, so they don't count as task failures; metadata still records them, along with the derived rates.

The publisher also no longer refuses arms with infrastructure failures. A two-arm run with 3 infrastructure failures out of 258 trials published nothing; those trials now just get no scores, and each arm's completeness is still recorded in the experiment metadata and the job summary.

Trial errors now keep their last 2,000 characters instead of their first 400. A traceback's cause comes last, and the old cut hid it.

This adds about 5 scores per trial (about 650 per 129-task arm) to Braintrust usage.

Validation

bun test (21), tsc, and prettier pass. Building events for a 129-trial job directory succeeds.

@rgarcia rgarcia changed the title Publish suite rewards as Braintrust scores Publish suite rewards as Braintrust scores, including incomplete arms Oct 13, 2026
@rgarcia
rgarcia merged commit 59f0cde into main Oct 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant