Repository navigation
Publish suite rewards as Braintrust scores, including incomplete arms - #5
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Braintrust experiments showed only a list of traces. The summary bar, score distributions, and the comparison view's improvements/regressions were all empty, because the publisher put every reward in
metadata. Braintrust's summarize API returned"scores": {}for the full ClawBench run.Each trial's suite rewards in [0, 1] are now Braintrust
scores. For ClawBench that meansreward_lenient,reward_strict,intercepted,kernel_mcp_valid, andreward. Infrastructure failures get no scores, so they don't count as task failures; metadata still records them, along with the derived rates.The publisher also no longer refuses arms with infrastructure failures. A two-arm run with 3 infrastructure failures out of 258 trials published nothing; those trials now just get no scores, and each arm's completeness is still recorded in the experiment metadata and the job summary.
Trial errors now keep their last 2,000 characters instead of their first 400. A traceback's cause comes last, and the old cut hid it.
This adds about 5 scores per trial (about 650 per 129-task arm) to Braintrust usage.
Validation
bun test(21),tsc, and prettier pass. Building events for a 129-trial job directory succeeds.