Skip to content

Fix Braintrust redaction false positives - #2

Merged
rgarcia merged 1 commit into
mainfrom
hypeship/fix-braintrust-redaction
Oct 13, 2026
Merged

rgarcia merged 1 commit into
mainfrom
hypeship/fix-braintrust-redaction

Conversation

@rgarcia

@rgarcia rgarcia commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Summary

The first full ClawBench run graded all 129 trials but published nothing to Braintrust: the publisher's safety check rejected the payload. The redactor had three false positives, all from code inherited from kernel-mcp-server:

  • Labels followed by a line break. In a page snapshot, "Email address:" followed by a new line was read as an address field whose value was the next lines.
  • Method calls. Agent code like const cookie=page.getByRole(...) was read as a cookie= credential.
  • Quoted labels with spaces. In 'Work email':'[REDACTED]', the checker took the label's closing quote for an opening one, then skipped [REDACTED] as if it were a snapshot attribute.

In each case the redactor and the safety check disagreed, and one disagreement aborted the whole experiment.

Changes:

  • Fix all three cases in both the redactor and the checker.
  • Before the final check, the publisher now replaces any string that fails the check with [WITHHELD] and logs a count. An unexpected value costs that value, not the experiment. The safety check still runs on the final payload.

Validation

  • bun test passes (21 tests, including regressions for each case and for withholding), along with tsc and prettier.
  • Building events for every local Harbor job directory now succeeds with nothing withheld, including a previous 129-trial run (9,894 events). Before this change, each of them failed.

@rgarcia

rgarcia commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Validated in CI: run 38440542932 published 129 rows to Braintrust with nothing withheld. In the next run, 4 values were withheld and the experiment still published.

@rgarcia
rgarcia merged commit d4b8050 into main Oct 13, 2026
3 checks passed
@rgarcia
rgarcia deleted the hypeship/fix-braintrust-redaction branch October 13, 2026 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant