Skip to content

Allow ClawBench agents to read task files with a shell - #3

Closed
rgarcia wants to merge 1 commit into
hypeship/fix-braintrust-redactionfrom
hypeship/allow-reading-task-files
Closed

rgarcia wants to merge 1 commit into
hypeship/fix-braintrust-redactionfrom
hypeship/allow-reading-task-files

Conversation

@rgarcia

@rgarcia rgarcia commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Summary

In the first full run of codex-kernel-mcp (129 tasks), 38 trials ended with the agent saying it couldn't read the task's credential files. 3 of those were still intercepted, and only 41 trials were intercepted in total.

The cause:

  • ClawBench's instruction says the task's files are under ./my-info/, but also says "Do NOT use command-line tools". Codex can only read files through its shell.
  • The previous harness in kernel-mcp-server resolved this conflict by telling the agent to use the credentials under ./my-info/. This repo dropped that line.
  • The kernel-mcp-local instruction then made it worse: it allowed reading "this one file" (the session-ID file), which implies other files are off limits.

Changes:

  • The ClawBench suite replaces its removed runtime block with one sentence: reading files with shell commands is allowed, and everything else goes through the browser. Reading ./my-info/ is a ClawBench concern, so the fix lives in the suite.
  • The toolset no longer limits file reading to the session-ID file.

Stacked on #2.

Validation

  • pytest (asserts the new sentence is in generated instructions), ruff, and bench prepare pass.
  • Full-arm rerun is linked in the comments.

@rgarcia

rgarcia commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Full-arm rerun of codex-kernel-mcp from this branch (run, Braintrust): file-access refusals fell from 38 to 1, intercepted rose from 41 to 56 of 129, and lenient rose from 30 to 36 of 129. 129/129 trials were graded with no infrastructure errors.

@rgarcia
rgarcia deleted the branch hypeship/fix-braintrust-redaction October 13, 2026 19:16
@rgarcia rgarcia closed this Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant