Add nightly pinned dependency updates and bench smoke

[rgarcia]

Summary

Adds a nightly job that keeps every pinned dependency current and merges the update itself, plus bench smoke, an offline check for the agent-harness changes unit tests can't see.

Stacked on #6: the smoke task reuses that PR's finalize-mcp server and mcp-call-log wrapper. Rebase onto main after #6 lands.

bench smoke

uv run bench smoke                                    # one arm per distinct agent, version, model and agent kwargs
uv run bench smoke codex-repl --agent-version 0.120.0 # one arm at another version

Runs the arm's agent, version, model and agent_kwargs through Harbor on a local Docker task (no Kernel browser). The task's only tool is the bench stdio MCP server's finalize, wrapped in mcp-call-log. A run passes when:

• the trial has no exception and the agent reports the requested version;
• the MCP server logged the finalize call, and the verifier scored it 1;
• Harbor converted the session into trajectory.json with tool calls, token counts and cost, which is what the report and Braintrust publishing read.

Harbor swallows conversion errors, so the trajectory check is the one that catches a format change. The summary lists the tool names the trajectory shows and, for Claude Code, the built-in tools from its init event. Comparing two versions makes behavior changes visible. Current arms:

Arm	Agent	Version	Result	Trajectory tools	Server calls	Cost
claude-haiku-repl-xwebmcp	claude-code	2.1.301	pass	ToolSearch, mcp__bench__finalize	finalize	$0.0008
claude-kernel-mcp	claude-code	2.1.238	pass	ToolSearch, mcp__bench__finalize	finalize	$0.0209
codex-kernel-mcp-site-config	codex	0.120.0	pass	mcp__bench__finalize	finalize	$0.0031
codex-playwright-mcp	codex	0.163.0	pass	exec	finalize	$0.0016

The Codex rows show the 0.163.0 change: the MCP call is nested in code-mode exec and no longer appears as mcp__bench__finalize in the trajectory. All four run in parallel in about a minute.

Nightly pinned dependency updates

nightly-pinned-deps.yml (daily and workflow_dispatch) runs a Pi agent with nightly-pinned-deps-prompt.md, authenticated as the kernel-internal GitHub App through a renewable installation token. The agent:

1. resumes or resets the evergreen branch automation/nightly-pinned-deps;
2. updates every stale pin:
   • agent_version for Codex and Claude Code arms;
   • Harbor: the head of the fork's kernel-environment branch, or the upstream release once one contains Add Kernel as an environment provider harbor-framework/harbor#2122 and provides the kernel extra. It switches only after checking that the release keeps what the harness uses, starting with browser_create_overrides;
   • the ClawBench commit, the locked Kernel SDK, and Playwright MCP (arm and toolset default);
   • toolset image packages (bun, mcp-remote) and the uv image;
   • workflow actions (SHA-pinned) and their uv/Bun versions, and report packages;
3. adapts code to breaking changes and runs CI's checks plus a targeted check per changed pin: bench smoke at the head and at main for harness or Harbor changes, a stdio tools/list for Playwright MCP, a local header-forwarding check for mcp-remote, and a docker build of affected prepared images;
4. opens or updates the PR, waits for checks, and squash-merges.

Models, agent_kwargs, browser settings and the main refs toolsets build from are out of scope. The agent never runs bench run or creates Kernel browsers.

Traceability. The PR body and the squash commit body open with Behavior changes: every arm whose agent, Harbor or Playwright MCP version changed, any built-in tool newly available to an arm without disable_web_search, and changes in how tool calls are recorded or in the ClawBench task set. The report's configuration column already records each arm's agent version.

Merge gate. The gh wrapper refuses gh pr merge unless the harness and report checks and every other check on the head commit are green. It also runs smoke-harnesses itself, which is bench smoke for every agent configuration in a clean worktree of the head commit, with results cached per commit. The outer workflow then confirms the outcome file says no-updates or merged:<n> and that the PR really merged.

Unlike the workflow this is modeled on, there is no BugBot gate, because BugBot doesn't run on this repository.

Setup

• Secrets: ADMIN_APP_ID, ADMIN_APP_PRIVATE_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY. Variable: PI_PREFERRED_MODEL.
• The kernel-internal App must be installed on this repository.

Testing

• uv run pytest (33 tests): new tests for the smoke checks, task layout and harness dedup, plus one that Harbor's KernelEnvironment still takes browser_create_overrides. The existing environment test mocks __init__, so it couldn't catch a Harbor upgrade that drops it.
• ruff check and ruff format --check.
• bench prepare for both benchmarks, as in CI.
• bench smoke live, locally, for all four agent configurations (table above), and for Codex 0.120.0 vs 0.163.0 on the same arm. With an empty Anthropic key, the Claude rows fail (Not logged in) even though Harbor records no trial exception.
• The workflow's auth step, run in a scratch clone: smoke-harnesses passes, reuses its cached result for the same commit, cleans up its worktree, and fails with per-check reasons when API keys are bad. actionlint and shellcheck pass on the workflow and generated scripts.
• Not run: the workflow itself on Actions, and gh pr merge through the wrapper. The required-check logic was tested against Add xwebmcp-repl benchmark: browser REPL with and without xwebmcp tools #6's real check runs.