Add xwebmcp-repl benchmark: browser REPL with and without xwebmcp tools

[rgarcia]

Summary

Adds xwebmcp-repl, a benchmark of how much a site's kernel/xwebmcp tools help an agent that drives a Kernel browser through Kernel MCP's browser_repl. It serves two purposes:

1. Measure the boost. Compare success and agent time with and without the tools, on a held-out test split.
2. Improve the tools. Iterate on a train split and gate changes on a validation split, without overfitting to the tasks they are scored on.

Pieces

• Suite xwebmcp-tasks. 56 tasks on 14 sites with xwebmcp tools that work signed out: airlines, Google Flights, AMC, Ticketmaster, Resy, Target, Etsy and Blue Bottle. Each site has four tasks: two train, one validation, one test. Setup opens the task's start URL before the agent starts, so both arms start on the same loaded page.

• finalize tool. A small stdio MCP server, bench, with one tool: finalize(evidence, reasoning, outcome).

  • outcome is success, failed (the agent's own failure, or the site doesn't offer what the task asks), website_error (the site errored or blocked the agent: third party) or kernel_error (the browser, REPL or a WebMCP tool failed: first party).
  • evidence and reasoning come before outcome in the schema, and both must be non-empty. The first call wins; later calls are refused.
  • reward is 1 for success. outcome_<name> and finalized record the rest. The report is kept as verifier/finalize.json, so an external judge can grade the same trials later.

• Arms. All use a stealth Kernel browser and have the agent's own web search/fetch tools off (disable_web_search):

  • codex-repl / codex-repl-xwebmcp: Codex 0.163.0, gpt-6-luna, with browser_repl only / plus the site's xwebmcp tools registered in the browser.
  • claude-haiku-repl / claude-haiku-repl-xwebmcp: the same pair with Claude Code 2.1.301 and claude-haiku-5-5.
  • codex-playwright-mcp: Codex with Playwright MCP 0.0.83 connected over CDP to the same Kernel browser, to compare Playwright MCP's tools with browser_repl. Playwright MCP cannot call WebMCP tools, so it has no xwebmcp counterpart.

  The instruction is identical within each pair: list the page's tools with webmcp.listTools() and use one when it fits. Only the page's tool list differs. The treatment arm gets the site's read-only tools, plus any state-changing tools a task names in actions (e.g. add_to_cart). Tools marked broken are skipped.

• Call logging. A stdio wrapper (mcp-call-log) logs every tool call the browser MCP server receives, with timing. Validity checks, WebMCP-invocation counts and the report's call counts read it, so they work the same for every harness, including Codex 0.163.0, which nests MCP calls inside a code-mode exec tool.

• Toolsets.

  • kernel-mcp-local takes toolsets = ["repl"], which sets KERNEL_MCP_ENABLED_TOOLSETS, the instruction and the validity check. It now appends to mcp_servers instead of replacing them. The default (playwright) produces the same instruction and image as before.
  • The new xwebmcp toolset applies kernel-mcp-local, then copies the site's tool files and runs xwebmcp-register in step setup.
  • kernel-mcp-verify also adds webmcp_invoked (any webmcp.invokeTool in a REPL cell), so adoption is visible in both arms.

• Harness.

  • Arms take [toolset_args] and benchmarks take [suite_args].
  • New flags: --suite-arg KEY=VALUE, --toolset-arg KEY=VALUE (overrides only the arms that set that key) and --attempts (Harbor's --n-attempts).
  • Suite modules are registered in sys.modules before they run, so dataclasses work in them.

• Report. When trials have no reward_lenient, the report shows:

  • success and finalize rate;
  • median agent time (agent execution only, no setup);
  • WebMCP adoption;
  • a per-task table and the median speedup over tasks both arms completed.

  ClawBench output is unchanged.

• Workflow. xwebmcp-repl.yml with inputs for split, attempts, xwebmcp_ref and kernel_mcp_server_ref.

First run

All 56 tasks, both arms, one attempt each, run locally on the benchmark project. Each site's four tasks mix three kinds: a search with several constraints (two dates, a time window, a fare type), a flow one step deeper (a round trip to the trip total, two items in a cart, a seat map), and a task that needs every result or a second page.

Arm	Success	Finalized	Median agent time	Median calls	Cost
codex-repl	53/56	56/56	117s	29.5	$2.53
codex-repl-xwebmcp	51/56	53/56	75s	15.5	$1.44

• Outcomes. This run used the earlier success boolean; later runs report the four outcomes above.
• Speed and calls. Median speedup 1.67× over the 50 tasks both arms completed. The per-task table now has median tool calls too.
• Timeouts are agent stalls. In every timeout in both runs and both arms, the last browser call returned within seconds. Codex then started its next model turn and produced nothing for 10–15 minutes until the 900s cap. Kernel's REPL and WebMCP layers are not involved; the cause (OpenAI stream vs. Codex) is under investigation.
• Arms disagree. On frontier-las-den-evening the control arm found an evening nonstop and the treatment arm reported there was none.
• Live data. On delta-evening-trip-summary both arms failed honestly: the qualifying fare was sold out in one arm, and Delta errored on the way to Trip Summary in the other.

Test tasks replaced after the five-arm run

• ticketmaster-msg-cheapest → ticketmaster-rangers-cheapest: every concert at Madison Square Garden in the next two months was one sold-out residency, so no arm could report a price. Rangers home games have resale and primary tickets for 2 (checked live).
• delta-evening-trip-summary (DTW→MSP) → delta-atl-lga-evening-summary: DTW→MSP has one nonstop after 5 PM, and its Main fare sold out. ATL→LGA has four, all with Main fares (checked live), and the task now says to skip sold-out fares.

Caveats

• Self-reported success is still near the ceiling (53/56 without the tools). Agent time and tool calls are the signal. Comparing answers across arms, or an external judge over finalize.json, would make success discriminate.
• Small samples. One task per site in validation and test is small, so use --attempts 3 or more and compare arms within one run.
• Signed-in sites are not covered yet. Instacart, Kroger carts, Uber Eats, DoorDash and Purelymail would need Kernel profiles.

Testing

• uv run pytest (21 tests): suite splits and fixtures, finalize over stdio, scoring, both toolsets, and --toolset-arg scoping.
• ruff check and ruff format.
• Report: bun test, tsc and prettier for the new tables.
• bench prepare for both benchmarks, also added to CI.
• A live run of both arms on all 56 tasks (above).