feat(evaluations)!: route handlers by provider and mode

[donei003]

Summary

This change implements the handler routing rules of the shared spec, TESTING.md §8, in launchdarkly/ai-sdks-monorepo#52. It lets one evaluation run use different providers or modes for generation and for judges. For example, generation uses OpenAI and a judge uses Anthropic, or generation uses an agent handler and a judge uses a messages handler.

This is a breaking change to the experimental evaluations API.

handler or handlers

• run() takes one handler as handler or a list as handlers, not both. judge_handlers is removed, with no alias.
• Every handler must declare provides_for. Before this change, a handler without it received every judge config with no warning.
• Two handlers with the same provider and mode raise an error.

One selection rule

• Generation and every judge select a handler with utils.select_handler, the same rule as config(). The evals code has no copy of it.
• There is no mode fallback. A messages-mode judge never runs on an agent handler, so collapse_messages is removed from the evals path. The online judges.py path does not change.

generation.mode

• mode is required: "completion" or "agent". There is no default.
• Other values, such as "messages", "judge" or a typo, raise before any network I/O. They do not go through normalize_mode.
• GenerationConfig stays a TypedDict with Required[...], so callers pass a plain dict and import no type. The new GenerationOverrides TypedDict is the type for overrides with ai_config. Two @overload signatures on run() select between them.
• The GenerationConfig docstring and the README tell the user how to choose a mode.
• The mode is not sent to LaunchDarkly.

Prompt must match the handler mode

• A messages handler with instructions raises: "the generation prompt needs messages".
• An agent handler with messages raises: "the generation prompt needs instructions".
• The same check applies to each judge. All judge problems are reported in one error, before any record is created.

Judges get no tools

A judge call gets an empty tool map, and tools is removed from the judge config.

ai_config

• The run first reads GET ai-configs/<key> to get the mode. The mode is on the AI Config, not on the variation.
• Then it reads the variation, and then the model config.
• Each check runs right after the read that supplies its input:
  • mode coverage: after 1 GET
  • prompt: after 2 GETs
  • provider: after the model config read
• The mode selects the variation's prompt field: instructions in agent mode, messages in completion and judge mode.
• When the mode's field is empty, the other field is kept, so that the prompt check rejects it.
• Overrides cannot set mode.

Migration

# Before
await evals.run(..., handler=openai_handler, judge_handlers=[claude_handler],
                generation={"provider": "OpenAI", "model": "gpt-4o"})

# After
await evals.run(..., handlers=[openai_handler, claude_handler],
                generation={"provider": "OpenAI", "model": "gpt-4o", "mode": "completion"})

To evaluate your own code, wrap it: create_handler(("OpenAI", "messages"), my_app).

Tests

• The existing tests now use tagged handlers and set mode. The judge fixtures now use messages, which is what judge configs carry.
• New tests cover:
  • handler argument validation
  • mode validation
  • prompt and mode mismatches
  • selection by provider and mode, and exact provider over wildcard
  • no mode fallback for judges
  • one error for all judge problems
  • judges with no tools
  • the request count at each ai_config failure
  • prompt field selection by mode
• make test after the merge with main: 2583 passed, 14 skipped. make lint, make format-check and make typecheck are clean.

Review fixes

• utils.select_handler now compares a handler's provider and its mode after normalize_mode. Before this change, a handler tagged ("OpenAI", "completion") or with a list passed validation but was never selected. The fix is in the shared function, so the online config() path is fixed too. Tests for both paths fail without the fix.
• GenerationMode and GenerationOverrides are exported from launchdarkly_ai_server.experimental.evaluations, which is the public path since feat(evaluations)!: mark evaluations from code as experimental #157. They are also in its surface test.
• The branch is merged with main. The packages/ai/README.md conflict takes the side of main.

Out of scope

• The spec's tools map shape and its GET ai-tools step at run time. Python uses list[EvalTool] and tools.get(), which was an earlier divergence.

BEGIN_COMMIT_OVERRIDE
feat(evaluations)!: route handlers by provider and mode
fix(client): match a handler's provides_for after normalizing its mode
END_COMMIT_OVERRIDE

🤖 Generated with Claude Code

---

Note

Overview
Breaking change to the experimental evaluations.run() API: handler routing for generation and judges now follows the same provider-and-mode rules as config(), with upfront validation and no silent fallbacks.

judge_handlers is removed. Callers pass a single handler or a handlers list; every handler must declare provides_for, and duplicate provider/mode pairs are rejected. Generation requires mode ("completion" or "agent") with no default, plus messages vs instructions must match the selected handler mode—errors fire before network I/O. Judges are matched from the same pool with exact mode (no agent collapsing messages in the evals path), prompt mismatches and missing handlers are aggregated into one error, and judge invocations get no tools.

ai_config runs now read the AI Config mode first (GET ai-configs/<key>), then the variation and model config, validating handler coverage, prompt shape, and provider as each step completes; generation.mode cannot be overridden. Types add GenerationMode, GenerationOverrides, and stricter GenerationConfig; shared handler_key normalizes completion/messages tags for registry, select_handler, and online judge matching.

^{Reviewed by Cursor Bugbot for commit 16e2194. Bugbot is set up for automated code reviews on this repo. Configure here.}

[jeffdupont]

Reviewed at 26eafe03, against spec §8.4 as merged in monorepo #52. The routing rules match: one select_handler rule for generation and judges, no mode fallback, generation.mode checked before normalize_mode (module.py:852-856, with 'judge' and 'agnet' tested), prompt/mode checks, and judges with no tools. make test: 1565 passed, 11 skipped. This also closes the judge_handlers divergence that monorepo #56 had deleted the note for.

1. A handler tagged "completion", or with a list, is accepted and then never selected (inline at runner.py:117; same finding as the open Devin and Bugbot threads).
   • _validate_handlers normalizes the tag, so ("OpenAI", "completion") counts as ("OpenAI", "messages"). test_evaluations_run.py:4300 even relies on this to reject a duplicate.
   • But utils.select_handler compares the raw tag: h.provides_for == (provider, mode) (utils.py:516), and [1] == mode for wildcards (:524).
   • I ran it. create_handler(("OpenAI", "completion"), fn) and ["OpenAI", "messages"] both pass validation, but select_eval_handler returns None. The error then lists the handler as ('OpenAI', 'messages'), the very pair it needed.
   • This PR tells users to write mode: "completion", so that tag is the natural one to reach for. Online config() misses it the same way, because it shares select_handler.
   • Fix it once in select_handler (utils.py, not in this diff): compare against (h.provides_for[0], normalize_mode(h.provides_for[1])), and use the same normalized value in the wildcard branch. That's about 3 lines, and it fixes config() too. A test with a lone ("OpenAI", "completion") handler would cover it.
2. The two new types need to reach #157's public path (inline at evaluations/__init__.py:36). I trial-merged this onto #157. The only conflict is packages/ai/README.md: take #157's side, since it removes that example. The merged suite passes (2418 passed). But GenerationMode and GenerationOverrides end up only in launchdarkly_ai_server.evaluations, which #157 makes internal. They aren't in launchdarkly_ai_server.experimental.evaluations. Whichever of the two lands second should add both names to experimental/evaluations.py __all__ and to #157's public-surface test. I'll add + GenerationMode and + GenerationOverrides to the API list in monorepo #51.

Not checked: the ai_config GET order (I went by the tests, I didn't probe it).