Export offline evaluation spans through the SDK exporter

[ccschmitz-launchdarkly]

launchdarkly/python-ai-sdk

EvaluationsModule.run exports each dataset row as one trace, through the tracer provider init_client already registers. That provider's launchdarkly.project_id and highlight.project_id are both the SDK key, not the project's BSON id. $ld:ai:offline-evals:generation and :criterion are unchanged and still carry no transcript or tool payload.

Spec: launchdarkly/ai-sdks-monorepo#60 (TESTING.md §8.10).

Frontend: launchdarkly/gonfalon#74117.

Staging evaluation you can open

Project default, environment ai-o-11-y. The test-only AI Config is offline-eval-trace-verify and the judge is offline-eval-trace-verify-judge. The judge has a user message with {{message_history}} after its system rubric, which is how library judges are written. The row is the canned "What is the capital of France?" prompt with a lookup_capital tool. Custom events went to https://events-stg.launchdarkly.com, the events URI in gonfalon's staging e2e config (packages/e2e/playwright/config.staging.json). The run summary reached passed=1, which only happens after those events are ingested.

https://ld-stg.launchdarkly.com/projects/default/ai/evaluations/178760f1-2873-440a-ab43-91f5dd17cc0b/runs/a7a2bd3d-8f1a-4194-8187-7353aabd79dd?env=ai-o-11-y

Preview of gonfalon#74117 on that same run (frontend-version is the PR head, b6338c27c):

https://ld-stg.launchdarkly.com/projects/default/ai/evaluations/178760f1-2873-440a-ab43-91f5dd17cc0b/runs/a7a2bd3d-8f1a-4194-8187-7353aabd79dd?env=ai-o-11-y&frontend-version=b6338c27c

• Run a7a2bd3d-8f1a-4194-8187-7353aabd79dd
• Row 0 conversationId be7c123c-c9bb-50ed-aee1-88af53750ca7, status COMPLETE, judge score 0.9 (LaunchDarkly ruled it a pass)
• Trace: https://ld-stg.launchdarkly.com/projects/default/traces?query=gen_ai.conversation.id%3D%22be7c123c-c9bb-50ed-aee1-88af53750ca7%22&env=ai-o-11-y

Turn the spans flag on, or the panel shows a different waterfall

enable-evaluation-run-conversation shows the panel. enable-evaluation-run-conversation-spans gates whether it queries the exported spans, and it is off in default / staging. With it off the panel renders evaluationResultToSpans, which rebuilds a transcript from the stored evaluation result instead. That fallback mints one trace id per phase (<conversationId>:generation and <conversationId>:evaluation, so the header reads "2 traces") and takes each duration from a stored millisecond latency. On this canned run those are generationLatencyMs: 1 and the criterion's meta.usage.latencyMs: 0, which formatDuration renders as 1ms, 0ns, and 0ns. None of those numbers come from the exported spans.

With the flag on, the backend's trace_conversation query returns what the SDK actually sent:

summary: trace_count=1  span_count=5  total_duration_ns=1165197

launchdarkly.dataset_row                     parent=-         1165197ns -> 1ms
  chat gpt-4o                                parent=root       103760ns -> 103µs
  execute_tool lookup_capital                parent=root        23783ns -> 23µs
  evaluate offline-eval-trace-verify-judge   parent=root       265362ns -> 265µs
    chat gpt-4o                              parent=evaluate    17053ns -> 17µs

One trace id, the parenting §8.10 specifies, and each span's window inside its parent's: the tool call sits inside the generation chat, and the judge chat inside evaluate. The trace-count chip does not render at all, because it only appears above one trace.

The durations are microseconds because the handler is a canned in-memory function that makes no model call. The spans are stamped from timestamps captured around the call, and nothing is padded to make the waterfall look busier. A real provider call produces real wall time in the same spans.

What each row exports

• launchdarkly.dataset_row is the row's root span. It carries the shared UUID5 gen_ai.conversation.id plus launchdarkly.evaluation_run.id and .row_index. It has no gen_ai operation, so the conversation view treats it as an envelope. Its status is OK, or ERROR when the handler raised.
• The generation chat {model} span is a child of the root. It carries the rendered gen_ai.system_instructions, gen_ai.input.messages, and gen_ai.output.messages (a tool_call part for each recorded call, then the answer), plus provider, model, usage, and tool definitions.
• execute_tool {name} is a child of the root, one per recorded tool call.
• On a row that generated, evaluate {criterion} is a child of the root and carries a gen_ai.evaluation.result with no verdict. A row whose handler raised is never scored (§8.8.6), so it has no evaluate span.
• The judge's chat {model} is a child of evaluate, which keeps it out of the transcript. It carries the judge's system instructions, then its other messages rendered with the judge variables, then the judged output.

The runner writes the chat and execute_tool spans only when the handler opened no SDK span of its own, so a provider handler that traces itself gets no duplicates.

Where the code lives (§0.3)

Core conversation.py doesn't know offline evaluations exist. Its span processor exposes two feature-neutral primitives. stamp_span_attributes writes a caller-supplied str/int attribute set, write-if-absent, onto SDK spans started in the block. count_sdk_spans counts SDK spans started in the block. The launchdarkly.evaluation_run.* keys, the ai-evaluator UUID5 namespace, conversation_id_for_evaluation_row, bind_evaluation_row, and record_evaluation_result all live in evaluations/spans.py. The runner uses count_sdk_spans to decide whether to describe a plain function's call. This layout doesn't change behavior, so the staging run and links still apply.

capture_content stays false (SDKOptions.capture_content, introduced by Alexis Georges). Synthetic evaluation spans still export the transcript: row input and messages, rendered system and judge instructions, model output, tool arguments and results, and the judge completion and explanation. The public warning is in packages/client/README.md, under "Run an evaluation from code". A normal handler span written with the resolved default still omits that content. test_offline_spans_export_content_when_capture_content_is_false covers both sides. Staging was not rerun.

This change does not touch the generation_incomplete criterion event that Python still emits for a failed row. §8.8.6 tracks that as a separate divergence. Here the runner just stops wrapping it in a span.

Known view-side gap: gonfalon shows gen_ai.system_instructions as its raw JSON part array, because it reads that attribute as a plain string.

Neither repo has a pull request template (pull_request_template is unset). This stays a draft.