Repository navigation
telemetry: send a session summary in the recording header - #8069
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ssion summary The report's events are not uploaded, so the header is the only way agent evals can see the close reason and error counts. Drop the unused "version" key: every key is optional and new keys are additive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The summary is now the report's to_dict without chat history, events and the local recording path, plus agent_name, tags, duration_ms, user_turns, errors and close_reason as the livekit.agent.CloseReason proto enum name. Usage is the report's list with each entry's type added back. SessionReport.to_dict gets include_chat_history and include_events flags; defaults are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
to_dict appends agent_name, tags, duration_ms, user_turns, errors and close_reason after sdk_version, and each usage entry gains its type. The recording header's session summary is now to_dict without chat history, events and the local recording path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ol#1887) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…helper The local audio_recording_path is harmless, so the summary is exactly to_dict(include_chat_history=False, include_events=False). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| for event in self.events: | ||
| if event.type == "error": | ||
| key = _ERROR_KEYS.get(getattr(event.error, "type", "")) | ||
| if key is not None: | ||
| errors[key] += 1 |
There was a problem hiding this comment.
🟡 Interruption failures always count as zero
When interruption detection fails, errors.interruption remains zero. AgentActivity._on_error returns without emitting an ErrorEvent, leaving the summary without a failure to count.
Learn more
AgentActivity._on_error emits ErrorEvent for LLM, STT, TTS and realtime failures. For InterruptionDetectionError, it instead falls back to VAD if needed and returns without emitting anything. _error_counts sees only recorded ErrorEvent objects, so the advertised interruption category never reflects detector failures in normal sessions.
Example: An adaptive interruption detector fails twice during a call and the activity falls back to VAD. The uploaded session summary reports "interruption": 0, hiding both failures.
Recommended fix: Emit a corresponding ErrorEvent from the interruption-error branch in AgentActivity._on_error, including its source, before any fallback or return. Verify that the new event does not alter existing shutdown handling.
Was this helpful? React with 👍 or 👎 to provide feedback.
…eason Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CR_<member name>, checked against livekit.agent.CloseReason; TASK_COMPLETED is the one exception (CR_COMPLETED). Replaces the hand-written mapping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ing-session-summary
…s its tags Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What
_upload_session_reportsetsMetricsRecordingHeader.session_summary, a compact JSON object:SessionReport.to_dict(include_chat_history=False, include_events=False). The upload adds no other logic.SessionReport.to_dictgets keyword-onlyinclude_chat_historyandinclude_eventsflags (defaultTrue).Falseomits the key.SessionReportgainsagent_name(default"") andtags(default[]).JobContext.make_session_reportfills them from the job and the job'sTagger(sorted).to_dictchanges are additive. Every existing key keeps its value and order. These keys are appended aftersdk_version:agent_name,tagsduration_ms: int ms,nullwhen the report has no durationuser_turns: userChatMessageitems in the chat historyerrors: error event counts byllm,stt,tts,realtime,interruption(always all five keys)close_reason: thelivekit.agent.CloseReasonproto enum name for the last close event (e.g.CR_PARTICIPANT_DISCONNECTED),nullwithout one. Unknown reasons map toCR_UNKNOWN.usageentry into_dictnow includes itstype(llm_usage,tts_usage, ...). The other keys are unchanged (model_dump(exclude_defaults=True)).durationis now set from the report (ms).Why
Agent evals choose which sessions to score from facts the SDK knows at upload time. The collector stores the summary (livekit/cloud-observability#656) and agent-evals reads it (livekit/agents-private#745). Key contract:
evalschema/session-summary.schema.jsonin livekit/cloud-protocol (livekit/cloud-protocol#1415).nullmeans "not sent". The computed keys live into_dictso the logic exists in one place, and local session reports get them too.How
voice/report.pyholds the error-key mapping (_ERROR_KEYS) and the close-reason mapping (_CLOSE_REASON_NAMES, keyed byCloseReasonmembers)._session_summary(report)intelemetry/traces.pyis only the trimmedto_dictcall.errorsandclose_reasonreach agent evals only through the header.Tests
tests/test_session_options_report.py:to_dictdefault keys (old keys in order, then the new keys); each flag omits only its key; computed keys for a full session and an empty session; drift test: everyCloseReasonmember maps to a name in the generated proto enumlivekit.protocol.agent_pb.agent_session.CloseReason.tests/test_recording.py: the header carriesto_dict(include_chat_history=False, include_events=False).tests/test_realtime/test_openai_realtime_model.py: the usage entry expectation includestype.uv run pytest --unit --audio_eotpasses onlivekit-protocol1.1.28 (5780 passed; the only errors aretests/test_room.py, which needs a locallivekit-server).make type-checkpasses.Depends on
livekit-protocol1.1.28 (livekit-protocol v1.1.28 python-sdks#895). This PR requireslivekit-protocol>=1.1.28.🤖 Generated with Claude Code