Python: fix tools from context providers

[sophia-ramsey]

Motivation & Context

The agent evaluation previously omitted tools injected by context providers, causing Foundry tool-selection and tool-input-accuracy evaluators to return null scores because tool definitions were missing. This change captures the tool definitions available during execution and preserves schema-only snapshots across response serialization, so they remain available for evaluation. It supports evaluating agents with dynamically provided tools, both immediately after execution and from saved responses. Tool snapshots are preserved through streaming reconstruction, approval continuations, and loops.

Description & Review Guide

• What are the major changes? Capturing run-specific tool definitions, including provider-injected and dynamically exposed tools, and preserving schema-only snapshots across JSON/pickle serialization and agent-loop aggregation. Evaluation automatically uses these definitions, with an optional tools= override.
• What is the impact of these changes? Evaluation receiving the definitions needed to score tool calls without persisting executable callbacks or connections. Serialized responses gain tool-schema metadata, and schema capture introduces earlier JSON-compatibility validation that can reject non-serializable custom schemas.
• What do you want reviewers to focus on? Potential changes in serialization, specifically whether the metadata and schema validation introduces compatibility changes. The addition of tools= was not explicitly required to resolve this issue, but it seemed like it would improve the developer experience. Similarly, response additional_properties metadata now contains agent_framework.evaluation with version and tools (which can include an empty "tools": []).

Related Issue

Fixes #9160

AI Assistance

• No material AI assistance was used.
• This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: Used copilot with GPT-6.1 Sol for solution planning, design evaluation, implementation, and testing.

Contribution Checklist

• The code builds clean without any errors or warnings
• All unit tests pass, and I have added new tests where possible
• The PR follows the Contribution Guidelines
• This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
• This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

[github-code-quality]

Code coverage overview

Languages: Python

Python / code-coverage/python

main 54b558b	sramsey/python-tool-... c1acc93	+/-
92%	92%	0%

View coverage of the 8 most impacted files.

File	main 54b558b	sramsey/python-tool-... c1acc93	+/-
packages/core/a...ework/_types.py	96%	96%	0%
packages/ag-ui/...i/_agent_run.py	90%	90%	0%
packages/core/a.../_evaluation.py	88%	88%	0%
packages/core/a...ool_approval.py	91%	91%	0%
packages/core/a...work/_skills.py	96%	96%	0%
packages/core/a...arness/_loop.py	98%	98%	0%
packages/core/a...ork/_clients.py	93%	93%	0%
packages/core/a...work/_agents.py	95%	96%	+1%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.

---

_{Updated October 19, 2026 21:26 UTC}

[Copilot]

🟡 Changes recommended

Serialization and supported streaming/client paths can still lose or misrepresent tool definitions.

4 open findings

• Preserve tool snapshots in middleware streaming responses · New
• Capture dynamically added tools at each model call · New
• Preserve tuple values in serialized snapshots · New
• Persist empty tool snapshots across serialization · New

What changed in this PR

Addresses #9160 by capturing run-specific tool definitions for evaluation, including tools supplied by context providers.

Changes:

• Captures schema-only tool snapshots and preserves them through serialization.
• Adds an explicit tools= evaluation override and loop aggregation support.
• Adds regression tests and documents snapshot behavior.

File	Description
python/​packages/​core/​tests/​core/​test_evaluation.py	Tests tool discovery, overrides, and serialization.
python/​packages/​core/​AGENTS.md	Documents evaluation snapshots.
python/​packages/​core/​agent_framework/​_types.py	Stores and restores tool schemas.
python/​packages/​core/​agent_framework/​_harness/​_loop.py	Aggregates definitions across loop iterations.
python/​packages/​core/​agent_framework/​_evaluation.py	Uses captured definitions and explicit overrides.
python/​packages/​core/​agent_framework/​_agents.py	Captures tools during execution.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

MAF Automated Review — Iteration 1

Result: Findings reported
Scope: full PR (2 commit(s)): 813d7c0994f1, 36a84a8608a6
Model: gpt-5.6-sol

Overview

The change captures run-specific tool declarations with deep-copied, JSON-validated schemas, restores them as non-executable tools, and carries them through normal response serialization and loop aggregation. Per-run capture, malformed-metadata validation, explicit overrides, and broad streaming/non-streaming tests are strong guardrails. Three serialization and lifecycle gaps remain: empty snapshots are erased, same-name progressive replacements retain stale schemas, and update-only streaming round trips omit the snapshots.

Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
3 verified findings remained after source verification (3 medium) across 2 files. Details are attached to the affected lines below.

Affected areas: python/packages/core/agent_framework/_agents.py, python/packages/core/agent_framework/_types.py