Python: fix(eval): include ContextProvider and SkillsProvider tools in evaluation definitions

[Nithin (nithin42)]

Closes #9160

Description

When running evaluate_agent() (and evaluating metrics like tool_selection and tool_input_accuracy), tool definitions were only extracted from agent.default_options["tools"] and agent.mcp_tools[*].functions. Tools contributed by agent.context_providers (such as SkillsProvider or VectorCollectionContextProvider) were omitted.

As a result, agents equipped with SkillsProvider (load_skill, read_skill_resource, run_skill_script) or other context providers ended up with tools=None (or incomplete tool lists) in the generated EvalItem, breaking evaluator scoring on tool selection and tool input accuracy.

Changes

1. SkillsProvider.tools: Added @property def tools(self) -> list[FunctionTool] returning the canonical skill tools (load_skill, read_skill_resource, run_skill_script), matching providers like VectorCollectionContextProvider.
2. _evaluation.py:
   • Updated _to_eval_item to extract and normalize tools from getattr(agent, "context_providers", []) (tools or get_tools()), deduplicating against existing tools by name.
   • Updated legacy AgentEvalConverter.extract_tools to also include context provider tools.
3. Tests:
   • Added unit test in test_skills.py verifying SkillsProvider.tools.
   • Added unit tests in test_evaluation.py verifying context provider tool extraction, SkillsProvider integration, and deduplication across both _to_eval_item and AgentEvalConverter.extract_tools.

[Copilot]

🟡 Changes recommended

Both extractors mistake CodeAct sandbox helpers for model-facing tools and omit the required execute_code definition.

1 open finding

• Fallback tool extraction omits required execute_code definition · New

What changed in this PR

Extends Python evaluation tool discovery to include context-provider tools, addressing missing definitions reported in #9160.

Changes:

• Exposes canonical skill tool definitions through SkillsProvider.tools.
• Adds provider-tool extraction and deduplication to both evaluation converters.
• Adds tests for provider tools, skills integration, and deduplication.

File	Description
python/​packages/​core/​tests/​core/​test_skills.py	Tests canonical skill tool exposure.
python/​packages/​core/​tests/​core/​test_evaluation.py	Tests provider-tool extraction and deduplication.
python/​packages/​core/​agent_framework/​_skills.py	Adds the public tools property.
python/​packages/​core/​agent_framework/​_evaluation.py	Includes context-provider tools in evaluation definitions.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Both extraction paths still omit tools from get_tools()-only providers, including SecureAgentConfig.

1 open finding

• Support get_tools()-only providers in evaluation tool extraction · New

1 resolved since last review

• Fallback tool extraction omits required execute_code definition

🧠 Review effort: Balanced

[Copilot]

🔵 Needs a closer look

Both extraction paths can invoke a provider-exposed FunctionTool instead of collecting its definition.

0 open findings

1 resolved since last review

• Support get_tools()-only providers in evaluation tool extraction

Previously missed (1)

In code that hasn't changed since last review

Avoid invoking single FunctionTool instances during tool extraction

python/​packages/​core/​agent_framework/​_evaluation.py:920

If a custom provider exposes a single FunctionTool through .tools, this branch calls it instead of collecting its definition. FunctionTool.__call__ invokes the wrapped function, so a synchronous no-argument tool can cause side effects during evaluation; a tool requiring arguments is silently omitted by the exception handler. normalize_tools already accepts single FunctionTool instances. Exclude them from this accessor-call branch and apply the same guard in AgentEvalConverter.extract_tools at line 847. Add regression coverage for both extraction paths to verify the tool is collected without being invoked.

🧠 Review effort: Balanced

[Eduard van Valkenburg (eavanvalkenburg)]

Nithin (@nithin42) Thanks for the contribution and for working through the evaluation edge cases. The team has decided to implement #9160 internally in #9403, so we are closing this overlapping PR to keep a single implementation path.