Repository navigation
TEST-201 docs: bring Simulations pages up to date with Composer test generation and the run API - #1342
Open
scott-lowe-vapi wants to merge 2 commits into
Open
TEST-201 docs: bring Simulations pages up to date with Composer test generation and the run API#1342scott-lowe-vapi wants to merge 2 commits into
scott-lowe-vapi wants to merge 2 commits into
Conversation
…generation and the run API Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 19, 2026
Lightsage docs evalsLightsage could not queue docs evals for this PR. Docs URL: https://vapi-preview-01a1527c-c7e7-742f-bb13-ed300ead57a9.docs.buildwithfern.com |
Contributor
|
🌿 Preview your docs: https://vapi-preview-01a1526d-3cb5-753e-be8d-81136127992c.docs.buildwithfern.com |
Contributor
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
🌿 Preview your docs: https://vapi-preview-01a1527c-c7e7-742f-bb13-ed300ead57a9.docs.buildwithfern.com |
|
|
||
| <Steps> | ||
| <Step title="Review the requirements"> | ||
| Composer drafts a numbered list of **Test requirements**. Each one is a behavior a simulated caller could trigger. Composer tags a requirement **Assumption** when the prompt implies it without stating it, and **Can't be simulated** when a chat simulation can't check it, such as pronunciation or the exact wording of the first message. It also asks up to three open questions when an answer would change a requirement. |
There was a problem hiding this comment.
This is a little rough to read for me, I feel like:
Requirements are intended to outline simulated caller behaviors. We tag requirements with "assumption" when the prompt implies it without stating it, or "can't be simulated" when simulations can't test the behavior (e.g. pronunciation or exact wording). It will ask clarifying questions about the requirements.
is maybe more readable? Not sure
| Composer drafts a numbered list of **Test requirements**. Each one is a behavior a simulated caller could trigger. Composer tags a requirement **Assumption** when the prompt implies it without stating it, and **Can't be simulated** when a chat simulation can't check it, such as pronunciation or the exact wording of the first message. It also asks up to three open questions when an answer would change a requirement. | ||
| </Step> | ||
| <Step title="Accept or change them"> | ||
| Select **Generate coverage** to accept the draft, or type the changes you want in the chat. Composer revises the draft and asks you to review it again. It never generates a suite from requirements you haven't agreed to. |
There was a problem hiding this comment.
Suggested change
| Select **Generate coverage** to accept the draft, or type the changes you want in the chat. Composer revises the draft and asks you to review it again. It never generates a suite from requirements you haven't agreed to. | |
| Select **Generate coverage** to accept the draft, or iterate on them with Composer in the chat. Composer will revise the draft and ask you to review it again. It never generates a suite from requirements you haven't agreed to. |
| Composer generates the suite and shows it on a card. Each simulation covers one caller goal, and each requirement it covers becomes one of its success criteria. The card shows how many requirements are covered and why any aren't, and lists the handoffs and transfers the suite doesn't test as **Not tested**, with the reason. Every tool in the assistant's tool list that would reach an external system is [mocked by default](/observability/simulations-advanced#mock-tool-responses), and each simulation lists the tools it mocks. Tools that only a hook calls aren't mocked. | ||
| </Step> | ||
| <Step title="Run the suite"> | ||
| Select **Run suite** on the card, or ask Composer to run it. The suite runs once, in chat, against the assistant it was generated for. Composer doesn't wait for the run: progress and results appear at the run's dashboard URL. When the run ends, ask Composer about the results. |
There was a problem hiding this comment.
Suggested change
| Select **Run suite** on the card, or ask Composer to run it. The suite runs once, in chat, against the assistant it was generated for. Composer doesn't wait for the run: progress and results appear at the run's dashboard URL. When the run ends, ask Composer about the results. | |
| Select **Run suite** on the card, or ask Composer to run it. The suite runs once, in chat, against the assistant it was generated for. Composer doesn't wait for the run: progress and results appear on the assistant's test tab or the [runs page](https://dashboard.vapi.ai/simulations/runs). When the run ends, ask Composer about the results. |
| ### Deletion | ||
|
|
||
| Composer can delete resources such as assistants, tools, phone numbers, squads, and files. Deletion follows your approval mode: unless you have selected Bypass, Composer asks for your explicit approval before it deletes anything. Because deletion is permanent, keep an approval mode on if you want a confirmation step before a resource is removed. | ||
| Composer does not delete resources. Delete assistants, tools, phone numbers, squads, and files from the dashboard yourself. The approval settings include a delete category, so a policy you set now also applies if Composer gains deletion later. When Composer removes a simulation from a suite it generated, the simulation is unlinked from the suite, not deleted. |
There was a problem hiding this comment.
Does this mean we're gonna have a bunch of orphan suites?
| - **Cannot access external systems**: Composer operates within Vapi and cannot reach your CRM, database, or other third-party systems directly. | ||
| - **Cannot make business logic decisions**: You provide the domain knowledge; Composer handles the technical configuration. | ||
| - **Cannot test your agent**: You need to test the agent yourself and provide feedback. | ||
| - **Cannot test every behavior**: Suites Composer generates run in chat, so they can't check voice-only behavior such as pronunciation, and [some handoffs and transfers](/observability/simulations-advanced#handoffs-and-transfers-in-simulations) aren't tested. Make test calls yourself for these. |
There was a problem hiding this comment.
Three ideas are chained with "so" and "and". Splitting them up:
Suggested change
| - **Cannot test every behavior**: Suites Composer generates run in chat, so they can't check voice-only behavior such as pronunciation, and [some handoffs and transfers](/observability/simulations-advanced#handoffs-and-transfers-in-simulations) aren't tested. Make test calls yourself for these. | |
| - **Cannot test every behavior**: Generated suites run in chat, so they can't check voice-only behavior such as pronunciation. They also skip [some handoffs and transfers](/observability/simulations-advanced#handoffs-and-transfers-in-simulations). Test these with real calls. |
|
|
||
| ## Generate tests for an assistant | ||
|
|
||
| Ask Composer to generate tests for an assistant, for example "Generate a test suite for my dental office agent." Composer reads the assistant's prompt and tools, agrees a list of testable requirements with you, then builds a [simulation suite](/observability/simulations-overview) that covers them. |
There was a problem hiding this comment.
"agrees a list of testable requirements with you" is hard to parse.
Suggested change
| Ask Composer to generate tests for an assistant, for example "Generate a test suite for my dental office agent." Composer reads the assistant's prompt and tools, agrees a list of testable requirements with you, then builds a [simulation suite](/observability/simulations-overview) that covers them. | |
| Ask Composer to generate tests for an assistant, for example "Generate a test suite for my dental office agent." Composer reads the assistant's prompt and tools and drafts a list of testable requirements. Once you agree on them, it builds a [simulation suite](/observability/simulations-overview) that covers them. |
| Select **Generate coverage** to accept the draft, or type the changes you want in the chat. Composer revises the draft and asks you to review it again. It never generates a suite from requirements you haven't agreed to. | ||
| </Step> | ||
| <Step title="Review the suite"> | ||
| Composer generates the suite and shows it on a card. Each simulation covers one caller goal, and each requirement it covers becomes one of its success criteria. The card shows how many requirements are covered and why any aren't, and lists the handoffs and transfers the suite doesn't test as **Not tested**, with the reason. Every tool in the assistant's tool list that would reach an external system is [mocked by default](/observability/simulations-advanced#mock-tool-responses), and each simulation lists the tools it mocks. Tools that only a hook calls aren't mocked. |
There was a problem hiding this comment.
This sentence carries five facts. A list makes the card easy to scan.
Suggested change
| Composer generates the suite and shows it on a card. Each simulation covers one caller goal, and each requirement it covers becomes one of its success criteria. The card shows how many requirements are covered and why any aren't, and lists the handoffs and transfers the suite doesn't test as **Not tested**, with the reason. Every tool in the assistant's tool list that would reach an external system is [mocked by default](/observability/simulations-advanced#mock-tool-responses), and each simulation lists the tools it mocks. Tools that only a hook calls aren't mocked. | |
| Composer generates the suite and shows it on a card. The card shows: | |
| - Each simulation. A simulation covers one caller goal, and the requirements it covers become its success criteria. | |
| - How many requirements are covered, and why any aren't. | |
| - Handoffs and transfers the suite doesn't test, marked **Not tested**, with the reason. | |
| - The tools each simulation mocks. | |
| Composer [mocks](/observability/simulations-advanced#mock-tool-responses) every tool that would reach an external system. Tools that only a hook calls aren't mocked. |
| - **Rewrite a simulation**: Describe the change, for example "make the caller refuse to give a date of birth." | ||
| - **Retry failed simulations**: When some simulations couldn't be generated, Composer offers to try them again. | ||
|
|
||
| The requirements you agreed to are saved with the suite, and edits always work from that saved set. Revising the requirements later doesn't change an existing suite: Composer generates a new suite from the new requirements. |
There was a problem hiding this comment.
Passive voice plus a colon; simpler:
Suggested change
| The requirements you agreed to are saved with the suite, and edits always work from that saved set. Revising the requirements later doesn't change an existing suite: Composer generates a new suite from the new requirements. | |
| Composer saves the agreed requirements with the suite, and every edit works from them. If you change the requirements later, Composer builds a new suite. The existing suite doesn't change. |
| The requirements you agreed to are saved with the suite, and edits always work from that saved set. Revising the requirements later doesn't change an existing suite: Composer generates a new suite from the new requirements. | ||
|
|
||
| <Tip> | ||
| One passing run shows the assistant **can** succeed, not that it does so reliably. Run five or more iterations before you trust a pass rate: ask Composer, or [rerun the suite](/observability/simulations-manage#rerun-a-suite) with more **Iterations**. See [testing best practices](/test/simulations-best-practices). |
There was a problem hiding this comment.
Optional tightening:
Suggested change
| One passing run shows the assistant **can** succeed, not that it does so reliably. Run five or more iterations before you trust a pass rate: ask Composer, or [rerun the suite](/observability/simulations-manage#rerun-a-suite) with more **Iterations**. See [testing best practices](/test/simulations-best-practices). | |
| One passing run shows the assistant **can** succeed, not that it will every time. Run five or more iterations before you trust the pass rate. Ask Composer to run more, or [rerun the suite](/observability/simulations-manage#rerun-a-suite) with more **Iterations**. See [testing best practices](/test/simulations-best-practices). |
| | --- | --- | | ||
| | `GET /eval/simulation/run` | `suiteId` (repeat for up to 100 suites), `targetType` (`assistant` or `squad`), `targetId`, and `status` (`queued`, `running`, or `ended`). `targetType` and `targetId` each work alone. | | ||
| | `GET /eval/simulation/run/<run-id>/item` | `status` (`queued`, `running`, `evaluating`, `passed`, `failed`, or `canceled`) and `simulationId`. | | ||
| | `GET /eval/simulation/suite` | `assistantId` and `squadId` (repeat each for up to 100 IDs; sending both requires both to match) and `name`, which matches part of the suite name. | |
There was a problem hiding this comment.
Maybe replace "sending both requires both to match" with "if you send both, a suite must match both".
| Each list also accepts `limit` (default 100, maximum 1,000) and `page`. | ||
|
|
||
| <Note> | ||
| The response shape depends on the query. With no query parameters, these endpoints return a plain array. With any query parameter, the run and run item lists return a paginated object, `{ "metadata": { ... }, "results": [ ... ] }`, with at most `limit` records. The suite list does the same when you send `limit`, `page`, `name`, `assistantId`, or `squadId`. If your integration parses the response as an array, read `results` whenever you send a query parameter. |
There was a problem hiding this comment.
Dense note; a short lead plus bullets:
Suggested change
| The response shape depends on the query. With no query parameters, these endpoints return a plain array. With any query parameter, the run and run item lists return a paginated object, `{ "metadata": { ... }, "results": [ ... ] }`, with at most `limit` records. The suite list does the same when you send `limit`, `page`, `name`, `assistantId`, or `squadId`. If your integration parses the response as an array, read `results` whenever you send a query parameter. | |
| Adding query parameters changes the response shape: | |
| - With no query parameters, these endpoints return a plain array. | |
| - The run and run item lists return `{ "metadata": { ... }, "results": [ ... ] }` when you send any query parameter. | |
| - The suite list returns the same object when you send `limit`, `page`, `name`, `assistantId`, or `squadId`. | |
| If your code expects an array, read `results` whenever you send a query parameter. |
|
|
||
| - A retry with the same key and the same body returns the run the first request created, with `reused: true`. It doesn't start or bill the run again. | ||
| - Reusing a key with a different body returns `409`. Use a new key for a different run. | ||
| - If a request returns `503` with `code: SIMULATION_WORK_DISABLED`, simulations are paused across Vapi. Retry later. Any other `503` can be retried with the same key. |
There was a problem hiding this comment.
Active voice for the last sentence:
Suggested change
| - If a request returns `503` with `code: SIMULATION_WORK_DISABLED`, simulations are paused across Vapi. Retry later. Any other `503` can be retried with the same key. | |
| - If a request returns `503` with `code: SIMULATION_WORK_DISABLED`, simulations are paused across Vapi. Retry later. Retry any other `503` with the same key. |
| - Reusing a key with a different body returns `409`. Use a new key for a different run. | ||
| - If a request returns `503` with `code: SIMULATION_WORK_DISABLED`, simulations are paused across Vapi. Retry later. Any other `503` can be retried with the same key. | ||
| - A key is up to 255 printable ASCII characters and is scoped to your organization. An empty or invalid key returns `400`. | ||
| - A key stays bound to its run for as long as the run exists. It does not expire after a set time. |
There was a problem hiding this comment.
Suggested change
| - A key stays bound to its run for as long as the run exists. It does not expire after a set time. | |
| - Keys don't expire. A key stays tied to its run for as long as the run exists. |
| ### Edge-case tests | ||
|
|
||
| Test realistic variations such as an ambiguous request, an impatient customer, an unavailable appointment, a tool error, an interruption, or a handoff. Base AI tester personalities on the customer types the [**assistant**](/assistants) or [**squad**](/squads) handles. Synthetic callers don't reliably reproduce every interruption, silence, or background-noise condition. Keep controlled real-call checks for problems the simulation can't reproduce. | ||
| Test realistic variations such as an ambiguous request, an impatient customer, an unavailable appointment, a tool error, an interruption, or a handoff. Some handoffs and transfers [behave differently in simulations](/observability/simulations-advanced#handoffs-and-transfers-in-simulations). Base AI tester personalities on the customer types the [**assistant**](/assistants) or [**squad**](/squads) handles. Synthetic callers don't reliably reproduce every interruption, silence, or background-noise condition. Keep controlled real-call checks for problems the simulation can't reproduce. |
There was a problem hiding this comment.
The inserted sentence about handoffs breaks the paragraph's flow. Consider moving it to the end of the paragraph.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Linear: TEST-201. Companion in-repo docs: VapiAI/vapi#23428.
/composer#generate-tests-for-an-assistant): requirements draft → Generate coverage → suite card with mocked tools and not-tested gaps → Run suite, plus edits and a tip to run five or more iterations.discard-draft, for your own unpublished draft.customer.speech.timeoutand the tester hang-up ending ascustomer-ended-call.Idempotency-Keyheader onPOST /eval/simulation/run.suiteId,targetType,targetId,status,simulationId,assistantId,squadId,name).{ metadata, results }.Evidence
Every behavior was checked against VapiAI/vapi
main:features/simulationsOnboarding/constants.ts.simulationRunIdempotencyKeyParse.tsandsimulationRunIdempotentReplayGet.ts.simulationRun.types.tsandGetSimulationSuitesDTO.ts.handoffSimulationRefusalGet.tsandsimulationTargetReengagementHooks/.Testing Steps
fern check(5.151.1) passes: 0 errors. The 14 warnings were already on main.