Repository navigation
TEST-203 docs: add the Simulations onboarding API guide - #1344
Draft
scott-lowe-vapi wants to merge 2 commits into
Draft
scott-lowe-vapi wants to merge 2 commits into
scott-lowe-vapi wants to merge 2 commits into
Conversation
Contributor
|
🌿 Preview your docs: https://vapi-preview-01a1526d-5544-7429-b82e-095836bfb9eb.docs.buildwithfern.com |
Contributor
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Oct 19, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scott-lowe-vapi
force-pushed
the
scottlowe/test-203-onboarding-api-guide
branch
from
October 19, 2026 04:46
ef56047 to
b6e5fd4
Compare
Contributor
|
🌿 Preview your docs: https://vapi-preview-01a1527c-90fc-733c-869a-5f48d4798f2d.docs.buildwithfern.com |
Contributor
|
🌿 Preview your docs: https://vapi-preview-01a1527c-d56f-7645-8012-59af756e398c.docs.buildwithfern.com |
| - **Cannot make business logic decisions**: You provide the domain knowledge; Composer handles the technical configuration. | ||
| - **Cannot test every behavior**: Suites Composer generates run in chat, so they can't check voice-only behavior such as pronunciation, and [some handoffs and transfers](/observability/simulations-advanced#handoffs-and-transfers-in-simulations) aren't tested. Make test calls yourself for these. | ||
| - **Cannot deploy to production automatically**: You control when and how changes go live. | ||
| - **Daily limits on test generation**: Requirement drafts, suite generations, and suite edits share a free daily allowance per organization with the [Simulations onboarding API](/observability/simulations-generate-api#limits). When your organization reaches a limit, Composer tells you and doesn't retry. The limits reset at 00:00 UTC. |
There was a problem hiding this comment.
The subject is three nouns, and "with the API" comes at the end.
Suggested change
| - **Daily limits on test generation**: Requirement drafts, suite generations, and suite edits share a free daily allowance per organization with the [Simulations onboarding API](/observability/simulations-generate-api#limits). When your organization reaches a limit, Composer tells you and doesn't retry. The limits reset at 00:00 UTC. | |
| - **Daily limits on test generation**: Each organization gets a free daily allowance of requirement drafts, suite generations, and suite edits. Composer and the [Simulations onboarding API](/observability/simulations-generate-api#limits) share it. When you reach a limit, Composer tells you and doesn't retry. Limits reset at 00:00 UTC. |
| slug: observability/simulations-generate-api | ||
| --- | ||
|
|
||
| The Simulations onboarding API builds a [**simulation suite**](/observability/simulations-overview) for an assistant from its prompt, tools, and handoffs. It's the same flow [**Composer**](/composer#generate-tests-for-an-assistant) runs when you ask it to generate tests, exposed as three endpoints you can call from a script, a CI job, or a coding agent: |
There was a problem hiding this comment.
Suggested change
| The Simulations onboarding API builds a [**simulation suite**](/observability/simulations-overview) for an assistant from its prompt, tools, and handoffs. It's the same flow [**Composer**](/composer#generate-tests-for-an-assistant) runs when you ask it to generate tests, exposed as three endpoints you can call from a script, a CI job, or a coding agent: | |
| The Simulations onboarding API builds a [**simulation suite**](/observability/simulations-overview) for an assistant from its prompt, tools, and handoffs. It runs the same flow as [**Composer**](/composer#generate-tests-for-an-assistant). Call these three endpoints from a script, a CI job, or a coding agent: |
| You then run the suite with the existing [`POST /eval/simulation/run`](/observability/simulations-quickstart) endpoint. | ||
|
|
||
| <Note> | ||
| **Beta.** Private API keys can call these endpoints only for organizations enabled during the beta. For any other organization they return `404`. [Contact support](/support) to enable your organization. Composer and the Dashboard work without this. |
There was a problem hiding this comment.
Suggested change
| **Beta.** Private API keys can call these endpoints only for organizations enabled during the beta. For any other organization they return `404`. [Contact support](/support) to enable your organization. Composer and the Dashboard work without this. | |
| **Beta.** During the beta, these endpoints only work for organizations we've enabled. Other organizations get a `404`. [Contact support](/support) to enable yours. You don't need this to use Composer or the Dashboard. |
| - An HTTP client with a long timeout. Suite generation can take up to 3 minutes, so set a client timeout of at least 200 seconds. | ||
|
|
||
| <Warning> | ||
| Generated simulations mock every tool in the assistant's tool list that would reach an external system, and the response lists them in `mockedTools`. Tools that only an assistant hook calls aren't mocked, and the assistant's server messages, such as the end-of-call report, still go to its server URL. Tools you add to a simulation later, or mocks you turn off, run for real. See [**Mock tool responses**](/observability/simulations-advanced#mock-tool-responses). |
There was a problem hiding this comment.
Four ideas in two sentences (and it repeats Advanced). Bullets:
Suggested change
| Generated simulations mock every tool in the assistant's tool list that would reach an external system, and the response lists them in `mockedTools`. Tools that only an assistant hook calls aren't mocked, and the assistant's server messages, such as the end-of-call report, still go to its server URL. Tools you add to a simulation later, or mocks you turn off, run for real. See [**Mock tool responses**](/observability/simulations-advanced#mock-tool-responses). | |
| Generated simulations mock every tool in the assistant's tool list that would reach an external system. The response lists them in `mockedTools`. These still run for real: | |
| - Tools that only an assistant hook calls. | |
| - Server messages, such as the end-of-call report, which still go to the assistant's server URL. | |
| - Tools you add to a simulation later, or mocks you turn off. | |
| See [**Mock tool responses**](/observability/simulations-advanced#mock-tool-responses). |
| } | ||
| ``` | ||
|
|
||
| Review the requirements and answer the open questions. A requirement tagged `assumption` rests on a guess; one tagged `not-testable` describes something a chat simulation can't check, such as pronunciation. |
There was a problem hiding this comment.
Split at the semicolon:
Suggested change
| Review the requirements and answer the open questions. A requirement tagged `assumption` rests on a guess; one tagged `not-testable` describes something a chat simulation can't check, such as pronunciation. | |
| Review the requirements and answer the open questions. A requirement tagged `assumption` is based on a guess. A requirement tagged `not-testable` is something a chat simulation can't check, such as pronunciation. |
| | `destination-truncated` | The destination wasn't checked because the assistant has too many handoff destinations to check in time, or it didn't load in time. The handoff into it is mocked. | | ||
| | `transfer-external` | A `transferCall` can leave Vapi, to a phone number, a SIP address, or a destination your server chooses, so it is mocked. | | ||
| | `destination-name-unscoped` | A handoff names a destination assistant outside a squad, which can't be resolved at runtime, so that destination is never reached. The handoff isn't mocked. | | ||
| | `native-tool-shadowed` | A tool that normally runs as is (such as `endCall` or `transferCall`) shares its name with a mocked tool, so the mock answers it too. | |
There was a problem hiding this comment.
"A tool that normally runs as is" is awkward. Maybe: "A built-in tool, such as endCall or transferCall, has the same name as a mocked tool, so the mock answers it too."
|
|
||
| A retry that returns a saved suite sets `reused` to `true` and doesn't count toward your limits. To generate a new suite from a revised draft, use a new key. | ||
|
|
||
| **Edits.** Send a new `patchKey` for each edit. Replaying a key that was already applied changes nothing and returns the current suite, with `failedEditSimulationIds` listing what that edit couldn't rewrite. So you can retry a timeout or a `5xx` with the same key. A suite remembers its last 50 patch keys, so never reuse one. |
There was a problem hiding this comment.
This contradicts itself: it says to retry a timeout with the same key, then ends "so never reuse one".
Suggested change
| **Edits.** Send a new `patchKey` for each edit. Replaying a key that was already applied changes nothing and returns the current suite, with `failedEditSimulationIds` listing what that edit couldn't rewrite. So you can retry a timeout or a `5xx` with the same key. A suite remembers its last 50 patch keys, so never reuse one. | |
| **Edits.** Send a new `patchKey` for each edit, and the same key to retry it. Replaying a key that was already applied changes nothing. It returns the current suite, with `failedEditSimulationIds` listing what that edit couldn't rewrite. A suite remembers only its last 50 patch keys, so never reuse a key for a different edit. |
|
|
||
| ## Limits | ||
|
|
||
| Each organization has a free daily allowance per UTC day, shared by the API, the Dashboard, and Composer. Limits reset at 00:00 UTC. |
There was a problem hiding this comment.
"a free daily allowance per UTC day" says "daily" twice. Maybe: "Each organization has a free daily allowance, shared by the API, the Dashboard, and Composer. Limits reset at 00:00 UTC."
|
|
||
| | Status | When | What to do | | ||
| | -- | -- | -- | | ||
| | `400` | The body is invalid; `previousDraft` and `feedback` were not sent together; or the generated or edited suite uses something your organization's data-region policy doesn't allow, so it isn't saved. | Fix the request. | |
There was a problem hiding this comment.
This cell (and the 422 row below) lists several causes separated by semicolons and parentheses. Consider one row per cause, or <br /> between causes.
| | --- | --- | | ||
| | Every iteration fails | Review the transcript, then verify the evaluation's description, data type, comparator, and expected value. | | ||
| | A run stays queued or running | Check the assistant or squad configuration and its provider credentials. Fetch the run and its items to find a failure reason. | | ||
| | Creating a voice run returns `400 Target assistant would fail to start a call: <message>` | The inline assistant in `target` has a configuration a call would reject, such as a tool type the model doesn't support. Fix what the message names, then create the run again. Vapi checks this only for voice runs with an inline assistant target, not for a saved `assistantId`, a squad, or a chat run. | |
There was a problem hiding this comment.
Optional: "Vapi checks this only for voice runs…" → "Vapi runs this check only for voice runs with an inline assistant target. It doesn't check a saved assistantId, a squad, or a chat run."
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Draft: merge when VapiAI/vapi#23422 (TEST-200) deploys. That PR publishes the onboarding endpoints and lists this guide as a merge gate. Linear: TEST-203. Stacked on #1342.
/observability/simulations-generate-api), added to the nav after Advanced. It covers:idempotencyKey, edit withpatchKey, run withIdempotency-KeyFailed to create target callfailure reason (VapiAI/vapi#23400).Sources are the open stack's diffs (#23400, #23407, #23418–#23422). The 524 in the retry list comes from the endpoint's own operation description.
Before merge:
fern api updatepulls the routes in.What's New blurb:
Testing Steps
fern check(5.151.1) passes: 0 errors. The 14 warnings were already on main.