Skip to content

TEST-203 docs: add the Simulations onboarding API guide #1344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
scott-lowe-vapi wants to merge 2 commits into
base: scottlowe/test-201-simulations-docs-catch-up
Choose a base branch
from
+377 −0
Draft

TEST-203 docs: add the Simulations onboarding API guide #1344

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5309b88
TEST-203 docs: add the Simulations onboarding API guide
scott-lowe-vapi Oct 19, 2026
b6e5fd4
TEST-203 docs: note unmocked hook tools in the API guide
scott-lowe-vapi Oct 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions fern/composer.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ Composer understands voice agent architecture and Vapi's capabilities. It can cr
- **Cannot make business logic decisions**: You provide the domain knowledge; Composer handles the technical configuration.
- **Cannot test every behavior**: Suites Composer generates run in chat, so they can't check voice-only behavior such as pronunciation, and [some handoffs and transfers](/observability/simulations-advanced#handoffs-and-transfers-in-simulations) aren't tested. Make test calls yourself for these.
- **Cannot deploy to production automatically**: You control when and how changes go live.
- **Daily limits on test generation**: Requirement drafts, suite generations, and suite edits share a free daily allowance per organization with the [Simulations onboarding API](/observability/simulations-generate-api#limits). When your organization reaches a limit, Composer tells you and doesn't retry. The limits reset at 00:00 UTC.

@chris-garber-vapi chris-garber-vapi Oct 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The subject is three nouns, and "with the API" comes at the end.

Suggested change
- **Daily limits on test generation**: Requirement drafts, suite generations, and suite edits share a free daily allowance per organization with the [Simulations onboarding API](/observability/simulations-generate-api#limits). When your organization reaches a limit, Composer tells you and doesn't retry. The limits reset at 00:00 UTC.
- **Daily limits on test generation**: Each organization gets a free daily allowance of requirement drafts, suite generations, and suite edits. Composer and the [Simulations onboarding API](/observability/simulations-generate-api#limits) share it. When you reach a limit, Composer tells you and doesn't retry. Limits reset at 00:00 UTC.


## Common use cases

Expand Down
3 changes: 3 additions & 0 deletions fern/docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -692,6 +692,9 @@ navigation:
- page: Advanced
path: observability/simulations-advanced.mdx
icon: fa-light fa-flask-vial
- page: Generate with the API
path: observability/simulations-generate-api.mdx
icon: fa-light fa-wand-magic-sparkles
- page: Configure an AI tester
path: observability/simulations-configure-ai-tester.mdx
icon: fa-light fa-robot
Expand Down
2 changes: 2 additions & 0 deletions fern/observability/simulations-advanced.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -379,6 +379,8 @@ For a transient scenario, set the same `targetOverrides` on the inline `scenario
| --- | --- |
| Every iteration fails | Review the transcript, then verify the evaluation's description, data type, comparator, and expected value. |
| A run stays queued or running | Check the assistant or squad configuration and its provider credentials. Fetch the run and its items to find a failure reason. |
| Creating a voice run returns `400 Target assistant would fail to start a call: <message>` | The inline assistant in `target` has a configuration a call would reject, such as a tool type the model doesn't support. Fix what the message names, then create the run again. Vapi checks this only for voice runs with an inline assistant target, not for a saved `assistantId`, a squad, or a chat run. |

@chris-garber-vapi chris-garber-vapi Oct 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: "Vapi checks this only for voice runs…" → "Vapi runs this check only for voice runs with an inline assistant target. It doesn't check a saved assistantId, a squad, or a chat run."

@stephenvapiai stephenvapiai Oct 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same actor point as on the API page.

Suggested change
| Creating a voice run returns `400 Target assistant would fail to start a call: <message>` | The inline assistant in `target` has a configuration a call would reject, such as a tool type the model doesn't support. Fix what the message names, then create the run again. Vapi checks this only for voice runs with an inline assistant target, not for a saved `assistantId`, a squad, or a chat run. |
| Creating a voice run returns `400 Target assistant would fail to start a call: <message>` | The inline assistant in `target` has a configuration a call would reject, such as a tool type the model doesn't support. Fix what the message names, then create the run again. This check runs only for voice runs with an inline assistant target, not for a saved `assistantId`, a squad, or a chat run. |

| A run item fails with `Failed to create target call: <message>` | The assistant or squad under test failed call validation when the run item started its call. Fix what the message names, then rerun. No AI tester call is started for that item. |
| Results vary between runs | Make the intent and evaluations more specific, then run multiple iterations to measure consistency. |
| There is no audio or recording | Check whether the run used chat mode. Use voice mode to test or record audio. |
| Start or end webhooks do not trigger | Confirm the corresponding hook and its URL are configured on the scenario. |
Expand Down
Loading
Loading