diff --git a/fern/composer.mdx b/fern/composer.mdx index 80f62d5a0..07f53e4d7 100644 --- a/fern/composer.mdx +++ b/fern/composer.mdx @@ -54,15 +54,16 @@ Composer understands voice agent architecture and Vapi's capabilities. It can cr - Configure integrations and webhooks - Update agent prompts and settings - Troubleshoot technical issues +- Create and run [simulations](/observability/simulations-overview), and [generate a test suite](#generate-tests-for-an-assistant) for an assistant - Answer questions about Vapi features - Recommend best practices for your use case ### Limitations -- **Cannot delete resources**: Composer cannot delete assistants, tools, phone numbers, or any other resources. This is an intentional safety measure. Use the dashboard sidebar to delete resources manually. +- **Cannot delete resources**: Composer cannot delete assistants, tools, phone numbers, or any other resources. This is an intentional safety measure. Use the dashboard sidebar to delete resources manually. Composer can discard your own unpublished assistant or tool draft, which never changes the live resource. - **Cannot access external systems**: Composer operates within Vapi and cannot reach your CRM, database, or other third-party systems directly. - **Cannot make business logic decisions**: You provide the domain knowledge; Composer handles the technical configuration. -- **Cannot test your agent**: You need to test the agent yourself and provide feedback. +- **Cannot test every behavior**: Generated suites run in chat, so they can't check voice-only behavior such as pronunciation. They also skip [some handoffs and transfers](/observability/simulations-advanced#handoffs-and-transfers-in-simulations). Test these with real calls. - **Cannot deploy to production automatically**: You control when and how changes go live. ## Common use cases @@ -171,6 +172,44 @@ Composer understands voice agent architecture and Vapi's capabilities. It can cr +## Generate tests for an assistant + +Ask Composer to generate tests for an assistant, for example "Generate a test suite for my dental office agent." Composer reads the assistant's prompt and tools and drafts a list of testable requirements. Once you agree on them, it builds a [simulation suite](/observability/simulations-overview) that covers them. + + + + Composer drafts a numbered list of **Test requirements**. Each one is a behavior a simulated caller could trigger. Composer tags a requirement **Assumption** when the prompt implies it without stating it. It tags a requirement **Can't be simulated** when a chat simulation can't check it, such as pronunciation or the exact wording of the first message. It also asks up to three open questions when an answer would change a requirement. + + + Select **Generate coverage** to accept the draft, or describe the changes you want in the chat. Composer revises the draft and asks you to review it again. It never generates a suite from requirements you haven't agreed to. + + + Composer generates the suite and shows it on a card. The card shows: + + - Each simulation. A simulation covers one caller goal, and the requirements it covers become its success criteria. + - How many requirements are covered, and why any aren't. + - Handoffs and transfers the suite doesn't test, marked **Not tested**, with the reason. + - The tools each simulation mocks. + + Every tool in the assistant's tool list that would reach an external system is [mocked by default](/observability/simulations-advanced#mock-tool-responses). Tools that only a hook calls aren't mocked. + + + Select **Run suite** on the card, or ask Composer to run it. The suite runs once, in chat, against the assistant it was generated for. Composer doesn't wait for the run. Follow progress and results on the [runs page](https://dashboard.vapi.ai/simulations/runs). When the run ends, ask Composer about the results. + + + +After generating a suite, ask Composer to change it in the same conversation: + +- **Remove simulations**: Removed simulations are unlinked from the suite, not deleted. +- **Rewrite a simulation**: Describe the change, for example "make the caller refuse to give a date of birth." +- **Retry failed simulations**: When some simulations couldn't be generated, Composer offers to try them again. + +Composer saves the agreed requirements with the suite, and every edit works from them. If you change the requirements later, Composer builds a new suite. The existing suite doesn't change. It remains in your account until you delete it from the dashboard. + + +One passing run shows the assistant **can** succeed, not that it will every time. Run five or more iterations before you trust the pass rate. Ask Composer to run more, or [rerun the suite](/observability/simulations-manage#rerun-a-suite) with more **Iterations**. See [testing best practices](/test/simulations-best-practices). + + ## Safety features Composer includes safeguards to prevent accidental or irreversible changes to your account. You control how much Composer can do on its own with per-conversation approval modes. @@ -188,7 +227,7 @@ Approval enforcement always happens on the server, and read operations never req ### Deletion -Composer can delete resources such as assistants, tools, phone numbers, squads, and files. Deletion follows your approval mode: unless you have selected Bypass, Composer asks for your explicit approval before it deletes anything. Because deletion is permanent, keep an approval mode on if you want a confirmation step before a resource is removed. +Composer does not delete resources. Delete assistants, tools, phone numbers, squads, and files from the dashboard yourself. When Composer removes a simulation from a suite it generated, the simulation is unlinked from the suite, not deleted. ### How the approval flow works diff --git a/fern/observability/simulations-advanced.mdx b/fern/observability/simulations-advanced.mdx index 591986a63..0d7183c9f 100644 --- a/fern/observability/simulations-advanced.mdx +++ b/fern/observability/simulations-advanced.mdx @@ -41,6 +41,17 @@ Open a suite and select **Edit**, then **Next**. The review step contains the ** During a simulation, the [**assistant**](/assistants) or [**squad**](/squads) under test runs its real [**tools**](/tools). Mock a tool to return fixed text instead, so that tool never calls its live service or API. A mock intercepts only the tool it names; every other tool the assistant or squad calls still runs for real. + +Suites that [Composer generates](/composer#generate-tests-for-an-assistant) come with mocks already set. Every tool in the assistant's tool list that would reach an external system is mocked, including handoffs and transfers that can't run safely. Tools that only end or route the conversation run normally, such as end call, voicemail, DTMF, and static handoffs. + +To change a generated mock, edit the suite and open **Tool mocks & webhooks**. Change the **Mock result**, or turn off **Enabled** to run the tool live. Suites you build yourself start with no mocks, so every tool runs for real until you add one. + + + + +Generated mocks don't cover everything. Tools that only an assistant [**hook**](/assistants/assistant-hooks) calls aren't mocked. Explicit [scenario webhooks](#send-webhooks-on-run-start-and-end), unmocked tools, and destination requests can still contact your server. Simulation lifecycle [server messages](/server-url), including the end-of-call report, are suppressed; that doesn't disable these other requests. Check them before running against production systems. + + @@ -126,6 +137,34 @@ Return plain text to test how the assistant handles a service that is unavailabl These strings replace the selected tool's output. They do not produce an HTTP error, add network latency, or cause a network timeout. +## Handoffs and transfers in simulations + +In a simulation, a [**handoff**](/squads/handoff) with a fixed list of destinations runs only when the model picks one of them. If the model names no destination, asks for `dynamic`, or names an assistant that isn't on the list, the handoff is refused. The assistant doesn't switch, and the model is asked to try again with a listed destination. Your server isn't asked for a destination. On a live call, a missing or `dynamic` destination can instead trigger a request to your server. + +A [**dynamic handoff**](/squads/handoff/destinations#dynamic-handoffs) runs as it does on a live call, so it asks your server for a destination unless you [mock it](#mock-tool-responses). + +In a suite you build yourself, `transferCall` isn't mocked automatically. In a voice simulation, an external transfer ends the simulation without dialing the phone number or SIP destination, so it doesn't test that destination. If the tool asks your server to choose a destination, that lookup can still run. Mock the tool to avoid that request, and use a real test call to test the external transfer itself. + +### Gaps in generated suites + +When [Composer generates a suite](/composer#generate-tests-for-an-assistant), it lists each handoff or transfer the suite doesn't exercise as **Not tested**, with the reason: + +| Reason | What the suite does | +| --- | --- | +| Your server chooses the destination (a dynamic handoff) | Mocks the handoff. | +| It transfers to a phone number or your server | Mocks the transfer. | +| The destination assistant couldn't be checked. It uses MCP tools or a pinned version, it didn't load, or Composer reached its limit on how many assistants it reads. | Mocks the handoff into it. | +| It hands off by assistant name outside a [**squad**](/squads) | Doesn't mock it. A name-based handoff can't resolve outside a squad, so the simulation never reaches it. | +| A built-in tool, such as `endCall`, has the same name as a mocked tool | The mock answers both. | + +Other supported handoffs and assistant-to-assistant transfers run normally. Cover the paths above with test calls against sandbox systems. + +## Hooks and call endings + +Simulations run the [**hooks**](/assistants/assistant-hooks) on the assistant or squad under test, just like a real call. For example, a `call.ending` hook runs in a voice simulation. The exception is `customer.speech.timeout`. Those hooks are removed from the assistant and every squad member so the assistant doesn't prompt the AI tester when the tester pauses. To test re-engagement, set `targetOverrides.hooks` on the scenario. Those hooks are used exactly as set. + +If the AI tester hangs up first in a voice simulation, the call ends with `customer-ended-call`, just like a real caller hanging up. Anything that depends on a caller hang-up runs normally. + ## Send webhooks on run start and end Turn on webhooks to notify your server when a simulation iteration starts or ends. Each webhook sends a `POST` request to the URL you configure. @@ -351,6 +390,10 @@ For a transient scenario, set the same `targetOverrides` on the inline `scenario | There is no audio or recording | Check whether the run used chat mode. Use voice mode to test or record audio. | | Start or end webhooks do not trigger | Confirm the corresponding hook and its URL are configured on the scenario. | | A tool mock does not apply | Confirm that `toolName` matches the configured tool name exactly and that `enabled` is `true`. | +| A handoff is refused in the transcript | The model gave no destination, `dynamic`, or an assistant the handoff doesn't list. Check the handoff's destinations and the prompt's routing instructions. See [**Handoffs and transfers in simulations**](#handoffs-and-transfers-in-simulations). | +| A generated suite lists a handoff or transfer as **Not tested** | That path couldn't run safely, so Composer mocked it, or the simulation can't reach it. Cover it with a test call. See [**Gaps in generated suites**](#gaps-in-generated-suites). | +| The assistant never asks whether the AI tester is still there | Simulations remove `customer.speech.timeout` hooks. To test them, set `targetOverrides.hooks` on the scenario. See [**Hooks and call endings**](#hooks-and-call-endings). | +| A voice simulation's assistant call ends as `customer-ended-call` | The AI tester hung up first, which matches a real caller hanging up. | | A tool looks up the wrong caller | Set `customer.number` in the scenario's variable values. See [**Simulate a call from a specific phone number**](#simulate-a-call-from-a-specific-phone-number). | | A latency limit fails without a measured value | No turn in the call measured that metric. Confirm the run used voice mode and that the conversation completed at least one exchange. | | Latency limits show as skipped | Check whether the run used chat mode or the assistant under test uses GPT-Live. Both skip latency limits. | diff --git a/fern/observability/simulations-manage.mdx b/fern/observability/simulations-manage.mdx index 9bfb4c14f..97f549c2e 100644 --- a/fern/observability/simulations-manage.mdx +++ b/fern/observability/simulations-manage.mdx @@ -92,6 +92,31 @@ curl -X GET "https://api.vapi.ai/eval/simulation/run/" \ The run includes `status` and `itemCounts` (`total`, `passed`, `failed`, and so on). Fetch the result for each simulation iteration with `GET /eval/simulation/run//item`. Each item's `results` holds its `evaluations` and, when the scenario sets latency limits, its `latencyEvaluations`. See [**Review latency results**](/observability/simulations-advanced#review-latency-results). +Filter the lists with query parameters: + +| Endpoint | Filters | +| --- | --- | +| `GET /eval/simulation/run` | `suiteId` (repeat for up to 100 suites), `targetType` (`assistant` or `squad`), `targetId`, and `status` (`queued`, `running`, or `ended`). You can send `targetType` or `targetId` without the other. | +| `GET /eval/simulation/run//item` | `status` (`queued`, `running`, `evaluating`, `passed`, `failed`, or `canceled`) and `simulationId`. | +| `GET /eval/simulation/suite` | `assistantId` and `squadId` (repeat each for up to 100 IDs; if you send both, a suite must match both) and `name`, which matches part of the suite name. | + +```bash +curl -X GET "https://api.vapi.ai/eval/simulation/run?suiteId=&status=ended" \ + -H "Authorization: Bearer $VAPI_API_KEY" +``` + +Each list also accepts `limit` (default 100, maximum 1,000) and `page`. + + + Adding query parameters changes the response shape: + + - With no query parameters, these endpoints return a plain array. + - The run and run item lists return `{ "metadata": { ... }, "results": [ ... ] }` when you send any query parameter. + - The suite list returns the same object when you send `limit`, `page`, `name`, `assistantId`, or `squadId`. + + If your code expects an array, read `results` whenever you send a query parameter. + + @@ -121,6 +146,27 @@ curl -X POST "https://api.vapi.ai/eval/simulation/run" \ Use `"provider": "vapi.websocket"` for voice. +To retry a run request without starting a second run, send an `Idempotency-Key` header with a value you generate, such as a UUID: + +```bash +curl -X POST "https://api.vapi.ai/eval/simulation/run" \ + -H "Authorization: Bearer $VAPI_API_KEY" \ + -H "Content-Type: application/json" \ + -H "Idempotency-Key: " \ + -d '{ + "simulations": [{ "type": "simulationSuite", "simulationSuiteId": "" }], + "target": { "type": "assistant", "assistantId": "" }, + "transport": { "provider": "vapi.webchat" }, + "iterations": 1 + }' +``` + +- A retry with the same key and the same body returns the run the first request created, with `reused: true`. It doesn't start or bill the run again. +- Reusing a key with a different body returns `409`. Use a new key for a different run. +- If a request returns `503` with the code `SIMULATION_WORK_DISABLED`, simulations are paused across Vapi. Retry later. For any other `503`, retry with the same key. +- A key is up to 255 printable ASCII characters and is scoped to your organization. An empty or invalid key returns `400`. +- Keys don't expire. A key stays tied to its run for as long as the run exists. + @@ -138,7 +184,7 @@ Create a regression test when you fix a defect or find an unexpected response. R ### Edge-case tests -Test realistic variations such as an ambiguous request, an impatient customer, an unavailable appointment, a tool error, an interruption, or a handoff. Base AI tester personalities on the customer types the [**assistant**](/assistants) or [**squad**](/squads) handles. Synthetic callers don't reliably reproduce every interruption, silence, or background-noise condition. Keep controlled real-call checks for problems the simulation can't reproduce. +Test realistic variations such as an ambiguous request, an impatient customer, an unavailable appointment, a tool error, an interruption, or a handoff. Base AI tester personalities on the customer types the [**assistant**](/assistants) or [**squad**](/squads) handles. Synthetic callers don't reliably reproduce every interruption, silence, or background-noise condition. Keep controlled real-call checks for problems the simulation can't reproduce. Some handoffs and transfers [behave differently in simulations](/observability/simulations-advanced#handoffs-and-transfers-in-simulations). ### Design evaluations diff --git a/fern/observability/simulations-overview.mdx b/fern/observability/simulations-overview.mdx index 264be58d4..987264758 100644 --- a/fern/observability/simulations-overview.mdx +++ b/fern/observability/simulations-overview.mdx @@ -89,7 +89,7 @@ Simulations test live interactions. An AI tester with a defined personality and Choose Evals when you want to lock down a specific response or verify a tool call's arguments with fast, rerunnable checks. -Choose Simulations when you want to see how your assistant or squad performs against an AI tester with different personalities, multi-turn flows, squad handoffs, or voice. +Choose Simulations when you want to see how your assistant or squad performs against an AI tester with different personalities, multi-turn flows, squad handoffs, or voice. Some handoffs and transfers [behave differently in simulations](/observability/simulations-advanced#handoffs-and-transfers-in-simulations). See [testing voice agents](/test/voice-testing) for guidance on combining both tools into one coverage plan. diff --git a/fern/observability/simulations-quickstart.mdx b/fern/observability/simulations-quickstart.mdx index 511f73ffe..436a8b0d3 100644 --- a/fern/observability/simulations-quickstart.mdx +++ b/fern/observability/simulations-quickstart.mdx @@ -18,7 +18,7 @@ A reusable **simulation suite** that asks an AI tester to book an appointment wi You create Riley and the structured output during this quickstart. - Simulations run unmocked tools for real. Before running, inspect Riley's tools and use sandbox integrations or [mock every tool](/observability/simulations-advanced#mock-tool-responses) that could create a booking, send a message, or affect a real customer. + In a suite you build yourself, every tool without a [mock](/observability/simulations-advanced#mock-tool-responses) runs for real. Before running, inspect Riley's tools and use sandbox integrations or mock every tool that could create a booking, send a message, or affect a real customer. Suites that [Composer generates](/composer#generate-tests-for-an-assistant) mock tools that reach external systems by default. ## Build and run a simulation @@ -232,10 +232,13 @@ The response returns the created suite, including its `id`. Save the `id` to pas This request starts a suite run against your [**assistant**](/assistants) or [**squad**](/squads). Pass the suite, the assistant or squad to test, the transport provider (`vapi.websocket` for voice or `vapi.webchat` for chat), and the number of iterations. +Send an `Idempotency-Key` header with a unique value for this run. Reuse the key and the same body when retrying, so a lost response cannot start and bill a second run. This applies to the first run and every rerun. See [**Run request idempotency**](/observability/simulations-manage#rerun-a-suite) for replay and error behavior. + ```bash curl -X POST "https://api.vapi.ai/eval/simulation/run" \ -H "Authorization: Bearer $VAPI_API_KEY" \ -H "Content-Type: application/json" \ + -H "Idempotency-Key: " \ -d '{ "simulations": [{ "type": "simulationSuite", "simulationSuiteId": "" }], "target": { "type": "assistant", "assistantId": "" }, diff --git a/fern/test/run-and-maintain-tests.mdx b/fern/test/run-and-maintain-tests.mdx index f887e4d87..d7efcaf60 100644 --- a/fern/test/run-and-maintain-tests.mdx +++ b/fern/test/run-and-maintain-tests.mdx @@ -38,6 +38,7 @@ Rerun affected tests whenever you change prompts, tools, models, voices, transcr AI conversations can take different paths from the same starting point. One passing Simulation shows that one path worked, not that the journey is reliable. - Use one iteration while you build or debug a Simulation. +- Run five or more iterations before you trust a pass rate. - Repeat release checks based on the impact of failure and how much the result has varied before. - Use more iterations for critical journeys and intermittent failures when the extra evidence justifies the time and cost. diff --git a/fern/test/simulations-best-practices.mdx b/fern/test/simulations-best-practices.mdx index 5fc6aa2b8..38d1e514a 100644 --- a/fern/test/simulations-best-practices.mdx +++ b/fern/test/simulations-best-practices.mdx @@ -117,7 +117,7 @@ Build and debug a scenario in chat mode, then run important paths in voice mode ## Repeat critical tests and review the conversation -Agent behavior can vary between runs. One pass shows that the agent **can** succeed; repeated passes provide stronger evidence that it succeeds consistently. Use more iterations for outcomes where a failure would harm a customer or the business. +Agent behavior can vary between runs. One pass shows that the agent **can** succeed; repeated passes provide stronger evidence that it succeeds consistently. Use one iteration while you set up a test, then run five or more iterations before you trust a pass rate. Use more iterations for outcomes where a failure would harm a customer or the business. Read every failed transcript and sample passed transcripts. Ask: diff --git a/fern/test/voice-testing.mdx b/fern/test/voice-testing.mdx index 2d978bcac..b12949ba7 100644 --- a/fern/test/voice-testing.mdx +++ b/fern/test/voice-testing.mdx @@ -47,7 +47,7 @@ voice agents need both. | **Core question** | At this point in the conversation, did the agent make the right next decision? | By the end of the conversation, did the agent reach the right outcome? | | **Scope** | Best kept focused on one or a few decision checkpoints, with a known conversation history | A complete conversation with an AI tester | | **Conversation path** | Fixed context; checks what happens next | Flexible; the tester adapts as the conversation unfolds | -| **Best for** | Tool selection and arguments, routing, refusals, required questions, and specific responses | Task completion, recovery from problems, handoffs, caller behavior, and the overall experience | +| **Best for** | Tool selection and arguments, routing, refusals, required questions, and specific responses | Task completion, recovery from problems, handoffs (see [handoffs in simulations](/observability/simulations-advanced#handoffs-and-transfers-in-simulations)), caller behavior, and the overall experience | | **Mode** | Text-based mock conversation | Chat or voice | For example, use an **Eval** to check that an agent calls the rescheduling tool only after it verifies the caller. Use a **Simulation** to check that a caller can reschedule, hear the correct new time, and finish the conversation successfully. The Simulation should pass even if the agent asks the questions in a different valid order.