From 3bbe0acda0862ee7cd67646171e7156597bfdbbc Mon Sep 17 00:00:00 2001 From: Sayan- <1415138+Sayan-@users.noreply.github.com> Date: Wed, 14 Oct 2026 00:48:10 +0000 Subject: [PATCH 1/4] Document captcha telemetry fields and outcomes --- browsers/telemetry/categories.mdx | 50 +++-- .../telemetry/pausing-for-captcha-solves.mdx | 196 ++++++++++++------ browsers/telemetry/streaming.mdx | 2 +- 3 files changed, 174 insertions(+), 74 deletions(-) diff --git a/browsers/telemetry/categories.mdx b/browsers/telemetry/categories.mdx index 0a20b1cf..f07a9066 100644 --- a/browsers/telemetry/categories.mdx +++ b/browsers/telemetry/categories.mdx @@ -83,25 +83,45 @@ A visible captcha challenge can require multiple solver tasks. When the identifi | --- | --- | --- | | `captcha_solve_started` | Solver task | The solver accepted a task. This doesn't mean a solve is currently in flight. | | `captcha_solve_result` | Solver task | A task ended with `success`, `failure`, `timeout`, or `abandoned`. A successful task means the solver returned a usable solution, not that the visible challenge cleared. | -| `captcha_challenge_result` | Visible challenge | When emitted, the challenge reached its overall `solved`, `failure`, `timeout`, or `abandoned` outcome. | +| `captcha_challenge_result` | Visible challenge or inferred token result | An observed challenge reached its overall `solved`, `failure`, `timeout`, or `abandoned` outcome, or a successful token task produced an inferred `solved` result without page observation. | -Challenge results are currently emitted for visible reCAPTCHA v2, hCaptcha, and press-and-hold challenges. Other captcha types can emit task events without a challenge result. Telemetry delivery is best-effort and isn't ordered: a start can arrive after its result, and any event can be absent. When you need a challenge-level outcome, wait for a bounded interval and fall back to available task results and the current page state. +`captcha_provider` names the product: `hcaptcha`, `recaptcha_v2`, `recaptcha_v3`, `turnstile`, `geetest`, `arkose` (FunCaptcha), `human` (press-and-hold), or `other`. Enterprise reCAPTCHA uses its version bucket. `task_kind` says what a solver task produced: `token` is a response for the whole widget, `image_challenge` is one image round that may be followed by another, and `press_and_hold` is one attempt. `task_kind` is on task events only and can be absent when the task can't be classified. -What each `captcha_type` emits today: +`captcha_type` is deprecated but still emitted. Use `captcha_provider` to identify the product and `task_kind` to decide what a task result means. Its values are frozen: image-challenge rounds report `other`, and press-and-hold reports `press_and_hold`. Older events can lack the new fields, so fall back to `captcha_type` when you read stored events. -| `captcha_type` | Task events | `captcha_challenge_result` | `challenge_id` on task events | +What each `captcha_provider` emits today: + +| `captcha_provider` | `task_kind` on classified tasks | Challenge result | `challenge_id` on task events | | --- | --- | --- | --- | -| `hcaptcha` | Yes | Yes | Yes | -| `press_and_hold` | Yes | Yes | Yes | -| `recaptcha_v2` | Yes | Yes | Can be absent | -| `turnstile` | Yes | No | Can be absent | -| `geetest` | Yes | No | Can be absent | -| `recaptcha_v3` | Yes | No | Can be absent | -| `other` | Yes | No | Can be absent | +| `hcaptcha` | `token` or `image_challenge` | Observed | Can be absent | +| `recaptcha_v2` | `token` or `image_challenge` | Observed | Can be absent | +| `human` | `press_and_hold` | Observed | Can be absent | +| `turnstile` | `token` | None | Can be absent | +| `geetest` | `token` | Inferred on successful token tasks | Can be absent | +| `recaptcha_v3` | `token` | Inferred on successful token tasks | Can be absent | +| `arkose` | `token` or `image_challenge` | Inferred on successful token tasks | Can be absent | +| `other` | `token`, `image_challenge`, or absent | Inferred on successful token tasks | Can be absent | + +An observed result has a `challenge_id` and covers the visible challenge across its solver tasks. An inferred result has `inferred: true` and a `task_id` instead of `challenge_id`; join it to the task events on `task_id`. Each successful eligible `token` task can produce an inferred result, so multiple inferred results can belong to one challenge. Failed tasks and image rounds don't produce inferred results. A returned Turnstile token isn't treated as a challenge outcome. Telemetry delivery is best-effort and isn't ordered: a start can arrive after its result, and any event can be absent. When you need a challenge-level outcome, wait for a bounded interval and fall back to available task results and the current page state. -Challenge results for `recaptcha_v2` and `hcaptcha` report `solved` or `abandoned`. `failure` and `timeout` are reported for `press_and_hold`, whose solver has an explicit attempt budget. +Challenge results for observed `recaptcha_v2` and `hcaptcha` report `solved` or `abandoned`. `failure` and `timeout` are reported for `human`, whose solver has an explicit attempt budget. Inferred results report `solved` from the task without observing the page. -Task events carry a `task_id`, but not all of them carry a `challenge_id`. Kernel sets `challenge_id` from tracking the challenge widget in the page - the same tracking that produces `captcha_challenge_result` - so a task the automatic solver runs without it reports a `task_id` only. Those task events can't be joined to a challenge result for the same page, even when one is emitted. Pair a start with its result on `task_id`, and use `challenge_id` for grouping only when it's there. +Every captcha event has `ts` (Unix microseconds), `type`, `category: captcha`, and `source`. The `data` payload holds the event details, and optional `truncated: true` means that payload was cut to the size limit. The fields inside `data` are: + +| Field | Events | Meaning | +| --- | --- | --- | +| `captcha_provider` | All three | Captcha product. Optional on older events. | +| `task_kind` | Task start and result | What the task produces. Optional when the task can't be classified. | +| `captcha_type` | All three | Deprecated legacy classification. Still emitted. | +| `task_id` | Task start and result; inferred challenge result | Opaque task identifier. Pair task events on it; an inferred result uses it instead of `challenge_id`. | +| `challenge_id` | Task start and result; observed challenge result | Opaque identifier for one visible challenge. It can span multiple tasks or continue across a reload. | +| `status` | Task and challenge results | A task reports `success`, `failure`, `timeout`, or `abandoned`; a challenge reports `solved`, `failure`, `timeout`, or `abandoned`. | +| `duration_ms` | Task and challenge results | Wall-clock duration of the task or challenge. An inferred result uses its task's duration. | +| `website_host`, `website_path` | All three | Host and path of the page where the task ran or challenge appeared. The path excludes the query string; the host can be empty when a task has no page URL. | +| `error_code` | Task result | Solver-specific failure code; absent on success. | +| `inferred` | Challenge result | True when a successful token task produced the result without page observation. Absent on observed results. | + +Task events don't always carry `task_id` or `challenge_id`. Pair a start with its result on `task_id` when present, and use `challenge_id` to group tasks for a visible challenge only when it's there. Use each result event's `duration_ms` as the authoritative duration rather than calculating it from event timestamps. Challenge duration covers every solver attempt from when the challenge appeared until its terminal outcome. @@ -109,7 +129,7 @@ Challenge outcomes have these meanings: | Status | Meaning | | --- | --- | -| `solved` | The page observed the challenge clear after a solver attempt. This doesn't prove the site accepted the solution or that the guarded action succeeded. | +| `solved` | The page observed the challenge clear, or an inferred result reports a token without page observation. Neither proves the site accepted the solution or that the guarded action succeeded. | | `failure` | A terminal solver failure occurred, or all attempts ended while the challenge remained. | | `timeout` | The challenge-level wait budget expired while the challenge remained. | | `abandoned` | Observation ended without an attributable terminal challenge outcome. This includes a dismissed widget, a page unload, or a token appearing while multiple challenges from the same provider are open. In the last case, a task can report `success` while the challenge with the same `challenge_id` reports `abandoned`. | @@ -170,7 +190,7 @@ Telemetry is off by default. The default set isn't limited to session metadata: | `interaction` | Text of clicked elements and typed keys, which can include personal data entered into forms. | | `screenshot` | A full rendered image of the page - the broadest exposure, capturing anything visible on screen. | | `control` | The source you submit to the Playwright code-execution endpoint, on the `code` field of `api_call`, capped at 8 KB and marked with `...[truncated]` when cut. Whatever your script embeds is captured with it, so a literal password or token in the snippet is captured too. `cdp_command` carries sanitized arguments such as the method and phase, coordinates, counts, flags, and named keys such as `Enter` and `Tab`. Typed text, file paths, scripts, templates, dialog input, and autofill values aren't captured; navigation commands retain only the URL scheme, not the host, path, query, or fragment. | -| `captcha` | Captcha type, task and challenge outcomes and durations, and opaque `task_id` and `challenge_id` values. It can include the host and path of the page where a task ran or a challenge appeared; the query string is excluded. Failed tasks can include a solver-specific error code. | +| `captcha` | Captcha provider and task kind, task and challenge outcomes and durations, and opaque `task_id` and `challenge_id` values. It can include the host and path of the page where a task ran or a challenge appeared; the query string is excluded. Failed tasks can include a solver-specific error code. | | `platform`, `connection`, `system`, `monitor` | Session and VM metadata only (VM-management calls, connection and health events). No page content. | Captured events are persisted and can be replayed by [resuming the stream](/browsers/telemetry/streaming#resuming-after-a-disconnect), so this sensitivity applies to the data at rest, not just the live stream. Events are retained for 30 days, then expired (see [Retention](/browsers/telemetry/overview#retention)). Treat captured telemetry - and anywhere you forward or store it - with the same care as the underlying content. For how Kernel encrypts, retains, and processes data overall, see [Security](/security) and the [Data Processing Addendum](/dpa). diff --git a/browsers/telemetry/pausing-for-captcha-solves.mdx b/browsers/telemetry/pausing-for-captcha-solves.mdx index 5f21af34..66316618 100644 --- a/browsers/telemetry/pausing-for-captcha-solves.mdx +++ b/browsers/telemetry/pausing-for-captcha-solves.mdx @@ -17,11 +17,12 @@ The design follows one split: telemetry decides **what to say**, while the live | --- | --- | --- | | `captcha_solve_started` | Solver task | The solver accepted a task. It does **not** mean a solve is currently in flight. | | `captcha_solve_result` | Solver task | A task ended `success`, `failure`, `timeout`, or `abandoned`. Success means the solver returned a usable answer, not that the challenge cleared. | -| `captcha_challenge_result` | Visible challenge | The challenge reached its overall outcome. Only emitted for challenge types Kernel tracks as a widget. | +| `captcha_challenge_result` | Visible challenge or inferred token result | An observed challenge reached its overall outcome, or a successful token task produced an inferred result without page observation. | Three rules from [Correlate captcha tasks and challenges](/browsers/telemetry/categories#correlate-captcha-tasks-and-challenges) shape everything below: -- **`task_id` is the only join.** Pair a start with its result on it. `challenge_id` groups tasks for one visible challenge and is present only when Kernel tracked the widget — tasks without one can never be joined to a `captcha_challenge_result`, even when one is emitted for the same page. +- **Use the identifier the result carries.** Pair a start with its task result on `task_id`. An observed challenge result joins on `challenge_id`; an inferred result has `inferred: true` and joins on `task_id` instead. +- **A task result isn't always the end of a challenge.** `task_kind: token` returns an answer for the whole widget, while `image_challenge` completes only one image round. On older events without `task_kind`, the gate falls back to `captcha_type` where it can and leaves ambiguous tasks unclassified. - **Delivery is best-effort and unordered.** A start can arrive after its result, and any event can be absent. Nothing may depend on arrival order, and every wait needs a deadline. - **Fall back to the page.** When you need a challenge-level outcome and don't have one, use the available task results and the current page state. @@ -51,7 +52,7 @@ The four snippets below are one file, `captcha-gate.ts`, in order. - Three maps, one per thing the telemetry can tell you, all keyed so nothing depends on arrival order. `joinable` is the important one: it holds only the `challenge_id`s that actually appeared on a task event, which are the only ones a challenge result may be attributed to. + Three maps, one per thing the telemetry can tell you, all keyed so nothing depends on arrival order. `joinable` holds both task IDs and challenge IDs seen on task events. That lets the gate join an inferred result on `task_id` and an observed result on `challenge_id`. A task record created by a *result* already has a status, so a `captcha_solve_started` that arrives afterwards finds a closed task and leaves it closed. That only works while the task is still in `tasks`, though — once `reset` has cleared it, a late start for the same `task_id` would otherwise look like a brand-new one. `settledAt` remembers when `reset` cleared each ID, so a late arrival within `TASK_SETTLE_MS` of that is dropped instead of reopening an episode the agent has already been told about. Past that window the same ID is treated as fresh again, since Kernel can reuse a `challenge_id` when an episode continues across a page reload. @@ -84,7 +85,10 @@ interface PageState { interface Outcome { status?: string; durationMs?: number; - captchaType?: string; + captchaProvider?: string; + taskKind?: string; + challengeId?: string; + inferred?: boolean; } interface Verdict extends Outcome { @@ -98,15 +102,25 @@ interface Verdict extends Outcome { function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => Promise) { const tasks = new Map(); const challenges = new Map(); - // Only challenge_ids that appeared on a task event may be attributed to it. const joinable = new Set(); - // task_ids and challenge_ids reported and cleared by reset(), each stamped + // Task and challenge keys reported and cleared by reset(), each stamped // with when that happened. A late-arriving duplicate within TASK_SETTLE_MS // of that -- delivery is unordered, a start can arrive after its own // result -- is dropped instead of reopening an already-told episode. Past // that window it's treated as fresh, since Kernel can reuse the same // challenge_id when an episode continues across a page reload. const settledAt = new Map(); + const taskKey = (id: string) => `task:${id}`; + const challengeKey = (id: string) => `challenge:${id}`; + const providerOf = (data: { captcha_provider?: string; captcha_type?: string }) => + data.captcha_provider ?? (data.captcha_type === "press_and_hold" ? "human" : data.captcha_type); + const kindOf = (data: { task_kind?: string; captcha_type?: string }) => + data.task_kind ?? + (data.captcha_type === "press_and_hold" + ? "press_and_hold" + : ["turnstile", "geetest", "recaptcha_v3"].includes(data.captcha_type ?? "") + ? "token" + : undefined); const isSettled = (id: string) => { const at = settledAt.get(id); return at !== undefined && Date.now() - at < TASK_SETTLE_MS; @@ -119,25 +133,38 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P for await (const { event } of stream) { if (event.category !== "captcha") continue; - if (event.type === "captcha_solve_started" && event.data?.task_id && !isSettled(event.data.task_id)) { + if (event.type === "captcha_solve_started" && event.data?.task_id && !isSettled(taskKey(event.data.task_id))) { // A result for an unseen task lands already closed, so a start that // arrives after its own result never reopens it. const task = tasks.get(event.data.task_id) ?? { openedAt: Date.now() }; - task.captchaType = event.data.captcha_type ?? task.captchaType; + task.captchaProvider = providerOf(event.data) ?? task.captchaProvider; + task.taskKind = kindOf(event.data) ?? task.taskKind; + task.challengeId = event.data.challenge_id ?? task.challengeId; tasks.set(event.data.task_id, task); - if (event.data.challenge_id) joinable.add(event.data.challenge_id); - } else if (event.type === "captcha_solve_result" && event.data?.task_id && !isSettled(event.data.task_id)) { + joinable.add(taskKey(event.data.task_id)); + if (event.data.challenge_id) joinable.add(challengeKey(event.data.challenge_id)); + } else if (event.type === "captcha_solve_result" && event.data?.task_id && !isSettled(taskKey(event.data.task_id))) { const task = tasks.get(event.data.task_id) ?? { openedAt: Date.now() }; task.status = event.data.status; task.durationMs = event.data.duration_ms; - task.captchaType = event.data.captcha_type ?? task.captchaType; + task.captchaProvider = providerOf(event.data) ?? task.captchaProvider; + task.taskKind = kindOf(event.data) ?? task.taskKind; + task.challengeId = event.data.challenge_id ?? task.challengeId; tasks.set(event.data.task_id, task); - if (event.data.challenge_id) joinable.add(event.data.challenge_id); - } else if (event.type === "captcha_challenge_result" && event.data?.challenge_id && !isSettled(event.data.challenge_id)) { - challenges.set(event.data.challenge_id, { + joinable.add(taskKey(event.data.task_id)); + if (event.data.challenge_id) joinable.add(challengeKey(event.data.challenge_id)); + } else if (event.type === "captcha_challenge_result" && event.data) { + const key = event.data.challenge_id + ? challengeKey(event.data.challenge_id) + : event.data.inferred && event.data.task_id + ? taskKey(event.data.task_id) + : undefined; + if (!key || isSettled(key)) continue; + challenges.set(key, { status: event.data.status, durationMs: event.data.duration_ms, - captchaType: event.data.captcha_type, + captchaProvider: providerOf(event.data), + inferred: event.data.inferred, }); } } @@ -152,6 +179,11 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P const isOpen = (t: Outcome & { openedAt: number }) => !t.status && Date.now() - t.openedAt < TASK_SETTLE_MS; const openTasks = () => [...tasks.values()].filter(isOpen); const joinedResult = () => [...challenges].find(([id]) => joinable.has(id)); + const expectsChallengeResult = () => [...tasks.values()].some((task) => + task.challengeId || + (task.status === "success" && task.taskKind === "token" && + !["turnstile", "hcaptcha", "recaptcha_v2"].includes(task.captchaProvider ?? "")) + ); async function until(done: () => boolean, timeoutMs: number) { const deadline = Date.now() + timeoutMs; @@ -161,7 +193,7 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P - `resolve` settles what it can, waits a bounded interval for a challenge-level result *only* when a task actually carried a `challenge_id`, then reads the page and picks the best available source: an attributable challenge result, then any challenge result (labelled as a page observation rather than a join), then task results, then the page alone. + `resolve` settles what it can, waits a bounded interval for a challenge-level result when a task carried a `challenge_id` or a successful token task can produce an inferred result, then reads the page. It picks an attributable challenge result first, followed by an unjoined challenge result (labelled as an observation or uncorrelated token result), task results, and the page alone. It doesn't wait for an inferred result from an image round or a Turnstile token. `holding` and `pending` are separate on purpose. `pending` covers a terminal outcome that lands while no action is in flight — without it, a challenge result arriving between tool calls is recorded and never told to anyone. It has to catch *every* settled task, not just failures: a successful task-only result (Turnstile's usual path) and a task whose deadline lapsed with no result at all are both settled outcomes the agent needs to hear about. @@ -169,9 +201,9 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P ```ts captcha-gate.ts async function resolve(): Promise { - // A challenge result covers every task under it, so stop waiting once one lands. + // Stop waiting when a result joins to a task or visible challenge. await until(() => openTasks().length === 0 || Boolean(joinedResult()), TASK_SETTLE_MS); - if (joinable.size > 0 && !joinedResult()) await until(() => Boolean(joinedResult()), CHALLENGE_GRACE_MS); + if (expectsChallengeResult() && !joinedResult()) await until(() => Boolean(joinedResult()), CHALLENGE_GRACE_MS); const page = await probePage(); const challenge = joinedResult() ?? [...challenges][0]; @@ -191,16 +223,15 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P pending: () => challenges.size > 0 || [...tasks.values()].some((t) => !isOpen(t)), resolve, reset: () => { - // A challenge result covers every task under it, open or not, so an - // open task isn't preserved once its challenge already resolved -- - // only when there's no other signal covering it yet, meaning the - // solver may still be working via a follow-up task resolve() hasn't - // seen close out. - const covered = Boolean(joinedResult()); + // Keep an open task unless a result joined to that task or its + // challenge covers it. + const result = joinedResult(); const now = Date.now(); for (const [id, t] of tasks) { + const covered = result && (result[0] === taskKey(id) || + (t.challengeId && result[0] === challengeKey(t.challengeId))); if (isOpen(t) && !covered) continue; - settledAt.set(id, now); + settledAt.set(taskKey(id), now); tasks.delete(id); } for (const id of challenges.keys()) settledAt.set(id, now); @@ -214,25 +245,33 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P function describe(source: Verdict["source"], status: string, joined: boolean, outcome: Outcome, page: PageState): Verdict { const took = outcome.durationMs ? ` after ${(outcome.durationMs / 1000).toFixed(1)}s` : ""; - const kind = outcome.captchaType ? `${outcome.captchaType} ` : ""; + const kind = outcome.captchaProvider ? `${outcome.captchaProvider} ` : ""; const headline = source === "page" ? "No terminal captcha telemetry arrived." : source === "task" ? status === "success" - ? `The solver returned an answer for a ${kind}task${took}. No challenge-level outcome was reported, so this is not a cleared challenge.` + ? outcome.taskKind === "image_challenge" + ? `The solver completed one ${kind}image round${took}. The challenge may serve another round.` + : outcome.taskKind === "token" + ? `The solver returned a token for a ${kind}challenge${took}. No challenge-level outcome was reported, so this is not a cleared challenge.` + : `The solver returned an answer for a ${kind}task${took}. No challenge-level outcome was reported, so this is not a cleared challenge.` : `A ${kind}solver task ended as "${status}"${took}.` + : outcome.inferred && status === "solved" + ? `Kernel inferred a ${kind}challenge result from a successful token task${took}. The page did not confirm it cleared.` : status === "solved" ? `Kernel observed the ${kind}challenge clear${took}. That is not proof the site accepted the solution.` : `Kernel reported the ${kind}challenge as "${status}"${took}.`; const caveat = source === "challenge" && !joined - ? " It could not be joined to this page's solver tasks, so treat it as an observation about the page." + ? outcome.inferred + ? " It could not be joined to this page's solver tasks, so treat it as an uncorrelated token result." + : " It could not be joined to this page's solver tasks, so treat it as an observation about the page." : ""; const where = page.widgets.length ? `A captcha widget is still on the page (${page.widgets.join(", ")}).` : "No captcha widget is visible on the page."; - return { source, status, joined, captchaType: outcome.captchaType, durationMs: outcome.durationMs, page, message: `${headline}${caveat} ${where}` }; + return { source, status, joined, captchaProvider: outcome.captchaProvider, taskKind: outcome.taskKind, inferred: outcome.inferred, durationMs: outcome.durationMs, page, message: `${headline}${caveat} ${where}` }; } ``` @@ -352,7 +391,10 @@ interface PageState { interface Outcome { status?: string; durationMs?: number; - captchaType?: string; + captchaProvider?: string; + taskKind?: string; + challengeId?: string; + inferred?: boolean; } interface Verdict extends Outcome { @@ -366,15 +408,25 @@ interface Verdict extends Outcome { function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => Promise) { const tasks = new Map(); const challenges = new Map(); - // Only challenge_ids that appeared on a task event may be attributed to it. const joinable = new Set(); - // task_ids and challenge_ids reported and cleared by reset(), each stamped + // Task and challenge keys reported and cleared by reset(), each stamped // with when that happened. A late-arriving duplicate within TASK_SETTLE_MS // of that -- delivery is unordered, a start can arrive after its own // result -- is dropped instead of reopening an already-told episode. Past // that window it's treated as fresh, since Kernel can reuse the same // challenge_id when an episode continues across a page reload. const settledAt = new Map(); + const taskKey = (id: string) => `task:${id}`; + const challengeKey = (id: string) => `challenge:${id}`; + const providerOf = (data: { captcha_provider?: string; captcha_type?: string }) => + data.captcha_provider ?? (data.captcha_type === "press_and_hold" ? "human" : data.captcha_type); + const kindOf = (data: { task_kind?: string; captcha_type?: string }) => + data.task_kind ?? + (data.captcha_type === "press_and_hold" + ? "press_and_hold" + : ["turnstile", "geetest", "recaptcha_v3"].includes(data.captcha_type ?? "") + ? "token" + : undefined); const isSettled = (id: string) => { const at = settledAt.get(id); return at !== undefined && Date.now() - at < TASK_SETTLE_MS; @@ -387,25 +439,38 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P for await (const { event } of stream) { if (event.category !== "captcha") continue; - if (event.type === "captcha_solve_started" && event.data?.task_id && !isSettled(event.data.task_id)) { + if (event.type === "captcha_solve_started" && event.data?.task_id && !isSettled(taskKey(event.data.task_id))) { // A result for an unseen task lands already closed, so a start that // arrives after its own result never reopens it. const task = tasks.get(event.data.task_id) ?? { openedAt: Date.now() }; - task.captchaType = event.data.captcha_type ?? task.captchaType; + task.captchaProvider = providerOf(event.data) ?? task.captchaProvider; + task.taskKind = kindOf(event.data) ?? task.taskKind; + task.challengeId = event.data.challenge_id ?? task.challengeId; tasks.set(event.data.task_id, task); - if (event.data.challenge_id) joinable.add(event.data.challenge_id); - } else if (event.type === "captcha_solve_result" && event.data?.task_id && !isSettled(event.data.task_id)) { + joinable.add(taskKey(event.data.task_id)); + if (event.data.challenge_id) joinable.add(challengeKey(event.data.challenge_id)); + } else if (event.type === "captcha_solve_result" && event.data?.task_id && !isSettled(taskKey(event.data.task_id))) { const task = tasks.get(event.data.task_id) ?? { openedAt: Date.now() }; task.status = event.data.status; task.durationMs = event.data.duration_ms; - task.captchaType = event.data.captcha_type ?? task.captchaType; + task.captchaProvider = providerOf(event.data) ?? task.captchaProvider; + task.taskKind = kindOf(event.data) ?? task.taskKind; + task.challengeId = event.data.challenge_id ?? task.challengeId; tasks.set(event.data.task_id, task); - if (event.data.challenge_id) joinable.add(event.data.challenge_id); - } else if (event.type === "captcha_challenge_result" && event.data?.challenge_id && !isSettled(event.data.challenge_id)) { - challenges.set(event.data.challenge_id, { + joinable.add(taskKey(event.data.task_id)); + if (event.data.challenge_id) joinable.add(challengeKey(event.data.challenge_id)); + } else if (event.type === "captcha_challenge_result" && event.data) { + const key = event.data.challenge_id + ? challengeKey(event.data.challenge_id) + : event.data.inferred && event.data.task_id + ? taskKey(event.data.task_id) + : undefined; + if (!key || isSettled(key)) continue; + challenges.set(key, { status: event.data.status, durationMs: event.data.duration_ms, - captchaType: event.data.captcha_type, + captchaProvider: providerOf(event.data), + inferred: event.data.inferred, }); } } @@ -414,6 +479,11 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P const isOpen = (t: Outcome & { openedAt: number }) => !t.status && Date.now() - t.openedAt < TASK_SETTLE_MS; const openTasks = () => [...tasks.values()].filter(isOpen); const joinedResult = () => [...challenges].find(([id]) => joinable.has(id)); + const expectsChallengeResult = () => [...tasks.values()].some((task) => + task.challengeId || + (task.status === "success" && task.taskKind === "token" && + !["turnstile", "hcaptcha", "recaptcha_v2"].includes(task.captchaProvider ?? "")) + ); async function until(done: () => boolean, timeoutMs: number) { const deadline = Date.now() + timeoutMs; @@ -421,9 +491,9 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P } async function resolve(): Promise { - // A challenge result covers every task under it, so stop waiting once one lands. + // Stop waiting when a result joins to a task or visible challenge. await until(() => openTasks().length === 0 || Boolean(joinedResult()), TASK_SETTLE_MS); - if (joinable.size > 0 && !joinedResult()) await until(() => Boolean(joinedResult()), CHALLENGE_GRACE_MS); + if (expectsChallengeResult() && !joinedResult()) await until(() => Boolean(joinedResult()), CHALLENGE_GRACE_MS); const page = await probePage(); const challenge = joinedResult() ?? [...challenges][0]; @@ -443,16 +513,15 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P pending: () => challenges.size > 0 || [...tasks.values()].some((t) => !isOpen(t)), resolve, reset: () => { - // A challenge result covers every task under it, open or not, so an - // open task isn't preserved once its challenge already resolved -- - // only when there's no other signal covering it yet, meaning the - // solver may still be working via a follow-up task resolve() hasn't - // seen close out. - const covered = Boolean(joinedResult()); + // Keep an open task unless a result joined to that task or its + // challenge covers it. + const result = joinedResult(); const now = Date.now(); for (const [id, t] of tasks) { + const covered = result && (result[0] === taskKey(id) || + (t.challengeId && result[0] === challengeKey(t.challengeId))); if (isOpen(t) && !covered) continue; - settledAt.set(id, now); + settledAt.set(taskKey(id), now); tasks.delete(id); } for (const id of challenges.keys()) settledAt.set(id, now); @@ -466,25 +535,33 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P function describe(source: Verdict["source"], status: string, joined: boolean, outcome: Outcome, page: PageState): Verdict { const took = outcome.durationMs ? ` after ${(outcome.durationMs / 1000).toFixed(1)}s` : ""; - const kind = outcome.captchaType ? `${outcome.captchaType} ` : ""; + const kind = outcome.captchaProvider ? `${outcome.captchaProvider} ` : ""; const headline = source === "page" ? "No terminal captcha telemetry arrived." : source === "task" ? status === "success" - ? `The solver returned an answer for a ${kind}task${took}. No challenge-level outcome was reported, so this is not a cleared challenge.` + ? outcome.taskKind === "image_challenge" + ? `The solver completed one ${kind}image round${took}. The challenge may serve another round.` + : outcome.taskKind === "token" + ? `The solver returned a token for a ${kind}challenge${took}. No challenge-level outcome was reported, so this is not a cleared challenge.` + : `The solver returned an answer for a ${kind}task${took}. No challenge-level outcome was reported, so this is not a cleared challenge.` : `A ${kind}solver task ended as "${status}"${took}.` + : outcome.inferred && status === "solved" + ? `Kernel inferred a ${kind}challenge result from a successful token task${took}. The page did not confirm it cleared.` : status === "solved" ? `Kernel observed the ${kind}challenge clear${took}. That is not proof the site accepted the solution.` : `Kernel reported the ${kind}challenge as "${status}"${took}.`; const caveat = source === "challenge" && !joined - ? " It could not be joined to this page's solver tasks, so treat it as an observation about the page." + ? outcome.inferred + ? " It could not be joined to this page's solver tasks, so treat it as an uncorrelated token result." + : " It could not be joined to this page's solver tasks, so treat it as an observation about the page." : ""; const where = page.widgets.length ? `A captcha widget is still on the page (${page.widgets.join(", ")}).` : "No captcha widget is visible on the page."; - return { source, status, joined, captchaType: outcome.captchaType, durationMs: outcome.durationMs, page, message: `${headline}${caveat} ${where}` }; + return { source, status, joined, captchaProvider: outcome.captchaProvider, taskKind: outcome.taskKind, inferred: outcome.inferred, durationMs: outcome.durationMs, page, message: `${headline}${caveat} ${where}` }; } const PROBE = ` @@ -569,9 +646,12 @@ Every verdict pairs a telemetry claim with the page state it was checked against | Source | Message | | --- | --- | -| Attributable challenge result | "Kernel observed the challenge clear after 18.0s. That is not proof the site accepted the solution. No captcha widget is visible on the page." | -| Unjoinable challenge result | Same, plus "It could not be joined to this page's solver tasks, so treat it as an observation about the page." | -| Task result only | "The solver returned an answer for a turnstile task after 3.9s. No challenge-level outcome was reported, so this is not a cleared challenge. A captcha widget is still on the page (turnstile)." | +| Observed challenge result | "Kernel observed the challenge clear after 18.0s. That is not proof the site accepted the solution. No captcha widget is visible on the page." | +| Inferred challenge result | "Kernel inferred a geetest challenge result from a successful token task after 3.9s. The page did not confirm it cleared. No captcha widget is visible on the page." | +| Unjoinable observed result | Same, plus "It could not be joined to this page's solver tasks, so treat it as an observation about the page." | +| Unjoinable inferred result | Same, plus "It could not be joined to this page's solver tasks, so treat it as an uncorrelated token result." | +| Token task result only | "The solver returned a token for a turnstile challenge after 3.9s. No challenge-level outcome was reported, so this is not a cleared challenge. A captcha widget is still on the page (turnstile)." | +| Image-round task result only | "The solver completed one recaptcha_v2 image round after 3.9s. The challenge may serve another round. A captcha widget is still on the page (recaptcha)." | | Nothing terminal arrived | "No terminal captcha telemetry arrived. A captcha widget is still on the page (turnstile)." | Durations come straight from each event's `duration_ms`, which is authoritative; don't compute them from event timestamps. @@ -579,7 +659,7 @@ Durations come straight from each event's `duration_ms`, which is authoritative; ## Limits - **`TASK_SETTLE_MS` and `CHALLENGE_GRACE_MS` are the safety net.** Events can be absent, so both waits are bounded and the gate falls through to the page rather than stalling. Raise `TASK_SETTLE_MS` if your solves routinely run longer than 20s. -- **Challenge results aren't emitted for every widget type.** Turnstile, for instance, reports task events only, so the task-plus-page path is the one that runs there. +- **Challenge results aren't emitted for every widget type.** Turnstile reports task events only, so the task-plus-page path is the one that runs there. An inferred result for another provider reports a returned token, not an observed page clear. - **One episode at a time.** The gate resets after each verdict; overlapping visible challenges from the same provider aren't split apart, and the telemetry docs note Kernel's own event model can't always attribute a result in that case either. - **The page probe is best-effort.** It reads the DOM for known widget markers and a response token. A site that renders its challenge somewhere those selectors miss won't be held past the logged message alone. - **GeeTest has no selector either.** The solver reports `geetest` task events, but `PROBE`'s `groups` has no GeeTest entry and `tokenPresent` checks no GeeTest field, so `widgets` never reports one and the agent proceeds as soon as the task settles, even while the challenge is still open. Add a selector for your target's GeeTest integration if you need it to hold. diff --git a/browsers/telemetry/streaming.mdx b/browsers/telemetry/streaming.mdx index f096f856..8bd77414 100644 --- a/browsers/telemetry/streaming.mdx +++ b/browsers/telemetry/streaming.mdx @@ -35,7 +35,7 @@ with kernel.browsers.telemetry.stream(session_id) as stream: ``` -To filter, check `event.category` and `event.type` in your loop. If the stream drops, re-open it with the last `seq` you processed as `last_event_id` to resume without gaps. +To filter, check `event.category` and `event.type` in your loop, or pass `type` to the stream endpoint to receive only selected event types. Repeat `type` or pass comma-separated values for multiple types. The archived events endpoint also accepts `type`. If the stream drops, re-open it with the last `seq` you processed as `last_event_id` to resume without gaps. ## Via CLI From 09b036480677041eea74655ae9e26b19802a6077 Mon Sep 17 00:00:00 2001 From: Sayan- <1415138+Sayan-@users.noreply.github.com> Date: Wed, 14 Oct 2026 00:51:25 +0000 Subject: [PATCH 2/4] Simplify captcha telemetry docs and example --- browsers/telemetry/categories.mdx | 6 +- .../telemetry/pausing-for-captcha-solves.mdx | 92 +++++++++---------- 2 files changed, 46 insertions(+), 52 deletions(-) diff --git a/browsers/telemetry/categories.mdx b/browsers/telemetry/categories.mdx index f07a9066..664370d9 100644 --- a/browsers/telemetry/categories.mdx +++ b/browsers/telemetry/categories.mdx @@ -102,9 +102,11 @@ What each `captcha_provider` emits today: | `arkose` | `token` or `image_challenge` | Inferred on successful token tasks | Can be absent | | `other` | `token`, `image_challenge`, or absent | Inferred on successful token tasks | Can be absent | -An observed result has a `challenge_id` and covers the visible challenge across its solver tasks. An inferred result has `inferred: true` and a `task_id` instead of `challenge_id`; join it to the task events on `task_id`. Each successful eligible `token` task can produce an inferred result, so multiple inferred results can belong to one challenge. Failed tasks and image rounds don't produce inferred results. A returned Turnstile token isn't treated as a challenge outcome. Telemetry delivery is best-effort and isn't ordered: a start can arrive after its result, and any event can be absent. When you need a challenge-level outcome, wait for a bounded interval and fall back to available task results and the current page state. +An observed result has a `challenge_id` and covers the visible challenge across its solver tasks. An inferred result has `inferred: true` and a `task_id` instead; join it to task events on `task_id`. Each eligible successful `token` task can produce an inferred result, so one challenge may have several. Failed tasks and image rounds don't produce inferred results. Turnstile tokens don't produce challenge results. -Challenge results for observed `recaptcha_v2` and `hcaptcha` report `solved` or `abandoned`. `failure` and `timeout` are reported for `human`, whose solver has an explicit attempt budget. Inferred results report `solved` from the task without observing the page. +Telemetry delivery is best-effort and unordered: a start can arrive after its result, and any event can be absent. When you need a challenge outcome, wait for a bounded interval, then check available task results and the current page state. + +Observed `recaptcha_v2` and `hcaptcha` results report `solved` or `abandoned`. `human` can also report `failure` and `timeout`. Inferred results report `solved` without observing the page. Every captcha event has `ts` (Unix microseconds), `type`, `category: captcha`, and `source`. The `data` payload holds the event details, and optional `truncated: true` means that payload was cut to the size limit. The fields inside `data` are: diff --git a/browsers/telemetry/pausing-for-captcha-solves.mdx b/browsers/telemetry/pausing-for-captcha-solves.mdx index 66316618..3f9dd8ea 100644 --- a/browsers/telemetry/pausing-for-captcha-solves.mdx +++ b/browsers/telemetry/pausing-for-captcha-solves.mdx @@ -52,7 +52,7 @@ The four snippets below are one file, `captcha-gate.ts`, in order. - Three maps, one per thing the telemetry can tell you, all keyed so nothing depends on arrival order. `joinable` holds both task IDs and challenge IDs seen on task events. That lets the gate join an inferred result on `task_id` and an observed result on `challenge_id`. + The gate records task and challenge events by ID, so delivery order doesn't matter. It joins inferred results on `task_id` and observed results on `challenge_id`. A task record created by a *result* already has a status, so a `captcha_solve_started` that arrives afterwards finds a closed task and leaves it closed. That only works while the task is still in `tasks`, though — once `reset` has cleared it, a late start for the same `task_id` would otherwise look like a brand-new one. `settledAt` remembers when `reset` cleared each ID, so a late arrival within `TASK_SETTLE_MS` of that is dropped instead of reopening an episode the agent has already been told about. Past that window the same ID is treated as fresh again, since Kernel can reuse a `challenge_id` when an episode continues across a page reload. @@ -114,13 +114,11 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P const challengeKey = (id: string) => `challenge:${id}`; const providerOf = (data: { captcha_provider?: string; captcha_type?: string }) => data.captcha_provider ?? (data.captcha_type === "press_and_hold" ? "human" : data.captcha_type); - const kindOf = (data: { task_kind?: string; captcha_type?: string }) => - data.task_kind ?? - (data.captcha_type === "press_and_hold" - ? "press_and_hold" - : ["turnstile", "geetest", "recaptcha_v3"].includes(data.captcha_type ?? "") - ? "token" - : undefined); + const kindOf = (data: { task_kind?: string; captcha_type?: string }) => { + if (data.task_kind) return data.task_kind; + if (data.captcha_type === "press_and_hold") return "press_and_hold"; + if (["turnstile", "geetest", "recaptcha_v3"].includes(data.captcha_type ?? "")) return "token"; + }; const isSettled = (id: string) => { const at = settledAt.get(id); return at !== undefined && Date.now() - at < TASK_SETTLE_MS; @@ -193,7 +191,7 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P - `resolve` settles what it can, waits a bounded interval for a challenge-level result when a task carried a `challenge_id` or a successful token task can produce an inferred result, then reads the page. It picks an attributable challenge result first, followed by an unjoined challenge result (labelled as an observation or uncorrelated token result), task results, and the page alone. It doesn't wait for an inferred result from an image round or a Turnstile token. + `resolve` waits briefly for a challenge result when one can follow the task, then reads the page. It prefers a joined challenge result, then an unjoined result, a task result, or the page alone. Image rounds and Turnstile tokens don't trigger the extra wait. `holding` and `pending` are separate on purpose. `pending` covers a terminal outcome that lands while no action is in flight — without it, a challenge result arriving between tool calls is recorded and never told to anyone. It has to catch *every* settled task, not just failures: a successful task-only result (Turnstile's usual path) and a task whose deadline lapsed with no result at all are both settled outcomes the agent needs to hear about. @@ -201,7 +199,6 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P ```ts captcha-gate.ts async function resolve(): Promise { - // Stop waiting when a result joins to a task or visible challenge. await until(() => openTasks().length === 0 || Boolean(joinedResult()), TASK_SETTLE_MS); if (expectsChallengeResult() && !joinedResult()) await until(() => Boolean(joinedResult()), CHALLENGE_GRACE_MS); @@ -246,22 +243,21 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P function describe(source: Verdict["source"], status: string, joined: boolean, outcome: Outcome, page: PageState): Verdict { const took = outcome.durationMs ? ` after ${(outcome.durationMs / 1000).toFixed(1)}s` : ""; const kind = outcome.captchaProvider ? `${outcome.captchaProvider} ` : ""; - const headline = - source === "page" - ? "No terminal captcha telemetry arrived." - : source === "task" - ? status === "success" - ? outcome.taskKind === "image_challenge" - ? `The solver completed one ${kind}image round${took}. The challenge may serve another round.` - : outcome.taskKind === "token" - ? `The solver returned a token for a ${kind}challenge${took}. No challenge-level outcome was reported, so this is not a cleared challenge.` - : `The solver returned an answer for a ${kind}task${took}. No challenge-level outcome was reported, so this is not a cleared challenge.` - : `A ${kind}solver task ended as "${status}"${took}.` - : outcome.inferred && status === "solved" - ? `Kernel inferred a ${kind}challenge result from a successful token task${took}. The page did not confirm it cleared.` - : status === "solved" - ? `Kernel observed the ${kind}challenge clear${took}. That is not proof the site accepted the solution.` - : `Kernel reported the ${kind}challenge as "${status}"${took}.`; + let headline: string; + if (source === "page") { + headline = "No terminal captcha telemetry arrived."; + } else if (source === "task") { + if (status !== "success") headline = `A ${kind}solver task ended as "${status}"${took}.`; + else if (outcome.taskKind === "image_challenge") headline = `The solver completed one ${kind}image round${took}. The challenge may serve another round.`; + else if (outcome.taskKind === "token") headline = `The solver returned a token for a ${kind}challenge${took}. No challenge-level outcome was reported, so this is not a cleared challenge.`; + else headline = `The solver returned an answer for a ${kind}task${took}. No challenge-level outcome was reported, so this is not a cleared challenge.`; + } else if (outcome.inferred) { + headline = `Kernel inferred a ${kind}challenge result from a successful token task${took}. The page did not confirm it cleared.`; + } else if (status === "solved") { + headline = `Kernel observed the ${kind}challenge clear${took}. That is not proof the site accepted the solution.`; + } else { + headline = `Kernel reported the ${kind}challenge as "${status}"${took}.`; + } const caveat = source === "challenge" && !joined ? outcome.inferred @@ -420,13 +416,11 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P const challengeKey = (id: string) => `challenge:${id}`; const providerOf = (data: { captcha_provider?: string; captcha_type?: string }) => data.captcha_provider ?? (data.captcha_type === "press_and_hold" ? "human" : data.captcha_type); - const kindOf = (data: { task_kind?: string; captcha_type?: string }) => - data.task_kind ?? - (data.captcha_type === "press_and_hold" - ? "press_and_hold" - : ["turnstile", "geetest", "recaptcha_v3"].includes(data.captcha_type ?? "") - ? "token" - : undefined); + const kindOf = (data: { task_kind?: string; captcha_type?: string }) => { + if (data.task_kind) return data.task_kind; + if (data.captcha_type === "press_and_hold") return "press_and_hold"; + if (["turnstile", "geetest", "recaptcha_v3"].includes(data.captcha_type ?? "")) return "token"; + }; const isSettled = (id: string) => { const at = settledAt.get(id); return at !== undefined && Date.now() - at < TASK_SETTLE_MS; @@ -491,7 +485,6 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P } async function resolve(): Promise { - // Stop waiting when a result joins to a task or visible challenge. await until(() => openTasks().length === 0 || Boolean(joinedResult()), TASK_SETTLE_MS); if (expectsChallengeResult() && !joinedResult()) await until(() => Boolean(joinedResult()), CHALLENGE_GRACE_MS); @@ -536,22 +529,21 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P function describe(source: Verdict["source"], status: string, joined: boolean, outcome: Outcome, page: PageState): Verdict { const took = outcome.durationMs ? ` after ${(outcome.durationMs / 1000).toFixed(1)}s` : ""; const kind = outcome.captchaProvider ? `${outcome.captchaProvider} ` : ""; - const headline = - source === "page" - ? "No terminal captcha telemetry arrived." - : source === "task" - ? status === "success" - ? outcome.taskKind === "image_challenge" - ? `The solver completed one ${kind}image round${took}. The challenge may serve another round.` - : outcome.taskKind === "token" - ? `The solver returned a token for a ${kind}challenge${took}. No challenge-level outcome was reported, so this is not a cleared challenge.` - : `The solver returned an answer for a ${kind}task${took}. No challenge-level outcome was reported, so this is not a cleared challenge.` - : `A ${kind}solver task ended as "${status}"${took}.` - : outcome.inferred && status === "solved" - ? `Kernel inferred a ${kind}challenge result from a successful token task${took}. The page did not confirm it cleared.` - : status === "solved" - ? `Kernel observed the ${kind}challenge clear${took}. That is not proof the site accepted the solution.` - : `Kernel reported the ${kind}challenge as "${status}"${took}.`; + let headline: string; + if (source === "page") { + headline = "No terminal captcha telemetry arrived."; + } else if (source === "task") { + if (status !== "success") headline = `A ${kind}solver task ended as "${status}"${took}.`; + else if (outcome.taskKind === "image_challenge") headline = `The solver completed one ${kind}image round${took}. The challenge may serve another round.`; + else if (outcome.taskKind === "token") headline = `The solver returned a token for a ${kind}challenge${took}. No challenge-level outcome was reported, so this is not a cleared challenge.`; + else headline = `The solver returned an answer for a ${kind}task${took}. No challenge-level outcome was reported, so this is not a cleared challenge.`; + } else if (outcome.inferred) { + headline = `Kernel inferred a ${kind}challenge result from a successful token task${took}. The page did not confirm it cleared.`; + } else if (status === "solved") { + headline = `Kernel observed the ${kind}challenge clear${took}. That is not proof the site accepted the solution.`; + } else { + headline = `Kernel reported the ${kind}challenge as "${status}"${took}.`; + } const caveat = source === "challenge" && !joined ? outcome.inferred From c5a5e007736e4da2fc66c97c5d5e544cc164a09f Mon Sep 17 00:00:00 2001 From: Sayan- <1415138+Sayan-@users.noreply.github.com> Date: Wed, 14 Oct 2026 18:34:57 +0000 Subject: [PATCH 3/4] Clarify captcha telemetry coverage and correlation --- browsers/telemetry/categories.mdx | 74 ++++++++++--------- .../telemetry/pausing-for-captcha-solves.mdx | 20 +++-- 2 files changed, 49 insertions(+), 45 deletions(-) diff --git a/browsers/telemetry/categories.mdx b/browsers/telemetry/categories.mdx index 664370d9..df3c0fbf 100644 --- a/browsers/telemetry/categories.mdx +++ b/browsers/telemetry/categories.mdx @@ -18,7 +18,7 @@ These categories report on the session itself rather than page content. | `platform` | Calls that manage the VM rather than drive the browser: recording lifecycle, filesystem, process execution, log streaming, scale-to-zero, telemetry, display and browser configuration, and extension uploads | `platform_api_call` | | `connection` | CDP and live view connect/disconnect activity | `cdp_connect`, `cdp_disconnect`, `live_view_connect`, `live_view_disconnect` | | `system` | VM-level failures | `system_oom_kill`, `service_crashed` | -| `captcha` | Solver task lifecycle and visible challenge outcomes | `captcha_solve_started`, `captcha_solve_result`, `captcha_challenge_result` | +| `captcha` | Solver task and challenge outcomes | `captcha_solve_started`, `captcha_solve_result`, `captcha_challenge_result` | `control` answers "what did my agent do." `platform` is mostly Kernel acting on the VM on your behalf - saving a profile, capturing a replay, polling a recorder - so it is off by default even though the rest of this group is on. Enable it when you are debugging a profile save, a replay, or a session-setup step rather than the agent itself. @@ -75,59 +75,65 @@ browser = kernel.browsers.create( Exclusion affects telemetry only; the commands still reach the browser. On `cdp_disconnect`, `telemetry_excluded` counts configured exclusions. Treat a nonzero `telemetry_dropped` as a telemetry-loss signal rather than using it to reconstruct the missing command sequence; the browser commands themselves still reach the browser. -### Correlate captcha tasks and challenges +### Captcha event fields -A visible captcha challenge can require multiple solver tasks. When the identifiers are present, use `task_id` to pair a `captcha_solve_started` event with its terminal `captcha_solve_result`, and use `challenge_id` to group the tasks for the same visible challenge. A `challenge_id` can continue across a page reload when the same challenge episode continues; it doesn't indicate task order or completion. +A visible captcha challenge can require multiple solver tasks. These events report task progress and challenge outcomes separately: | Event type | Scope | Meaning | | --- | --- | --- | | `captcha_solve_started` | Solver task | The solver accepted a task. This doesn't mean a solve is currently in flight. | | `captcha_solve_result` | Solver task | A task ended with `success`, `failure`, `timeout`, or `abandoned`. A successful task means the solver returned a usable solution, not that the visible challenge cleared. | -| `captcha_challenge_result` | Visible challenge or inferred token result | An observed challenge reached its overall `solved`, `failure`, `timeout`, or `abandoned` outcome, or a successful token task produced an inferred `solved` result without page observation. | +| `captcha_challenge_result` | Challenge | An observed challenge reached an outcome, or a successful token task produced an inferred `solved` result without page observation. | + +Every captcha event has `ts` (Unix microseconds), `type`, `category: captcha`, and `source`. The `data` payload holds the event details, and optional `truncated: true` means the payload was cut to the size limit. + +| Classification field | Values | Meaning | +| --- | --- | --- | +| `captcha_provider` | `hcaptcha`, `recaptcha_v2`, `recaptcha_v3`, `turnstile`, `geetest`, `arkose`, `human`, `other` | Captcha product. FunCaptcha uses `arkose`, press-and-hold uses `human`, and Enterprise reCAPTCHA uses its version bucket. | +| `task_kind` | `token`, `image_challenge`, `press_and_hold` | A `token` answers the whole widget; `image_challenge` completes one round that may be followed by another; `press_and_hold` is one attempt. Task events only; can be absent when the task can't be classified. | +| `captcha_type` | `hcaptcha`, `recaptcha_v2`, `recaptcha_v3`, `turnstile`, `geetest`, `press_and_hold`, `other` | Deprecated classification, still emitted on every captcha event. | -`captcha_provider` names the product: `hcaptcha`, `recaptcha_v2`, `recaptcha_v3`, `turnstile`, `geetest`, `arkose` (FunCaptcha), `human` (press-and-hold), or `other`. Enterprise reCAPTCHA uses its version bucket. `task_kind` says what a solver task produced: `token` is a response for the whole widget, `image_challenge` is one image round that may be followed by another, and `press_and_hold` is one attempt. `task_kind` is on task events only and can be absent when the task can't be classified. +Use `captcha_provider` for the product and `task_kind` for what a task result means. `captcha_type` values are frozen: image rounds sent through the captcha relay report `other`, while press-and-hold reports `press_and_hold`. Older events may lack the new fields, so fall back to `captcha_type` when reading stored events. -`captcha_type` is deprecated but still emitted. Use `captcha_provider` to identify the product and `task_kind` to decide what a task result means. Its values are frozen: image-challenge rounds report `other`, and press-and-hold reports `press_and_hold`. Older events can lack the new fields, so fall back to `captcha_type` when you read stored events. +| Other `data` field | Events | Meaning | +| --- | --- | --- | +| `task_id` | Task start and result; inferred challenge result | Opaque task identifier. Pair task events on it. | +| `challenge_id` | Task start and result when available; observed challenge result | Opaque identifier for one visible challenge. It can span tasks or continue across a reload. | +| `status` | Task and challenge results | Task: `success`, `failure`, `timeout`, `abandoned`. Challenge: `solved`, `failure`, `timeout`, `abandoned`. | +| `duration_ms` | Task and challenge results | Wall-clock duration of the task or challenge. An inferred result uses its task's duration. | +| `website_host` | All three | Host of the page; can be empty when a task has no page URL. | +| `website_path` | All three | Path of the page, excluding the query string. | +| `error_code` | Task result | Solver-specific failure code; absent on success. | +| `inferred` | Challenge result | True when a successful token task produced the result without page observation; absent on observed results. | -What each `captcha_provider` emits today: +### What each captcha provider emits | `captcha_provider` | `task_kind` on classified tasks | Challenge result | `challenge_id` on task events | | --- | --- | --- | --- | -| `hcaptcha` | `token` or `image_challenge` | Observed | Can be absent | -| `recaptcha_v2` | `token` or `image_challenge` | Observed | Can be absent | -| `human` | `press_and_hold` | Observed | Can be absent | -| `turnstile` | `token` | None | Can be absent | -| `geetest` | `token` | Inferred on successful token tasks | Can be absent | -| `recaptcha_v3` | `token` | Inferred on successful token tasks | Can be absent | -| `arkose` | `token` or `image_challenge` | Inferred on successful token tasks | Can be absent | -| `other` | `token`, `image_challenge`, or absent | Inferred on successful token tasks | Can be absent | +| `hcaptcha` | `token` or `image_challenge` | Observed | Sometimes | +| `recaptcha_v2` | `token` or `image_challenge` | Observed | No | +| `human` | `press_and_hold` | Observed | Yes | +| `turnstile` | `token` | None | No | +| `geetest` | `token` | Inferred on successful token tasks | No | +| `recaptcha_v3` | `token` | Inferred on successful token tasks | No | +| `arkose` | `token` or `image_challenge` | Inferred on successful token tasks | No | +| `other` | `image_challenge` or absent | None | No | -An observed result has a `challenge_id` and covers the visible challenge across its solver tasks. An inferred result has `inferred: true` and a `task_id` instead; join it to task events on `task_id`. Each eligible successful `token` task can produce an inferred result, so one challenge may have several. Failed tasks and image rounds don't produce inferred results. Turnstile tokens don't produce challenge results. +An observed result covers the visible challenge across its solver tasks. An inferred result reports a returned token without observing the page. Each eligible successful `token` task can produce one, so a challenge may have several inferred results. Failed tasks and image rounds don't produce inferred results. Turnstile tokens don't produce challenge results. -Telemetry delivery is best-effort and unordered: a start can arrive after its result, and any event can be absent. When you need a challenge outcome, wait for a bounded interval, then check available task results and the current page state. +### Correlate captcha tasks and challenges -Observed `recaptcha_v2` and `hcaptcha` results report `solved` or `abandoned`. `human` can also report `failure` and `timeout`. Inferred results report `solved` without observing the page. +Use `task_id` to pair a `captcha_solve_started` event with its `captcha_solve_result`. Use `challenge_id` to group tasks for a visible challenge when it's present. It can continue across a page reload; it doesn't indicate task order or completion. -Every captcha event has `ts` (Unix microseconds), `type`, `category: captcha`, and `source`. The `data` payload holds the event details, and optional `truncated: true` means that payload was cut to the size limit. The fields inside `data` are: +Tasks sent through the captcha relay don't have `challenge_id` because it sees individual tasks, not the visible challenge. In-browser hCaptcha image rounds and press-and-hold attempts do have it. This means an observed `recaptcha_v2` result can't be joined to its task events, even when both arrive. Join an inferred result on its `task_id` instead; it has no `challenge_id`. -| Field | Events | Meaning | -| --- | --- | --- | -| `captcha_provider` | All three | Captcha product. Optional on older events. | -| `task_kind` | Task start and result | What the task produces. Optional when the task can't be classified. | -| `captcha_type` | All three | Deprecated legacy classification. Still emitted. | -| `task_id` | Task start and result; inferred challenge result | Opaque task identifier. Pair task events on it; an inferred result uses it instead of `challenge_id`. | -| `challenge_id` | Task start and result; observed challenge result | Opaque identifier for one visible challenge. It can span multiple tasks or continue across a reload. | -| `status` | Task and challenge results | A task reports `success`, `failure`, `timeout`, or `abandoned`; a challenge reports `solved`, `failure`, `timeout`, or `abandoned`. | -| `duration_ms` | Task and challenge results | Wall-clock duration of the task or challenge. An inferred result uses its task's duration. | -| `website_host`, `website_path` | All three | Host and path of the page where the task ran or challenge appeared. The path excludes the query string; the host can be empty when a task has no page URL. | -| `error_code` | Task result | Solver-specific failure code; absent on success. | -| `inferred` | Challenge result | True when a successful token task produced the result without page observation. Absent on observed results. | +Telemetry delivery is best-effort and unordered: a start can arrive after its result, and any event can be absent. When you need a challenge outcome, wait for a bounded interval, then check available task results and the current page state. -Task events don't always carry `task_id` or `challenge_id`. Pair a start with its result on `task_id` when present, and use `challenge_id` to group tasks for a visible challenge only when it's there. +### Captcha challenge outcomes -Use each result event's `duration_ms` as the authoritative duration rather than calculating it from event timestamps. Challenge duration covers every solver attempt from when the challenge appeared until its terminal outcome. +Observed `recaptcha_v2` and `hcaptcha` results report `solved` or `abandoned`. `human` can also report `failure` and `timeout`. Inferred results report `solved` without observing the page. -Challenge outcomes have these meanings: +Use each result event's `duration_ms` as the authoritative duration rather than calculating it from event timestamps. An observed challenge's duration covers every solver attempt from when it appeared until its terminal outcome. | Status | Meaning | | --- | --- | diff --git a/browsers/telemetry/pausing-for-captcha-solves.mdx b/browsers/telemetry/pausing-for-captcha-solves.mdx index 3f9dd8ea..8d1690b8 100644 --- a/browsers/telemetry/pausing-for-captcha-solves.mdx +++ b/browsers/telemetry/pausing-for-captcha-solves.mdx @@ -19,9 +19,9 @@ The design follows one split: telemetry decides **what to say**, while the live | `captcha_solve_result` | Solver task | A task ended `success`, `failure`, `timeout`, or `abandoned`. Success means the solver returned a usable answer, not that the challenge cleared. | | `captcha_challenge_result` | Visible challenge or inferred token result | An observed challenge reached its overall outcome, or a successful token task produced an inferred result without page observation. | -Three rules from [Correlate captcha tasks and challenges](/browsers/telemetry/categories#correlate-captcha-tasks-and-challenges) shape everything below: +Four rules from [Correlate captcha tasks and challenges](/browsers/telemetry/categories#correlate-captcha-tasks-and-challenges) shape everything below: -- **Use the identifier the result carries.** Pair a start with its task result on `task_id`. An observed challenge result joins on `challenge_id`; an inferred result has `inferred: true` and joins on `task_id` instead. +- **Use the identifier the result carries.** Pair a start with its task result on `task_id`. An observed challenge result joins on `challenge_id` when the task carries it; an inferred result has `inferred: true` and joins on `task_id` instead. - **A task result isn't always the end of a challenge.** `task_kind: token` returns an answer for the whole widget, while `image_challenge` completes only one image round. On older events without `task_kind`, the gate falls back to `captcha_type` where it can and leaves ambiguous tasks unclassified. - **Delivery is best-effort and unordered.** A start can arrive after its result, and any event can be absent. Nothing may depend on arrival order, and every wait needs a deadline. - **Fall back to the page.** When you need a challenge-level outcome and don't have one, use the available task results and the current page state. @@ -178,9 +178,8 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P const openTasks = () => [...tasks.values()].filter(isOpen); const joinedResult = () => [...challenges].find(([id]) => joinable.has(id)); const expectsChallengeResult = () => [...tasks.values()].some((task) => - task.challengeId || - (task.status === "success" && task.taskKind === "token" && - !["turnstile", "hcaptcha", "recaptcha_v2"].includes(task.captchaProvider ?? "")) + task.taskKind !== "image_challenge" && + (task.challengeId || (task.status === "success" && task.captchaProvider !== "turnstile")) ); async function until(done: () => boolean, timeoutMs: number) { @@ -191,7 +190,7 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P - `resolve` waits briefly for a challenge result when one can follow the task, then reads the page. It prefers a joined challenge result, then an unjoined result, a task result, or the page alone. Image rounds and Turnstile tokens don't trigger the extra wait. + `resolve` waits briefly for a challenge result when one can follow the task, then reads the page. It prefers a joined challenge result, then an unjoined result, a task result, or the page alone. Known image rounds and Turnstile tokens don't trigger the extra wait. An observed reCAPTCHA result can be used without joining it to a task. `holding` and `pending` are separate on purpose. `pending` covers a terminal outcome that lands while no action is in flight — without it, a challenge result arriving between tool calls is recorded and never told to anyone. It has to catch *every* settled task, not just failures: a successful task-only result (Turnstile's usual path) and a task whose deadline lapsed with no result at all are both settled outcomes the agent needs to hear about. @@ -200,7 +199,7 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P ```ts captcha-gate.ts async function resolve(): Promise { await until(() => openTasks().length === 0 || Boolean(joinedResult()), TASK_SETTLE_MS); - if (expectsChallengeResult() && !joinedResult()) await until(() => Boolean(joinedResult()), CHALLENGE_GRACE_MS); + if (expectsChallengeResult() && challenges.size === 0) await until(() => challenges.size > 0, CHALLENGE_GRACE_MS); const page = await probePage(); const challenge = joinedResult() ?? [...challenges][0]; @@ -474,9 +473,8 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P const openTasks = () => [...tasks.values()].filter(isOpen); const joinedResult = () => [...challenges].find(([id]) => joinable.has(id)); const expectsChallengeResult = () => [...tasks.values()].some((task) => - task.challengeId || - (task.status === "success" && task.taskKind === "token" && - !["turnstile", "hcaptcha", "recaptcha_v2"].includes(task.captchaProvider ?? "")) + task.taskKind !== "image_challenge" && + (task.challengeId || (task.status === "success" && task.captchaProvider !== "turnstile")) ); async function until(done: () => boolean, timeoutMs: number) { @@ -486,7 +484,7 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P async function resolve(): Promise { await until(() => openTasks().length === 0 || Boolean(joinedResult()), TASK_SETTLE_MS); - if (expectsChallengeResult() && !joinedResult()) await until(() => Boolean(joinedResult()), CHALLENGE_GRACE_MS); + if (expectsChallengeResult() && challenges.size === 0) await until(() => challenges.size > 0, CHALLENGE_GRACE_MS); const page = await probePage(); const challenge = joinedResult() ?? [...challenges][0]; From d9f0738d4caecbe9c6642f305841dcba81f05d95 Mon Sep 17 00:00:00 2001 From: Sayan- <1415138+Sayan-@users.noreply.github.com> Date: Wed, 14 Oct 2026 18:50:24 +0000 Subject: [PATCH 4/4] Polish captcha telemetry guidance --- browsers/telemetry/categories.mdx | 10 +++++----- browsers/telemetry/pausing-for-captcha-solves.mdx | 4 ++-- browsers/telemetry/streaming.mdx | 2 +- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/browsers/telemetry/categories.mdx b/browsers/telemetry/categories.mdx index df3c0fbf..5e5ec0f1 100644 --- a/browsers/telemetry/categories.mdx +++ b/browsers/telemetry/categories.mdx @@ -93,7 +93,7 @@ Every captcha event has `ts` (Unix microseconds), `type`, `category: captcha`, a | `task_kind` | `token`, `image_challenge`, `press_and_hold` | A `token` answers the whole widget; `image_challenge` completes one round that may be followed by another; `press_and_hold` is one attempt. Task events only; can be absent when the task can't be classified. | | `captcha_type` | `hcaptcha`, `recaptcha_v2`, `recaptcha_v3`, `turnstile`, `geetest`, `press_and_hold`, `other` | Deprecated classification, still emitted on every captcha event. | -Use `captcha_provider` for the product and `task_kind` for what a task result means. `captcha_type` values are frozen: image rounds sent through the captcha relay report `other`, while press-and-hold reports `press_and_hold`. Older events may lack the new fields, so fall back to `captcha_type` when reading stored events. +Use `captcha_provider` for the product and `task_kind` for what a task result means. `captcha_type` values are frozen: `other` covers Arkose, unclassified tasks, and some image rounds; press-and-hold reports `press_and_hold`. An image round can also report `hcaptcha`, so don't infer its task kind from `captcha_type`. Older events may lack the new fields, so fall back to `captcha_type` when reading stored events. | Other `data` field | Events | Meaning | | --- | --- | --- | @@ -108,9 +108,9 @@ Use `captcha_provider` for the product and `task_kind` for what a task result me ### What each captcha provider emits -| `captcha_provider` | `task_kind` on classified tasks | Challenge result | `challenge_id` on task events | +| `captcha_provider` | `task_kind` on classified tasks | Challenge result | `challenge_id` on emitted task events | | --- | --- | --- | --- | -| `hcaptcha` | `token` or `image_challenge` | Observed | Sometimes | +| `hcaptcha` | `token` or `image_challenge` | Observed | In-browser image rounds only | | `recaptcha_v2` | `token` or `image_challenge` | Observed | No | | `human` | `press_and_hold` | Observed | Yes | | `turnstile` | `token` | None | No | @@ -125,13 +125,13 @@ An observed result covers the visible challenge across its solver tasks. An infe Use `task_id` to pair a `captcha_solve_started` event with its `captcha_solve_result`. Use `challenge_id` to group tasks for a visible challenge when it's present. It can continue across a page reload; it doesn't indicate task order or completion. -Tasks sent through the captcha relay don't have `challenge_id` because it sees individual tasks, not the visible challenge. In-browser hCaptcha image rounds and press-and-hold attempts do have it. This means an observed `recaptcha_v2` result can't be joined to its task events, even when both arrive. Join an inferred result on its `task_id` instead; it has no `challenge_id`. +Most solver tasks identify an individual solve, so they have `task_id` but no `challenge_id`. In-browser hCaptcha image rounds and press-and-hold attempts also identify the visible challenge and carry `challenge_id`. This means an observed `recaptcha_v2` result can't be joined to its task events, even when both arrive. Join an inferred result on its `task_id` instead; it has no `challenge_id`. Telemetry delivery is best-effort and unordered: a start can arrive after its result, and any event can be absent. When you need a challenge outcome, wait for a bounded interval, then check available task results and the current page state. ### Captcha challenge outcomes -Observed `recaptcha_v2` and `hcaptcha` results report `solved` or `abandoned`. `human` can also report `failure` and `timeout`. Inferred results report `solved` without observing the page. +Observed `recaptcha_v2` and `hcaptcha` results report `solved` or `abandoned`. `human` can also report `failure` when its solve budget is exhausted. Inferred results report `solved` without observing the page. Use each result event's `duration_ms` as the authoritative duration rather than calculating it from event timestamps. An observed challenge's duration covers every solver attempt from when it appeared until its terminal outcome. diff --git a/browsers/telemetry/pausing-for-captcha-solves.mdx b/browsers/telemetry/pausing-for-captcha-solves.mdx index 8d1690b8..aebca297 100644 --- a/browsers/telemetry/pausing-for-captcha-solves.mdx +++ b/browsers/telemetry/pausing-for-captcha-solves.mdx @@ -636,7 +636,7 @@ Every verdict pairs a telemetry claim with the page state it was checked against | Source | Message | | --- | --- | -| Observed challenge result | "Kernel observed the challenge clear after 18.0s. That is not proof the site accepted the solution. No captcha widget is visible on the page." | +| Observed challenge result | "Kernel observed the recaptcha_v2 challenge clear after 18.0s. That is not proof the site accepted the solution. No captcha widget is visible on the page." | | Inferred challenge result | "Kernel inferred a geetest challenge result from a successful token task after 3.9s. The page did not confirm it cleared. No captcha widget is visible on the page." | | Unjoinable observed result | Same, plus "It could not be joined to this page's solver tasks, so treat it as an observation about the page." | | Unjoinable inferred result | Same, plus "It could not be joined to this page's solver tasks, so treat it as an uncorrelated token result." | @@ -659,7 +659,7 @@ Durations come straight from each event's `duration_ms`, which is authoritative; ## Next steps -- [Telemetry Categories](/browsers/telemetry/categories#correlate-captcha-tasks-and-challenges) — the full captcha event schema and correlation rules +- [Telemetry Categories](/browsers/telemetry/categories#captcha-event-fields) — captcha event fields and correlation rules - [Stream Telemetry](/browsers/telemetry/streaming) — resuming a dropped stream, filtering by category - [Stealth mode](/browsers/bot-detection/stealth) — what the automatic captcha solver covers - [Playwright with Computer Use Fallback](/browsers/playwright-computer-use-fallback) — holding and redirecting an agent mid-run for a different reason diff --git a/browsers/telemetry/streaming.mdx b/browsers/telemetry/streaming.mdx index 8bd77414..3fcb1b18 100644 --- a/browsers/telemetry/streaming.mdx +++ b/browsers/telemetry/streaming.mdx @@ -35,7 +35,7 @@ with kernel.browsers.telemetry.stream(session_id) as stream: ``` -To filter, check `event.category` and `event.type` in your loop, or pass `type` to the stream endpoint to receive only selected event types. Repeat `type` or pass comma-separated values for multiple types. The archived events endpoint also accepts `type`. If the stream drops, re-open it with the last `seq` you processed as `last_event_id` to resume without gaps. +To filter, check `event.category` and `event.type` in your loop, or pass `type` when you open the stream. For direct HTTP requests, repeat `type` or pass comma-separated values for multiple types. The [Read telemetry events](https://kernel.sh/docs/api-reference/browser-telemetry/read-telemetry-events-for-a-browser-session) endpoint accepts the same filter. If the stream drops, re-open it with the last `seq` you processed as `last_event_id` to resume without gaps. ## Via CLI