diff --git a/browsers/telemetry/categories.mdx b/browsers/telemetry/categories.mdx index 0a20b1cf..5e5ec0f1 100644 --- a/browsers/telemetry/categories.mdx +++ b/browsers/telemetry/categories.mdx @@ -18,7 +18,7 @@ These categories report on the session itself rather than page content. | `platform` | Calls that manage the VM rather than drive the browser: recording lifecycle, filesystem, process execution, log streaming, scale-to-zero, telemetry, display and browser configuration, and extension uploads | `platform_api_call` | | `connection` | CDP and live view connect/disconnect activity | `cdp_connect`, `cdp_disconnect`, `live_view_connect`, `live_view_disconnect` | | `system` | VM-level failures | `system_oom_kill`, `service_crashed` | -| `captcha` | Solver task lifecycle and visible challenge outcomes | `captcha_solve_started`, `captcha_solve_result`, `captcha_challenge_result` | +| `captcha` | Solver task and challenge outcomes | `captcha_solve_started`, `captcha_solve_result`, `captcha_challenge_result` | `control` answers "what did my agent do." `platform` is mostly Kernel acting on the VM on your behalf - saving a profile, capturing a replay, polling a recorder - so it is off by default even though the rest of this group is on. Enable it when you are debugging a profile save, a replay, or a session-setup step rather than the agent itself. @@ -75,41 +75,69 @@ browser = kernel.browsers.create( Exclusion affects telemetry only; the commands still reach the browser. On `cdp_disconnect`, `telemetry_excluded` counts configured exclusions. Treat a nonzero `telemetry_dropped` as a telemetry-loss signal rather than using it to reconstruct the missing command sequence; the browser commands themselves still reach the browser. -### Correlate captcha tasks and challenges +### Captcha event fields -A visible captcha challenge can require multiple solver tasks. When the identifiers are present, use `task_id` to pair a `captcha_solve_started` event with its terminal `captcha_solve_result`, and use `challenge_id` to group the tasks for the same visible challenge. A `challenge_id` can continue across a page reload when the same challenge episode continues; it doesn't indicate task order or completion. +A visible captcha challenge can require multiple solver tasks. These events report task progress and challenge outcomes separately: | Event type | Scope | Meaning | | --- | --- | --- | | `captcha_solve_started` | Solver task | The solver accepted a task. This doesn't mean a solve is currently in flight. | | `captcha_solve_result` | Solver task | A task ended with `success`, `failure`, `timeout`, or `abandoned`. A successful task means the solver returned a usable solution, not that the visible challenge cleared. | -| `captcha_challenge_result` | Visible challenge | When emitted, the challenge reached its overall `solved`, `failure`, `timeout`, or `abandoned` outcome. | +| `captcha_challenge_result` | Challenge | An observed challenge reached an outcome, or a successful token task produced an inferred `solved` result without page observation. | + +Every captcha event has `ts` (Unix microseconds), `type`, `category: captcha`, and `source`. The `data` payload holds the event details, and optional `truncated: true` means the payload was cut to the size limit. -Challenge results are currently emitted for visible reCAPTCHA v2, hCaptcha, and press-and-hold challenges. Other captcha types can emit task events without a challenge result. Telemetry delivery is best-effort and isn't ordered: a start can arrive after its result, and any event can be absent. When you need a challenge-level outcome, wait for a bounded interval and fall back to available task results and the current page state. +| Classification field | Values | Meaning | +| --- | --- | --- | +| `captcha_provider` | `hcaptcha`, `recaptcha_v2`, `recaptcha_v3`, `turnstile`, `geetest`, `arkose`, `human`, `other` | Captcha product. FunCaptcha uses `arkose`, press-and-hold uses `human`, and Enterprise reCAPTCHA uses its version bucket. | +| `task_kind` | `token`, `image_challenge`, `press_and_hold` | A `token` answers the whole widget; `image_challenge` completes one round that may be followed by another; `press_and_hold` is one attempt. Task events only; can be absent when the task can't be classified. | +| `captcha_type` | `hcaptcha`, `recaptcha_v2`, `recaptcha_v3`, `turnstile`, `geetest`, `press_and_hold`, `other` | Deprecated classification, still emitted on every captcha event. | -What each `captcha_type` emits today: +Use `captcha_provider` for the product and `task_kind` for what a task result means. `captcha_type` values are frozen: `other` covers Arkose, unclassified tasks, and some image rounds; press-and-hold reports `press_and_hold`. An image round can also report `hcaptcha`, so don't infer its task kind from `captcha_type`. Older events may lack the new fields, so fall back to `captcha_type` when reading stored events. -| `captcha_type` | Task events | `captcha_challenge_result` | `challenge_id` on task events | +| Other `data` field | Events | Meaning | +| --- | --- | --- | +| `task_id` | Task start and result; inferred challenge result | Opaque task identifier. Pair task events on it. | +| `challenge_id` | Task start and result when available; observed challenge result | Opaque identifier for one visible challenge. It can span tasks or continue across a reload. | +| `status` | Task and challenge results | Task: `success`, `failure`, `timeout`, `abandoned`. Challenge: `solved`, `failure`, `timeout`, `abandoned`. | +| `duration_ms` | Task and challenge results | Wall-clock duration of the task or challenge. An inferred result uses its task's duration. | +| `website_host` | All three | Host of the page; can be empty when a task has no page URL. | +| `website_path` | All three | Path of the page, excluding the query string. | +| `error_code` | Task result | Solver-specific failure code; absent on success. | +| `inferred` | Challenge result | True when a successful token task produced the result without page observation; absent on observed results. | + +### What each captcha provider emits + +| `captcha_provider` | `task_kind` on classified tasks | Challenge result | `challenge_id` on emitted task events | | --- | --- | --- | --- | -| `hcaptcha` | Yes | Yes | Yes | -| `press_and_hold` | Yes | Yes | Yes | -| `recaptcha_v2` | Yes | Yes | Can be absent | -| `turnstile` | Yes | No | Can be absent | -| `geetest` | Yes | No | Can be absent | -| `recaptcha_v3` | Yes | No | Can be absent | -| `other` | Yes | No | Can be absent | +| `hcaptcha` | `token` or `image_challenge` | Observed | In-browser image rounds only | +| `recaptcha_v2` | `token` or `image_challenge` | Observed | No | +| `human` | `press_and_hold` | Observed | Yes | +| `turnstile` | `token` | None | No | +| `geetest` | `token` | Inferred on successful token tasks | No | +| `recaptcha_v3` | `token` | Inferred on successful token tasks | No | +| `arkose` | `token` or `image_challenge` | Inferred on successful token tasks | No | +| `other` | `image_challenge` or absent | None | No | + +An observed result covers the visible challenge across its solver tasks. An inferred result reports a returned token without observing the page. Each eligible successful `token` task can produce one, so a challenge may have several inferred results. Failed tasks and image rounds don't produce inferred results. Turnstile tokens don't produce challenge results. + +### Correlate captcha tasks and challenges + +Use `task_id` to pair a `captcha_solve_started` event with its `captcha_solve_result`. Use `challenge_id` to group tasks for a visible challenge when it's present. It can continue across a page reload; it doesn't indicate task order or completion. + +Most solver tasks identify an individual solve, so they have `task_id` but no `challenge_id`. In-browser hCaptcha image rounds and press-and-hold attempts also identify the visible challenge and carry `challenge_id`. This means an observed `recaptcha_v2` result can't be joined to its task events, even when both arrive. Join an inferred result on its `task_id` instead; it has no `challenge_id`. -Challenge results for `recaptcha_v2` and `hcaptcha` report `solved` or `abandoned`. `failure` and `timeout` are reported for `press_and_hold`, whose solver has an explicit attempt budget. +Telemetry delivery is best-effort and unordered: a start can arrive after its result, and any event can be absent. When you need a challenge outcome, wait for a bounded interval, then check available task results and the current page state. -Task events carry a `task_id`, but not all of them carry a `challenge_id`. Kernel sets `challenge_id` from tracking the challenge widget in the page - the same tracking that produces `captcha_challenge_result` - so a task the automatic solver runs without it reports a `task_id` only. Those task events can't be joined to a challenge result for the same page, even when one is emitted. Pair a start with its result on `task_id`, and use `challenge_id` for grouping only when it's there. +### Captcha challenge outcomes -Use each result event's `duration_ms` as the authoritative duration rather than calculating it from event timestamps. Challenge duration covers every solver attempt from when the challenge appeared until its terminal outcome. +Observed `recaptcha_v2` and `hcaptcha` results report `solved` or `abandoned`. `human` can also report `failure` when its solve budget is exhausted. Inferred results report `solved` without observing the page. -Challenge outcomes have these meanings: +Use each result event's `duration_ms` as the authoritative duration rather than calculating it from event timestamps. An observed challenge's duration covers every solver attempt from when it appeared until its terminal outcome. | Status | Meaning | | --- | --- | -| `solved` | The page observed the challenge clear after a solver attempt. This doesn't prove the site accepted the solution or that the guarded action succeeded. | +| `solved` | The page observed the challenge clear, or an inferred result reports a token without page observation. Neither proves the site accepted the solution or that the guarded action succeeded. | | `failure` | A terminal solver failure occurred, or all attempts ended while the challenge remained. | | `timeout` | The challenge-level wait budget expired while the challenge remained. | | `abandoned` | Observation ended without an attributable terminal challenge outcome. This includes a dismissed widget, a page unload, or a token appearing while multiple challenges from the same provider are open. In the last case, a task can report `success` while the challenge with the same `challenge_id` reports `abandoned`. | @@ -170,7 +198,7 @@ Telemetry is off by default. The default set isn't limited to session metadata: | `interaction` | Text of clicked elements and typed keys, which can include personal data entered into forms. | | `screenshot` | A full rendered image of the page - the broadest exposure, capturing anything visible on screen. | | `control` | The source you submit to the Playwright code-execution endpoint, on the `code` field of `api_call`, capped at 8 KB and marked with `...[truncated]` when cut. Whatever your script embeds is captured with it, so a literal password or token in the snippet is captured too. `cdp_command` carries sanitized arguments such as the method and phase, coordinates, counts, flags, and named keys such as `Enter` and `Tab`. Typed text, file paths, scripts, templates, dialog input, and autofill values aren't captured; navigation commands retain only the URL scheme, not the host, path, query, or fragment. | -| `captcha` | Captcha type, task and challenge outcomes and durations, and opaque `task_id` and `challenge_id` values. It can include the host and path of the page where a task ran or a challenge appeared; the query string is excluded. Failed tasks can include a solver-specific error code. | +| `captcha` | Captcha provider and task kind, task and challenge outcomes and durations, and opaque `task_id` and `challenge_id` values. It can include the host and path of the page where a task ran or a challenge appeared; the query string is excluded. Failed tasks can include a solver-specific error code. | | `platform`, `connection`, `system`, `monitor` | Session and VM metadata only (VM-management calls, connection and health events). No page content. | Captured events are persisted and can be replayed by [resuming the stream](/browsers/telemetry/streaming#resuming-after-a-disconnect), so this sensitivity applies to the data at rest, not just the live stream. Events are retained for 30 days, then expired (see [Retention](/browsers/telemetry/overview#retention)). Treat captured telemetry - and anywhere you forward or store it - with the same care as the underlying content. For how Kernel encrypts, retains, and processes data overall, see [Security](/security) and the [Data Processing Addendum](/dpa). diff --git a/browsers/telemetry/pausing-for-captcha-solves.mdx b/browsers/telemetry/pausing-for-captcha-solves.mdx index 5f21af34..aebca297 100644 --- a/browsers/telemetry/pausing-for-captcha-solves.mdx +++ b/browsers/telemetry/pausing-for-captcha-solves.mdx @@ -17,11 +17,12 @@ The design follows one split: telemetry decides **what to say**, while the live | --- | --- | --- | | `captcha_solve_started` | Solver task | The solver accepted a task. It does **not** mean a solve is currently in flight. | | `captcha_solve_result` | Solver task | A task ended `success`, `failure`, `timeout`, or `abandoned`. Success means the solver returned a usable answer, not that the challenge cleared. | -| `captcha_challenge_result` | Visible challenge | The challenge reached its overall outcome. Only emitted for challenge types Kernel tracks as a widget. | +| `captcha_challenge_result` | Visible challenge or inferred token result | An observed challenge reached its overall outcome, or a successful token task produced an inferred result without page observation. | -Three rules from [Correlate captcha tasks and challenges](/browsers/telemetry/categories#correlate-captcha-tasks-and-challenges) shape everything below: +Four rules from [Correlate captcha tasks and challenges](/browsers/telemetry/categories#correlate-captcha-tasks-and-challenges) shape everything below: -- **`task_id` is the only join.** Pair a start with its result on it. `challenge_id` groups tasks for one visible challenge and is present only when Kernel tracked the widget — tasks without one can never be joined to a `captcha_challenge_result`, even when one is emitted for the same page. +- **Use the identifier the result carries.** Pair a start with its task result on `task_id`. An observed challenge result joins on `challenge_id` when the task carries it; an inferred result has `inferred: true` and joins on `task_id` instead. +- **A task result isn't always the end of a challenge.** `task_kind: token` returns an answer for the whole widget, while `image_challenge` completes only one image round. On older events without `task_kind`, the gate falls back to `captcha_type` where it can and leaves ambiguous tasks unclassified. - **Delivery is best-effort and unordered.** A start can arrive after its result, and any event can be absent. Nothing may depend on arrival order, and every wait needs a deadline. - **Fall back to the page.** When you need a challenge-level outcome and don't have one, use the available task results and the current page state. @@ -51,7 +52,7 @@ The four snippets below are one file, `captcha-gate.ts`, in order. - Three maps, one per thing the telemetry can tell you, all keyed so nothing depends on arrival order. `joinable` is the important one: it holds only the `challenge_id`s that actually appeared on a task event, which are the only ones a challenge result may be attributed to. + The gate records task and challenge events by ID, so delivery order doesn't matter. It joins inferred results on `task_id` and observed results on `challenge_id`. A task record created by a *result* already has a status, so a `captcha_solve_started` that arrives afterwards finds a closed task and leaves it closed. That only works while the task is still in `tasks`, though — once `reset` has cleared it, a late start for the same `task_id` would otherwise look like a brand-new one. `settledAt` remembers when `reset` cleared each ID, so a late arrival within `TASK_SETTLE_MS` of that is dropped instead of reopening an episode the agent has already been told about. Past that window the same ID is treated as fresh again, since Kernel can reuse a `challenge_id` when an episode continues across a page reload. @@ -84,7 +85,10 @@ interface PageState { interface Outcome { status?: string; durationMs?: number; - captchaType?: string; + captchaProvider?: string; + taskKind?: string; + challengeId?: string; + inferred?: boolean; } interface Verdict extends Outcome { @@ -98,15 +102,23 @@ interface Verdict extends Outcome { function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => Promise) { const tasks = new Map(); const challenges = new Map(); - // Only challenge_ids that appeared on a task event may be attributed to it. const joinable = new Set(); - // task_ids and challenge_ids reported and cleared by reset(), each stamped + // Task and challenge keys reported and cleared by reset(), each stamped // with when that happened. A late-arriving duplicate within TASK_SETTLE_MS // of that -- delivery is unordered, a start can arrive after its own // result -- is dropped instead of reopening an already-told episode. Past // that window it's treated as fresh, since Kernel can reuse the same // challenge_id when an episode continues across a page reload. const settledAt = new Map(); + const taskKey = (id: string) => `task:${id}`; + const challengeKey = (id: string) => `challenge:${id}`; + const providerOf = (data: { captcha_provider?: string; captcha_type?: string }) => + data.captcha_provider ?? (data.captcha_type === "press_and_hold" ? "human" : data.captcha_type); + const kindOf = (data: { task_kind?: string; captcha_type?: string }) => { + if (data.task_kind) return data.task_kind; + if (data.captcha_type === "press_and_hold") return "press_and_hold"; + if (["turnstile", "geetest", "recaptcha_v3"].includes(data.captcha_type ?? "")) return "token"; + }; const isSettled = (id: string) => { const at = settledAt.get(id); return at !== undefined && Date.now() - at < TASK_SETTLE_MS; @@ -119,25 +131,38 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P for await (const { event } of stream) { if (event.category !== "captcha") continue; - if (event.type === "captcha_solve_started" && event.data?.task_id && !isSettled(event.data.task_id)) { + if (event.type === "captcha_solve_started" && event.data?.task_id && !isSettled(taskKey(event.data.task_id))) { // A result for an unseen task lands already closed, so a start that // arrives after its own result never reopens it. const task = tasks.get(event.data.task_id) ?? { openedAt: Date.now() }; - task.captchaType = event.data.captcha_type ?? task.captchaType; + task.captchaProvider = providerOf(event.data) ?? task.captchaProvider; + task.taskKind = kindOf(event.data) ?? task.taskKind; + task.challengeId = event.data.challenge_id ?? task.challengeId; tasks.set(event.data.task_id, task); - if (event.data.challenge_id) joinable.add(event.data.challenge_id); - } else if (event.type === "captcha_solve_result" && event.data?.task_id && !isSettled(event.data.task_id)) { + joinable.add(taskKey(event.data.task_id)); + if (event.data.challenge_id) joinable.add(challengeKey(event.data.challenge_id)); + } else if (event.type === "captcha_solve_result" && event.data?.task_id && !isSettled(taskKey(event.data.task_id))) { const task = tasks.get(event.data.task_id) ?? { openedAt: Date.now() }; task.status = event.data.status; task.durationMs = event.data.duration_ms; - task.captchaType = event.data.captcha_type ?? task.captchaType; + task.captchaProvider = providerOf(event.data) ?? task.captchaProvider; + task.taskKind = kindOf(event.data) ?? task.taskKind; + task.challengeId = event.data.challenge_id ?? task.challengeId; tasks.set(event.data.task_id, task); - if (event.data.challenge_id) joinable.add(event.data.challenge_id); - } else if (event.type === "captcha_challenge_result" && event.data?.challenge_id && !isSettled(event.data.challenge_id)) { - challenges.set(event.data.challenge_id, { + joinable.add(taskKey(event.data.task_id)); + if (event.data.challenge_id) joinable.add(challengeKey(event.data.challenge_id)); + } else if (event.type === "captcha_challenge_result" && event.data) { + const key = event.data.challenge_id + ? challengeKey(event.data.challenge_id) + : event.data.inferred && event.data.task_id + ? taskKey(event.data.task_id) + : undefined; + if (!key || isSettled(key)) continue; + challenges.set(key, { status: event.data.status, durationMs: event.data.duration_ms, - captchaType: event.data.captcha_type, + captchaProvider: providerOf(event.data), + inferred: event.data.inferred, }); } } @@ -152,6 +177,10 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P const isOpen = (t: Outcome & { openedAt: number }) => !t.status && Date.now() - t.openedAt < TASK_SETTLE_MS; const openTasks = () => [...tasks.values()].filter(isOpen); const joinedResult = () => [...challenges].find(([id]) => joinable.has(id)); + const expectsChallengeResult = () => [...tasks.values()].some((task) => + task.taskKind !== "image_challenge" && + (task.challengeId || (task.status === "success" && task.captchaProvider !== "turnstile")) + ); async function until(done: () => boolean, timeoutMs: number) { const deadline = Date.now() + timeoutMs; @@ -161,7 +190,7 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P - `resolve` settles what it can, waits a bounded interval for a challenge-level result *only* when a task actually carried a `challenge_id`, then reads the page and picks the best available source: an attributable challenge result, then any challenge result (labelled as a page observation rather than a join), then task results, then the page alone. + `resolve` waits briefly for a challenge result when one can follow the task, then reads the page. It prefers a joined challenge result, then an unjoined result, a task result, or the page alone. Known image rounds and Turnstile tokens don't trigger the extra wait. An observed reCAPTCHA result can be used without joining it to a task. `holding` and `pending` are separate on purpose. `pending` covers a terminal outcome that lands while no action is in flight — without it, a challenge result arriving between tool calls is recorded and never told to anyone. It has to catch *every* settled task, not just failures: a successful task-only result (Turnstile's usual path) and a task whose deadline lapsed with no result at all are both settled outcomes the agent needs to hear about. @@ -169,9 +198,8 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P ```ts captcha-gate.ts async function resolve(): Promise { - // A challenge result covers every task under it, so stop waiting once one lands. await until(() => openTasks().length === 0 || Boolean(joinedResult()), TASK_SETTLE_MS); - if (joinable.size > 0 && !joinedResult()) await until(() => Boolean(joinedResult()), CHALLENGE_GRACE_MS); + if (expectsChallengeResult() && challenges.size === 0) await until(() => challenges.size > 0, CHALLENGE_GRACE_MS); const page = await probePage(); const challenge = joinedResult() ?? [...challenges][0]; @@ -191,16 +219,15 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P pending: () => challenges.size > 0 || [...tasks.values()].some((t) => !isOpen(t)), resolve, reset: () => { - // A challenge result covers every task under it, open or not, so an - // open task isn't preserved once its challenge already resolved -- - // only when there's no other signal covering it yet, meaning the - // solver may still be working via a follow-up task resolve() hasn't - // seen close out. - const covered = Boolean(joinedResult()); + // Keep an open task unless a result joined to that task or its + // challenge covers it. + const result = joinedResult(); const now = Date.now(); for (const [id, t] of tasks) { + const covered = result && (result[0] === taskKey(id) || + (t.challengeId && result[0] === challengeKey(t.challengeId))); if (isOpen(t) && !covered) continue; - settledAt.set(id, now); + settledAt.set(taskKey(id), now); tasks.delete(id); } for (const id of challenges.keys()) settledAt.set(id, now); @@ -214,25 +241,32 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P function describe(source: Verdict["source"], status: string, joined: boolean, outcome: Outcome, page: PageState): Verdict { const took = outcome.durationMs ? ` after ${(outcome.durationMs / 1000).toFixed(1)}s` : ""; - const kind = outcome.captchaType ? `${outcome.captchaType} ` : ""; - const headline = - source === "page" - ? "No terminal captcha telemetry arrived." - : source === "task" - ? status === "success" - ? `The solver returned an answer for a ${kind}task${took}. No challenge-level outcome was reported, so this is not a cleared challenge.` - : `A ${kind}solver task ended as "${status}"${took}.` - : status === "solved" - ? `Kernel observed the ${kind}challenge clear${took}. That is not proof the site accepted the solution.` - : `Kernel reported the ${kind}challenge as "${status}"${took}.`; + const kind = outcome.captchaProvider ? `${outcome.captchaProvider} ` : ""; + let headline: string; + if (source === "page") { + headline = "No terminal captcha telemetry arrived."; + } else if (source === "task") { + if (status !== "success") headline = `A ${kind}solver task ended as "${status}"${took}.`; + else if (outcome.taskKind === "image_challenge") headline = `The solver completed one ${kind}image round${took}. The challenge may serve another round.`; + else if (outcome.taskKind === "token") headline = `The solver returned a token for a ${kind}challenge${took}. No challenge-level outcome was reported, so this is not a cleared challenge.`; + else headline = `The solver returned an answer for a ${kind}task${took}. No challenge-level outcome was reported, so this is not a cleared challenge.`; + } else if (outcome.inferred) { + headline = `Kernel inferred a ${kind}challenge result from a successful token task${took}. The page did not confirm it cleared.`; + } else if (status === "solved") { + headline = `Kernel observed the ${kind}challenge clear${took}. That is not proof the site accepted the solution.`; + } else { + headline = `Kernel reported the ${kind}challenge as "${status}"${took}.`; + } const caveat = source === "challenge" && !joined - ? " It could not be joined to this page's solver tasks, so treat it as an observation about the page." + ? outcome.inferred + ? " It could not be joined to this page's solver tasks, so treat it as an uncorrelated token result." + : " It could not be joined to this page's solver tasks, so treat it as an observation about the page." : ""; const where = page.widgets.length ? `A captcha widget is still on the page (${page.widgets.join(", ")}).` : "No captcha widget is visible on the page."; - return { source, status, joined, captchaType: outcome.captchaType, durationMs: outcome.durationMs, page, message: `${headline}${caveat} ${where}` }; + return { source, status, joined, captchaProvider: outcome.captchaProvider, taskKind: outcome.taskKind, inferred: outcome.inferred, durationMs: outcome.durationMs, page, message: `${headline}${caveat} ${where}` }; } ``` @@ -352,7 +386,10 @@ interface PageState { interface Outcome { status?: string; durationMs?: number; - captchaType?: string; + captchaProvider?: string; + taskKind?: string; + challengeId?: string; + inferred?: boolean; } interface Verdict extends Outcome { @@ -366,15 +403,23 @@ interface Verdict extends Outcome { function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => Promise) { const tasks = new Map(); const challenges = new Map(); - // Only challenge_ids that appeared on a task event may be attributed to it. const joinable = new Set(); - // task_ids and challenge_ids reported and cleared by reset(), each stamped + // Task and challenge keys reported and cleared by reset(), each stamped // with when that happened. A late-arriving duplicate within TASK_SETTLE_MS // of that -- delivery is unordered, a start can arrive after its own // result -- is dropped instead of reopening an already-told episode. Past // that window it's treated as fresh, since Kernel can reuse the same // challenge_id when an episode continues across a page reload. const settledAt = new Map(); + const taskKey = (id: string) => `task:${id}`; + const challengeKey = (id: string) => `challenge:${id}`; + const providerOf = (data: { captcha_provider?: string; captcha_type?: string }) => + data.captcha_provider ?? (data.captcha_type === "press_and_hold" ? "human" : data.captcha_type); + const kindOf = (data: { task_kind?: string; captcha_type?: string }) => { + if (data.task_kind) return data.task_kind; + if (data.captcha_type === "press_and_hold") return "press_and_hold"; + if (["turnstile", "geetest", "recaptcha_v3"].includes(data.captcha_type ?? "")) return "token"; + }; const isSettled = (id: string) => { const at = settledAt.get(id); return at !== undefined && Date.now() - at < TASK_SETTLE_MS; @@ -387,25 +432,38 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P for await (const { event } of stream) { if (event.category !== "captcha") continue; - if (event.type === "captcha_solve_started" && event.data?.task_id && !isSettled(event.data.task_id)) { + if (event.type === "captcha_solve_started" && event.data?.task_id && !isSettled(taskKey(event.data.task_id))) { // A result for an unseen task lands already closed, so a start that // arrives after its own result never reopens it. const task = tasks.get(event.data.task_id) ?? { openedAt: Date.now() }; - task.captchaType = event.data.captcha_type ?? task.captchaType; + task.captchaProvider = providerOf(event.data) ?? task.captchaProvider; + task.taskKind = kindOf(event.data) ?? task.taskKind; + task.challengeId = event.data.challenge_id ?? task.challengeId; tasks.set(event.data.task_id, task); - if (event.data.challenge_id) joinable.add(event.data.challenge_id); - } else if (event.type === "captcha_solve_result" && event.data?.task_id && !isSettled(event.data.task_id)) { + joinable.add(taskKey(event.data.task_id)); + if (event.data.challenge_id) joinable.add(challengeKey(event.data.challenge_id)); + } else if (event.type === "captcha_solve_result" && event.data?.task_id && !isSettled(taskKey(event.data.task_id))) { const task = tasks.get(event.data.task_id) ?? { openedAt: Date.now() }; task.status = event.data.status; task.durationMs = event.data.duration_ms; - task.captchaType = event.data.captcha_type ?? task.captchaType; + task.captchaProvider = providerOf(event.data) ?? task.captchaProvider; + task.taskKind = kindOf(event.data) ?? task.taskKind; + task.challengeId = event.data.challenge_id ?? task.challengeId; tasks.set(event.data.task_id, task); - if (event.data.challenge_id) joinable.add(event.data.challenge_id); - } else if (event.type === "captcha_challenge_result" && event.data?.challenge_id && !isSettled(event.data.challenge_id)) { - challenges.set(event.data.challenge_id, { + joinable.add(taskKey(event.data.task_id)); + if (event.data.challenge_id) joinable.add(challengeKey(event.data.challenge_id)); + } else if (event.type === "captcha_challenge_result" && event.data) { + const key = event.data.challenge_id + ? challengeKey(event.data.challenge_id) + : event.data.inferred && event.data.task_id + ? taskKey(event.data.task_id) + : undefined; + if (!key || isSettled(key)) continue; + challenges.set(key, { status: event.data.status, durationMs: event.data.duration_ms, - captchaType: event.data.captcha_type, + captchaProvider: providerOf(event.data), + inferred: event.data.inferred, }); } } @@ -414,6 +472,10 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P const isOpen = (t: Outcome & { openedAt: number }) => !t.status && Date.now() - t.openedAt < TASK_SETTLE_MS; const openTasks = () => [...tasks.values()].filter(isOpen); const joinedResult = () => [...challenges].find(([id]) => joinable.has(id)); + const expectsChallengeResult = () => [...tasks.values()].some((task) => + task.taskKind !== "image_challenge" && + (task.challengeId || (task.status === "success" && task.captchaProvider !== "turnstile")) + ); async function until(done: () => boolean, timeoutMs: number) { const deadline = Date.now() + timeoutMs; @@ -421,9 +483,8 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P } async function resolve(): Promise { - // A challenge result covers every task under it, so stop waiting once one lands. await until(() => openTasks().length === 0 || Boolean(joinedResult()), TASK_SETTLE_MS); - if (joinable.size > 0 && !joinedResult()) await until(() => Boolean(joinedResult()), CHALLENGE_GRACE_MS); + if (expectsChallengeResult() && challenges.size === 0) await until(() => challenges.size > 0, CHALLENGE_GRACE_MS); const page = await probePage(); const challenge = joinedResult() ?? [...challenges][0]; @@ -443,16 +504,15 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P pending: () => challenges.size > 0 || [...tasks.values()].some((t) => !isOpen(t)), resolve, reset: () => { - // A challenge result covers every task under it, open or not, so an - // open task isn't preserved once its challenge already resolved -- - // only when there's no other signal covering it yet, meaning the - // solver may still be working via a follow-up task resolve() hasn't - // seen close out. - const covered = Boolean(joinedResult()); + // Keep an open task unless a result joined to that task or its + // challenge covers it. + const result = joinedResult(); const now = Date.now(); for (const [id, t] of tasks) { + const covered = result && (result[0] === taskKey(id) || + (t.challengeId && result[0] === challengeKey(t.challengeId))); if (isOpen(t) && !covered) continue; - settledAt.set(id, now); + settledAt.set(taskKey(id), now); tasks.delete(id); } for (const id of challenges.keys()) settledAt.set(id, now); @@ -466,25 +526,32 @@ function createCaptchaGate(kernel: KERNEL, sessionId: string, probePage: () => P function describe(source: Verdict["source"], status: string, joined: boolean, outcome: Outcome, page: PageState): Verdict { const took = outcome.durationMs ? ` after ${(outcome.durationMs / 1000).toFixed(1)}s` : ""; - const kind = outcome.captchaType ? `${outcome.captchaType} ` : ""; - const headline = - source === "page" - ? "No terminal captcha telemetry arrived." - : source === "task" - ? status === "success" - ? `The solver returned an answer for a ${kind}task${took}. No challenge-level outcome was reported, so this is not a cleared challenge.` - : `A ${kind}solver task ended as "${status}"${took}.` - : status === "solved" - ? `Kernel observed the ${kind}challenge clear${took}. That is not proof the site accepted the solution.` - : `Kernel reported the ${kind}challenge as "${status}"${took}.`; + const kind = outcome.captchaProvider ? `${outcome.captchaProvider} ` : ""; + let headline: string; + if (source === "page") { + headline = "No terminal captcha telemetry arrived."; + } else if (source === "task") { + if (status !== "success") headline = `A ${kind}solver task ended as "${status}"${took}.`; + else if (outcome.taskKind === "image_challenge") headline = `The solver completed one ${kind}image round${took}. The challenge may serve another round.`; + else if (outcome.taskKind === "token") headline = `The solver returned a token for a ${kind}challenge${took}. No challenge-level outcome was reported, so this is not a cleared challenge.`; + else headline = `The solver returned an answer for a ${kind}task${took}. No challenge-level outcome was reported, so this is not a cleared challenge.`; + } else if (outcome.inferred) { + headline = `Kernel inferred a ${kind}challenge result from a successful token task${took}. The page did not confirm it cleared.`; + } else if (status === "solved") { + headline = `Kernel observed the ${kind}challenge clear${took}. That is not proof the site accepted the solution.`; + } else { + headline = `Kernel reported the ${kind}challenge as "${status}"${took}.`; + } const caveat = source === "challenge" && !joined - ? " It could not be joined to this page's solver tasks, so treat it as an observation about the page." + ? outcome.inferred + ? " It could not be joined to this page's solver tasks, so treat it as an uncorrelated token result." + : " It could not be joined to this page's solver tasks, so treat it as an observation about the page." : ""; const where = page.widgets.length ? `A captcha widget is still on the page (${page.widgets.join(", ")}).` : "No captcha widget is visible on the page."; - return { source, status, joined, captchaType: outcome.captchaType, durationMs: outcome.durationMs, page, message: `${headline}${caveat} ${where}` }; + return { source, status, joined, captchaProvider: outcome.captchaProvider, taskKind: outcome.taskKind, inferred: outcome.inferred, durationMs: outcome.durationMs, page, message: `${headline}${caveat} ${where}` }; } const PROBE = ` @@ -569,9 +636,12 @@ Every verdict pairs a telemetry claim with the page state it was checked against | Source | Message | | --- | --- | -| Attributable challenge result | "Kernel observed the challenge clear after 18.0s. That is not proof the site accepted the solution. No captcha widget is visible on the page." | -| Unjoinable challenge result | Same, plus "It could not be joined to this page's solver tasks, so treat it as an observation about the page." | -| Task result only | "The solver returned an answer for a turnstile task after 3.9s. No challenge-level outcome was reported, so this is not a cleared challenge. A captcha widget is still on the page (turnstile)." | +| Observed challenge result | "Kernel observed the recaptcha_v2 challenge clear after 18.0s. That is not proof the site accepted the solution. No captcha widget is visible on the page." | +| Inferred challenge result | "Kernel inferred a geetest challenge result from a successful token task after 3.9s. The page did not confirm it cleared. No captcha widget is visible on the page." | +| Unjoinable observed result | Same, plus "It could not be joined to this page's solver tasks, so treat it as an observation about the page." | +| Unjoinable inferred result | Same, plus "It could not be joined to this page's solver tasks, so treat it as an uncorrelated token result." | +| Token task result only | "The solver returned a token for a turnstile challenge after 3.9s. No challenge-level outcome was reported, so this is not a cleared challenge. A captcha widget is still on the page (turnstile)." | +| Image-round task result only | "The solver completed one recaptcha_v2 image round after 3.9s. The challenge may serve another round. A captcha widget is still on the page (recaptcha)." | | Nothing terminal arrived | "No terminal captcha telemetry arrived. A captcha widget is still on the page (turnstile)." | Durations come straight from each event's `duration_ms`, which is authoritative; don't compute them from event timestamps. @@ -579,7 +649,7 @@ Durations come straight from each event's `duration_ms`, which is authoritative; ## Limits - **`TASK_SETTLE_MS` and `CHALLENGE_GRACE_MS` are the safety net.** Events can be absent, so both waits are bounded and the gate falls through to the page rather than stalling. Raise `TASK_SETTLE_MS` if your solves routinely run longer than 20s. -- **Challenge results aren't emitted for every widget type.** Turnstile, for instance, reports task events only, so the task-plus-page path is the one that runs there. +- **Challenge results aren't emitted for every widget type.** Turnstile reports task events only, so the task-plus-page path is the one that runs there. An inferred result for another provider reports a returned token, not an observed page clear. - **One episode at a time.** The gate resets after each verdict; overlapping visible challenges from the same provider aren't split apart, and the telemetry docs note Kernel's own event model can't always attribute a result in that case either. - **The page probe is best-effort.** It reads the DOM for known widget markers and a response token. A site that renders its challenge somewhere those selectors miss won't be held past the logged message alone. - **GeeTest has no selector either.** The solver reports `geetest` task events, but `PROBE`'s `groups` has no GeeTest entry and `tokenPresent` checks no GeeTest field, so `widgets` never reports one and the agent proceeds as soon as the task settles, even while the challenge is still open. Add a selector for your target's GeeTest integration if you need it to hold. @@ -589,7 +659,7 @@ Durations come straight from each event's `duration_ms`, which is authoritative; ## Next steps -- [Telemetry Categories](/browsers/telemetry/categories#correlate-captcha-tasks-and-challenges) — the full captcha event schema and correlation rules +- [Telemetry Categories](/browsers/telemetry/categories#captcha-event-fields) — captcha event fields and correlation rules - [Stream Telemetry](/browsers/telemetry/streaming) — resuming a dropped stream, filtering by category - [Stealth mode](/browsers/bot-detection/stealth) — what the automatic captcha solver covers - [Playwright with Computer Use Fallback](/browsers/playwright-computer-use-fallback) — holding and redirecting an agent mid-run for a different reason diff --git a/browsers/telemetry/streaming.mdx b/browsers/telemetry/streaming.mdx index f096f856..3fcb1b18 100644 --- a/browsers/telemetry/streaming.mdx +++ b/browsers/telemetry/streaming.mdx @@ -35,7 +35,7 @@ with kernel.browsers.telemetry.stream(session_id) as stream: ``` -To filter, check `event.category` and `event.type` in your loop. If the stream drops, re-open it with the last `seq` you processed as `last_event_id` to resume without gaps. +To filter, check `event.category` and `event.type` in your loop, or pass `type` when you open the stream. For direct HTTP requests, repeat `type` or pass comma-separated values for multiple types. The [Read telemetry events](https://kernel.sh/docs/api-reference/browser-telemetry/read-telemetry-events-for-a-browser-session) endpoint accepts the same filter. If the stream drops, re-open it with the last `seq` you processed as `last_event_id` to resume without gaps. ## Via CLI