diff --git a/.changeset/convex-collection.md b/.changeset/convex-collection.md new file mode 100644 index 0000000000..b85a855f9a --- /dev/null +++ b/.changeset/convex-collection.md @@ -0,0 +1,8 @@ +--- +'@tanstack/convex-db-collection': minor +'@tanstack/offline-transactions': minor +--- + +Add a Convex collection adapter with typed query subscriptions, full-row sync, automatic reactive pagination, durable offline mutation replay, optimistic mutation handlers, and support for both Convex JavaScript and React clients. + +Add opt-in mutation codecs to the durable outbox while preserving the default wire format and retaining unreadable codec records for recovery. diff --git a/.github/workflows/e2e-tests.yml b/.github/workflows/e2e-tests.yml index 6a3d2ec495..4d3a6e3b39 100644 --- a/.github/workflows/e2e-tests.yml +++ b/.github/workflows/e2e-tests.yml @@ -56,6 +56,7 @@ jobs: --filter '@tanstack/react-router-with-db...' --filter '@tanstack/electric-db-collection...' --filter '@tanstack/query-db-collection...' + --filter '@tanstack/convex-db-collection...' --filter '@tanstack/db...' build @@ -88,6 +89,10 @@ jobs: POSTGRES_PASSWORD: password POSTGRES_DB: e2e_test + - name: Run Convex backend E2E tests + if: matrix.group == 'browser' + run: pnpm --filter @tanstack/convex-db-collection test:e2e + - name: Run Query E2E tests if: matrix.group == 'browser' run: | diff --git a/.prettierignore b/.prettierignore index 9a9a6363b3..20eda14e54 100644 --- a/.prettierignore +++ b/.prettierignore @@ -7,3 +7,5 @@ **/snap pnpm-lock.yaml **/routeTree.gen.ts +packages/convex-db-collection/e2e/backend/.convex/ +packages/convex-db-collection/e2e/backend/convex/_generated/ diff --git a/docs/collections/convex-collection.md b/docs/collections/convex-collection.md new file mode 100644 index 0000000000..8d6e63c7f7 --- /dev/null +++ b/docs/collections/convex-collection.md @@ -0,0 +1,283 @@ +--- +title: Convex Collection +--- + +`@tanstack/convex-db-collection` subscribes to a Convex query and keeps its rows in a TanStack DB collection. Use DB for local filters, joins, and optimistic writes. Convex still runs your server functions and handles authentication, reconnects, and access control. + +## Install + +```sh +npm install @tanstack/convex-db-collection @tanstack/db convex +``` + +## Subscribe to a query + +Use a public Convex query that returns an array of documents. The adapter infers document and argument types from your generated function reference. + +```ts +import { createCollection } from '@tanstack/db' +import { convexCollectionOptions } from '@tanstack/convex-db-collection' +import { ConvexClient } from 'convex/browser' +import { api } from './convex/_generated/api' + +const client = new ConvexClient(import.meta.env.VITE_CONVEX_URL) + +export const tasks = createCollection( + convexCollectionOptions({ + id: 'tasks', + client, + query: api.tasks.list, + queryArgs: { projectId }, + getKey: (task) => task._id, + }), +) + +await tasks.preload() +``` + +An existing `ConvexReactClient`, including the client from `useConvex()`, also works. The adapter imports neither React nor the Convex runtime. Keep the client and collection stable across renders. + +Create fresh adapter options for each direct `createCollection` call. Reusing the same options, including a shallow copy, throws. A `collectionOptions` descriptor can still materialize independent collections in separate `DbClient` instances. + +Each result replaces the subscribed query's rows. Missing documents are deleted, and missing fields are removed. An empty array is a ready, empty collection. Keys must be unique strings or finite numbers. Duplicate keys reject the complete result and preserve the last valid snapshot. + +The adapter uses eager sync. DB filters, ordering, and limits apply locally to the subscribed result. They do not change your Convex query or fetch documents outside its result. To change the query arguments, create a new collection with a distinct ID. + +## Write through Convex mutations + +Supply handlers for the operations your application supports. Await mutations on the same client used for the subscription. + +```ts +const tasks = createCollection( + convexCollectionOptions({ + id: 'tasks', + client, + query: api.tasks.list, + queryArgs: { projectId }, + getKey: (task) => task._id, + onUpdate: async ({ transaction }) => { + await Promise.all( + transaction.mutations.map(({ original, changes }) => + client.mutation(api.tasks.update, { + id: original._id, + ...changes, + }), + ), + ) + }, + onDelete: async ({ transaction }) => { + await Promise.all( + transaction.mutations.map(({ original }) => + client.mutation(api.tasks.remove, { id: original._id }), + ), + ) + }, + }), +) + +const transaction = tasks.update(taskId, (task) => { + task.completed = true +}) +await transaction.isPersisted.promise +``` + +DB applies optimistic writes immediately. Successful Convex mutation promises wait for the client's queries to catch up. Before a handler settles, the adapter reads the latest cached result and waits for DB to accept its sync writes. It does not wait for visibility held by that same optimistic transaction. If a wrapping persistence layer rejects source acceptance, that sync run stays failed. Call `collection.cleanup()`, then `collection.preload()`, before retrying replay. A provider query error can recover when a later valid result arrives. A rejected handler rolls back its optimistic writes. + +Awaiting an HTTP mutation, an action, a different client's mutation, or a detached promise does not provide this query synchronization guarantee. Avoid applying Convex optimistic updates to the same query, because the adapter would receive those local results as source data. + +A DB transaction does not make several Convex calls atomic. For atomic batches, send the complete batch through one Convex mutation. That mutation defines argument validation, authorization, and server write behavior. + +### Inserts and generated IDs + +Convex assigns `_id` on the server. DB needs a stable key before an optimistic insert. For optimistic inserts, add an application key such as `clientId` to your schema and documents. Use that key in `getKey` and send it through your insert mutation. Use `select` to omit server-only fields from the local row type. + +```ts +select: (tasks) => tasks.map((task) => ({ + clientId: task.clientId, + projectId: task.projectId, + title: task.title, +})), +getKey: (task) => task.clientId, +onInsert: async ({ transaction }) => { + await Promise.all( + transaction.mutations.map(({ modified }) => + client.mutation(api.tasks.create, { + clientId: modified.clientId, + projectId: modified.projectId, + title: modified.title, + }), + ), + ) +}, +``` + +Insert a local row with `tasks.insert({ clientId: crypto.randomUUID(), projectId, title })`. The server query must return the same `clientId`. + +Alternatively, call `client.mutation(api.tasks.create, args)` directly. The subscription adds the resulting document with its server ID. The adapter does not invent Convex IDs or silently replace temporary keys. + +## Project query results + +For an envelope or a nullable document, supply `select`. Its output is the complete set of rows for that collection. + +```ts +convexCollectionOptions({ + client, + query: api.tasks.listWithMetadata, + queryArgs: { projectId }, + select: (result) => result.tasks, + getKey: (task) => task._id, +}) + +convexCollectionOptions({ + client, + query: api.tasks.get, + queryArgs: { id: taskId }, + select: (task) => (task === null ? [] : [task]), + getKey: (task) => task._id, +}) +``` + +Keep projections synchronous and treat query results as immutable. A projection can change the row shape. A standard schema can validate local writes through the collection's `schema` option. + +## Complete eager pagination + +Use `convexPaginatedCollectionOptions` for a complete eager mirror of a finite query that returns Convex's `PaginationResult`. The adapter supplies `paginationOpts`, loads every page automatically, and keeps the full result and its page subscriptions in memory. `preload()` waits for the whole result. Set a server-side scope such as a project or workspace before using it with a large table. + +```ts +import { convexPaginatedCollectionOptions } from '@tanstack/convex-db-collection' + +const tasks = createCollection( + convexPaginatedCollectionOptions({ + id: 'project-tasks', + client, + query: api.tasks.paginated, + queryArgs: { projectId }, + pageSize: 100, + maxRows: 5_000, + getKey: (task) => task._id, + }), +) + +await tasks.preload() +``` + +Your server query must pass `paginationOpts` through to the same ordered Convex database query on every call. Use Convex's `paginationOptsValidator` for argument validation. + +The adapter keeps each loaded cursor range subscribed. Fixed `endCursor` boundaries prevent gaps when rows move or are deleted. Convex split hints replace a range with two subscriptions. The adapter publishes a replacement only after every range is available and the final page reports exhaustion. `preload()` waits for this complete initial result. During later page acquisition, the last complete snapshot stays visible. + +`pageSize` defaults to 100 and must be a positive safe integer. It sets the initial request size. Reactive ranges can contain more rows after inserts. Both supported clients use their public query subscription APIs, without experimental pagination methods. + +`maxRows` defaults to 10,000 and must be a positive safe integer. It caps the accepted complete result. An oversized result rejects with `RangeError` before the adapter publishes it or continues past a known oversized prefix. Initial `preload()` rejects; later growth preserves the last accepted snapshot and reports the error through `onError` and `getSyncError()`. Raise the finite cap only for a source size your application can hold. + +The cap does not limit the bytes in a server response or guarantee peak memory use. A single reactive page can already exceed it. `pageSize` does not bound total memory or initial load time. Initial acquisition can inspect earlier cached pages repeatedly as it extends the cursor chain. + +This API uses eager sync and does not translate DB's `loadSubset` requests. Demand-driven loading remains a separate API decision because it needs an application-supplied filter and ordering translation, ownership of overlapping windows, and rejection of unsupported requests. Until that API exists, narrow the server query or use another adapter when fetching only a requested window is required. + +For paginated collections, `select` receives the complete row array. DB still applies local filters, joins, ordering, and limits. Pagination loads a finite query completely into memory, it does not push DB predicates to your server. + +## Durable offline writes + +Use the existing durable outbox with the Convex bridge: + +```sh +npm install @tanstack/offline-transactions +``` + +Start from a stable Collection that uses an application key such as `clientId`. Keep its ID, registry key, and named mutation function stable across reloads. + +```ts +import { IndexedDBAdapter, startOfflineExecutor } from '@tanstack/offline-transactions' +import { convexMutationCodec, convexOfflineMutation } from '@tanstack/convex-db-collection/offline' + +type Task = { clientId: string; title: string; completed: boolean } + +const executor = startOfflineExecutor({ + mutationCodec: convexMutationCodec, + collections: { tasks }, + storage: new IndexedDBAdapter('project-task-outbox'), + mutationFns: { + saveTask: convexOfflineMutation({ + client, + mutation: api.tasks.saveIdempotently, + collections: [tasks], + getArgs: ({ transaction }) => { + const task = transaction.mutations[0].modified as Task + return { clientId: task.clientId, title: task.title, completed: task.completed } + }, + }), + }, +}) + +await executor.waitForInit() + +const addTask = executor.createOfflineAction({ + mutationFnName: 'saveTask', + onMutate: (task) => { tasks.insert(task) }, +}) + +const transaction = addTask({ + clientId: crypto.randomUUID(), + title: 'Works after a reload', + completed: false, +}) + +void transaction.when('settled').catch(reportError) +``` + +Use the offline action to create durable writes. Calling `tasks.insert()` directly uses its normal handler and does not enter the durable outbox. The example's `tasks` Collection must project the same `Task` shape and use `clientId` as its key. + +The opt-in codec preserves Convex BigInt, binary data, special floats, and undefined fields in mutation changes. Keep the same codec configured across reloads. Missing or incompatible codecs stop replay and retain the stored work. Transaction metadata keeps the outbox’s existing JSON/Date/Temporal behavior. + +The outbox records the transaction before sending it. After a reload, the executor restores pending optimistic writes and replays their named mutation functions. The bridge supplies the original `idempotencyKey`, waits for the Convex mutation, then waits for each configured Collection to accept its source snapshot. It does not wait for visibility held by that same optimistic transaction. If a wrapping persistence layer rejects source acceptance, that sync run stays failed. Call `collection.cleanup()`, then `collection.preload()`, before retrying replay. A provider query error can recover when a later valid result arrives. + +Convex handles WebSocket interruptions without rejecting the mutation promise. Argument-construction errors and rejected mutation calls are terminal failures. The bridge turns them into `NonRetriableError`, so the executor rolls back instead of retrying invalid writes indefinitely. A later source-sync failure can retry with the original key. + +### Deduplicate on the server + +Your Convex mutation must accept `idempotencyKey: v.string()`. In that same mutation transaction: + +1. Look up the key in an indexed receipts table. +2. If a receipt exists, return its saved result. +3. Apply the complete write or batch. +4. Store the key and result in the receipts table. + +Convex executes these writes atomically. A client can lose an acknowledgement after the server commits, so SDK request deduplication alone cannot replace this durable receipt. Keep receipts for at least the full replay lifetime. Scope receipt keys to the authenticated owner, and apply authorization before returning a saved result. + +The [real backend test fixture](https://github.com/TanStack/db/blob/main/packages/convex-db-collection/e2e/backend/convex/tasks.ts) shows an idempotent write mutation. It is an isolated test backend without application authorization, so add your own access checks. + +For atomic batches, `getArgs` must include every operation in the offline transaction and your server mutation must apply them together. The example handles one insert per action. Supply the platform's storage and coordination adapters when using the executor outside a browser. See the [offline transactions guide](../guides/offline-transactions.md) for leadership, storage failures, retry policy, and disposal. + +The outbox stores pending writes, not a full offline read cache. Pair the Collection with DB's persistence support when it must show previously synced rows after a reload without network access. + +## Errors and cleanup + +```ts +convexCollectionOptions({ + client, + query: api.tasks.list, + queryArgs: { projectId }, + getKey: (task) => task._id, + onError: (error) => reportError(error), +}) +``` + +An initial subscription error rejects `preload()` with the original error. Later errors preserve existing rows and call `onError`. `collection.utils.getSyncError()` returns the latest error. This getter is not reactive. A valid query result clears provider or projection errors and allows initial sync to recover. + +Collection cleanup unsubscribes and invalidates callbacks from that sync run. It does not close the shared Convex client. After a restart, the first result replaces retained data, including when the result is empty. Close the client separately when your application no longer needs it. + +## Scope + +This adapter uses the supported `onUpdate` and `watchQuery` APIs, tested with Convex 1.47.0 and a real isolated local Convex backend. It does not access private client fields or implement a second Convex protocol. + +Convex only exposes data through your server functions. The adapter cannot bypass authorization, subscribe directly to an arbitrary table, or translate arbitrary DB predicates into Convex functions. Queries remain subject to Convex read and result limits. A paginated query result describes one page, not a complete table. Automatic reactive pagination uses `convexPaginatedCollectionOptions`. Arbitrary server predicate pushdown is not implemented. + +Convex manages connection recovery. Use the offline bridge for durable writes. A full read cache and an SSR snapshot fetch remain separate concerns. Reuse your existing Convex authentication setup. For SSR, use request-scoped `DbClient` instances and the DB hydration APIs. A module-scoped descriptor can wrap `convexCollectionOptions` with `collectionOptions`; each client receives fresh adapter state. Supply an appropriate Convex client for each request and close it when finished. + +## Run the backend tests + +```sh +pnpm --filter @tanstack/convex-db-collection... build +pnpm --filter @tanstack/convex-db-collection test:e2e +``` + +The test command starts an anonymous local Convex backend, deploys the fixture, runs both client integrations and generated process-crash histories, then stops the backend. It needs network access on the first run to download Convex's backend binary. It does not use your cloud deployment or deploy key. diff --git a/docs/config.json b/docs/config.json index 14ce831d6a..214305222b 100644 --- a/docs/config.json +++ b/docs/config.json @@ -67,6 +67,10 @@ { "label": "Collections", "children": [ + { + "label": "Convex Collection", + "to": "collections/convex-collection" + }, { "label": "Query Collection", "to": "collections/query-collection" diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index a085dd75d9..52c24c4188 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -11,6 +11,157 @@ local terms, but it must not silently rename a production concept. The [2026-10-10 owner-overlap review](oracle-reviews/2026-10-10-owner-overlap-audit.md) distinguishes canonical laws from their receiving witnesses across packages. +## Convex collection integration + +Exact-head follow-up records: [source and backend](oracle-reviews/2026-10-23-convex-source-and-backend.md) and [pagination and codecs](oracle-reviews/2026-10-23-convex-pagination-and-codecs.md), both bound to source revision `6511ee4ed0940cd54ee58ceeba7202ca859344b3`. + +The [public app receiving review](oracle-reviews/2026-10-24-convex-public-apps.md) is bound to test-source revision `4223c39a6d5c2b2e2fedb1a7eee3c0052b5e5013`. The [auth dependency follow-up](oracle-reviews/2026-10-24-convex-auth-dependency-update.md) records patched Auth.js 0.41.3 and Convex Auth 0.0.96 at revision `bae01f788134aa421a9018278ec6d2867519dae3`, with the full final 17-test backend suite passing. + +The Convex package has these independent judgments: + +- `packages/convex-db-collection/tests/snapshot-oracle.property.test.ts` compares + public rows with the latest valid provider array. It covers missing-row + deletion, full-row replacement, reintroduction, empty results, and atomic + duplicate-key rejection, through both raw results and successful select projections. +- `packages/convex-db-collection/tests/ownership-oracle.property.test.ts` owns + concrete options admission separately from snapshot refinement. Its identity + model crosses direct and shallow-copy reuse with idle, ready, and cleaned-up + owners. Reuse rejects before it can replace the first owner's cached mutation + confirmation. Fresh options and separate DbClient descriptor materializations + remain independent, including sibling cleanup and rightful-owner restart. + The original implementation failed admission at seed 20261022/path 2:0:0; + the original two-owner mutation witness settled the first owner to its stale + row. The claim repair passes both fixed and random 60-history campaigns and + direct replay. It follows the existing LocalStorage concrete-options policy. + This owner does not cover arbitrary Collection lifecycle or a real SDK's + multi-listener scheduling. SDK tests receive the cached-listener premise and exercise both real clients with either sibling cleaned up before the survivor receives another update. +- `packages/convex-db-collection/tests/pagination-oracle.property.test.ts` owns + complete finite snapshots across reactive cursor ranges. It varies page sizes + 1-4, forward/reverse callback delivery, row insertion/deletion/movement, and + required/recommended/no split hints. The reference is the latest input array; + the provider fixture independently answers numeric rank intervals. Comparison + checks provider row order after exhaustion and preserves the previous public snapshot while replacement ranges are pending. Split cursors stay inside their returned page ranges. The accepted-row bound rejects over-cap results without partial publication. + Its initial-materialization work law also varies 3-40 pages and 1-8 rows per + page. The original implementation copied incomplete prefixes and failed at + seed 20261022/path 0:0:0:0. The repaired fixed/random campaigns copy each + provider row once at first exhaustion. Cache scans still grow quadratically + with page count during sequential acquisition, this is not a total-work bound. +- `packages/convex-db-collection/tests/offline-oracle.property.test.ts` owns the + bridge's ordered argument/mutation/sync stages, durable key propagation, and + terminal application-error classification. Argument and mutation failures + reject with NonRetriableError; post-confirmation sync failures can retry. + The original argument-error path failed this oracle at seed 20261022, path + 5:1:1. The repair includes argument construction in terminal classification. + +- `packages/convex-db-collection/tests/values-oracle.property.test.ts` owns the + Convex receiving check for native scalar types, bytes, nested containers, and + removed fields across durable storage and fresh readers. + +Each owner runs matching fixed and random campaigns and accepts explicit +seed/path replay through their CONVEX_*_ORACLE_SEED/PATH environment variables. +Snapshot replay uses CONVEX_SNAPSHOT_ORACLE; pagination has separate CONVEX_PAGINATION_ROWS, CONVEX_PAGINATION_WORK, and CONVEX_PAGINATION_BOUND selectors; +offline bridge replay uses CONVEX_OFFLINE_ORACLE; native value replay uses +CONVEX_VALUES_ORACLE; ownership uses CONVEX_OWNERSHIP_ORACLE. + +`packages/convex-db-collection/e2e/backend-oracle.test.ts` receives these contracts +at an actual isolated local Convex backend using both Convex 1.47.0 SDK clients. +It compares pagination with a separately evaluated unpaginated server query, +including real server-generated split responses. Its durable driver runs fixed +and random insert/update/delete histories through SIGKILL after disk admission and, optionally, +a second SIGKILL after server acceptance but before recording outbox completion. +It requires no offline provider call, restored Collection rows, and one server +receipt after replay with the original key and the expected final server rows. +Native payload variation covers inserts and updates. A separate non-idempotent application counter must equal one, and the receipt key must match the on-disk durable key. Restored Collections are compared on every application field, including native payloads. Atomic batches and multi-tab leadership +remain outside this backend witness. Replay uses +CONVEX_BACKEND_INSERT_ORACLE_SEED/PATH, CONVEX_BACKEND_UPDATE_ORACLE_SEED/PATH, or CONVEX_BACKEND_DELETE_ORACLE_SEED/PATH, with matching operation test selection. The command owns backend startup and shutdown. +The offline engine's interruption, settlement, storage, and leadership owners +remain primary for those laws; this is their Convex receiving witness. + +`packages/convex-db-collection/e2e/public-pagination-oracle.test.ts` receives the +finite pagination contract through the actual public `get-convex/convex-demos` +functions at revision `2fac21e360bfcceae6065096f1b81480bd8c6fd5`. Its independent +append ledger predicts descending message order, indexed author membership, and +uppercase/author-initial projections. Both SDK clients exercise all three query +forms at page size three, live appends, real TCP connection loss, direct writes +while disconnected, and complete recovery after reconnect. Unique authored +bodies supply the stable key omitted by the transformed query. It observes +ordered input to select and public Collection rows after source acceptance. +This is six authored receiving histories, not a new primary pagination model or +an arbitrary transformed-key guarantee. Setup fetches the pinned source at run +time, with no upstream application source redistributed in this repository. + +`packages/convex-db-collection/e2e/auth-app-oracle.test.ts` receives the snapshot +contract through the official role-based Convex Auth app at revision +`45f08008e1257758560948db4280cc4e7d64e84a`. The app's schema, message functions, +permission checks, and role updates come from that pinned source. Setup changes +only the external sign-in providers to the official Anonymous provider, with +ephemeral signing keys and real issued JWTs. An independent set-valued permission +policy drives all six read/write/admin operation cells, including demotion and repeated equal-body sends, while a multiset of +authored message bodies predicts the public rows at each accepted source +checkpoint. Both SDKs must reject an unauthenticated Collection preload; an +invalid-token HTTP query is a separate authentication control. The app deliberately +allows self-service role selection, so this does not establish secure role +administration. The two authored receiving histories do not cover OAuth redirects, +email delivery, token refresh, browser auth components, or production hosting. +Both public-app fixtures deploy with this package's installed Convex 1.47 SDK. +The pagination app's upstream package declares 1.48, its business functions are +unchanged here but its original dependency installation is not being tested. + +`packages/convex-db-collection/e2e/backend-startup-oracle.test.ts` owns the +receiving fixture's occupied-port startup judgment. It holds the former fixed +pagination port, invokes the actual pinned public-app setup, and requires a +different loopback URL serving a fresh empty query. Expected availability comes +from an independently held listener, not a copy of CLI selection logic. An +already occupied port is also a valid precondition, and its external owner is +never closed. All three fixture setups delegate default port selection to the +supported Convex CLI, an explicit user `CONVEX_TEST_PORT` still requires that +port. This is a fixed native startup witness, not a concurrent-allocation proof +or a promise to start when no free ports exist. A CI run reproduced the original +mandatory-port failure at auth port 43330 before any tests ran. The [startup follow-up](oracle-reviews/2026-10-24-convex-backend-startup.md) records exact source revision `4b37df090402469cafed0cf9f5e16b9528dae60f`, the rejected fixed-port control, and the final 18-test run with the former auth port occupied. + +`packages/offline-transactions/tests/transaction-serializer-oracle.property.test.ts` +remains the primary wire-format owner. Its opt-in codec grammar crosses +insert/update/delete and arrays of signed 64-bit integers, special floats and +binary buffers in modified/original/changes. It uses an independently supplied +V8 codec, recreates the reader, and checks missing/mismatched/throwing codecs +through both get and getAll while preserving the exact stored bytes. Default +unversioned/v2/v3 histories retain their prior expectations. A version-4 record +requires its codec, and configured readers continue to accept old formats. +The original serializer failed at durable admission for seed 20261022/path +0:0:0:0, a single BigInt insert. The repaired codec path preserves the values. +The Convex values receiving owner, +`packages/convex-db-collection/tests/values-oracle.property.test.ts`, checks +bounded nested native trees, marker-like objects and undefined fields against +an explicit type/byte observation. The backend process witness also varies +native payloads. Transaction metadata is intentionally outside codec dispatch. + +The source-snapshot mutant that omits deletion and the partial-row mutant both +fail the snapshot owner at public row comparison. Disabling automatic continuation +or publishing an incomplete required split fails the pagination owner at public +row comparison; replay seeds and shrink paths are recorded beside the property. These bounded tests do not prove +continuous unbounded growth, cross-Collection atomic publication, cloud service +availability, or power-loss atomicity during a filesystem write. The test backend +has no application authorization; production authorization and atomic receipt +storage are requirements of each application's server mutation. + +The October 23 review adds real-Collection source-acceptance histories to the offline bridge owner: idle, pending first result, failed startup, provider failure, and cleanup. Successful empty or nonempty snapshots must actually be accepted. Original idle acceptance fails seed 20261023/path 0:0:0, the repaired fixed/random campaigns and direct replay pass. The same file keeps its ordered-stage model separate from the source-settlement and rejected-acceptance judgments. A rejected durable acceptance stays terminal for that sync run, same or changed cached rows cannot falsely certify a retry, and cleanup/restart with a fresh accepted snapshot recovers. The nonterminal mutant fails seed 20261023/path 0:0 under CONVEX_OFFLINE_ACCEPTANCE_ORACLE_SEED/PATH. Ordinary provider errors retain same-run valid-result recovery. Dedicated codec replay uses OFFLINE_CODEC_ORACLE_SEED/PATH/RUNS; legacy serializer properties keep OFFLINE_ORACLE selectors. + +The bounded eager API defaults maxRows to 10,000. It is not an on-demand window, a network-byte bound, a strict peak-memory bound, or a lifetime subscription-count bound. Cursor ranges remain subscribed, and cache scans still grow quadratically during acquisition. A separate on-demand API needs application-owned filter/order translation and demand-scoped publication/release witnesses before making those promises. + +Validation at the initial review: 43 adapter checks, 302 offline-engine tests, and seven real-backend tests passed. The October 23 strengthened backend suite passes nine tests with 36 generated crash histories. Final local validation passes 62 adapter checks and 302 offline-engine tests. Exact-head review records accompany this follow-up. The native-value receiving oracle rejects a +scalar-as-JSON decoder at the typed-value comparison (seed 20261022/path 0:0); +its repaired replay passes. `pnpm size:compare --packages +convex-db-collection,offline-transactions --files` measures 2,863 gzip / 2,594 +brotli bytes for the new adapter's combined exports, with dependencies external. +The shared outbox grows by 192 gzip / 174 brotli bytes. Kyle review repairs add 230 gzip / 213 brotli bytes over the published PR head for actual first-source acceptance and the accepted-row bound. Existing tree-shaken app scenarios remain unchanged. Reusing the sync config materialization hook avoids a parallel startup lifecycle, and the existing page loop enforces the bound without a second pager. A new adapter has no +baseline to subtract; normal and paginated sources share one snapshot writer, +and the existing outbox supplies durability instead of a second write queue. +The codec hook is needed to preserve native values without changing default +JSON semantics. Review fixes add 74 gzip / 69 brotli bytes over the initial +adapter implementation, paying for concrete-options ownership enforcement and +deferring row materialization. Tests and documentation are separate from these +shipped production measurements. + ## Scope of the oracle repair project [Issue #1808](https://github.com/TanStack/db/issues/1808) commissioned a bounded diff --git a/docs/contributing/oracle-reviews/2026-10-23-convex-pagination-and-codecs.md b/docs/contributing/oracle-reviews/2026-10-23-convex-pagination-and-codecs.md new file mode 100644 index 0000000000..147e2efcd4 --- /dev/null +++ b/docs/contributing/oracle-reviews/2026-10-23-convex-pagination-and-codecs.md @@ -0,0 +1,105 @@ +# Convex pagination and mutation codec review + +Reviewed source and test head: `6511ee4ed0940cd54ee58ceeba7202ca859344b3`. +This record reviews these owners against [ORC-001 through ORC-014](../oracle-rubric.md): + +- `packages/convex-db-collection/tests/pagination-oracle.property.test.ts` +- `packages/convex-db-collection/tests/values-oracle.property.test.ts` +- The opt-in codec property in `packages/offline-transactions/tests/transaction-serializer-oracle.property.test.ts` + +The pagination review is independent of its implementation author. The reviewer +also inspected the value models and codec dispatch independently, then repaired +replay routing and cleanup mechanics. That participation is not an independent +review of those harness repairs. This is not a new audit of every historical +serializer law. Source acceptance and snapshot ownership have separate reviews. + +No known product counterexample remains in the bounded domains below. Passing +these histories does not establish an on-demand API, a lifetime resource bound, +or arbitrary storage and process behavior. + +## Executed evidence + +At the reviewed head, the reviewer ran the pagination and Convex values owners, +including their normal fixed and random campaigns. The pagination file has ten +checks, including cleanup calibration. The Convex values property runs 100 +histories per campaign. The shared serializer file passes all 51 checks with +`OFFLINE_CODEC_ORACLE_SEED=20261022` and +`OFFLINE_CODEC_ORACLE_PATH=0:0:0:0`. This full-file command directly replays the +codec property without applying its path to unrelated properties. + +Before the selector repair, the same path under the shared `OFFLINE_ORACLE` +prefix made four unrelated properties fail with `Unable to replay, got wrong +path`. That is a replay-routing failure, not a product mismatch or checker kill. +The dedicated prefix repairs that failure. Other legacy serializer properties +keep their existing selectors. + +The pagination author recorded these assertion failures at their intended +observations, with repaired replays passing: + +| Wrong behavior | Seed and path | Observation | +| --- | --- | --- | +| Reverse concatenated pages | `20261022`, `0:1:0:2:3:1:1:1` | Ordered rows received by `select` | +| Publish a prefix while another range is missing | `20261022`, `0:1:3:1:2:5:7:3:3:0:1:1` | Previous public snapshot while acquisition is pending | +| Copy incomplete prefixes during initial acquisition | `20261022`, `0:0:0:0` | Provider row reads at first exhaustion | +| Omit the accepted-row cap | `20261023`, `0:0` | Error and retained public rows after growth beyond the cap | + +The prior codec calibration is recorded in the coverage map: a decoder that +leaves Convex scalar wire markers as JSON fails the typed-value observation at +`CONVEX_VALUES_ORACLE_SEED=20261022`, path `0:0`. This review inspected the +unchanged Convex encoding and observation boundary, and reran the repaired owner. +It does not claim to have rerun that production mutant after the head was frozen. + +## Requirement outcomes + +| Requirement | Outcome and evidence | +| --- | --- | +| ORC-001, authority and limits | Satisfied for the stated domains. Pagination follows the documented complete eager mirror, `maxRows`, and Convex's fixed `endCursor` and split semantics. The row-read check is a deliberately narrow acquisition cost guard, not a total complexity promise. Codec meaning follows the opt-in `MutationValueCodec` and version-4 wire contract. Metadata retains the existing version-3 contract. The Convex receiver uses its public scalar encoding. Limits are stated in the files and below. | +| ORC-002, independent judgment | Satisfied. Pagination expects authored rows in mathematical rank order, without using production cursor bookkeeping. The numeric interval provider answers requested intervals independently. Convex values are compared with their authored native trees through a type and byte observation. The shared codec fixture uses Node V8 serialization, while expected native values come directly from generated leaves. Neither codec model uses production dispatch to compute its expected result. | +| ORC-003, literate responsibilities | Satisfied within this scope. Pagination explains the complete snapshot law, range fixture, histories, public and select observations, pending cut, and work/cap comparisons beside the code. The Convex values opening distinguishes authored tree, storage driver, fresh reader, and typed comparison. The shared codec section separates its opt-in law, V8 fixture, native grammar, and storage/error comparisons from the older serializer model. | +| ORC-004, grammar controls | Satisfied within the declared bounds. Pagination varies 0-8 distinct rows, 1-10 snapshots, page sizes 1-4, both callback directions, and no/required/recommended splits. Work varies 3-40 pages and 1-8 rows per page. Cap histories vary limits 1-20, page sizes 1-6, and excess 1-6. Empty results distinguish readiness from nonempty data; row removal and movement distinguish complete replacement from append-only accumulation; split modes distinguish replacement from continuation; callback direction removes delivery-order dependence. Split cursors are asserted strictly inside the returned range, excluding the formerly illegal page-size-1 split. Convex values use depth-two trees, arrays up to three, bytes up to eight, bounded strings, signed int64s, special floats, undefined fields, and six marker-like field names. Removing a native leaf would remove evidence for that codec branch. Shared codec histories use 1-5 native leaves, bytes up to twelve, and each mutation kind, plus missing/wrong/throwing readers and a non-string encoder. Cycles, custom native objects, and arbitrary malformed server protocols are not legal histories in these grammars. | +| ORC-005, production path and observation | Satisfied. Pagination uses a real Collection and adapter with controlled provider cache callbacks. It compares public keyed rows and ordered `select` input after exhaustion, and both while a new range lacks a result. The pending pump drains scheduled flushes before checking, so it reaches incomplete acquisition rather than filling new ranges first. The work recorder counts numeric provider element reads before explicit cache refresh. Codec drivers use actual OutboxManager writes and fresh readers; all three mutation payloads are compared. The shared owner also checks registry identity, wire version/codec ID, metadata, mixed legacy records, and byte-for-byte retained storage after failed reads. No concrete missing-input hypothesis remains for these named cuts under the stated provider premises. | +| ORC-006, calibration | Satisfied with the recorded controls above. Pagination's wrong page order, premature prefix, extra row reads, and absent cap each fail the relevant assertion. The typed-value receiver rejects scalar-as-JSON normalization. The original default serializer's BigInt admission failure and repaired codec replay are recorded in the coverage map. Replay-routing failures are reported separately and are not counted as product calibration. | +| ORC-007, campaigns and replay | Satisfied. Each generated property normally runs matching fixed and random budgets. Pagination has separate `CONVEX_PAGINATION_ROWS`, `CONVEX_PAGINATION_WORK`, and `CONVEX_PAGINATION_BOUND` seed/path selectors, each with 60 runs. Values uses `CONVEX_VALUES_ORACLE`, with 100. New shared codec replay uses `OFFLINE_CODEC_ORACLE_SEED/PATH/RUNS`, defaulting to 60. Their direct replay branches bypass the normal two campaigns for the selected property. No reviewed property uses `fc.commands`, so command `replayPath` handling is not applicable. | +| ORC-008, state minimality | Satisfied for pagination. Its last accepted input array is necessary: a missing replacement range must preserve that array, and a later completed acquisition replaces it. Two different accepted snapshots are distinguished by the next pending-range observation. Provider slot/cache state drives production and is not the reference state machine. Not applicable to the value models, which compare one authored tree with one restored tree rather than reducing a stateful history. | +| ORC-009, vocabulary | Satisfied. Collection rows, public snapshot, source result, acceptance, and cleanup retain the glossary's meanings. Numeric rank intervals are an explicit fixture abstraction of opaque provider cursors, not production cursor parsing. The work recorder measures element reads, not network work or subscription count. Codec wire version and codec ID are distinct from the application's transaction version and named mutation function. No hidden production-state equivalence is introduced. | +| ORC-010, failure fidelity | Satisfied for the reviewed generated properties. Pagination uses the shared test cleanup instrument, whose double-fault calibration preserves the original comparison as AggregateError cause, keeps secondary failures, and attempts a later release after an earlier release throws. Values and new shared codec catch and rethrow the primary failure, then use the existing cleanupOfflineOracle instrument to attempt both Collection releases. That instrument logs secondary cleanup failures without replacing an existing primary. The reviewer inspected this control flow; the reviewed value properties do not inject cleanup failures themselves. | +| ORC-011, second formulation | Satisfied where a concrete shared-fault hypothesis applies. A numeric controlled provider could implement the same wrong cursor semantics as production, so the actual backend independently evaluates an unpaginated query and compares it with real cursor-range results, including actual split responses. A JSON-normalizing recorder could hide native value loss, so typed scalar/byte observation distinguishes that hypothesis, with native values also received by process/backend tests. No further specific shared semantic fault was identified that calls for another model. This is not an exhaustive absence claim. | +| ORC-012, review evidence | Satisfied by this versioned record tied to the exact reviewed source/test commit. It lists each requirement, concrete checks, controls, and remaining limits. The record reviews the pagination bug class at complete exhaustion, pending acquisition, ordered selection, initial element-read, and cap/error cuts. It reviews codecs at durable admission and fresh read, with all payloads and unreadable-record retention. Known limits below are not presented as closed classes. | +| ORC-013, distinguishing boundary witnesses | Satisfied within the promised boundaries. Generated cap histories first accept exactly `maxRows` rows, then reject excess, retain the previous snapshot, and recover after shrink. A separate pinned default test distinguishes 10,000 from 10,001, and invalid 0, negative, fractional, infinite, and NaN caps reject before subscription. Work's three-row/page-size-1 failure distinguishes copying an incomplete prefix from the one complete materialization. Codec histories distinguish valid binding from absent/wrong/throwing binding while requiring unchanged durable bytes. These are not claims about arbitrary resource thresholds. | +| ORC-014, controlled-premise handoff | Satisfied with explicit limits. The controlled pager assumes Convex fixed ranges and legal split responses; real local backend tests exercise both actual SDK clients, fixed ranges, and server-generated splits against an unpaginated query. The native tree/storage driver assumes preservation across serialization and fresh readers; backend crash tests receive native payloads on disk and across process death, now on insert and update. These receiving drivers use provider-specific controls and separate expected server queries/application state rather than literally reusing the numeric fixture grammar. Their shared law is documented, but arbitrary cloud scheduling and every generated native tree are not thereby received. | + +## Remaining limits + +The eager cap bounds accepted rows, not network bytes, peak memory, or lifetime +cursor ranges. A reactive range can itself return more than the cap. Old ranges +remain subscribed, so long-running insert/delete histories can accumulate ranges +even while the current row count remains small. Initial acquisition still scans +cached ranges repeatedly. Only row materialization, not total work, is bounded +by the work property. On-demand filtering, ordering, demand release, and a bounded +subscription policy need a separate API and receiving witnesses. + +The numeric interval grammar assumes coherent source revisions and finite +acquisition. It does not prove continuous growth, cross-Collection publication, +or arbitrary contradictory cursor responses. The backend is a real local Convex +backend, not evidence for cloud availability or network latency. + +The direct Convex value receiver uses one update-shaped mutation, checking +modified, original, and changes. Mutation kinds are crossed by the shared owner, +and CRUD is received by the backend process tests. This does not prove every +native tree through every operation and storage implementation. The reviewed +values are bounded acyclic trees; identity sharing, cycles, cross-realm natives, +custom objects, native Temporal, browser storage atomicity, and power-loss +atomicity are outside this evidence. These limits remain with the codec and +backend owners in the coverage map. + +## CI type-check follow-up + +CI exposed an omitted type-level selector entry in the shared config reader: +`OFFLINE_CODEC_ORACLE` worked at runtime but was absent from the accepted prefix +union. The offline package type check reproduced TS2820 before the repair. +Adding that exact prefix to the union makes the type check pass, and the same +full-file codec replay at seed 20261022/path 0:0:0:0 still passes all 51 checks. +This follow-up changes only a test configuration type, no production code, +model judgment, generated history, runtime selector routing, or comparison. +The exact-head semantic review above continues to refer to its stated source +revision rather than claiming a fresh independent audit of this type edit. diff --git a/docs/contributing/oracle-reviews/2026-10-23-convex-source-and-backend.md b/docs/contributing/oracle-reviews/2026-10-23-convex-source-and-backend.md new file mode 100644 index 0000000000..484b6af507 --- /dev/null +++ b/docs/contributing/oracle-reviews/2026-10-23-convex-source-and-backend.md @@ -0,0 +1,130 @@ +# Independent review of Convex snapshot, ownership, bridge, and backend oracles + +Reviewed implementation head: `6511ee4ed0940cd54ee58ceeba7202ca859344b3`. + +The reviewer did not author these semantic models. The reviewer changed their +cleanup harness after finding that cleanup could replace a primary mismatch. +Pagination is excluded from this independent verdict because the reviewer +implemented its changes; another reviewer covers that owner. + +## Scope and observations + +- Snapshot owner: `packages/convex-db-collection/tests/snapshot-oracle.property.test.ts`. + Last valid keyed source array, including absent fields and duplicate rejection, + through raw arrays and a successful uppercase text projection. Observations are + public keyed rows and error presence after synchronous delivery. +- Ownership owner: `packages/convex-db-collection/tests/ownership-oracle.property.test.ts`. + Concrete option identity admission and independent fresh/descriptor owners, + across idle, ready, and cleanup checkpoints. Observations are construction + rejection and public rows after cached mutation confirmation. +- Bridge owner: `packages/convex-db-collection/tests/offline-oracle.property.test.ts`. + Ordered stages and retry classification, plus cold real-Collection source + acceptance and rejected durable acceptance through cleanup/restart. Observations are stage traces, settlement, provider subscription + admission, and public rows before/after the first source delivery. +- Backend receiving owner: `packages/convex-db-collection/e2e/backend-oracle.test.ts`. + Real range pagination and durable insert/update/delete replay through process + death. The replay observations are durable receipt key, server application + counter, complete server fields, and complete replayed Collection fields. +- SDK receiving support: `packages/convex-db-collection/tests/sdk.test.ts`. + Both SDK clients receive subscriptions, mutation timestamp transitions, + reconnects, and sibling listener cleanup. These focused tests do not own + another independent state model. + +Fresh independent execution before the final freeze passed 28 checks across +snapshot, ownership, bridge, SDK, and pagination support. The author then ran +the new durable-acceptance rejection property with 60 fixed and 60 random +histories and direct replay after the independent finding described below. +The final source tree is pinned above; this record adds review evidence after +that implementation commit without changing its production or test sources. + +The backend author reports all nine final backend tests green after the adapter +freeze, including 36 generated process histories. Insert replay at seed 20261022, +path 0:0:0:1, and update replay at seed 20261022, path 0:0:0:0:0, pass with their +operation-specific selectors. The source acceptance, snapshot, ownership and +backend evidence below distinguishes independent source inspection and fresh +review execution from the author's hostile-control calibrations. + +## Requirement outcomes + +| Requirement | Snapshot | Ownership | Bridge, including cold source | Real backend receiving owner | +| --- | --- | --- | --- | --- | +| ORC-001, authority and limits | Complete keyed snapshots and select follow the public adapter guide. Transport and mutation timing are separate boundaries. | The concrete-options claim follows the established LocalStorage policy and documented adapter ownership. Arbitrary Collection lifecycle is excluded. | The guide promises ordered mutation then source acceptance, with terminal application errors and retryable post-confirmation sync failure. Storage/leadership stay with offline-engine owners. | Convex transactions and the application's atomic receipt contract authorize one application per durable key. Finite pagination must equal an independent unpaginated query. Cloud availability, multi-tab leadership, and power-loss writes are excluded. | +| ORC-002, independent judgment | Last valid input array and pairwise duplicate-key judgment do not compute adapter sync operations. The selected reference applies the explicit input transformation. | Identity and materialization form determine admission; no production claim flag or sync state is imported. | A dependency prefix determines stage traces. Cold source expectations depend on whether a valid snapshot was delivered, not adapter state. | Expected task state comes from the operation and draft. An unpaginated server query judges range results, and a counter distinguishes equal final rows after duplicate writes. | +| ORC-003, literate responsibilities | Opening prose explains contract, validity model, grammar, production path, comparison, and exclusions. | Opening prose maps identity forms and checkpoints to admission and mutation confirmation. | Separate prose explains stage-prefix judgment and cold source acceptance, their histories, cuts, and limits. | Opening prose separates pagination and process-replay judgments, SIGKILL cuts, server and Collection observations, and excluded boundaries. | +| ORC-004, grammar controls | Keys a/b/c, text up to 12, optional text up to 6, 0-5 rows and 1-30 snapshots. Empty, changed, absent, duplicate, reordered and reintroduced keys are reconstructible. Raw/projected variants distinguish missing projection. Duplicate snapshots are rejected atomically rather than admitted as valid rows. | Four identity forms cross idle/ready/cleaned-up and text up to 16. Direct/shallow forms must reject, fresh/descriptor forms must admit. Removing reuse or sibling cleanup loses distinct ownership witnesses. | Stage failure chooses none/args/mutation/sync, with keys/text up to 15. Cold histories choose idle/pending/startup-error and empty/rows/error/cleanup delivery. Rejected durable acceptance crosses empty/nonempty snapshots, 1-4 unchanged/changed callback retries, and a cleanup/restart cut. Delivery is irrelevant after failed startup, which is explicitly terminal at that cut. Empty success and rejected startup distinguish readiness from nonempty rows. | Insert/update/delete cross acknowledgement known/uncertain, text up to 12, rank -50..50, native signed int64/special float/0-12 bytes on insert/update. Delete has no payload. One pending transaction excludes batch and shared-key leadership histories. Child checkpoints enforce admission before offline SIGKILL and server acceptance before uncertain SIGKILL. | +| ORC-005, path and checkpoint | Real Collection plus adapter receive controlled callbacks; keyed public rows and error presence are checked after each delivery. | Real constructors, DbClient materialization, adapters, optimistic transactions and public rows run. Actual SDK sibling tests receive the provider multi-listener premise separately. | The stage driver runs the public bridge with explicit collaborators. The cold driver runs the public bridge with actual Collections and proves pending settlement and one subscription before delivering source data. The acceptance-rejection driver defers actual core commit behind durability, matching persistence's withAcceptedReceipt boundary, and checks rejection before and after retries, then successful fresh-run rows. | Both actual SDK clients run against the isolated local backend. Process replay uses fresh child processes and disk storage. Parent reads the durable key independently; child checks public fields only at replay completion. | +| ORC-006, calibration | Omitting projection is rejected at public row comparison, seed 20261022/path 0:1:1:0:3:4:3:7:7:7:7:7:6. Existing deletion and full-row mutants separately calibrate omitted keys/fields. | Original reuse failed at admission, seed 20261022/path 2:0:0; the two-owner confirmation regression additionally exposed stale first-owner rows. | Original argument failure escaped terminal classification, seed 20261022/path 5:1:1. Original cold acceptance resolved before delivery, seed 20261023/path 0:0:0. Removing terminal durable-failure guards fails at rejection observation, seed 20261023/path 0:0. | Duplicate effects with one receipt fail at applicationCount, insert seed 20261022/path 0:0:0:1. A substituted durable key fails at receipt-key equality, update path 0:0:0:0:0. Dropped replay-only payload fails child public-row deepStrictEqual at the same update path. | +| ORC-007, parity and replay | Identical property runs 150 fixed and 150 random histories; CONVEX_SNAPSHOT_ORACLE_SEED/PATH directly selects replay. | Identical property runs 60 fixed and 60 random histories; CONVEX_OWNERSHIP_ORACLE_SEED/PATH directly selects replay. | Stage, cold source, and rejected-acceptance properties each run 60 fixed and 60 random histories. CONVEX_OFFLINE_ORACLE, CONVEX_OFFLINE_SOURCE_ORACLE, and CONVEX_OFFLINE_ACCEPTANCE_ORACLE keep their seed/path bindings separate. | Each operation runs six fixed and six random histories against the same driver. CONVEX_BACKEND_INSERT_ORACLE, CONVEX_BACKEND_UPDATE_ORACLE, and CONVEX_BACKEND_DELETE_ORACLE each own their seed/path, preventing another operation's grammar from interpreting a recorded path. | +| ORC-008, model state | Retained last-valid rows are needed: a duplicate-key rejection followed by observation distinguishes replacement from preserving prior rows. No queue or transport state enters the model. | No stateful reference machine is introduced; admission is a direct identity-form judgment. Lifecycle checkpoints are production driver conditions. | The prefix computation is stateless. The cold source judgment distinguishes no valid source from a delivered valid source; these states allow different successful settlements. Rejected acceptance has failed-run/restarted-run reference states: cached or changed callbacks do not restore authority, while cleanup/restart does. | No stateful reference machine is introduced. The expected operation result and counter increment are recomputed independently from one logical write. | +| ORC-009, vocabulary | Source input and public Collection rows are named separately; retained expected rows mean the latest valid source snapshot. | Concrete owner identity, sync-run cleanup and DbClient materialization are explicitly distinct. | Source acceptance is distinct from visibility held by an optimistic transaction. Idle/pending/startup-error are declared driver starts, not a replacement Collection status enum. | Offline replay is distinguished from pagination replacement. Admission, server acceptance, replay completion, receipt key and application counter name concrete process/server observations. | +| ORC-010, failure fidelity | Shared withTestCleanup retains the primary failure and secondary release errors. | The helper enumerates admitted Collections at cleanup time and attempts every release. | The real-Collection driver's cleanup uses the same preservation helper; the stage driver owns no disposable resource. | The shared preservation helper protects backend bodies and attempts both client close and temporary storage removal. Child timeout/runtime failures are setup/process failures, not successful mutant kills. | +| ORC-011, second formulation | No concrete shared semantic-fault hypothesis remains after pairwise validity and direct source-array comparison; another copy is not needed. | No named shared semantic-fault hypothesis calls for another identity formulation. Real SDK support is a receiving witness, not a second ownership model. | No named shared semantic-fault hypothesis remains for the dependency prefix; the real-Collection receiving path repairs the prior stub assumption at a different boundary. | Equal final task rows could hide duplicate applications. The non-idempotent counter is the independent distinguishing formulation, while durable receipt-key equality also rejects key substitution. | + +## Bug-class boundary and calibration + +The original cold-replay defect violated source acceptance at +`bridge × idle/pending/failed startup × real Collection adapter × replay settlement`. +The repaired source driver rejects premature fulfillment before the first valid +snapshot, accepts empty snapshots, and distinguishes provider-error same-run +recovery from cleanup invalidation. Independent review found an adjacent hole: +a rejected durable proposal could leave the identity fast path clearing its +error on the same cached rows. The receiving fixture now defers core commit +behind durability rather than artificially rejecting after core acceptance. +Empty/nonempty proposals, one to four unchanged/changed retries, and a fresh +cleanup/restart distinguish terminal failed-base authority from provider recovery. +The nonterminal-guard mutant fails at the rejection checkpoint and repaired +fixed/random campaigns and direct replay pass. No known reachable counterexample +remains in these declared cells; no universal concurrency proof is claimed. + +Backend calibration rejected all three hostile controls at intended observations: + +- Applying a write twice while retaining one receipt fails the non-idempotent + application counter, insert seed 20261022/path 0:0:0:1. +- Substituting `wrong-durable-key` fails equality with the admission record's key, + update seed 20261022/path 0:0:0:0:0. +- Omitting payload only from the replay child's select projection fails its + complete public-row comparison, with server payload still correct, using that + same update replay path. + +These were assertion failures after the legal target checkpoint, rather than +setup failures or timeouts. The author restored every mutant in finally and +reran the repaired operation-specific direct replays. The parent independently +reads durable bytes and server effects; the replay child observes its Collection. + +## Remaining limits and verdict + +The cold source driver establishes real Collection acceptance with a controlled +provider, not a cloud or local-backend cold replay. The backend replay fixture +preloads its Collection. SDK listeners receive the actual SDK cache and listener +ownership premise. None of these observations claims cross-Collection atomic +publication, continuous unbounded pagination, native browser storage power-loss +atomicity, or arbitrary coordination histories. + +No confirmed contract defect remains within the reviewed boundaries. This is a +bounded verdict, not a proof of every legal provider or scheduler history. + +## Additional numbered requirements + +| Requirement | Snapshot | Ownership | Bridge, including cold source and rejected acceptance | Real backend receiving owner | +| --- | --- | --- | --- | --- | +| ORC-012, exact-head review evidence | This versioned record names the reviewed head and every applicable outcome above. Missing projection was challenged by a hostile control, not only by a new happy-path example. | The record separates concrete admission from SDK receiving evidence and bounds the claim to the generated identity forms and checkpoints. | The bug-class boundary, original early success, adjacent rejected-durable-base counterexample, wrong-design rejection and remaining provider limits are explicit. No unresolved reachable in-scope trace is known. | The record names the real host, legal SIGKILL cuts, independent effects, exact calibration observations, replay bindings and excluded coordination/power-loss boundaries. | +| ORC-013, distinguishing reusable boundaries | One valid key is admitted; two equal keys reject the complete snapshot and preserve the prior valid array. An absent optional field removes it after a prior present field, rejecting partial-row retention. Nonidentity projection rejects an identity-selector implementation. | Same concrete owner rejects before and after cleanup; fresh options and separate DbClients admit at those same checkpoints. The cleaned-up reuse case rejects the wrong rule that cleanup retires the options owner. | Pending first source cannot fulfill, while valid empty source does. Argument/mutation errors are terminal while post-confirmation sync errors remain retryable. Provider error can recover on a later valid result; rejected durable acceptance stays failed across unchanged/changed retries until a fresh run. Removing the terminal guards is rejected. | Two server attempts with one durable key produce one logical application. One receipt alone is not sufficient: the duplicate-effect mutant still leaves one receipt yet fails the counter. A same-count wrong-key receipt fails exact key comparison, and a server-correct/public-stale payload fails the child's public-row comparison. | +| ORC-014, controlled-premise handoff | Controlled arrays and successful projection exercise the adapter semantic boundary. SDK tests receive actual Convex decoding and listener delivery; backend select projection receives real documents. They are narrower receiving examples, not copies of this full generated source grammar. The source-array owner does not claim provider scheduling coverage. | The controlled fixture supplies cached data before the scheduled callback. Both actual SDK clients receive that cache window in sdk.test.ts. New sibling tests clean up either listener, then supply the next protocol transition and check the survivor, supplying the exact multi-listener premise the controlled fixture cannot. | Stub stage traces claim collaborator ordering/classification only. Cold and rejected-acceptance drivers use real Collections with controlled callback and receipt boundaries. The receipt fixture delays actual core commit behind durability, matching persisted.visibleAfterDurable. These drivers do not claim native SQLite failure or backend cold-replay coverage. SDK tests separately receive query/mutation timestamp ordering. | This owner receives actual local Convex query and mutation behavior and actual process death/storage reads. Its online and single-leader collaborators deliberately limit the coordination premise; native multi-tab election and OS online detection are not claimed. It does not reuse the bridge's stage-prefix model because durable restart, counter effects and server receipts are a different law and observation boundary. | + +The SDK receiving examples use fixed protocol histories to observe the SDK's +cache, mutation timestamp and listener ownership premises. They do not share the +snapshot generator because their purpose is to receive those timing/ownership +conditions rather than repeat the complete snapshot law. Their fixed expected +selected text is derived from the explicitly supplied server text and projection. +The drift risk is that these examples do not receive every generated snapshot +shape; the generated owner remains the authority for those shapes at its stated +adapter boundary. Real-backend pagination and durable rows add provider-boundary +coverage without claiming the SDK fixture proves cloud availability. + +The manually deferred receipt is limited to the documented acceptance interface. +A Convex Collection wrapped by a real SQLite persistence adapter failing its disk +write has not been exercised by this integration. That native combination belongs +to the persistence receiving boundary, not a claim made by this controlled driver. +The coverage map retains the persistence and offline-engine owners for their +storage, leadership and settlement laws. + diff --git a/docs/contributing/oracle-reviews/2026-10-24-convex-auth-dependency-update.md b/docs/contributing/oracle-reviews/2026-10-24-convex-auth-dependency-update.md new file mode 100644 index 0000000000..11f080a818 --- /dev/null +++ b/docs/contributing/oracle-reviews/2026-10-24-convex-auth-dependency-update.md @@ -0,0 +1,75 @@ +# Auth fixture dependency update + +Reviewed dependency commit: `bae01f788134aa421a9018278ec6d2867519dae3`. +This is an addendum to the [public application receiving review](2026-10-24-convex-public-apps.md). +That earlier record remains an accurate account of its historical source and +runtime versions. This update reviews the new dependency resolution and receiving +evidence, it does not repeat the full oracle audit or claim new oracle behavior. + +The private auth fixture now uses `@convex-dev/auth` 0.0.96 and `@auth/core` +0.41.3, replacing 0.0.90 and 0.37.4. Only its package manifest and corresponding +lockfile entries changed in the reviewed commit. The pinned upstream business +functions, provider substitution, models, assertions, and setup code are unchanged. +No dependency override, security exemption, or shipped production code change was +introduced. + +## Independent checks + +The reviewer inspected the complete dependency diff and queried published npm +metadata for `@convex-dev/auth` 0.0.96. Its declared peers are: + +| Peer | Declared range | Resolved fixture version | +| --- | --- | --- | +| `@auth/core` | `^0.41.1` | `0.41.3` | +| `convex` | `^1.17.0` | `1.47.0` | +| `react` | `^18.2.0 \|\| ^19.0.0-0` | `19.2.4` | + +All three resolutions satisfy the published peer ranges. The old Auth.js and +Convex Auth resolutions are absent from the lockfile. Auth.js now resolves its +own `jose` 6.2.12 dependency, while Convex Auth and the fixture's signing-key +setup retain their separate `jose` 5.10.0 resolution. These are ordinary package +resolutions, not a forced shared version. CI runs Node 22.13. + +The reviewer also queried both official GitHub advisory APIs: + +- [GHSA-7rqj-j65f-68wh](https://github.com/advisories/GHSA-7rqj-j65f-68wh) + identifies the Auth.js email-normalization issue and lists 0.41.3 as the first + patched `@auth/core` version. +- [GHSA-xmf8-cvqr-rfgj](https://github.com/advisories/GHSA-xmf8-cvqr-rfgj) + identifies malformed Bearer-header handling and likewise lists 0.41.3 as the + first patched version. + +The selected Auth.js version is the published patched version for both reported +advisories, and its companion Convex Auth version explicitly supports that peer +range. This establishes the dependency remediation for those alerts, not an +independent reproduction of either security exploit or an absence of other +vulnerabilities. + +## Receiving evidence and scope + +The root agent reports the complete 17-test backend suite passing after the +upgrade, including the final full auth permission matrix and equal-body message +multiplicity checks, with 34.02 seconds of test execution. Package typecheck also +passed. The reviewer inspected the dependency resolution independently, but did +not rerun this suite or the prior mutants. + +This fresh run receives the existing application laws with the updated auth +runtime: genuine Anonymous sign-in and JWT use, unauthenticated rejection, +invalid-token rejection, read/write/admin permission transitions, demotion, and +public Collection rows after source acceptance. The six public pagination +histories and existing nine backend tests also passed in that combined run. +The earlier record's mutant calibrations remain attributed to their original +runs, they are not presented as new controls executed against these versions. + +The upgrade does not change the models or their legal histories, so their +independence, bounded observations, and cleanup judgments remain as described in +the earlier record. The changed receiving premise is the installed auth runtime; +the new complete backend run supplies compatibility evidence for that premise. +This does not test email-provider normalization or every malformed Bearer-header +case, the patched-version claim comes from the official advisories. + +The fixture still adapts a pinned public app rather than installing its original +lockfile. The local Rust backend remains selected by the installed Convex CLI, +its binary version was not recorded. Cloud infrastructure, third-party OAuth, +email delivery, browser components, token refresh/revocation, authenticated +reconnect, and arbitrary network schedules remain outside these receiving tests. diff --git a/docs/contributing/oracle-reviews/2026-10-24-convex-backend-startup.md b/docs/contributing/oracle-reviews/2026-10-24-convex-backend-startup.md new file mode 100644 index 0000000000..c5c702e7ca --- /dev/null +++ b/docs/contributing/oracle-reviews/2026-10-24-convex-backend-startup.md @@ -0,0 +1,87 @@ +# Local backend startup review + +Reviewed source and test commit: `4b37df090402469cafed0cf9f5e16b9528dae60f`. +This follows the [public-app receiving review](2026-10-24-convex-public-apps.md) +and [auth dependency update](2026-10-24-convex-auth-dependency-update.md). +It concerns test setup and workspace discovery, not an adapter production defect. +There is no shipped adapter production code change. + +The root agent reported that the browser CI job exited before test collection +with the official Convex CLI error `Requested port 43330 is not available`. +That establishes an unavailable requested port. It does not establish which +process occupied it or why. The fixture had imposed that machine-wide port +requirement itself. + +## Contract and repair + +Without a user port override, the fixture should start an isolated local backend +on available ports rather than require its former fixed numbers. The reviewer +inspected all three setup call sites and the installed Convex 1.47 CLI source. +Its anonymous deployment path calls `chooseLocalBackendPorts`, which delegates +to `detect-port`: explicit requested ports are mandatory, while defaults and +suggestions can select other available ports. The default search starts at +3210. The test does not calculate or promise the allocator's exact result. + +All three setups now omit mandatory port arguments by default. The existing +`CONVEX_TEST_PORT` override still supplies an explicit port to the main fixture. +The shared launcher retains strict loopback URL validation and verifies the +exact requested port when one was supplied. No custom free-port allocator, +retry loop, or silent fallback for an explicit override was added. + +A separate workspace discovery issue was also repaired. Auth setup creates an +ignored `auth-app/runtime/package.json`, which the broad `packages/**` workspace +glob could discover during concurrent pnpm checks. An exact exclusion now keeps +that generated runtime out of the workspace project set. The real auth fixture +remains a workspace package. This changes configuration, not dependency versions +or lockfile resolutions. + +## Witness and execution evidence + +`e2e/backend-startup-oracle.test.ts` supplies an independent resource judgment: +while the old public fixture port 43320 is occupied, setup must fulfill and +provide another loopback port serving a fresh empty upstream messages query. +It invokes the actual pinned public-app setup and queries the real backend. +Expected startup success and empty data do not come from copying CLI selection +logic. An explicit startup-fulfillment assertion precedes the URL and query +observations. + +The reviewer inspected that test, the setup changes, cleanup paths, workspace +exclusion, and installed CLI implementation. The following execution evidence +comes from the author and root agent, it is not presented as an independent +rerun by this reviewer: + +- Restoring the former fixed 43320 setup argument failed the startup-fulfillment + assertion in 292 ms with `Requested port 43320 is not available`. This is the + intended startup outcome comparison. The later query checkpoint was not + reached, so this is not a failure at a data comparison. +- Restoring default allocation passed in 1.05 seconds and reached the actual + upstream empty-page query, including `isDone: true`. +- The first combined 18-test run passed with normal allocation. A second run + held port 43330 from before pnpm and global setup began, matching the occupied + auth-port condition from CI, and all 18 tests passed in 33.69 seconds. The root + agent verified and terminated its own blocker afterward. Only this second + run establishes the specifically occupied auth-port condition. +- Final package typecheck and ESLint passed. With a synthetic generated runtime + package present, `pnpm list -r` succeeded and excluded it. The synthetic runtime + was removed afterward. + +The fixed startup witness runs in the normal backend suite. It releases its own +listener and backend independently through the existing failure-preserving +cleanup instrument. `EADDRINUSE` is accepted as an observation of an existing +external owner at bind time, that owner's lifetime is not leased by the test, +and the test never closes it. + +## Scope + +This closes the fixture's mandatory-default-port assumption for the reviewed +serial setup paths. The public fixture regression observes the shared startup +boundary directly; the complete suite also receives the auth setup under its +formerly required occupied port. The source inspection confirms that all three +call sites delegate their default choice to the same supported CLI path. + +This is a fixed native startup witness, not a generated port-allocation model +or a proof against every concurrent bind race. It does not promise startup when +no ports are available, override an occupied port explicitly requested by a user, +or identify the owner of the original CI port. Network access and backend +availability remain setup prerequisites. Existing public-app, auth, pagination, +and offline contracts and their recorded limits are unchanged. diff --git a/docs/contributing/oracle-reviews/2026-10-24-convex-public-apps.md b/docs/contributing/oracle-reviews/2026-10-24-convex-public-apps.md new file mode 100644 index 0000000000..615a36aaad --- /dev/null +++ b/docs/contributing/oracle-reviews/2026-10-24-convex-public-apps.md @@ -0,0 +1,125 @@ +# Public Convex application receiving review + +Reviewed source and test commit: `4223c39a6d5c2b2e2fedb1a7eee3c0052b5e5013`. +This independent review covers the new public pagination and authenticated app +receiving tests, their pinned-source setup, `local-backend.ts`, and +`network-gate.ts`. It found no adapter defect. This commit changes test and +integration setup, dependencies, and documentation, with no shipped production +code change. + +The reviewer inspected the committed test and setup files and fetched the pinned +upstream message functions, schema, auth functions, and permission implementation +through GitHub's API. The execution results below are attributed to the authors +and root agent. The reviewer did not rerun the backend suites or the production +mutants for this review. + +## Sources and observed evidence + +The pagination source is [get-convex/convex-demos at +2fac21e360bfcceae6065096f1b81480bd8c6fd5](https://github.com/get-convex/convex-demos/tree/2fac21e360bfcceae6065096f1b81480bd8c6fd5/pagination). +The actual functions are deployed unchanged from `git archive` of that commit. +The optional local checkout must have that HEAD, and archive reads the pinned +object rather than dirty worktree files. The local app shell and SDK dependency +are supplied by the test. Upstream declares Convex 1.48, while this fixture +intentionally deploys with the package's installed Convex 1.47 SDK/CLI and the +CLI-selected local Rust backend release. The binary version was not recorded, +so this record does not assign it the SDK's version. + +The auth source is [get-convex/convex-auth-with-role-based-permissions at +45f08008e1257758560948db4280cc4e7d64e84a](https://github.com/get-convex/convex-auth-with-role-based-permissions/tree/45f08008e1257758560948db4280cc4e7d64e84a). +Setup verifies the fetched HEAD and copies the message functions, schema, auth +routes, and permission library into an ignored runtime. The one deliberate +application adaptation replaces GitHub/Resend providers with Convex Auth's +Anonymous provider. The replacement checks the expected pinned source text. +Server permission checks and role updates retain upstream behavior. The fixture +uses `@convex-dev/auth` 0.0.90 and `@auth/core` 0.37.4, rather than the upstream +app's older declared 0.0.71 and ^0.34.1. It does not install the upstream lockfile. +Fresh signing keys allow a genuine sign-in and JWT without third-party +credentials. + +Reported execution evidence: + +- All 17 combined backend tests passed before the final auth history extension. + That run included the existing nine backend tests and all eight new public-app + histories, six pagination and two authentication. +- After the last extension, both auth client histories passed with every + role/operation pair, two successful equal-body sends, and two deletions. This + is targeted evidence for the final auth changes, not a claim that the reviewer + reran the full combined suite at the reviewed commit. +- The root agent reported 62 adapter checks, package typecheck, and lint passing + for the final source. The author reported all six public pagination histories + passing after the continuation control was restored, using default fresh + GitHub fetch/archive setup rather than the local-source override. +- A stale runtime sentinel was removed by actual auth preparation. This checks + the fresh-runtime prerequisite rather than assuming a previous teardown ran. + +Calibration evidence is distinguished from successful execution: + +| Control | Reported result and checkpoint | +| --- | --- | +| Disable automatic continuation | With the proper selector `mirrors list through`, both actual SDK histories failed the immediate ordered `select` comparison after preload. JavaScript exposed three rows instead of the ledger's 17, React three instead of 34, because successful writes from the earlier selected history remained in the shared backend. These were assertion failures at the named checkpoint, not timeouts or setup errors. | +| Initial selector containing a literal `>` | It matched no tests. This was an unreached path and is not counted as a mutant kill. | +| Bypass upstream role checks in a temporary prototype | The author changed the pinned permission function's final numeric comparison to `return true`, preserving user/role existence checks, and let the real backend redeploy. The first read/send denial assertion received a resolved `null` instead of rejection, one failed/one skipped test, with no timeout or setup failure. The original source was restored and both SDK histories passed through their own pinned-source setup. This control used the earlier five-step history; the final expanded history retains that first read/send cell. It was not rerun as a mutant at the frozen commit. | + +## Requirement outcomes + +| Requirement | Outcome and evidence | +| --- | --- | +| ORC-001, authority and limits | Satisfied for these receiving claims. Upstream pagination functions establish newest-first order, indexed author membership, and uppercase body/author-initial transformation. The upstream permission function and message endpoints establish read/write/admin access. The adapter's existing source snapshot and complete eager pagination contracts establish the Collection observations. The tests do not authorize a new provider or adapter behavior. Pins and adaptations are explicit. | +| ORC-002, independent judgment | Satisfied. Pagination's expected rows come from an authored ledger of successful sends, reversed and optionally filtered/transformed according to the upstream query contract. Expected rows do not come from the adapter, a paged query, or SDK cache. Auth uses a set-valued role policy rather than copying upstream's numeric role comparison, and a body multiset rather than server-generated IDs. Actual IDs are used only to target deletion, not to compute the expected message count. | +| ORC-003, literate responsibilities | Satisfied. Both files explain the upstream authority, reference computation, authored histories, real SDK/Collection path, public observations, and limitations before mechanics. Comments identify the shared-backend ledger prerequisite, stable keys for transformed rows, role-update adaptation, denied operations, and reconnect cut. Setup and README describe which upstream code is deployed and which authentication provider is substituted. | +| ORC-004, generated grammar controls | Not applicable, these are fixed authored receiving histories rather than generated properties. Their finite scope is still explicit: both SDKs cross all three pagination query forms at page size three; the auth history crosses all six role/operation cells, demotion, two equal-body successful sends, and deletion of one occurrence at a time. Unique pagination bodies remain unique after uppercase conversion, excluding the transformed-key ambiguity that would make those histories invalid. Arbitrary operation sequences, page sizes, role values, token lifetimes, and malformed protocols are not generated or claimed. | +| ORC-005, production path and observation | Satisfied. Both drivers use actual SDK clients, actual deployed upstream functions, and real Collections. Pagination compares ordered input to `select` immediately after initial preload, then compares keyed public rows after source acceptance. Later checkpoints poll for the expected source revision before checking the public Collection. Auth waits for each real mutation outcome, then source acceptance, and compares marker multiplicity after both permitted and denied operations. Unauthenticated Collection preload and invalid-token HTTP query are distinct controls. No concrete missing-input hypothesis remains for the stated cuts. | +| ORC-006, calibration | Satisfied with the attributed controls above. Disabled continuation fails immediate source-order comparison on both clients. Bypassed server permission checks fail denial observations. The skipped initial selector is excluded from calibration. These controls target two specific wrong designs, they are not an exhaustive mutation campaign. | +| ORC-007, generated campaigns and replay | Not applicable, there is no generated property or `fc.commands` history in these new receiving files. Each named query/client history is directly selectable. Pinned source commits and fresh local deployments support reconstruction, but they are not seed/path replay or random campaigns. Existing primary pagination and offline owners retain those generated campaigns. | +| ORC-008, state minimality | Satisfied for the reference histories. Pagination retains successful messages in insertion order because later queries need their values, author membership, and reverse order, including writes from earlier histories against the shared backend. Counts alone cannot predict those observations. Auth's multiset must retain multiplicity because two identical sends create two rows and deleting one leaves one. Collapsing them to a set predicts the wrong intermediate public snapshot. Server IDs and timestamps are deliberately omitted from the model because the compared projection and legal deletion targets do not require predicting them. | +| ORC-009, vocabulary and abstraction | Satisfied. Collection rows, source acceptance, public snapshot, and cleanup retain the glossary meanings. The pagination ledger models successful application sends, not provider subscriptions or core sync transactions. Auth bodies abstract away server IDs while preserving the multiplicity that the observations need. TCP gate state is transport availability, not Collection readiness. The invalid-token HTTP control does not establish invalid-token WebSocket behavior. | +| ORC-010, failure fidelity and cleanup | Satisfied for the exercised harness. Registered Collection, SDK, TCP gate, backend, checkout, and runtime cleanup uses the existing calibrated withTestCleanup instrument. It preserves a primary comparison or setup error while reporting secondary release failures and attempts subsequent releases. The local backend startup catch now preserves the startup cause if stopping fails. Auth records its runtime path before preparation, clears stale contents before copying, and can clean a partial preparation. The runtime is scoped and ignored. This review inspected those paths; it does not claim fault injection at every construction instruction or process-kill cleanup. | +| ORC-011, independent second formulation | Satisfied for the concrete shared-fault concern motivating these witnesses. Adapter-authored backend functions could reproduce the same assumptions as the adapter and primary numeric range fixture. These tests instead deploy independently maintained public application functions at verified pins and compare them with an authored operation ledger or permission table. Auth's permission table differs structurally from upstream numeric hierarchy. No additional concrete shared semantic fault was identified that requires another model. This is not an exhaustive absence claim. | +| ORC-012, review evidence | Satisfied by this versioned record tied to the exact source/test commit. It distinguishes independent inspection from author/root execution, records the unreached control, reports the harness/provenance repairs, and retains scope limits. No adapter bug-class closure is claimed by these test-only changes. They receive existing laws at additional application and transport boundaries. | +| ORC-013, distinguishing boundary witnesses | Satisfied within the fixed cases. Authentication distinguishes no identity, an invalid token, and a genuine signed token accepted by server functions. The role history distinguishes read from write for send, write from admin for delete, and demotion after a successful write; every role/operation cell is exercised. Two equal-body sends followed by single deletions distinguish multiplicity from set membership. Pagination crosses one page into several with 17 initial appends at page size three, and includes another author to distinguish indexed membership from an unfiltered mirror. These fixed witnesses do not establish arbitrary thresholds or workloads. | +| ORC-014, controlled-premise handoff | Satisfied for the named receiving premises. Both actual SDK clients and a real local backend now supply cursor pagination, query transformation, successful/denied authenticated mutations, and subscribed source updates. A real auth action issues the tested JWT, neither setAdminAuth nor a fabricated identity supplies the tested user. The TCP gate forwards uninterpreted bytes, destroys live connections, and refuses new ones while closed. The test waits for the adapter client's public connection state to become disconnected before an independent direct writer commits. Old rows remain public; later ledger rows must arrive after reopening the gate. This receives real SDK reconnection, without protocol-message simulation. These app-specific ledgers and histories do not literally reuse the primary numeric fixture grammar, they require different expected application fields and auth policy. Their relationship is source snapshot refinement and complete finite acquisition; shared-law drift remains a reason to keep the primary owners and pinned-source review. | + +## Repairs and remaining limits + +Review identified setup and evidence gaps, with no adapter defect: + +- Auth runtime contents needed an ignore rule and a fresh-copy prerequisite. + Both are present at the reviewed commit, and the known runtime path is + registered before copying so partial setup can be removed. +- Local backend startup could lose its original error if stopping also failed. + It now uses the existing failure-preserving cleanup instrument. +- The first auth model was a set and the history admitted only one successful + send. The final history covers the full role/operation table and uses a + multiset, so repeated equal-body sends and one-at-a-time deletion are explicit. + +The upstream auth demo deliberately lets an authenticated user choose a role. +This is its documented demo behavior, not evidence of secure role administration. +The test does not establish tenant isolation, sign-out or token revocation, +refresh-token rotation, browser auth components, OAuth redirects, email delivery, +or production identity infrastructure. Token corruption is tested through HTTP; +unauthenticated preload and genuine token use are tested through both WebSocket +SDKs. Authenticated reconnect is not exercised. The reconnect witness uses the +separate unauthenticated public pagination app. + +The pagination observations project `author` and `body`; they deliberately omit +upstream `_id` and `_creationTime`, which the transformed query does not return. +Auth compares marked body multiplicity, not complete raw message records or +user/author fields. These are business-ledger observations, not full raw-row +fidelity. The existing backend owner checks IDs and application fields separately. + +Pagination observes complete finite results and live appends, not arbitrary +updates/deletes, continuous growth, transport work bounds, bounded lifetime +subscriptions, or browser rendering. Transformed rows have stable keys only +because the authored bodies are unique, including after case conversion. The +ledger assumes a fresh deployment and sequential execution, both supplied by +setup and test configuration. Auth compares test-marked messages within the +upstream query's latest-100 window, not every application row in an arbitrary +large database. + +The public applications are deployed locally with the package's installed SDK +and backend, with an independently supplied app shell. They are not unchanged +full frontend deployments or evidence for their originally declared dependency +matrix. GitHub availability, local backend download, and operating-system TCP +behavior remain test prerequisites. No production cloud service, third-party +OAuth system, power-loss atomicity, or arbitrary network scheduling is proved. diff --git a/docs/guides/offline-transactions.md b/docs/guides/offline-transactions.md index 5ea40eae44..945d8c56f4 100644 --- a/docs/guides/offline-transactions.md +++ b/docs/guides/offline-transactions.md @@ -353,3 +353,11 @@ The final `refetch()` asks the server for current rows after a successful write. The [React Native offline transactions example](https://github.com/TanStack/db/tree/main/examples/react-native/offline-transactions) shows the full app structure. Its API client does not yet send an idempotency key, so use the request pattern above when you adapt it. The [shopping list example](https://github.com/TanStack/db/tree/main/examples/react-native/shopping-list) also shows Electric sync. The [Electron example](https://github.com/TanStack/db/tree/main/examples/electron/offline-first) uses a custom SQLite-backed `StorageAdapter` for the outbox. That adapter is separate from the persisted Collection tables. + +## Mutation value codecs + +The default outbox format preserves its existing JSON, Date, and Temporal behavior. For other mutation value domains, set `mutationCodec` to a codec with a stable `id`, synchronous `encode(value): string`, and synchronous `decode(encoded): unknown`. + +Encoded transactions use wire version 4 and retain the codec ID. Configure the same codec on restart. Missing or mismatched codecs, and decoder failures, reject the outbox read with the transaction ID while keeping its storage entry. Readers without version 4 support cannot replay these records. Default writers still emit version 3. A configured codec can read older records through their original decoder. + +The codec applies to `modified`, `original`, and `changes`. Transaction metadata keeps the default encoding. Codec functions must not perform network calls. TanStack's Convex adapter supplies `convexMutationCodec` for Convex values and removed fields. diff --git a/docs/reference/convex-db-collection/functions/convexCollectionOptions.md b/docs/reference/convex-db-collection/functions/convexCollectionOptions.md new file mode 100644 index 0000000000..474b327313 --- /dev/null +++ b/docs/reference/convex-db-collection/functions/convexCollectionOptions.md @@ -0,0 +1,40 @@ +--- +id: convexCollectionOptions +title: convexCollectionOptions +--- + +```ts +function convexCollectionOptions(config): CollectionConfig & object & [TSchema] extends [never] ? unknown : object; +``` + +Defined in: [index.ts:48](https://github.com/tanstack/db/blob/main/packages/convex-db-collection/src/index.ts#L48) + +Mirror one authoritative Convex query result. Filters and joins run in DB. + +## Type Parameters + +### TQuery + +`TQuery` *extends* `FunctionReference`\<`"query"`\> + +### TItem + +`TItem` *extends* `object` = `QueryRow`\<`TQuery`\> + +### TKey + +`TKey` *extends* `string` \| `number` = `string` \| `number` + +### TSchema + +`TSchema` *extends* `StandardSchemaV1`\<`TItem`, `TItem`\> = `never` + +## Parameters + +### config + +[`ConvexCollectionConfig`](../type-aliases/ConvexCollectionConfig.md)\<`TQuery`, `TItem`, `TKey`, `TSchema`\> + +## Returns + +`CollectionConfig`\<`TItem`, `TKey`, `TSchema`, [`ConvexCollectionUtils`](../interfaces/ConvexCollectionUtils.md)\> & `object` & \[`TSchema`\] *extends* \[`never`\] ? `unknown` : `object` diff --git a/docs/reference/convex-db-collection/functions/convexPaginatedCollectionOptions.md b/docs/reference/convex-db-collection/functions/convexPaginatedCollectionOptions.md new file mode 100644 index 0000000000..1cbe74a6d8 --- /dev/null +++ b/docs/reference/convex-db-collection/functions/convexPaginatedCollectionOptions.md @@ -0,0 +1,40 @@ +--- +id: convexPaginatedCollectionOptions +title: convexPaginatedCollectionOptions +--- + +```ts +function convexPaginatedCollectionOptions(config): CollectionConfig & object & [TSchema] extends [never] ? unknown : object; +``` + +Defined in: [paginated-options.ts:53](https://github.com/tanstack/db/blob/main/packages/convex-db-collection/src/paginated-options.ts#L53) + +Automatically subscribe to every page, retaining fixed reactive cursor ranges. + +## Type Parameters + +### TQuery + +`TQuery` *extends* `FunctionReference`\<`"query"`\> + +### TItem + +`TItem` *extends* `object` = `PaginatedRow`\<`TQuery`\> + +### TKey + +`TKey` *extends* `string` \| `number` = `string` \| `number` + +### TSchema + +`TSchema` *extends* `StandardSchemaV1`\<`TItem`, `TItem`\> = `never` + +## Parameters + +### config + +[`ConvexPaginatedCollectionConfig`](../type-aliases/ConvexPaginatedCollectionConfig.md)\<`TQuery`, `TItem`, `TKey`, `TSchema`\> + +## Returns + +`CollectionConfig`\<`TItem`, `TKey`, `TSchema`, [`ConvexCollectionUtils`](../interfaces/ConvexCollectionUtils.md)\> & `object` & \[`TSchema`\] *extends* \[`never`\] ? `unknown` : `object` diff --git a/docs/reference/convex-db-collection/index.md b/docs/reference/convex-db-collection/index.md new file mode 100644 index 0000000000..8650166963 --- /dev/null +++ b/docs/reference/convex-db-collection/index.md @@ -0,0 +1,19 @@ +--- +id: "@tanstack/convex-db-collection" +title: "@tanstack/convex-db-collection" +--- + +## Interfaces + +- [ConvexCollectionUtils](interfaces/ConvexCollectionUtils.md) + +## Type Aliases + +- [ConvexCollectionClient](type-aliases/ConvexCollectionClient.md) +- [ConvexCollectionConfig](type-aliases/ConvexCollectionConfig.md) +- [ConvexPaginatedCollectionConfig](type-aliases/ConvexPaginatedCollectionConfig.md) + +## Functions + +- [convexCollectionOptions](functions/convexCollectionOptions.md) +- [convexPaginatedCollectionOptions](functions/convexPaginatedCollectionOptions.md) diff --git a/docs/reference/convex-db-collection/interfaces/ConvexCollectionUtils.md b/docs/reference/convex-db-collection/interfaces/ConvexCollectionUtils.md new file mode 100644 index 0000000000..c2a658d997 --- /dev/null +++ b/docs/reference/convex-db-collection/interfaces/ConvexCollectionUtils.md @@ -0,0 +1,48 @@ +--- +id: ConvexCollectionUtils +title: ConvexCollectionUtils +--- + +Defined in: [options.ts:15](https://github.com/tanstack/db/blob/main/packages/convex-db-collection/src/options.ts#L15) + +## Extends + +- `UtilsRecord` + +## Indexable + +```ts +[key: string]: any +``` + +## Properties + +### awaitSync() + +```ts +awaitSync: () => Promise; +``` + +Defined in: [options.ts:19](https://github.com/tanstack/db/blob/main/packages/convex-db-collection/src/options.ts#L19) + +Refresh the source cache and wait for sync acceptance, safe inside mutation handlers. + +#### Returns + +`Promise`\<`void`\> + +*** + +### getSyncError() + +```ts +getSyncError: () => unknown; +``` + +Defined in: [options.ts:17](https://github.com/tanstack/db/blob/main/packages/convex-db-collection/src/options.ts#L17) + +Last subscription or conversion error, cleared by the next valid result. + +#### Returns + +`unknown` diff --git a/docs/reference/convex-db-collection/offline/functions/convexOfflineMutation.md b/docs/reference/convex-db-collection/offline/functions/convexOfflineMutation.md new file mode 100644 index 0000000000..c6aca1a944 --- /dev/null +++ b/docs/reference/convex-db-collection/offline/functions/convexOfflineMutation.md @@ -0,0 +1,30 @@ +--- +id: convexOfflineMutation +title: convexOfflineMutation +--- + +```ts +function convexOfflineMutation(config): OfflineMutationFn; +``` + +Defined in: [offline.ts:21](https://github.com/tanstack/db/blob/main/packages/convex-db-collection/src/offline.ts#L21) + +Use as a named mutationFn in startOfflineExecutor. The existing outbox owns +durability, restoration, leadership and retries. The server mutation must +atomically deduplicate idempotencyKey with its writes. + +## Type Parameters + +### TMutation + +`TMutation` *extends* `FunctionReference`\<`"mutation"`\> + +## Parameters + +### config + +`object` & `FunctionArgs`\<`TMutation`\> *extends* `object` ? `unknown` : `never` + +## Returns + +`OfflineMutationFn` diff --git a/docs/reference/convex-db-collection/offline/index.md b/docs/reference/convex-db-collection/offline/index.md new file mode 100644 index 0000000000..645dbb7fea --- /dev/null +++ b/docs/reference/convex-db-collection/offline/index.md @@ -0,0 +1,12 @@ +--- +id: "@tanstack/convex-db-collection" +title: "@tanstack/convex-db-collection" +--- + +## Variables + +- [convexMutationCodec](variables/convexMutationCodec.md) + +## Functions + +- [convexOfflineMutation](functions/convexOfflineMutation.md) diff --git a/docs/reference/convex-db-collection/offline/variables/convexMutationCodec.md b/docs/reference/convex-db-collection/offline/variables/convexMutationCodec.md new file mode 100644 index 0000000000..923ca3fdd3 --- /dev/null +++ b/docs/reference/convex-db-collection/offline/variables/convexMutationCodec.md @@ -0,0 +1,12 @@ +--- +id: convexMutationCodec +title: convexMutationCodec +--- + +```ts +const convexMutationCodec: MutationValueCodec; +``` + +Defined in: [offline.ts:59](https://github.com/tanstack/db/blob/main/packages/convex-db-collection/src/offline.ts#L59) + +Lossless, opt-in outbox encoding for Convex values and removed fields. diff --git a/docs/reference/convex-db-collection/type-aliases/ConvexCollectionClient.md b/docs/reference/convex-db-collection/type-aliases/ConvexCollectionClient.md new file mode 100644 index 0000000000..993938f0d0 --- /dev/null +++ b/docs/reference/convex-db-collection/type-aliases/ConvexCollectionClient.md @@ -0,0 +1,12 @@ +--- +id: ConvexCollectionClient +title: ConvexCollectionClient +--- + +```ts +type ConvexCollectionClient = + | Pick +| Pick; +``` + +Defined in: [subscription.ts:9](https://github.com/tanstack/db/blob/main/packages/convex-db-collection/src/subscription.ts#L9) diff --git a/docs/reference/convex-db-collection/type-aliases/ConvexCollectionConfig.md b/docs/reference/convex-db-collection/type-aliases/ConvexCollectionConfig.md new file mode 100644 index 0000000000..e5eb698e7e --- /dev/null +++ b/docs/reference/convex-db-collection/type-aliases/ConvexCollectionConfig.md @@ -0,0 +1,72 @@ +--- +id: ConvexCollectionConfig +title: ConvexCollectionConfig +--- + +```ts +type ConvexCollectionConfig = Omit, "syncMode" | "utils" | "schema"> & object & [TSchema] extends [never] ? object : object & + | { + select: (result) => ReadonlyArray; +} + | FunctionReturnType extends ReadonlyArray ? object : never; +``` + +Defined in: [index.ts:21](https://github.com/tanstack/db/blob/main/packages/convex-db-collection/src/index.ts#L21) + +## Type Declaration + +### client + +```ts +client: ConvexCollectionClient; +``` + +Share the client used for mutations, authentication, and reconnects. + +### onError()? + +```ts +optional onError: (error) => void; +``` + +Reports subscription and result-conversion failures, including after readiness. + +#### Parameters + +##### error + +`unknown` + +#### Returns + +`void` + +### query + +```ts +query: TQuery; +``` + +### queryArgs + +```ts +queryArgs: FunctionArgs; +``` + +## Type Parameters + +### TQuery + +`TQuery` *extends* `FunctionReference`\<`"query"`\> + +### TItem + +`TItem` *extends* `object` = `QueryRow`\<`TQuery`\> + +### TKey + +`TKey` *extends* `string` \| `number` = `string` \| `number` + +### TSchema + +`TSchema` *extends* `StandardSchemaV1`\<`TItem`, `TItem`\> = `never` diff --git a/docs/reference/convex-db-collection/type-aliases/ConvexPaginatedCollectionConfig.md b/docs/reference/convex-db-collection/type-aliases/ConvexPaginatedCollectionConfig.md new file mode 100644 index 0000000000..b9d3695d3a --- /dev/null +++ b/docs/reference/convex-db-collection/type-aliases/ConvexPaginatedCollectionConfig.md @@ -0,0 +1,82 @@ +--- +id: ConvexPaginatedCollectionConfig +title: ConvexPaginatedCollectionConfig +--- + +```ts +type ConvexPaginatedCollectionConfig = Omit, "syncMode" | "utils" | "schema"> & object & [TSchema] extends [never] ? object : object & FunctionReturnType extends PaginationResult ? unknown : never & FunctionArgs extends object ? unknown : never & + | { + select: (rows) => ReadonlyArray; +} + | PaginatedRow extends TItem ? object : never; +``` + +Defined in: [paginated-options.ts:20](https://github.com/tanstack/db/blob/main/packages/convex-db-collection/src/paginated-options.ts#L20) + +## Type Declaration + +### client + +```ts +client: ConvexCollectionClient; +``` + +### maxRows? + +```ts +optional maxRows: number; +``` + +Maximum accepted rows in the complete eager mirror. Defaults to 10,000. + +### onError()? + +```ts +optional onError: (error) => void; +``` + +#### Parameters + +##### error + +`unknown` + +#### Returns + +`void` + +### pageSize? + +```ts +optional pageSize: number; +``` + +### query + +```ts +query: TQuery; +``` + +### queryArgs + +```ts +queryArgs: Omit, "paginationOpts">; +``` + +## Type Parameters + +### TQuery + +`TQuery` *extends* `FunctionReference`\<`"query"`\> + +### TItem + +`TItem` *extends* `object` = `PaginatedRow`\<`TQuery`\> + +### TKey + +`TKey` *extends* `string` \| `number` = `string` \| `number` + +### TSchema + +`TSchema` *extends* `StandardSchemaV1`\<`TItem`, `TItem`\> = `never` diff --git a/eslint.config.js b/eslint.config.js index 4004fd1a5d..8a93d67d2f 100644 --- a/eslint.config.js +++ b/eslint.config.js @@ -6,6 +6,7 @@ export default [ { ignores: [ `**/dist/**`, + `packages/convex-db-collection/e2e/backend/convex/_generated/**`, `**/.output/**`, `**/.nitro/**`, `**/traildepot/**`, diff --git a/packages/convex-db-collection/README.md b/packages/convex-db-collection/README.md new file mode 100644 index 0000000000..b423be00f0 --- /dev/null +++ b/packages/convex-db-collection/README.md @@ -0,0 +1,7 @@ +# @tanstack/convex-db-collection + +Subscribe to Convex query results with TanStack DB. Supports `ConvexClient`, `ConvexReactClient`, generated query types, automatic reactive pagination, local queries, optimistic mutation handlers, and durable offline writes through TanStack’s outbox. + +See the [Convex collection guide](https://tanstack.com/db/latest/docs/collections/convex-collection) for setup, mutation handling, generated IDs, and supported behavior. + +Run `pnpm --filter @tanstack/convex-db-collection test:e2e` to start isolated local Convex backends and test both clients, server page splits, and durable replay after process death. The suite also fetches pinned official pagination and authenticated chat examples to check integration with application code maintained outside this adapter. It needs network access to GitHub and the Convex backend download, but no cloud project or production credentials. Authentication uses a real Anonymous-provider sign-in with temporary signing keys. These checks do not exercise production cloud infrastructure or third-party OAuth services. diff --git a/packages/convex-db-collection/e2e/auth-app-oracle.test.ts b/packages/convex-db-collection/e2e/auth-app-oracle.test.ts new file mode 100644 index 0000000000..d193e14b36 --- /dev/null +++ b/packages/convex-db-collection/e2e/auth-app-oracle.test.ts @@ -0,0 +1,152 @@ +/** + * Receiving witness for the official role-based Convex Auth application pinned + * in auth-app/README.md. Its server functions define read/write/admin access. + * The independent role table predicts which operations may change the messages: + * read permits observation, write permits send, admin also permits deletion. + * A genuine Anonymous signIn issues the JWT used by both SDK WebSocket clients. + * Rejected operations must leave the public Collection unchanged; successful + * sends and deletions must reach its rows after awaitSync accepts the source. + * + * These are bounded authored role histories through the upstream application, + * not a model of OAuth, token refresh, role administration security, or browsers. + * The upstream demo deliberately allows self-service role changes. A separate + * unauthenticated client and an invalid token distinguish server authentication + * from trusting an asserted client identity. No setAdminAuth is used. + */ +import { ConvexClient, ConvexHttpClient } from 'convex/browser' +import { ConvexReactClient } from 'convex/react' +import { makeFunctionReference } from 'convex/server' +import { createCollection } from '@tanstack/db' +import { expect, inject, it } from 'vitest' +import { convexCollectionOptions } from '../src/index' +import { withTestCleanup } from '../tests/helpers' + +declare module 'vitest' { + interface ProvidedContext { + authAppUrl: string + } +} +type Message = { _id: string; body: string; userId: string; author?: string } +const list = makeFunctionReference<`query`, {}, Array>(`messages:list`) +const send = makeFunctionReference<`mutation`, { body: string }, null>( + `messages:send`, +) +const remove = makeFunctionReference<`mutation`, { messageId: string }, null>( + `messages:deleteMessage`, +) +const role = makeFunctionReference< + `mutation`, + { role: `read` | `write` | `admin` }, + null +>(`auth:updateRole`) +const signIn = makeFunctionReference< + `action`, + { provider: string }, + { tokens: { token: string; refreshToken: string } | null } +>(`auth:signIn`) +const url = inject(`authAppUrl`) +type Role = `read` | `write` | `admin` +type Operation = `send` | `delete` +// This set-valued policy is independent of upstream's numeric role hierarchy. +// The message multiset contains authored bodies, never server-assigned IDs. +const permitted: Record> = { + read: [], + write: [`send`], + admin: [`send`, `delete`], +} +const history: ReadonlyArray<{ role: Role; operation: Operation }> = [ + { role: `read`, operation: `send` }, + { role: `write`, operation: `send` }, + { role: `write`, operation: `delete` }, + { role: `read`, operation: `send` }, + { role: `read`, operation: `delete` }, + { role: `admin`, operation: `send` }, + { role: `admin`, operation: `delete` }, + { role: `admin`, operation: `delete` }, +] + +it.each([`javascript`, `react`] as const)( + `receives genuine authenticated role transitions with the %s client`, + async (kind) => { + const http = new ConvexHttpClient(url) + await expect(http.query(list, {})).rejects.toThrow(`Not signed in`) + const result = await http.action(signIn, { provider: `anonymous` }) + expect(result.tokens?.token.split(`.`)).toHaveLength(3) + const token = result.tokens!.token + const invalid = new ConvexHttpClient(url) + invalid.setAuth(`${token.slice(0, -3)}bad`) + await expect(invalid.query(list, {})).rejects.toThrow() + const makeClient = () => + kind === `javascript` + ? new ConvexClient(url, { unsavedChangesWarning: false, logger: false }) + : new ConvexReactClient(url, { + unsavedChangesWarning: false, + logger: false, + }) + const unauthenticated = makeClient() + const denied = createCollection( + convexCollectionOptions({ + client: unauthenticated, + query: list, + queryArgs: {}, + getKey: (row) => row._id, + gcTime: 0, + }), + ) + await withTestCleanup( + async () => { + await expect(denied.preload()).rejects.toThrow(`Not signed in`) + expect([...denied.values()]).toEqual([]) + }, + () => denied.cleanup(), + () => unauthenticated.close(), + ) + const client = makeClient() + client.setAuth(() => Promise.resolve(token)) + const collection = createCollection( + convexCollectionOptions({ + client, + query: list, + queryArgs: {}, + getKey: (row) => row._id, + gcTime: 0, + }), + ) + const marker = `auth-${kind}-${crypto.randomUUID()}` + await withTestCleanup( + async () => { + await collection.preload() + const model: Array = [] + for (const step of history) { + // This is the upstream demo's public role selector, not admin auth. + await client.mutation(role, { role: step.role }) + const allowed = permitted[step.role].includes(step.operation) + const inserted = [...collection.values()].find( + (row) => row.body === marker, + ) + const attempt = + step.operation === `send` + ? client.mutation(send, { body: marker }) + : client.mutation(remove, { messageId: inserted!._id }) + if (allowed) { + await attempt + if (step.operation === `send`) model.push(marker) + else model.splice(model.indexOf(marker), 1) + } else + await expect(attempt).rejects.toThrow(`Insufficient permissions`) + await collection.utils.awaitSync() + // Compare at every accepted source checkpoint, including denials and + // demotion. Two successful equal-body sends remain two model rows. + expect( + [...collection.values()] + .filter((row) => row.body === marker) + .map((row) => row.body) + .sort(), + ).toEqual([...model].sort()) + } + }, + () => collection.cleanup(), + () => client.close(), + ) + }, +) diff --git a/packages/convex-db-collection/e2e/auth-app/.gitignore b/packages/convex-db-collection/e2e/auth-app/.gitignore new file mode 100644 index 0000000000..712f264ddd --- /dev/null +++ b/packages/convex-db-collection/e2e/auth-app/.gitignore @@ -0,0 +1 @@ +runtime/ diff --git a/packages/convex-db-collection/e2e/auth-app/README.md b/packages/convex-db-collection/e2e/auth-app/README.md new file mode 100644 index 0000000000..56b5563dd6 --- /dev/null +++ b/packages/convex-db-collection/e2e/auth-app/README.md @@ -0,0 +1,7 @@ +# Authenticated official app fixture + +Adapted from [get-convex/convex-auth-with-role-based-permissions](https://github.com/get-convex/convex-auth-with-role-based-permissions/tree/45f08008e1257758560948db4280cc4e7d64e84a), pinned at `45f08008e1257758560948db4280cc4e7d64e84a`. + +Test setup loads the schema, message functions, permission checks, role update, and HTTP authentication routes from that pinned public checkout into an ignored runtime directory, no upstream source is vendored here. Its GitHub/Resend providers are replaced with Convex Auth's official Anonymous provider so a real sign-in can issue a JWT without external credentials. No admin authentication shortcut supplies the tested client identity. Setup generates fresh signing keys for an isolated local backend. + +The example's public `updateRole` allows signed-in users to choose their own role, as its source comment explains. This fixture checks that message access follows those roles, it does not claim that role administration is secure. OAuth redirects, email delivery, browser auth-provider components, refresh-token rotation, and production infrastructure are outside this witness. diff --git a/packages/convex-db-collection/e2e/auth-app/configure.ts b/packages/convex-db-collection/e2e/auth-app/configure.ts new file mode 100644 index 0000000000..21b43d397a --- /dev/null +++ b/packages/convex-db-collection/e2e/auth-app/configure.ts @@ -0,0 +1,13 @@ +import { exportJWK, exportPKCS8, generateKeyPair } from 'jose' + +/** Ephemeral keys only, passed to the isolated backend's environment setup. */ +export async function authEnvironment(): Promise> { + const { privateKey, publicKey } = await generateKeyPair(`RS256`) + return { + JWT_PRIVATE_KEY: (await exportPKCS8(privateKey)).trimEnd(), + JWKS: JSON.stringify({ + keys: [{ use: `sig`, ...(await exportJWK(publicKey)) }], + }), + SITE_URL: `http://127.0.0.1:3000`, + } +} diff --git a/packages/convex-db-collection/e2e/auth-app/package.json b/packages/convex-db-collection/e2e/auth-app/package.json new file mode 100644 index 0000000000..4e10d8985f --- /dev/null +++ b/packages/convex-db-collection/e2e/auth-app/package.json @@ -0,0 +1,11 @@ +{ + "name": "convex-auth-app-test-backend", + "private": true, + "type": "module", + "dependencies": { + "@convex-dev/auth": "0.0.96", + "@auth/core": "0.41.3", + "convex": "^1.47.0", + "jose": "5.10.0" + } +} diff --git a/packages/convex-db-collection/e2e/auth-app/prepare.ts b/packages/convex-db-collection/e2e/auth-app/prepare.ts new file mode 100644 index 0000000000..3a3a2a1308 --- /dev/null +++ b/packages/convex-db-collection/e2e/auth-app/prepare.ts @@ -0,0 +1,48 @@ +import { cp, mkdir, readFile, rm, writeFile } from 'node:fs/promises' +import { join } from 'node:path' +import { fileURLToPath } from 'node:url' + +/** Fetching and verifying the pinned public checkout belongs to test setup. */ +export async function prepareAuthApp(checkout: string): Promise { + const runtime = fileURLToPath(new URL(`./runtime`, import.meta.url)) + await rm(runtime, { recursive: true, force: true }) + const destination = join(runtime, `convex`) + await mkdir(destination, { recursive: true }) + for (const source of [ + `messages.ts`, + `schema.ts`, + `auth.config.ts`, + `http.ts`, + `tsconfig.json`, + `lib`, + ]) + await cp(join(checkout, `convex`, source), join(destination, source), { + recursive: true, + }) + const upstream = await readFile(join(checkout, `convex`, `auth.ts`), `utf8`) + if ( + !upstream.includes( + `import GitHub from "@auth/core/providers/github";\nimport Resend from "@auth/core/providers/resend";`, + ) || + !upstream.includes(`providers: [GitHub, Resend],`) + ) + throw new Error(`Pinned upstream auth provider configuration changed`) + const auth = upstream + .replace( + `import GitHub from "@auth/core/providers/github";\nimport Resend from "@auth/core/providers/resend";`, + `import { Anonymous } from "@convex-dev/auth/providers/Anonymous";`, + ) + .replace(`providers: [GitHub, Resend],`, `providers: [Anonymous],`) + if (auth === upstream || !auth.includes(`providers: [Anonymous],`)) + throw new Error(`Pinned upstream auth provider configuration changed`) + await writeFile(join(destination, `auth.ts`), auth) + await writeFile( + join(runtime, `package.json`), + JSON.stringify({ + private: true, + type: `module`, + dependencies: { convex: `^1.47.0` }, + }), + ) + return runtime +} diff --git a/packages/convex-db-collection/e2e/auth-app/setup.ts b/packages/convex-db-collection/e2e/auth-app/setup.ts new file mode 100644 index 0000000000..0f51fa756e --- /dev/null +++ b/packages/convex-db-collection/e2e/auth-app/setup.ts @@ -0,0 +1,76 @@ +import { execFile } from 'node:child_process' +import { mkdtemp, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { fileURLToPath, pathToFileURL } from 'node:url' +import { promisify } from 'node:util' +import { startLocalBackend } from '../local-backend' +import { withTestCleanup } from '../../tests/helpers' +import { prepareAuthApp } from './prepare' +import { authEnvironment } from './configure' +import type { TestProject } from 'vitest/node' + +const run = promisify(execFile) +const upstream = `https://github.com/get-convex/convex-auth-with-role-based-permissions.git` +const revision = `45f08008e1257758560948db4280cc4e7d64e84a` + +/** The provider issues genuine JWTs, no client is given deployment admin auth. */ +export default async function setup(project: TestProject) { + const checkout = await mkdtemp(join(tmpdir(), `convex-auth-upstream-`)) + let stop: (() => Promise) | undefined + const runtime = fileURLToPath(new URL(`./runtime/`, import.meta.url)) + try { + await run(`git`, [`init`, checkout]) + await run(`git`, [`-C`, checkout, `remote`, `add`, `origin`, upstream]) + await run(`git`, [`-C`, checkout, `fetch`, `--depth=1`, `origin`, revision]) + await run(`git`, [`-C`, checkout, `checkout`, `--detach`, `FETCH_HEAD`]) + const { stdout } = await run(`git`, [`-C`, checkout, `rev-parse`, `HEAD`]) + if (stdout.trim() !== revision) + throw new Error(`Wrong upstream auth revision`) + await prepareAuthApp(checkout) + const backend = await startLocalBackend(pathToFileURL(`${runtime}/`)) + stop = backend.stop + if (!backend.url.startsWith(`http://127.0.0.1:`)) + throw new Error(`Authentication setup requires an isolated local backend`) + const { + CONVEX_DEPLOY_KEY: _key, + CONVEX_SELF_HOSTED_URL: _url, + CONVEX_SELF_HOSTED_ADMIN_KEY: _admin, + CONVEX_DEPLOYMENT: _deployment, + ...environment + } = process.env + const cli = fileURLToPath( + new URL(`./node_modules/convex/bin/main.js`, import.meta.url), + ) + for (const [name, value] of Object.entries(await authEnvironment())) { + try { + await run(process.execPath, [cli, `env`, `set`, name, `--`, value], { + cwd: runtime, + env: environment, + }) + } catch { + // execFile errors contain argv, which contains the ephemeral private key. + throw new Error( + `Failed to configure local authentication variable ${name}`, + ) + } + } + project.provide(`authAppUrl`, backend.url) + } catch (cause) { + await withTestCleanup( + () => { + throw cause + }, + () => stop?.(), + () => rm(checkout, { recursive: true, force: true }), + () => rm(runtime, { recursive: true, force: true }), + ) + } + return async () => + withTestCleanup( + () => {}, + () => stop?.(), + () => rm(checkout, { recursive: true, force: true }), + () => rm(runtime, { recursive: true, force: true }), + ) +} diff --git a/packages/convex-db-collection/e2e/backend-oracle.test.ts b/packages/convex-db-collection/e2e/backend-oracle.test.ts new file mode 100644 index 0000000000..df9efada1e --- /dev/null +++ b/packages/convex-db-collection/e2e/backend-oracle.test.ts @@ -0,0 +1,405 @@ +/** + * Receiving oracle for the Convex provider boundary. Automatic pagination must + * match an independent unpaginated server query after each committed mutation. + * The first driver uses real server-assigned cursors, both SDK clients, and + * inserts, deletion at a boundary, and movement between ranges. Expected rows + * come from tasks:all, which does not paginate or use the adapter. + * + * The durable driver checks the outbox's documented process-restart law with + * actual disk storage and SIGKILL. Its independent model applies one logical + * insert, update, or delete, hence the draft row or its absence, one receipt + * containing the durable key, and one increment of a non-idempotent counter. + * Histories vary the operation and draft's text/rank, plus whether a second + * crash loses the server acknowledgement. Native values vary on inserted and updated rows. + * A server-accepted checkpoint occurs before the outbox phase can be recorded, + * so replay must reuse its key rather than rely on a completed local marker. + * Checkpoints are observed in child stdout before killing the process. Server + * rows, receipt keys, application count, and complete replayed Collection rows + * are checked after replay. Equal task rows alone cannot distinguish a duplicate + * server application, while the independently observed counter can. + * + * This is a receiving witness for the offline engine's interruption/settlement + * owners, not another model of leadership. It covers one pending mutation in one + * Collection against a real local backend. Shared-key/multi-tab histories stay + * with those primary owners. Cloud infrastructure and an OS power failure + * during a filesystem write are outside this process-crash test. + */ +import fc from 'fast-check' +import { convexToJson } from 'convex/values' +import { createCollection } from '@tanstack/db' +import { ConvexClient } from 'convex/browser' +import { ConvexReactClient } from 'convex/react' +import { makeFunctionReference } from 'convex/server' +import { describe, expect, inject, it } from 'vitest' +import { convexPaginatedCollectionOptions } from '../src/index' +import { withTestCleanup } from '../tests/helpers' +import type { Value } from 'convex/values' +import type { ConvexCollectionClient } from '../src/index' +import type { PaginationOptions, PaginationResult } from 'convex/server' + +type Task = { + _id: string + _creationTime: number + clientId: string + text: string + rank: number + payload?: Value +} +const seed = makeFunctionReference<`mutation`, { count: number }, null>( + `tasks:seed`, +) +const write = makeFunctionReference< + `mutation`, + { + idempotencyKey: string + clientId: string + text: string + rank: number + remove?: boolean + }, + null +>(`tasks:write`) +const all = makeFunctionReference<`query`, {}, Array>(`tasks:all`) +const url = inject(`convexUrl`) + +describe.each([`javascript`, `react`] as const)( + `real backend with %s client`, + (kind) => { + it(`rejects a mirror beyond its row bound without publishing a partial result`, async () => { + const client = + kind === `javascript` + ? new ConvexClient(url, { unsavedChangesWarning: false }) + : new ConvexReactClient(url, { unsavedChangesWarning: false }) + const collection = createCollection( + convexPaginatedCollectionOptions({ + client, + query: makeFunctionReference< + `query`, + { paginationOpts: PaginationOptions }, + PaginationResult + >(`tasks:list`), + queryArgs: {}, + pageSize: 4, + maxRows: 5, + getKey: (task) => task.clientId, + gcTime: 0, + }), + ) + await withTestCleanup( + async () => { + await client.mutation(seed, { count: 23 }) + await expect(collection.preload()).rejects.toBeInstanceOf(RangeError) + expect([...collection.values()]).toEqual([]) + }, + () => collection.cleanup(), + () => client.close(), + ) + }) + it.each([`list`, `split`] as const)( + `loads %s pages through inserts, moves, deletions, and server splits`, + async (functionName) => { + const client = + kind === `javascript` + ? new ConvexClient(url, { unsavedChangesWarning: false }) + : new ConvexReactClient(url, { unsavedChangesWarning: false }) + let splitResponses = 0 + const inspect = (value: unknown): void => { + if ( + value && + typeof value === `object` && + `pageStatus` in value && + (value.pageStatus === `SplitRequired` || + value.pageStatus === `SplitRecommended`) + ) + splitResponses++ + } + const source: ConvexCollectionClient = + `onUpdate` in client + ? { + onUpdate: (query, args, callback, onError) => + client.onUpdate( + query, + args, + (value) => { + inspect(value) + return callback(value) + }, + onError, + ), + } + : { + watchQuery: (query, args, options) => { + const watch = client.watchQuery(query, args, options) + return { + ...watch, + localQueryResult: () => { + const value = watch.localQueryResult() + inspect(value) + return value + }, + } + }, + } + + const collection = createCollection( + convexPaginatedCollectionOptions({ + id: `tasks`, + client: source, + query: makeFunctionReference< + `query`, + { paginationOpts: PaginationOptions }, + PaginationResult + >(`tasks:${functionName}`), + queryArgs: {}, + pageSize: 4, + getKey: (task) => task.clientId, + gcTime: 0, + }), + ) + const check = async () => { + const expected = await client.query(all, {}) + await collection.utils.awaitSync() + const actual = [...collection.values()] + .map(({ clientId, text, rank }) => ({ clientId, text, rank })) + .sort((a, b) => a.rank - b.rank) + expect(actual).toEqual( + expected.map(({ clientId, text, rank }) => ({ + clientId, + text, + rank, + })), + ) + } + await withTestCleanup( + async () => { + await client.mutation(seed, { count: 23 }) + await collection.preload() + expect(collection.size).toBe(23) + await check() + for (const operation of [ + { + clientId: `new-first`, + rank: -10, + text: `insert before all pages`, + }, + { + clientId: `new-last`, + rank: 100, + text: `append after exhaustion`, + }, + { + clientId: `3`, + rank: 3, + text: `delete a boundary`, + remove: true, + }, + { clientId: `12`, rank: -5, text: `move across cursor ranges` }, + { clientId: `8`, rank: 8, text: `update a middle page` }, + ]) { + await client.mutation(write, { + ...operation, + idempotencyKey: crypto.randomUUID(), + }) + await check() + } + if (functionName === `split`) + expect(splitResponses).toBeGreaterThan(0) + }, + () => collection.cleanup(), + () => client.close(), + ) + }, + ) + }, +) + +async function durableHistory( + draft: { clientId: string; text: string; rank: number; payload?: Value }, + uncertain: boolean, + operation: `insert` | `update` | `delete`, +) { + const { spawn } = await import('node:child_process') + const { mkdtemp, rm, readdir, readFile } = await import('node:fs/promises') + const { tmpdir } = await import('node:os') + const { join } = await import('node:path') + const { fileURLToPath } = await import('node:url') + const directory = await mkdtemp(join(tmpdir(), `convex-outbox-`)) + const client = new ConvexClient(url, { unsavedChangesWarning: false }) + async function run(mode: string, target: string): Promise> { + const child = spawn( + process.execPath, + [ + `--import`, + `tsx`, + fileURLToPath(new URL(`./offline-child.ts`, import.meta.url)), + mode, + directory, + url, + JSON.stringify(convexToJson(draft)), + operation, + ], + { stdio: [`ignore`, `pipe`, `pipe`] }, + ) + let output = `` + let errors = `` + await new Promise((resolve, reject) => { + const timeout = setTimeout(() => { + child.kill(`SIGKILL`) + reject(new Error(`Child ${mode} timed out: ${output}\n${errors}`)) + }, 15000) + child.stderr.on(`data`, (chunk) => { + errors += String(chunk) + }) + child.stdout.on(`data`, (chunk) => { + output += String(chunk) + if (output.includes(`CHECKPOINT:${target}\n`)) { + clearTimeout(timeout) + child.kill(`SIGKILL`) + resolve() + } + }) + child.on(`error`, reject) + child.on(`exit`, (code, signal) => { + if (code !== 0 && signal !== `SIGKILL`) { + clearTimeout(timeout) + reject(new Error(`Child ${mode} failed: ${errors}`)) + } + }) + }) + await new Promise((resolve) => { + if (child.exitCode !== null || child.signalCode !== null) resolve() + else child.once(`exit`, () => resolve()) + }) + return output + .split(`\n`) + .filter((line) => line.startsWith(`CHECKPOINT:`)) + .map((line) => line.slice(11)) + } + await withTestCleanup( + async () => { + await client.mutation(seed, { count: operation === `insert` ? 0 : 1 }) + const before = (await client.query(all, {})).map( + ({ clientId, text, rank, payload }) => ({ + clientId, + text, + rank, + payload, + }), + ) + // The public write contract gives one final draft for insert/update and no + // row for delete. These expectations do not inspect restored mutations. + const expected = + operation === `delete` ? [] : [{ ...draft, payload: draft.payload }] + const admitted = await run(`admit`, `durable`) + expect(admitted).not.toContain(`provider-call`) + // Read the actual admission record after SIGKILL, independently of the + // replay child. A fresh key on replay cannot pass a receipt-count check. + const records = (await readdir(directory)).filter((name) => + decodeURIComponent(name).startsWith(`tx:`), + ) + expect(records).toHaveLength(1) + const durable: { idempotencyKey: unknown } = JSON.parse( + await readFile(join(directory, records[0]!), `utf8`), + ) + expect(typeof durable.idempotencyKey).toBe(`string`) + expect( + (await client.query(all, {})).map( + ({ clientId, text, rank, payload }) => ({ + clientId, + text, + rank, + payload, + }), + ), + ).toEqual(before) + if (uncertain) { + expect(await run(`uncertain`, `server-accepted`)).toContain( + `provider-call`, + ) + expect( + (await client.query(all, {})).map((row) => row.clientId), + ).toEqual(expected.map((row) => row.clientId)) + } + expect(await run(`replay`, `replayed`)).toContain(`provider-call`) + const receipts = makeFunctionReference< + `query`, + {}, + Array<{ key: string }> + >(`tasks:receipts`) + expect((await client.query(receipts, {})).map(({ key }) => key)).toEqual([ + durable.idempotencyKey, + ]) + const applicationCount = makeFunctionReference<`query`, {}, number>( + `tasks:applicationCount`, + ) + expect(await client.query(applicationCount, {})).toBe(1) + expect( + (await client.query(all, {})).map( + ({ clientId, text, rank, payload }) => ({ + clientId, + text, + rank, + payload, + }), + ), + ).toEqual(expected) + }, + () => client.close(), + () => rm(directory, { recursive: true, force: true }), + ) +} + +it.each([`insert`, `update`, `delete`] as const)( + `restores durable %s and deduplicates uncertain acknowledgements across generated process histories`, + async (operation) => { + const property = fc.asyncProperty( + fc.record({ + text: + operation === `delete` + ? fc.constant(`task 0`) + : fc.string({ maxLength: 12 }), + rank: + operation === `delete` + ? fc.constant(0) + : fc.integer({ min: -50, max: 50 }), + uncertain: fc.boolean(), + payload: + operation !== `delete` + ? fc.oneof( + fc.bigInt({ min: -(2n ** 63n), max: 2n ** 63n - 1n }), + fc.constantFrom(NaN, Infinity, -Infinity, -0), + fc + .uint8Array({ maxLength: 12 }) + .map((bytes) => bytes.buffer as ArrayBuffer), + ) + : fc.constant(undefined), + }), + async ({ text, rank, uncertain, payload }) => + durableHistory( + { + clientId: operation === `insert` ? `offline-created` : `0`, + text, + rank, + ...(operation !== `delete` ? { payload } : {}), + }, + uncertain, + operation, + ), + ) + // Each grammar owns its seed/path pair. An insert shrink path is not a + // delete path, whose text/rank/payload dimensions are intentionally fixed. + const replayPrefix = `CONVEX_BACKEND_${operation.toUpperCase()}_ORACLE` + const replaySeed = process.env[`${replayPrefix}_SEED`] + if (replaySeed) + await fc.assert(property, { + numRuns: 6, + seed: Number(replaySeed), + path: process.env[`${replayPrefix}_PATH`], + }) + else { + await fc.assert(property, { numRuns: 6, seed: 20261022 }) + await fc.assert(property, { numRuns: 6 }) + } + }, + 120000, +) diff --git a/packages/convex-db-collection/e2e/backend-startup-oracle.test.ts b/packages/convex-db-collection/e2e/backend-startup-oracle.test.ts new file mode 100644 index 0000000000..3d4810e1c0 --- /dev/null +++ b/packages/convex-db-collection/e2e/backend-startup-oracle.test.ts @@ -0,0 +1,92 @@ +/** + * The real public-app setup must start on a free loopback port when no port + * override was requested, even if the fixture's former fixed port is occupied. + * Authority: Convex CLI's supported default local-backend port allocation and + * this suite's promise to own an isolated backend instead of requiring a + * particular machine-wide port. Explicit user port overrides remain fail-fast. + * + * The independent judgment is resource based: a listener already owns 43320, + * so the newly provided URL must name another port and serve an empty instance + * of the pinned upstream messages query. The test never calculates which free + * port the CLI should choose or reads the launcher's selection logic. + * + * This fixed history holds the old port, calls the actual public-pagination + * global setup, observes its provided URL, then executes the public query. + * EADDRINUSE is also a valid occupied-port precondition, and its external owner + * is never closed. All resources acquired here are released independently while + * preserving a primary mismatch. This is native startup conformance, not a + * generated port-allocation or concurrent-startup proof. External occupancy is + * observed at bind time, not held by a lease owned by this test. + * + * Calibration: restoring the former fixed 43320 setup argument rejects this + * startup-fulfillment assertion with the actual CLI's "Requested port 43320 is + * not available" failure. The public query checkpoint is not reached in that + * control. Removing the fixed argument permits startup and the empty query. + */ +import { createServer } from 'node:net' +import { ConvexHttpClient } from 'convex/browser' +import { makeFunctionReference } from 'convex/server' +import { expect, it, vi } from 'vitest' +import { withTestCleanup } from '../tests/helpers' +import setupPublicPagination from './public-pagination-setup' +import type { PaginationOptions, PaginationResult } from 'convex/server' +import type { TestProject } from 'vitest/node' + +it(`starts the actual public app when its former fixed port is occupied`, async () => { + const blocker = createServer((socket) => socket.destroy()) + let ownsBlocker = false + let stopBackend: (() => Promise) | undefined + let providedUrl: string | undefined + const project: Pick = { + provide: (key, value) => { + expect(key).toBe(`publicPaginationUrl`) + if (typeof value !== `string`) + throw new TypeError(`Expected the public app's loopback URL`) + providedUrl = value + }, + } + await withTestCleanup( + async () => { + vi.stubEnv(`CONVEX_TEST_PORT`, undefined) + ownsBlocker = await new Promise((resolve, reject) => { + blocker.once(`error`, (error: NodeJS.ErrnoException) => { + if (error.code === `EADDRINUSE`) resolve(false) + else reject(error) + }) + blocker.listen(43320, `127.0.0.1`, () => resolve(true)) + }) + const startup = setupPublicPagination(project).then((cleanup) => { + stopBackend = cleanup + return cleanup + }) + await expect(startup).resolves.toBeTypeOf(`function`) + if (!providedUrl) + throw new Error(`Public app setup did not provide a URL`) + const url = new URL(providedUrl) + expect(url.hostname).toBe(`127.0.0.1`) + expect(Number(url.port)).not.toBe(43320) + const query = makeFunctionReference< + `query`, + { paginationOpts: PaginationOptions }, + PaginationResult<{ author: string; body: string }> + >(`messages:list`) + const response = await new ConvexHttpClient(providedUrl).query(query, { + paginationOpts: { cursor: null, numItems: 1 }, + }) + expect(response.page).toEqual([]) + expect(response.isDone).toBe(true) + }, + async () => { + await stopBackend?.() + }, + async () => { + if (ownsBlocker) + await new Promise((resolve, reject) => + blocker.close((error) => (error ? reject(error) : resolve())), + ) + }, + () => { + vi.unstubAllEnvs() + }, + ) +}, 180000) diff --git a/packages/convex-db-collection/e2e/backend/.gitignore b/packages/convex-db-collection/e2e/backend/.gitignore new file mode 100644 index 0000000000..4db599fa11 --- /dev/null +++ b/packages/convex-db-collection/e2e/backend/.gitignore @@ -0,0 +1,3 @@ +.env.local +.convex/ +convex/_generated/ diff --git a/packages/convex-db-collection/e2e/backend/convex.json b/packages/convex-db-collection/e2e/backend/convex.json new file mode 100644 index 0000000000..a74ec106c4 --- /dev/null +++ b/packages/convex-db-collection/e2e/backend/convex.json @@ -0,0 +1 @@ +{ "functions": "convex/" } diff --git a/packages/convex-db-collection/e2e/backend/convex/schema.ts b/packages/convex-db-collection/e2e/backend/convex/schema.ts new file mode 100644 index 0000000000..3129c58112 --- /dev/null +++ b/packages/convex-db-collection/e2e/backend/convex/schema.ts @@ -0,0 +1,15 @@ +import { defineSchema, defineTable } from 'convex/server' +import { v } from 'convex/values' + +export default defineSchema({ + tasks: defineTable({ + clientId: v.string(), + text: v.string(), + rank: v.number(), + payload: v.optional(v.any()), + }) + .index('by_rank', ['rank']) + .index('by_clientId', ['clientId']), + applications: defineTable({ count: v.number() }), + receipts: defineTable({ key: v.string() }).index('by_key', ['key']), +}) diff --git a/packages/convex-db-collection/e2e/backend/convex/tasks.ts b/packages/convex-db-collection/e2e/backend/convex/tasks.ts new file mode 100644 index 0000000000..10045a4e5d --- /dev/null +++ b/packages/convex-db-collection/e2e/backend/convex/tasks.ts @@ -0,0 +1,96 @@ +import { + mutationGeneric, + paginationOptsValidator, + queryGeneric, +} from 'convex/server' +import { v } from 'convex/values' + +export const list = queryGeneric({ + args: { paginationOpts: paginationOptsValidator }, + handler: (ctx, args) => + ctx.db.query('tasks').withIndex('by_rank').paginate(args.paginationOpts), +}) +export const all = queryGeneric({ + args: {}, + handler: (ctx) => ctx.db.query('tasks').withIndex('by_rank').collect(), +}) +export const seed = mutationGeneric({ + args: { count: v.number() }, + handler: async (ctx, { count }) => { + for (const row of await ctx.db.query('tasks').collect()) + await ctx.db.delete(row._id) + for (const row of await ctx.db.query('receipts').collect()) + await ctx.db.delete(row._id) + for (const row of await ctx.db.query('applications').collect()) + await ctx.db.delete(row._id) + for (let i = 0; i < count; i++) + await ctx.db.insert('tasks', { + clientId: String(i), + text: `task ${i}`, + rank: i, + }) + }, +}) +export const write = mutationGeneric({ + args: { + idempotencyKey: v.string(), + clientId: v.string(), + text: v.string(), + rank: v.number(), + payload: v.optional(v.any()), + remove: v.optional(v.boolean()), + }, + handler: async (ctx, args) => { + const receipt = await ctx.db + .query('receipts') + .withIndex('by_key', (q) => q.eq('key', args.idempotencyKey)) + .unique() + if (receipt) return + // This observable effect is deliberately non-idempotent. Reapplying an + // acknowledged write must increment it, even when the final task is equal. + const applications = await ctx.db.query('applications').unique() + if (applications) + await ctx.db.patch(applications._id, { count: applications.count + 1 }) + else await ctx.db.insert('applications', { count: 1 }) + const previous = await ctx.db + .query('tasks') + .withIndex('by_clientId', (q) => q.eq('clientId', args.clientId)) + .unique() + if (args.remove) { + if (previous) await ctx.db.delete(previous._id) + } else if (previous) + await ctx.db.patch(previous._id, { + text: args.text, + rank: args.rank, + payload: args.payload, + }) + else + await ctx.db.insert('tasks', { + clientId: args.clientId, + text: args.text, + rank: args.rank, + ...(args.payload === undefined ? {} : { payload: args.payload }), + }) + await ctx.db.insert('receipts', { key: args.idempotencyKey }) + }, +}) +export const receipts = queryGeneric({ + args: {}, + handler: (ctx) => ctx.db.query('receipts').collect(), +}) + +export const applicationCount = queryGeneric({ + args: {}, + handler: async (ctx) => + (await ctx.db.query('applications').unique())?.count ?? 0, +}) + +// A low read budget forces genuine server page splits during the range tests. +export const split = queryGeneric({ + args: { paginationOpts: paginationOptsValidator }, + handler: (ctx, args) => + ctx.db + .query('tasks') + .withIndex('by_rank') + .paginate({ ...args.paginationOpts, maximumRowsRead: 3 }), +}) diff --git a/packages/convex-db-collection/e2e/backend/convex/tsconfig.json b/packages/convex-db-collection/e2e/backend/convex/tsconfig.json new file mode 100644 index 0000000000..41bfbb9b9a --- /dev/null +++ b/packages/convex-db-collection/e2e/backend/convex/tsconfig.json @@ -0,0 +1,25 @@ +{ + /* This TypeScript project config describes the environment that + * Convex functions run in and is used to typecheck them. + * You can modify it, but some settings are required to use Convex. + */ + "compilerOptions": { + /* These settings are not required by Convex and can be modified. */ + "allowJs": true, + "strict": true, + "moduleResolution": "Bundler", + "jsx": "react-jsx", + "skipLibCheck": true, + "allowSyntheticDefaultImports": true, + + /* These compiler options are required by Convex */ + "target": "ESNext", + "lib": ["ES2023", "dom"], + "forceConsistentCasingInFileNames": true, + "module": "ESNext", + "isolatedModules": true, + "noEmit": true + }, + "include": ["./**/*"], + "exclude": ["./_generated"] +} diff --git a/packages/convex-db-collection/e2e/backend/package.json b/packages/convex-db-collection/e2e/backend/package.json new file mode 100644 index 0000000000..6d70a389ac --- /dev/null +++ b/packages/convex-db-collection/e2e/backend/package.json @@ -0,0 +1,8 @@ +{ + "name": "convex-adapter-backend", + "private": true, + "type": "module", + "dependencies": { + "convex": "^1.47.0" + } +} diff --git a/packages/convex-db-collection/e2e/local-backend.ts b/packages/convex-db-collection/e2e/local-backend.ts new file mode 100644 index 0000000000..99946edf57 --- /dev/null +++ b/packages/convex-db-collection/e2e/local-backend.ts @@ -0,0 +1,89 @@ +import { spawn } from 'node:child_process' +import { readFile } from 'node:fs/promises' +import { fileURLToPath } from 'node:url' +import { withTestCleanup } from '../tests/helpers' + +/** Own a dedicated local backend for the test run. Never select a cloud project. */ +export async function startLocalBackend(directory: URL, port?: number) { + const { + CONVEX_DEPLOY_KEY: _deployKey, + CONVEX_SELF_HOSTED_URL: _selfHostedUrl, + CONVEX_SELF_HOSTED_ADMIN_KEY: _adminKey, + CONVEX_DEPLOYMENT: _deployment, + ...environment + } = process.env + const backend = spawn( + process.execPath, + [ + fileURLToPath( + new URL(`../node_modules/convex/bin/main.js`, import.meta.url), + ), + `dev`, + `--typecheck`, + `disable`, + `--tail-logs`, + `disable`, + ...(port === undefined + ? [] + : [ + `--local-cloud-port`, + String(port), + `--local-site-port`, + String(port + 1), + ]), + ], + { + cwd: fileURLToPath(directory), + env: { ...environment, CONVEX_AGENT_MODE: `anonymous`, FORCE_COLOR: `0` }, + stdio: [`ignore`, `pipe`, `pipe`], + }, + ) + let output = `` + try { + await new Promise((resolve, reject) => { + const timeout = setTimeout( + () => reject(new Error(`Local Convex startup timed out:\n${output}`)), + 120000, + ) + const receive = (chunk: Buffer) => { + output = (output + String(chunk)).slice(-8000) + if (output.includes(`Convex functions ready!`)) { + clearTimeout(timeout) + resolve() + } + } + backend.stdout.on(`data`, receive) + backend.stderr.on(`data`, receive) + backend.once(`error`, (cause) => { + clearTimeout(timeout) + reject(cause) + }) + backend.once(`exit`, (code) => { + clearTimeout(timeout) + reject(new Error(`Local Convex exited (${code}):\n${output}`)) + }) + }) + const env = await readFile(new URL(`.env.local`, directory), `utf8`) + const url = /^CONVEX_URL=(.+)$/m.exec(env)?.[1] + if ( + !url || + !/^http:\/\/127\.0\.0\.1:\d+$/.test(url) || + (port !== undefined && url !== `http://127.0.0.1:${port}`) + ) + throw new Error(`Expected an isolated loopback Convex backend`) + return { url, stop } + } catch (cause) { + await withTestCleanup(() => { + throw cause + }, stop) + throw cause + } + async function stop() { + if (backend.exitCode !== null || backend.signalCode !== null) return + const exited = new Promise((resolve) => + backend.once(`exit`, () => resolve()), + ) + backend.kill(`SIGINT`) + await exited + } +} diff --git a/packages/convex-db-collection/e2e/network-gate.ts b/packages/convex-db-collection/e2e/network-gate.ts new file mode 100644 index 0000000000..caba0d077d --- /dev/null +++ b/packages/convex-db-collection/e2e/network-gate.ts @@ -0,0 +1,57 @@ +import { createConnection, createServer } from 'node:net' +import type { Socket } from 'node:net' + +/** + * Forward real HTTP and WebSocket bytes without interpreting Convex messages. + * Closing the gate destroys existing TCP connections and refuses new ones, so + * SDK recovery must use its public reconnect path against the actual backend. + */ +export async function createNetworkGate(backendUrl: string) { + const target = new URL(backendUrl) + if (target.hostname !== `127.0.0.1`) + throw new Error(`Network tests require an isolated loopback backend`) + const connections = new Set() + let online = true + const server = createServer((incoming) => { + if (!online) { + incoming.destroy() + return + } + const outgoing = createConnection(Number(target.port), target.hostname) + connections.add(incoming) + connections.add(outgoing) + const close = () => { + connections.delete(incoming) + connections.delete(outgoing) + incoming.destroy() + outgoing.destroy() + } + incoming.on(`error`, close).on(`close`, close) + outgoing.on(`error`, close).on(`close`, close) + incoming.pipe(outgoing).pipe(incoming) + }) + await new Promise((resolve, reject) => { + server.once(`error`, reject) + server.listen(0, `127.0.0.1`, resolve) + }) + const address = server.address() + if (!address || typeof address === `string`) + throw new Error(`Expected a TCP listening address`) + return { + url: `http://127.0.0.1:${address.port}`, + disconnect() { + online = false + for (const socket of connections) socket.destroy() + }, + reconnect() { + online = true + }, + async close() { + online = false + for (const socket of connections) socket.destroy() + await new Promise((resolve, reject) => + server.close((cause) => (cause ? reject(cause) : resolve())), + ) + }, + } +} diff --git a/packages/convex-db-collection/e2e/offline-child.ts b/packages/convex-db-collection/e2e/offline-child.ts new file mode 100644 index 0000000000..6a741e13b7 --- /dev/null +++ b/packages/convex-db-collection/e2e/offline-child.ts @@ -0,0 +1,197 @@ +import { deepStrictEqual } from 'node:assert' +import { + mkdir, + readFile, + readdir, + rename, + unlink, + writeFile, +} from 'node:fs/promises' +import { join } from 'node:path' +import { createCollection } from '@tanstack/db' +import { startOfflineExecutor } from '@tanstack/offline-transactions' +import { ConvexClient } from 'convex/browser' +import { makeFunctionReference } from 'convex/server' +import { jsonToConvex } from 'convex/values' +import { convexPaginatedCollectionOptions } from '../src/index' +import { convexMutationCodec, convexOfflineMutation } from '../src/offline' +import type { Value } from 'convex/values' +import type { StorageAdapter } from '@tanstack/offline-transactions' +import type { PaginationOptions, PaginationResult } from 'convex/server' + +type Task = { clientId: string; text: string; rank: number; payload?: Value } +const [mode, directory, url, draftJSON, operation] = process.argv.slice(2) as [ + string, + string, + string, + string, + `insert` | `update` | `delete`, +] +const draft = jsonToConvex(JSON.parse(draftJSON)) as Task +const checkpoint = (value: string) => { + process.stdout.write(`CHECKPOINT:${value}\n`) +} +await mkdir(directory, { recursive: true }) +const storage: StorageAdapter = { + get: async (key) => { + try { + return await readFile(join(directory, encodeURIComponent(key)), `utf8`) + } catch (error) { + if ((error as NodeJS.ErrnoException).code === `ENOENT`) return null + throw error + } + }, + set: async (key, value) => { + if ( + mode === `uncertain` && + JSON.parse(value).outboxPhase === `deletion-pending` + ) { + checkpoint(`server-accepted`) + await new Promise(() => {}) + } + const path = join(directory, encodeURIComponent(key)) + await writeFile(`${path}.tmp`, value) + await rename(`${path}.tmp`, path) + }, + delete: async (key) => { + await unlink(join(directory, encodeURIComponent(key))) + }, + keys: async () => + (await readdir(directory)) + .filter((name) => !name.endsWith(`.tmp`)) + .map(decodeURIComponent), + clear: async () => { + for (const name of await readdir(directory)) + await unlink(join(directory, name)) + }, +} +const client = new ConvexClient(url, { + unsavedChangesWarning: false, + logger: false, +}) +const query = makeFunctionReference< + `query`, + { paginationOpts: PaginationOptions }, + PaginationResult +>(`tasks:list`) +const mutation = makeFunctionReference< + `mutation`, + { + idempotencyKey: string + clientId: string + text: string + rank: number + payload?: Value + remove?: boolean + }, + null +>(`tasks:write`) +const collection = createCollection( + convexPaginatedCollectionOptions({ + id: `durable-tasks`, + client, + query, + queryArgs: {}, + pageSize: 3, + getKey: (row) => row.clientId, + gcTime: 0, + select: (rows) => + rows.map(({ clientId, text, rank, payload }) => ({ + clientId, + text, + rank, + ...(payload === undefined ? {} : { payload }), + })), + }), +) +await collection.preload() +const offline = mode === `admit` +if (offline) await client.close() +let leader = true +const persist = convexOfflineMutation({ + client, + mutation, + collections: [collection], + getArgs: ({ transaction }) => { + const change = transaction.mutations[0] + const row = change.modified as Task + return { + clientId: row.clientId, + text: row.text, + rank: row.rank, + ...(change.type === `delete` ? { remove: true } : {}), + ...(row.payload === undefined ? {} : { payload: row.payload }), + } + }, +}) +const executor = startOfflineExecutor({ + mutationCodec: convexMutationCodec, + collections: { tasks: collection }, + storage, + mutationFns: { + save: async (params) => { + checkpoint(`provider-call`) + await persist(params) + }, + }, + leaderElection: { + requestLeadership: () => Promise.resolve(true), + releaseLeadership: () => { + leader = false + }, + isLeader: () => leader, + onLeadershipChange: () => () => {}, + }, + onlineDetector: { + isOnline: () => !offline, + subscribe: () => () => {}, + notifyOnline: () => {}, + dispose: () => {}, + }, +}) +await executor.waitForInit() +if (offline) { + const action = executor.createOfflineAction({ + mutationFnName: `save`, + onMutate: (row) => { + if (operation === `insert`) collection.insert(row) + else if (operation === `update`) + collection.update(row.clientId, (current) => + Object.assign(current, row), + ) + else collection.delete(row.clientId) + }, + }) + action(draft) + const poll = setInterval(async () => { + if ((await executor.peekOutbox()).length === 1) { + clearInterval(poll) + checkpoint(`durable`) + } + }, 5) +} else if (mode === `replay`) { + const poll = setInterval(async () => { + if ((await executor.peekOutbox()).length === 0) { + clearInterval(poll) + await collection.utils.awaitSync() + // Observe complete public rows, including signed zero and binary bytes, + // after replay settles. Text alone cannot detect a stale native payload. + deepStrictEqual( + [...collection.values()].map(({ clientId, text, rank, payload }) => ({ + clientId, + text, + rank, + ...(payload === undefined ? {} : { payload }), + })), + operation === `delete` ? [] : [draft], + ) + checkpoint(`replayed`) + executor.dispose() + await collection.cleanup() + await client.close() + process.exit(0) + } + }, 5) +} +// Parent sends SIGKILL at the admission/uncertain checkpoint. No graceful +// executor disposal or in-memory persistence is allowed to help restoration. diff --git a/packages/convex-db-collection/e2e/public-pagination-oracle.test.ts b/packages/convex-db-collection/e2e/public-pagination-oracle.test.ts new file mode 100644 index 0000000000..693b17c59d --- /dev/null +++ b/packages/convex-db-collection/e2e/public-pagination-oracle.test.ts @@ -0,0 +1,169 @@ +/** + * Receiving oracle for the actual public get-convex/convex-demos pagination app + * at 2fac21e360bfcceae6065096f1b81480bd8c6fd5, deployed unchanged at runtime by + * public-pagination-setup.ts. Its messages:list orders newest first; + * listWithExtraArg restricts that order to an indexed author; and + * listWithTransformation returns author initials and uppercase body text. + * Source: https://github.com/get-convex/convex-demos/blob/2fac21e360bfcceae6065096f1b81480bd8c6fd5/pagination/convex/messages.ts + * + * The independent model is an append-only ledger of successful sends. Reverse + * insertion order gives the complete source, with an optional author predicate + * and the declared text transformation. It never asks the adapter or a paged + * server query for expected rows. Every body is unique, including after case + * conversion, to provide the stable key omitted by the transformed upstream + * query. These tests do not claim that arbitrary transformed rows have keys. + * + * Both actual SDK clients run all three public queries at pageSize 3. Each + * history adds enough messages for several ranges and includes another author, + * then appends while subscribed. A transparent TCP gate next destroys the + * adapter client's real connections; a separate direct writer commits while + * disconnected. The old public snapshot must remain until reconnect, then the + * complete Collection and select-boundary order must equal the ledger. + * + * This is finite public-app conformance, not a browser-render test, a generated + * history campaign, a total network-work bound, or a test of cloud hosting. + * Calibration: disabling adapter continuation is rejected immediately after + * initial preload at the select-order comparison, three received rows versus + * seventeen expected for JavaScript (and three versus thirty-four for React). + * The fixed replay selector is `-t 'mirrors list through'`; no timeout or setup + * failure is counted as that rejection. The repaired full six-case run passes. + * + * The upstream source declares Convex 1.48.0; this receiving fixture deliberately + * deploys its functions with this package's installed Convex 1.47 SDK/backend. + */ +import { createCollection } from '@tanstack/db' +import { ConvexClient } from 'convex/browser' +import { ConvexReactClient } from 'convex/react' +import { makeFunctionReference } from 'convex/server' +import { describe, expect, inject, it } from 'vitest' +import { convexPaginatedCollectionOptions } from '../src/index' +import { withTestCleanup } from '../tests/helpers' +import { createNetworkGate } from './network-gate' +import type { PaginationOptions, PaginationResult } from 'convex/server' + +type Message = { author: string; body: string } +const ledger: Array = [] +const send = makeFunctionReference<`mutation`, Message, null>(`messages:send`) +const backend = inject(`publicPaginationUrl`) +const queries = [`list`, `listWithExtraArg`, `listWithTransformation`] as const +type QueryName = (typeof queries)[number] + +function expectedRows(name: QueryName, author: string): Array { + return [...ledger] + .reverse() + .filter( + (message) => name !== `listWithExtraArg` || message.author === author, + ) + .map((message) => + name === `listWithTransformation` + ? { author: message.author.charAt(0), body: message.body.toUpperCase() } + : { ...message }, + ) +} + +// The backend starts with a new runtime directory and empty database. File and +// test execution are sequential, so this ledger includes all successful writes +// across the six histories, including rows outside the requested author scope. +describe.each([`javascript`, `react`] as const)( + `public pagination app with %s client`, + (kind) => { + it.each(queries)( + `mirrors %s through live writes and a real reconnect`, + async (name) => { + const gate = await createNetworkGate(backend) + const writer = new ConvexClient(backend, { + unsavedChangesWarning: false, + logger: false, + }) + const client = + kind === `javascript` + ? new ConvexClient(gate.url, { + unsavedChangesWarning: false, + logger: false, + }) + : new ConvexReactClient(gate.url, { + unsavedChangesWarning: false, + logger: false, + }) + const author = `author:${kind}:${name}` + const query = makeFunctionReference< + `query`, + { paginationOpts: PaginationOptions; author?: string }, + PaginationResult + >(`messages:${name}`) + let selected: ReadonlyArray = [] + const collection = createCollection( + convexPaginatedCollectionOptions({ + client, + query, + queryArgs: name === `listWithExtraArg` ? { author } : {}, + pageSize: 3, + select: (rows) => { + selected = rows.map(({ author: from, body }) => ({ + author: from, + body, + })) + return rows + }, + getKey: (row) => row.body, + gcTime: 0, + }), + ) + const publish = async (from: string) => { + const message = { + author: from, + body: `message ${ledger.length} from ${from}`, + } + await writer.mutation(send, message) + ledger.push(message) + } + const check = async (waitForUpdate = true) => { + const expected = expectedRows(name, author) + if (waitForUpdate) + await expect + .poll(() => selected, { timeout: 15000 }) + .toEqual(expected) + else expect(selected).toEqual(expected) + await collection.utils.awaitSync() + expect( + [...collection.values()] + .map(({ author: from, body }) => ({ author: from, body })) + .sort((a, b) => a.body.localeCompare(b.body)), + ).toEqual([...expected].sort((a, b) => a.body.localeCompare(b.body))) + } + await withTestCleanup( + async () => { + for (let index = 0; index < 17; index++) + await publish(index % 2 === 0 ? author : `other:${author}`) + await collection.preload() + await check(false) + await publish(author) + await check() + await publish(`other:${author}`) + await check() + const before = [...selected] + gate.disconnect() + await expect + .poll(() => client.connectionState().isWebSocketConnected) + .toBe(false) + await publish(author) + await publish(`other:${author}`) + expect(selected).toEqual(before) + expect( + [...collection.values()] + .map(({ author: from, body }) => ({ author: from, body })) + .sort((a, b) => a.body.localeCompare(b.body)), + ).toEqual([...before].sort((a, b) => a.body.localeCompare(b.body))) + gate.reconnect() + await check() + }, + () => collection.cleanup(), + () => client.close(), + () => writer.close(), + () => gate.close(), + ) + }, + 45000, + ) + }, +) diff --git a/packages/convex-db-collection/e2e/public-pagination-setup.ts b/packages/convex-db-collection/e2e/public-pagination-setup.ts new file mode 100644 index 0000000000..dc4a24fb20 --- /dev/null +++ b/packages/convex-db-collection/e2e/public-pagination-setup.ts @@ -0,0 +1,102 @@ +import { execFile } from 'node:child_process' +import { mkdtemp, rm, symlink, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { fileURLToPath, pathToFileURL } from 'node:url' +import { promisify } from 'node:util' +import { withTestCleanup } from '../tests/helpers' +import { startLocalBackend } from './local-backend' +import type { TestProject } from 'vitest/node' + +const run = promisify(execFile) +const revision = `2fac21e360bfcceae6065096f1b81480bd8c6fd5` +const repository = `https://github.com/get-convex/convex-demos.git` + +declare module 'vitest' { + interface ProvidedContext { + publicPaginationUrl: string + } +} + +/** + * Run the actual pinned public demo, without redistributing its unlicensed + * source. git archive reads the pinned tree, so local edits and untracked + * files cannot alter the deployed messages/schema functions. The app shell is + * independently supplied test configuration using this package's Convex SDK. + */ +export default async function setup(project: Pick) { + const directory = await mkdtemp(join(tmpdir(), `convex-public-pagination-`)) + let stop: (() => Promise) | undefined + try { + let source = process.env.CONVEX_PUBLIC_DEMOS_SOURCE + if (source) { + const head = ( + await run(`git`, [`-C`, source, `rev-parse`, `HEAD`]) + ).stdout.trim() + if (head !== revision) + throw new Error( + `Public pagination source must be pinned to ${revision}`, + ) + } else { + source = join(directory, `repository`) + await run(`git`, [`init`, source]) + await run(`git`, [ + `-C`, + source, + `fetch`, + `--depth=1`, + repository, + revision, + ]) + await run(`git`, [`-C`, source, `checkout`, `--detach`, `FETCH_HEAD`]) + } + const archive = join(directory, `source.tar`) + await run(`git`, [ + `-C`, + source, + `archive`, + `--format=tar`, + `--output=${archive}`, + revision, + `pagination/convex`, + ]) + await run(`tar`, [`-xf`, archive, `-C`, directory]) + const app = join(directory, `pagination`) + await writeFile( + join(app, `package.json`), + JSON.stringify({ + name: `convex-public-pagination-integration`, + private: true, + type: `module`, + dependencies: { convex: `1.47.0` }, + }), + ) + await symlink( + fileURLToPath(new URL(`../node_modules`, import.meta.url)), + join(app, `node_modules`), + `dir`, + ) + const backend = await startLocalBackend(pathToFileURL(`${app}/`)) + stop = backend.stop + project.provide(`publicPaginationUrl`, backend.url) + } catch (error) { + await withTestCleanup( + () => { + throw error + }, + async () => { + await stop?.() + }, + () => rm(directory, { recursive: true, force: true }), + ) + throw error + } + return () => + withTestCleanup( + () => {}, + async () => { + await stop() + }, + () => rm(directory, { recursive: true, force: true }), + ) +} diff --git a/packages/convex-db-collection/e2e/setup.ts b/packages/convex-db-collection/e2e/setup.ts new file mode 100644 index 0000000000..acfded356f --- /dev/null +++ b/packages/convex-db-collection/e2e/setup.ts @@ -0,0 +1,20 @@ +import { startLocalBackend } from './local-backend' +import type { TestProject } from 'vitest/node' + +declare module 'vitest' { + interface ProvidedContext { + convexUrl: string + } +} + +/** Own a dedicated local backend for the test run. Never select a cloud project. */ +export default async function setup(project: TestProject) { + const backend = await startLocalBackend( + new URL(`./backend/`, import.meta.url), + process.env.CONVEX_TEST_PORT === undefined + ? undefined + : Number(process.env.CONVEX_TEST_PORT), + ) + project.provide(`convexUrl`, backend.url) + return backend.stop +} diff --git a/packages/convex-db-collection/package.json b/packages/convex-db-collection/package.json new file mode 100644 index 0000000000..fd9ca5e232 --- /dev/null +++ b/packages/convex-db-collection/package.json @@ -0,0 +1,73 @@ +{ + "name": "@tanstack/convex-db-collection", + "version": "0.1.0", + "description": "Convex collection for TanStack DB", + "author": "Tanner Linsley", + "license": "MIT", + "repository": { + "type": "git", + "url": "git+https://github.com/TanStack/db.git", + "directory": "packages/convex-db-collection" + }, + "homepage": "https://tanstack.com/db", + "keywords": [ + "convex", + "tanstack", + "optimistic", + "typescript" + ], + "scripts": { + "build": "vite build", + "dev": "vite build --watch", + "lint": "eslint . --fix", + "test": "vitest run", + "test:oracles": "vitest run tests/snapshot-oracle.property.test.ts tests/ownership-oracle.property.test.ts tests/pagination-oracle.property.test.ts tests/offline-oracle.property.test.ts tests/values-oracle.property.test.ts", + "typecheck": "tsc --noEmit", + "test:e2e": "vitest run --config vitest.e2e.config.ts" + }, + "type": "module", + "main": "dist/cjs/index.cjs", + "module": "dist/esm/index.js", + "types": "dist/esm/index.d.ts", + "exports": { + ".": { + "import": { + "types": "./dist/esm/index.d.ts", + "default": "./dist/esm/index.js" + }, + "require": { + "types": "./dist/cjs/index.d.cts", + "default": "./dist/cjs/index.cjs" + } + }, + "./package.json": "./package.json", + "./offline": { + "import": { + "types": "./dist/esm/offline.d.ts", + "default": "./dist/esm/offline.js" + }, + "require": { + "types": "./dist/cjs/offline.d.cts", + "default": "./dist/cjs/offline.cjs" + } + } + }, + "sideEffects": false, + "files": [ + "dist", + "src" + ], + "dependencies": { + "@standard-schema/spec": "^1.1.0", + "@tanstack/db": "workspace:*", + "@tanstack/offline-transactions": "workspace:*" + }, + "peerDependencies": { + "convex": "^1.47.0", + "typescript": ">=4.7" + }, + "devDependencies": { + "@vitest/coverage-istanbul": "^3.2.4", + "convex": "^1.47.0" + } +} diff --git a/packages/convex-db-collection/src/index.ts b/packages/convex-db-collection/src/index.ts new file mode 100644 index 0000000000..67aa8ea84f --- /dev/null +++ b/packages/convex-db-collection/src/index.ts @@ -0,0 +1,67 @@ +import { createConvexOptions } from './options' +import { subscribeQuery } from './subscription' +import type { ConvexCollectionUtils } from './options' +import type { StandardSchemaV1 } from '@standard-schema/spec' +import type { ConvexCollectionClient } from './subscription' +import type { + FunctionArgs, + FunctionReference, + FunctionReturnType, +} from 'convex/server' +import type { BaseCollectionConfig, CollectionConfig } from '@tanstack/db' + +export type { ConvexCollectionUtils } from './options' +export type { ConvexCollectionClient } from './subscription' + +type QueryRow> = + FunctionReturnType extends ReadonlyArray + ? TRow + : never + +export type ConvexCollectionConfig< + TQuery extends FunctionReference<`query`>, + TItem extends object = QueryRow, + TKey extends string | number = string | number, + TSchema extends StandardSchemaV1 = never, +> = Omit< + BaseCollectionConfig, + `syncMode` | `utils` | `schema` +> & { + /** Share the client used for mutations, authentication, and reconnects. */ + client: ConvexCollectionClient + query: TQuery + queryArgs: FunctionArgs + /** Reports subscription and result-conversion failures, including after readiness. */ + onError?: (error: unknown) => void +} & ([TSchema] extends [never] ? { schema?: never } : { schema: TSchema }) & + ( + | { + /** Convert an envelope or nullable result into the complete set of rows. */ + select: (result: FunctionReturnType) => ReadonlyArray + } + | (FunctionReturnType extends ReadonlyArray + ? { select?: never } + : never) + ) + +/** Mirror one authoritative Convex query result. Filters and joins run in DB. */ +export function convexCollectionOptions< + TQuery extends FunctionReference<`query`>, + TItem extends object = QueryRow, + TKey extends string | number = string | number, + TSchema extends StandardSchemaV1 = never, +>( + config: ConvexCollectionConfig, +): CollectionConfig & { + utils: ConvexCollectionUtils +} & ([TSchema] extends [never] ? unknown : { schema: TSchema }) { + const { client, query, queryArgs, ...options } = config + return createConvexOptions(options, (receive, fail) => + subscribeQuery(client, query, queryArgs, receive, fail), + ) +} + +export { + convexPaginatedCollectionOptions, + type ConvexPaginatedCollectionConfig, +} from './paginated-options' diff --git a/packages/convex-db-collection/src/offline.ts b/packages/convex-db-collection/src/offline.ts new file mode 100644 index 0000000000..db0fd4437f --- /dev/null +++ b/packages/convex-db-collection/src/offline.ts @@ -0,0 +1,109 @@ +import { convexToJson, jsonToConvex } from 'convex/values' +import { NonRetriableError } from '@tanstack/offline-transactions' +import type { JSONValue, Value } from 'convex/values' +import type { + MutationValueCodec, + OfflineConfig, +} from '@tanstack/offline-transactions' +import type { ConvexClient } from 'convex/browser' +import type { ConvexReactClient } from 'convex/react' +import type { FunctionArgs, FunctionReference } from 'convex/server' +import type { ConvexCollectionUtils } from './options' + +type OfflineMutation = OfflineConfig[`mutationFns`][string] +type OfflineParams = Parameters[0] + +/** + * Use as a named mutationFn in startOfflineExecutor. The existing outbox owns + * durability, restoration, leadership and retries. The server mutation must + * atomically deduplicate idempotencyKey with its writes. + */ +export function convexOfflineMutation< + TMutation extends FunctionReference<`mutation`>, +>( + config: { + client: Pick | Pick + mutation: TMutation + getArgs: ( + params: OfflineParams, + ) => Omit, `idempotencyKey`> + collections: ReadonlyArray<{ + utils: Pick + }> + } & (FunctionArgs extends { idempotencyKey: string } + ? unknown + : never), +): OfflineMutation { + return async (params) => { + try { + const args = { + ...config.getArgs(params), + idempotencyKey: params.idempotencyKey, + } as FunctionArgs + await config.client.mutation(config.mutation, args) + } catch (cause) { + // The WebSocket clients retry transport interruptions themselves. A + // rejected mutation has a server/function or client configuration error. + const error = new NonRetriableError( + cause instanceof Error ? cause.message : String(cause), + ) + throw error + } + await Promise.all( + config.collections.map((collection) => collection.utils.awaitSync()), + ) + } +} + +/** Lossless, opt-in outbox encoding for Convex values and removed fields. */ +export const convexMutationCodec: MutationValueCodec = { + id: `tanstack-convex-values/1`, + encode: (value) => JSON.stringify(encodeValue(value)), + decode: (value) => decodeValue(JSON.parse(value)), +} + +function encodeValue(value: unknown): unknown { + if (value === undefined) return [`undefined`] + if (Array.isArray(value)) return [`array`, value.map(encodeValue)] + if ( + value !== null && + typeof value === `object` && + !(value instanceof ArrayBuffer) + ) { + const prototype = Object.getPrototypeOf(value) + if (prototype !== Object.prototype && prototype !== null) + throw new TypeError( + `Convex mutation codecs support plain objects and Convex scalar values`, + ) + return [ + `object`, + Object.entries(value).map(([key, field]) => [key, encodeValue(field)]), + ] + } + return [`scalar`, convexToJson(value as Value)] +} + +function decodeValue(value: unknown): unknown { + if (!Array.isArray(value)) + throw new TypeError(`Invalid Convex mutation codec record`) + if (value[0] === `undefined` && value.length === 1) return undefined + if (value.length !== 2) + throw new TypeError(`Invalid Convex mutation codec record`) + if (value[0] === `scalar`) return jsonToConvex(value[1] as JSONValue) + if (!Array.isArray(value[1])) + throw new TypeError(`Invalid Convex mutation codec container`) + if (value[0] === `array`) return value[1].map(decodeValue) + if (value[0] === `object`) + return Object.fromEntries( + value[1].map((entry: unknown) => { + if ( + !Array.isArray(entry) || + entry.length !== 2 || + typeof entry[0] !== `string` + ) + throw new TypeError(`Invalid Convex mutation codec field`) + return [entry[0], decodeValue(entry[1])] + }), + ) + throw new TypeError(`Invalid Convex mutation codec tag`) +} diff --git a/packages/convex-db-collection/src/options.ts b/packages/convex-db-collection/src/options.ts new file mode 100644 index 0000000000..d8e4c4edab --- /dev/null +++ b/packages/convex-db-collection/src/options.ts @@ -0,0 +1,276 @@ +import { + whenSyncAccepted, + withCollectionConfigFactory, + withCollectionSyncConfigFactory, +} from '@tanstack/db' +import type { StandardSchemaV1 } from '@standard-schema/spec' +import type { + BaseCollectionConfig, + CollectionConfig, + SyncConfig, + UtilsRecord, +} from '@tanstack/db' +import type { ConvexSubscription } from './subscription' + +export interface ConvexCollectionUtils extends UtilsRecord { + /** Last subscription or conversion error, cleared by the next valid result. */ + getSyncError: () => unknown + /** Refresh the source cache and wait for sync acceptance, safe inside mutation handlers. */ + awaitSync: () => Promise +} +export function createConvexOptions< + TResult, + TItem extends object, + TKey extends string | number, + TSchema extends StandardSchemaV1, +>( + config: Omit< + BaseCollectionConfig, + `syncMode` | `utils` + > & { + select?: (result: TResult) => ReadonlyArray + onError?: (error: unknown) => void + }, + subscribe: ( + receive: (result: TResult) => void, + fail: (error: unknown) => void, + ) => ConvexSubscription, +): CollectionConfig & { + utils: ConvexCollectionUtils +} & ([TSchema] extends [never] ? unknown : { schema: TSchema }) { + const { select, onError, ...baseConfig } = config + let syncError: unknown + let readCurrent: (() => Promise) | undefined + + const sync: SyncConfig = { + rowUpdateMode: `full`, + sync: ({ + begin, + write, + commit, + truncate, + markReady, + markError, + collection, + }) => { + let active = true + let previous = new Map() + let initialized = false + let visible: true | Promise = true + let acceptanceFailure: { error: unknown } | undefined + let resolveFirst!: () => void + let rejectFirst!: (error: unknown) => void + const firstResult = new Promise((resolve, reject) => { + resolveFirst = resolve + rejectFirst = reject + }) + // A subscription may fail before any caller asks to await its source. + void firstResult.catch(() => {}) + const accepting = new Set>() + const fail = (error: unknown): void => { + if (!active) return + syncError = error + rejectFirst(error) + if (collection.status === `loading` || collection.status === `error`) { + markError(error) + } + onError?.(error) + } + const accept = (result: TResult): void => { + if (!active) return + if (acceptanceFailure) throw acceptanceFailure.error + // Validate the entire result before opening a sync transaction. A bad + // snapshot must never leave a half-written transaction behind. + const rows: unknown = select ? select(result) : result + if (!Array.isArray(rows)) { + throw new TypeError( + `A Convex collection query must return an array, or supply select.`, + ) + } + const next = new Map() + for (const candidate of rows as Array) { + if ( + typeof candidate !== `object` || + candidate === null || + Array.isArray(candidate) + ) { + throw new TypeError(`Convex collection rows must be objects.`) + } + const row = candidate as TItem + const key = config.getKey(row) + if ( + (typeof key !== `string` && typeof key !== `number`) || + (typeof key === `number` && !Number.isFinite(key)) + ) { + throw new TypeError( + `Convex collection keys must be strings or finite numbers.`, + ) + } + if (next.has(key)) { + throw new Error(`Duplicate Convex collection key: ${String(key)}`) + } + next.set(key, row) + } + if ( + initialized && + next.size === previous.size && + [...next].every(([key, row]) => previous.get(key) === row) + ) { + syncError = undefined + if (visible === true) markReady() + return + } + begin() + // A restarted run replaces retained or hydrated data, including an + // empty first result. Later results use keyed changes. + if (!initialized) truncate() + for (const [key, row] of next) { + if (!previous.has(key) || !initialized) { + write({ type: `insert`, value: row }) + } else if (previous.get(key) !== row) { + write({ type: `update`, value: row }) + } + } + for (const key of previous.keys()) { + if (!next.has(key)) write({ type: `delete`, key }) + } + previous = next + initialized = true + syncError = undefined + const receipt = commit() + visible = receipt + // Never await visibility from a mutation handler: DB may hold it until + // that same optimistic transaction settles. + const accepted = whenSyncAccepted(receipt) + if (accepted !== true) { + accepting.add(accepted) + void accepted.then( + () => accepting.delete(accepted), + (error) => { + accepting.delete(accepted) + // Later diffs may depend on this rejected source proposal. + // A new sync run must establish an authoritative base again. + acceptanceFailure = { error } + fail(error) + }, + ) + } + resolveFirst() + if (receipt === true) markReady() + else + void receipt.then(() => { + if (active) { + visible = true + markReady() + } + }, fail) + } + const receive = (result: TResult): void => { + try { + accept(result) + } catch (error) { + fail(error) + } + } + let subscription: ConvexSubscription | undefined + const refresh = async (): Promise => { + if (acceptanceFailure) throw acceptanceFailure.error + if (!active || !subscription) { + if (syncError !== undefined) throw syncError + throw new Error(`The Convex collection has no active subscription.`) + } + try { + await subscription.waitForCurrent?.() + // Cleanup can run while the asynchronous page acquisition waits. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + if (!active) + throw new Error(`Convex sync ended before source acceptance.`) + const value = subscription.getCurrentValue() + if (value !== undefined) accept(value) + } catch (error) { + fail(error) + throw error + } + if (!initialized) await firstResult + if (syncError !== undefined) throw syncError + await Promise.all([...accepting]) + } + readCurrent = refresh + try { + subscription = subscribe(receive, fail) + const value = subscription.getCurrentValue() + if (value !== undefined) receive(value) + } catch (error) { + fail(error) + } + return () => { + active = false + rejectFirst(new Error(`Convex sync ended before source acceptance.`)) + if (readCurrent === refresh) readCurrent = undefined + subscription?.unsubscribe() + } + }, + } + + // Convex mutation promises resolve after the subscription has caught up. + // Read the cache as well: a newly registered cached listener may still be + // waiting for its scheduled initial callback when the mutation resolves. + function wrapHandler( + handler: (params: TParams) => Promise, + ) { + return async (params: TParams): Promise => { + await handler(params) + await awaitSource() + } + } + async function awaitSource(): Promise { + if (!readCurrent) { + throw new Error(`The Convex collection has no active subscription.`) + } + await readCurrent() + } + const options: CollectionConfig< + TItem, + TKey, + TSchema, + ConvexCollectionUtils + > & { utils: ConvexCollectionUtils } = { + ...baseConfig, + syncMode: `eager` as const, + sync: withCollectionSyncConfigFactory( + sync, + (source, utils, startSyncIfIdle) => { + ;(utils as ConvexCollectionUtils).awaitSync = async () => { + startSyncIfIdle() + await awaitSource() + } + return source + }, + ), + onInsert: config.onInsert && wrapHandler(config.onInsert), + onUpdate: config.onUpdate && wrapHandler(config.onUpdate), + onDelete: config.onDelete && wrapHandler(config.onDelete), + utils: { + getSyncError: () => syncError, + awaitSync: awaitSource, + }, + } + let claimed = false + return withCollectionConfigFactory( + options, + () => createConvexOptions(config, subscribe), + () => { + if (claimed) { + throw new Error( + `Convex options can create only one Collection. Create fresh options for each Collection.`, + ) + } + claimed = true + return () => { + claimed = false + } + }, + ) as CollectionConfig & { + utils: ConvexCollectionUtils + } & ([TSchema] extends [never] ? unknown : { schema: TSchema }) +} diff --git a/packages/convex-db-collection/src/paginated-options.ts b/packages/convex-db-collection/src/paginated-options.ts new file mode 100644 index 0000000000..1fd8b07291 --- /dev/null +++ b/packages/convex-db-collection/src/paginated-options.ts @@ -0,0 +1,75 @@ +import { createConvexOptions } from './options' +import { subscribePages } from './pagination' +import type { StandardSchemaV1 } from '@standard-schema/spec' +import type { BaseCollectionConfig } from '@tanstack/db' +import type { + FunctionArgs, + FunctionReference, + FunctionReturnType, + PaginationOptions, + PaginationResult, +} from 'convex/server' +import type { ConvexCollectionUtils } from './options' +import type { ConvexCollectionClient } from './subscription' + +type PaginatedRow> = + FunctionReturnType extends PaginationResult + ? TRow + : never + +export type ConvexPaginatedCollectionConfig< + TQuery extends FunctionReference<`query`>, + TItem extends object = PaginatedRow, + TKey extends string | number = string | number, + TSchema extends StandardSchemaV1 = never, +> = Omit< + BaseCollectionConfig, + `syncMode` | `utils` | `schema` +> & { + client: ConvexCollectionClient + query: TQuery + queryArgs: Omit, `paginationOpts`> + pageSize?: number + /** Maximum accepted rows in the complete eager mirror. Defaults to 10,000. */ + maxRows?: number + onError?: (error: unknown) => void +} & ([TSchema] extends [never] ? { schema?: never } : { schema: TSchema }) & + (FunctionReturnType extends PaginationResult + ? unknown + : never) & + (FunctionArgs extends { paginationOpts: PaginationOptions } + ? unknown + : never) & + ( + | { + select: ( + rows: ReadonlyArray>, + ) => ReadonlyArray + } + | (PaginatedRow extends TItem ? { select?: never } : never) + ) + +/** Automatically subscribe to every page, retaining fixed reactive cursor ranges. */ +export function convexPaginatedCollectionOptions< + TQuery extends FunctionReference<`query`>, + TItem extends object = PaginatedRow, + TKey extends string | number = string | number, + TSchema extends StandardSchemaV1 = never, +>(config: ConvexPaginatedCollectionConfig) { + const { + client, + query, + queryArgs, + pageSize = 100, + maxRows = 10_000, + ...options + } = config + return createConvexOptions< + ReadonlyArray>, + TItem, + TKey, + TSchema + >(options, (receive, fail) => + subscribePages(client, query, queryArgs, pageSize, maxRows, receive, fail), + ) +} diff --git a/packages/convex-db-collection/src/pagination.ts b/packages/convex-db-collection/src/pagination.ts new file mode 100644 index 0000000000..255f7988de --- /dev/null +++ b/packages/convex-db-collection/src/pagination.ts @@ -0,0 +1,192 @@ +import { subscribeQuery } from './subscription' +import type { ConvexCollectionClient, ConvexSubscription } from './subscription' +import type { + FunctionArgs, + FunctionReference, + PaginationResult, +} from 'convex/server' + +type Page = { + cursor: string | null + endCursor?: string + subscription?: ConvexSubscription> +} + +/** + * Each bounded subscription owns (cursor, endCursor]. A changing page cannot + * move its neighbour's boundary. Unbounded tails are sealed before continuing; + * split hints replace a range with its two disjoint halves. Only a complete + * chain ending in isDone can replace the Collection's authoritative snapshot. + */ +export function subscribePages, TRow>( + client: ConvexCollectionClient, + query: TQuery, + args: Omit, `paginationOpts`>, + pageSize: number, + maxRows: number, + receive: (rows: ReadonlyArray) => void, + fail: (error: unknown) => void, +): ConvexSubscription> { + if (!Number.isSafeInteger(pageSize) || pageSize <= 0) { + throw new RangeError(`Convex pageSize must be a positive safe integer.`) + } + if (!Number.isSafeInteger(maxRows) || maxRows <= 0) { + throw new RangeError(`Convex maxRows must be a positive safe integer.`) + } + let active = true + let scheduled = false + let flushing = false + let result: ReadonlyArray | undefined + let error: unknown + const waiters = new Set<{ + resolve: () => void + reject: (reason: unknown) => void + }>() + const pages: Array> = [] + const boundaries = new Set() + const reject = (reason: unknown): void => { + if (!active) return + error = reason + result = undefined + for (const waiter of waiters) waiter.reject(reason) + waiters.clear() + fail(reason) + } + const schedule = (): void => { + if (!active || scheduled) return + result = undefined + scheduled = true + queueMicrotask(() => { + scheduled = false + flush() + }) + } + const createPage = ( + cursor: string | null, + endCursor?: string, + ): Page => { + const page: Page = { cursor, endCursor } + const paginationOpts = { + cursor, + numItems: pageSize, + ...(endCursor === undefined ? {} : { endCursor }), + } + page.subscription = subscribeQuery( + client, + query, + { ...args, paginationOpts } as FunctionArgs, + schedule, + reject, + ) as ConvexSubscription> + return page + } + const replace = (index: number, page: Page, boundary: string): void => { + if ( + boundary === page.cursor || + boundary === page.endCursor || + boundaries.has(boundary) + ) { + throw new Error( + `Convex pagination returned a boundary that makes no progress.`, + ) + } + const left = createPage(page.cursor, boundary) + let right: Page + try { + right = createPage(boundary, page.endCursor) + } catch (cause) { + left.subscription?.unsubscribe() + throw cause + } + boundaries.add(boundary) + pages.splice(index, 1, left, right) + page.subscription?.unsubscribe() + schedule() + } + function flush(): void { + if (!active || flushing) return + flushing = true + result = undefined + try { + const completePages: Array> = [] + let rowCount = 0 + for (let index = 0; index < pages.length; index++) { + const page = pages[index]! + const value = page.subscription!.getCurrentValue() + if (value === undefined) return + if ( + !Array.isArray(value.page) || + typeof value.isDone !== `boolean` || + typeof value.continueCursor !== `string` + ) { + throw new TypeError( + `Convex paginated queries must return PaginationResult.`, + ) + } + rowCount += value.page.length + if (rowCount > maxRows) { + throw new RangeError( + `Convex eager mirror exceeds maxRows (${maxRows}).`, + ) + } + const split = + value.pageStatus === `SplitRequired` || + value.pageStatus === `SplitRecommended` + if (split) { + if (!value.splitCursor) + throw new Error( + `Convex requested a page split without a splitCursor.`, + ) + replace(index, page, value.splitCursor) + return + } + if (page.endCursor === undefined && !value.isDone) { + replace(index, page, value.continueCursor) + return + } + completePages.push(value.page) + } + // Acquisition can add many ranges. Materialize rows only once every + // range is available, rather than copying each incomplete prefix. + const rows = completePages.flat() + error = undefined + result = rows + receive(rows) + for (const waiter of waiters) waiter.resolve() + waiters.clear() + } catch (cause) { + reject(cause) + } finally { + flushing = false + } + } + try { + pages.push(createPage(null)) + schedule() + } catch (cause) { + for (const page of pages) page.subscription?.unsubscribe() + throw cause + } + return { + getCurrentValue: () => { + flush() + return result + }, + waitForCurrent: () => { + flush() + if (error !== undefined) return Promise.reject(error) + if (result !== undefined) return Promise.resolve() + return new Promise((resolve, rejectWaiter) => { + waiters.add({ resolve, reject: rejectWaiter }) + }) + }, + unsubscribe: () => { + active = false + for (const page of pages) page.subscription?.unsubscribe() + const cause = new Error(`Convex pagination was cleaned up.`) + cause.name = `AbortError` + for (const waiter of waiters) waiter.reject(cause) + waiters.clear() + }, + } +} diff --git a/packages/convex-db-collection/src/subscription.ts b/packages/convex-db-collection/src/subscription.ts new file mode 100644 index 0000000000..210dc06dab --- /dev/null +++ b/packages/convex-db-collection/src/subscription.ts @@ -0,0 +1,49 @@ +import type { ConvexClient } from 'convex/browser' +import type { ConvexReactClient } from 'convex/react' +import type { + FunctionArgs, + FunctionReference, + FunctionReturnType, +} from 'convex/server' + +export type ConvexCollectionClient = + Pick | Pick + +export interface ConvexSubscription { + getCurrentValue: () => T | undefined + unsubscribe: () => void + /** Wait for a complete authoritative snapshot, without waiting for DB visibility. */ + waitForCurrent?: () => Promise +} + +export function subscribeQuery>( + client: ConvexCollectionClient, + query: TQuery, + args: FunctionArgs, + receive: (result: FunctionReturnType) => void, + fail: (error: unknown) => void, +): ConvexSubscription> { + if (`disabled` in client && client.disabled === true) { + throw new Error(`A disabled Convex client cannot sync a collection.`) + } + if (`onUpdate` in client) { + if (`closed` in client && client.closed === true) { + throw new Error(`A closed Convex client cannot sync a collection.`) + } + const subscription = client.onUpdate(query, args, receive, fail) + return { + getCurrentValue: () => subscription.getCurrentValue(), + unsubscribe: subscription, + } + } + const watch = client.watchQuery(query, args) + const unsubscribe = watch.onUpdate(() => { + try { + const result = watch.localQueryResult() + if (result !== undefined) receive(result) + } catch (error) { + fail(error) + } + }) + return { getCurrentValue: () => watch.localQueryResult(), unsubscribe } +} diff --git a/packages/convex-db-collection/tests/collection.test.ts b/packages/convex-db-collection/tests/collection.test.ts new file mode 100644 index 0000000000..9d9309bfa9 --- /dev/null +++ b/packages/convex-db-collection/tests/collection.test.ts @@ -0,0 +1,346 @@ +import { + DbClient, + collectionOptions, + createCollection, + withAcceptedReceipt, +} from '@tanstack/db' +import { describe, expect, it, vi } from 'vitest' +import { makeFunctionReference } from 'convex/server' +import { convexCollectionOptions } from '../src/index' +import { controlledClient, query } from './helpers' +import type { Row } from './helpers' + +function setup( + handlers: Partial< + Pick< + Parameters>[0], + `onInsert` | `onUpdate` | `onDelete` | `onError` + > + > = {}, +) { + const transport = controlledClient() + const options = convexCollectionOptions({ + client: transport.client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + gcTime: 0, + ...handlers, + }) + const collection = createCollection(options) + collection.startSyncImmediate() + return { transport, collection, options } +} + +describe(`Convex collection`, () => { + it(`rejects concrete options reuse before it can steal cached mutation confirmation`, async () => { + const { transport, collection, options } = setup({ + onUpdate: () => Promise.resolve(), + }) + transport.emit([{ _id: `a`, text: `old` }]) + expect(() => createCollection(options)).toThrow(`only one Collection`) + transport.cache([{ _id: `a`, text: `server` }]) + const transaction = collection.update(`a`, (row) => { + row.text = `optimistic` + }) + await transaction.isPersisted.promise + expect(collection.get(`a`)?.text).toBe(`server`) + await collection.cleanup() + }) + + it(`waits for a snapshot and accepts an empty initial result`, async () => { + const { transport, collection } = setup() + expect(collection.status).toBe(`loading`) + transport.emit([]) + await collection.preload() + expect(collection.status).toBe(`ready`) + expect(collection.size).toBe(0) + await collection.cleanup() + }) + + it(`replaces full rows and removes rows missing from later snapshots`, async () => { + const { transport, collection } = setup() + transport.emit([ + { _id: `a`, text: `old`, optional: `remove` }, + { _id: `b`, text: `gone` }, + ]) + transport.emit([{ _id: `a`, text: `new` }]) + expect(collection.size).toBe(1) + expect(collection.get(`a`)).toMatchObject({ _id: `a`, text: `new` }) + expect(collection.get(`a`)).not.toHaveProperty(`optional`) + transport.emit([]) + expect(collection.size).toBe(0) + await collection.cleanup() + }) + + it(`rejects invalid snapshots before changing rows and recovers on a valid result`, async () => { + const onError = vi.fn() + const { transport, collection } = setup({ onError }) + transport.emit([{ _id: `a`, text: `valid` }]) + transport.emit([ + { _id: `b`, text: `one` }, + { _id: `b`, text: `two` }, + ]) + expect(collection.get(`a`)?.text).toBe(`valid`) + expect(collection.has(`b`)).toBe(false) + expect(collection.utils.getSyncError()).toBeInstanceOf(Error) + expect(onError).toHaveBeenCalledOnce() + transport.emit([{ _id: `b`, text: `recovered` }]) + expect(collection.utils.getSyncError()).toBeUndefined() + expect(collection.has(`a`)).toBe(false) + await collection.cleanup() + }) + + it(`rejects preload with the original subscription error`, async () => { + const { transport, collection } = setup() + const ready = collection.preload() + const error = new Error(`permission denied`) + transport.error(error) + await expect(ready).rejects.toBe(error) + transport.emit([]) + expect(collection.status).toBe(`ready`) + await collection.cleanup() + }) + + it(`invalidates retired callbacks and replaces retained data after restart`, async () => { + const { transport, collection } = setup() + transport.emit([{ _id: `a`, text: `old` }]) + await collection.cleanup() + transport.late([{ _id: `ghost`, text: `late` }]) + expect(collection.has(`ghost`)).toBe(false) + transport.cache([]) + collection.startSyncImmediate() + await collection.preload() + expect(collection.size).toBe(0) + await collection.cleanup() + expect(transport.counts()).toEqual({ subscriptions: 2, unsubscriptions: 2 }) + }) + + it(`keeps optimistic writes until the cached server result is accepted`, async () => { + let resolve!: () => void + const pending = new Promise((done) => { + resolve = done + }) + const { transport, collection } = setup({ onUpdate: async () => pending }) + transport.emit([{ _id: `a`, text: `old` }]) + const tx = collection.update(`a`, (row) => { + row.text = `optimistic` + }) + expect(collection.get(`a`)?.text).toBe(`optimistic`) + transport.cache([{ _id: `a`, text: `server` }]) + resolve() + await tx.isPersisted.promise + expect(collection.get(`a`)?.text).toBe(`server`) + await collection.cleanup() + }) + + it(`rolls back failed writes`, async () => { + const error = new Error(`rejected`) + const { transport, collection } = setup({ + onDelete: () => Promise.reject(error), + }) + transport.emit([{ _id: `a`, text: `keep` }]) + const tx = collection.delete(`a`) + await expect(tx.isPersisted.promise).rejects.toBe(error) + expect(collection.get(`a`)?.text).toBe(`keep`) + await collection.cleanup() + }) + + it(`supports typed envelope projections`, () => { + const transport = controlledClient() + const envelope = makeFunctionReference<`query`, {}, { rows: Array }>( + `tasks:envelope`, + ) + const options = convexCollectionOptions({ + client: transport.client, + query: envelope, + queryArgs: {}, + select: (result) => result.rows, + getKey: (row) => row._id, + }) + expect(options.syncMode).toBe(`eager`) + }) + + it(`creates independent adapter state for each DbClient`, async () => { + const transport = controlledClient() + transport.cache([{ _id: `a`, text: `cached` }]) + const options = collectionOptions( + convexCollectionOptions({ + id: `tasks`, + client: transport.client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + gcTime: 0, + }), + ) + const first = new DbClient() + const second = new DbClient() + const a = first.collection(options) + const b = second.collection(options) + await Promise.all([a.preload(), b.preload()]) + expect(a).not.toBe(b) + expect(transport.counts().subscriptions).toBe(2) + await a.cleanup() + expect(b.get(`a`)?.text).toBe(`cached`) + await b.cleanup() + }) +}) + +it(`waits for deferred sync acceptance without waiting for its own visibility`, async () => { + const transport = controlledClient() + let delay = false + let accept!: () => void + const accepted = new Promise((resolve) => { + accept = resolve + }) + const options = convexCollectionOptions({ + client: transport.client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + gcTime: 0, + onUpdate: () => Promise.resolve(), + }) + const collection = createCollection({ + ...options, + sync: { + ...options.sync, + sync: (params) => + options.sync.sync({ + ...params, + commit: () => { + const visible = params.commit() + return delay + ? withAcceptedReceipt( + Promise.resolve(visible).then(() => undefined), + accepted, + ) + : visible + }, + }), + }, + }) + collection.startSyncImmediate() + transport.emit([{ _id: `a`, text: `old` }]) + delay = true + const transaction = collection.update(`a`, (row) => { + row.text = `optimistic` + }) + transport.emit([{ _id: `a`, text: `server` }]) + await new Promise((resolve) => setTimeout(resolve, 0)) + expect(transaction.state).toBe(`persisting`) + expect(collection.get(`a`)?.text).toBe(`optimistic`) + accept() + await transaction.isPersisted.promise + expect(collection.get(`a`)?.text).toBe(`server`) + await collection.cleanup() +}) + +it.each([null, {}, [null], [1], [[`nested`]]])( + `rejects a malformed query result %j`, + async (result) => { + const { collection, transport } = setup() + const ready = collection.preload() + transport.emit(result as unknown as Array) + await expect(ready).rejects.toBeInstanceOf(TypeError) + transport.emit([]) + expect(collection.status).toBe(`ready`) + await collection.cleanup() + }, +) + +it.each([NaN, Infinity, -Infinity])( + `rejects the invalid numeric key %s`, + async (key) => { + const transport = controlledClient() + const collection = createCollection( + convexCollectionOptions({ + client: transport.client, + query, + queryArgs: { owner: `alice` }, + getKey: () => key, + gcTime: 0, + }), + ) + const ready = collection.preload() + transport.emit([{ _id: `a`, text: `bad key` }]) + await expect(ready).rejects.toThrow(`finite numbers`) + await collection.cleanup() + }, +) + +it(`recovers from a projection failure before opening a sync transaction`, async () => { + const transport = controlledClient() + const error = new Error(`bad projection`) + let broken = true + const collection = createCollection( + convexCollectionOptions({ + client: transport.client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + gcTime: 0, + select: (rows) => { + if (broken) throw error + return rows + }, + }), + ) + const ready = collection.preload() + transport.emit([{ _id: `a`, text: `valid` }]) + await expect(ready).rejects.toBe(error) + broken = false + transport.emit([{ _id: `a`, text: `valid` }]) + expect(collection.get(`a`)?.text).toBe(`valid`) + await collection.cleanup() +}) + +it(`confirms optimistic inserts and deletes from subscription results`, async () => { + const transport = controlledClient() + const collection = createCollection( + convexCollectionOptions({ + client: transport.client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + gcTime: 0, + onInsert: () => { + transport.emit([{ _id: `a`, text: `server insert` }]) + return Promise.resolve() + }, + onDelete: () => { + transport.emit([]) + return Promise.resolve() + }, + }), + ) + collection.startSyncImmediate() + transport.emit([]) + const inserted = collection.insert({ _id: `a`, text: `optimistic` }) + await inserted.isPersisted.promise + expect(collection.get(`a`)?.text).toBe(`server insert`) + const deleted = collection.delete(`a`) + await deleted.isPersisted.promise + expect(collection.size).toBe(0) + await collection.cleanup() +}) + +it(`reports a cached query error discovered while settling a mutation`, async () => { + const onError = vi.fn() + const { transport, collection } = setup({ + onError, + onUpdate: () => Promise.resolve(), + }) + transport.emit([{ _id: `a`, text: `old` }]) + const error = new Error(`cached query failed`) + transport.cacheError(error) + const transaction = collection.update(`a`, (row) => { + row.text = `optimistic` + }) + await expect(transaction.isPersisted.promise).rejects.toBe(error) + expect(collection.utils.getSyncError()).toBe(error) + expect(onError).toHaveBeenCalledExactlyOnceWith(error) + expect(collection.get(`a`)?.text).toBe(`old`) + await collection.cleanup() +}) diff --git a/packages/convex-db-collection/tests/helpers.ts b/packages/convex-db-collection/tests/helpers.ts new file mode 100644 index 0000000000..bb39b210d5 --- /dev/null +++ b/packages/convex-db-collection/tests/helpers.ts @@ -0,0 +1,104 @@ +import { makeFunctionReference } from 'convex/server' +import type { ConvexCollectionClient } from '../src/index' + +export type Row = { _id: string; text: string; optional?: string } +export const query = makeFunctionReference< + `query`, + { owner: string }, + Array +>(`tasks:list`) +export const mutation = makeFunctionReference< + `mutation`, + { text: string }, + null +>(`tasks:update`) + +export function controlledClient() { + let value: Array | undefined + let readError: Error | undefined + let receive: ((rows: Array) => void) | undefined + let fail: ((error: Error) => void) | undefined + let retired: ((rows: Array) => void) | undefined + let subscriptions = 0 + let unsubscriptions = 0 + // The assertion adapts this one-query transport fixture to the SDK's generic + // method. Real SDK compatibility is checked separately, without this cast. + const client = { + onUpdate: ( + _query: unknown, + _args: unknown, + callback: typeof receive, + onError: typeof fail, + ) => { + subscriptions++ + receive = callback + fail = onError + return Object.assign( + () => { + unsubscriptions++ + retired = receive + receive = undefined + }, + { + getCurrentValue: () => { + if (readError) throw readError + return value + }, + }, + ) + }, + } as ConvexCollectionClient + return { + client, + emit(rows: Array) { + value = rows + receive?.(rows) + }, + cache(rows: Array) { + value = rows + }, + cacheError(error: Error) { + readError = error + }, + error(error: Error) { + fail?.(error) + }, + late(rows: Array) { + retired?.(rows) + }, + counts: () => ({ subscriptions, unsubscriptions }), + } +} + +/** Preserve the law failure and still attempt every registered resource release. */ +export async function withTestCleanup( + body: () => void | Promise, + ...cleanups: Array<() => void | Promise> +): Promise { + let primary: { error: unknown } | undefined + try { + await body() + } catch (error) { + primary = { error } + } + const errors: Array = [] + for (const cleanup of cleanups) { + try { + await cleanup() + } catch (error) { + errors.push(error) + } + } + if (primary) { + if (errors.length) + throw new AggregateError( + [primary.error, ...errors], + `Comparison and cleanup failed`, + { cause: primary.error }, + ) + throw primary.error + } + if (errors.length === 1) throw errors[0] + if (errors.length) + throw new AggregateError(errors, `Cleanup failed`, { cause: errors[0] }) +} diff --git a/packages/convex-db-collection/tests/offline-oracle.property.test.ts b/packages/convex-db-collection/tests/offline-oracle.property.test.ts new file mode 100644 index 0000000000..51f2e1119e --- /dev/null +++ b/packages/convex-db-collection/tests/offline-oracle.property.test.ts @@ -0,0 +1,317 @@ +/** + * The Convex offline bridge sends one durable idempotency key, then waits for + * source acceptance. Argument construction and rejected server mutations are + * terminal application failures; a sync failure after server confirmation may + * be retried with that same key. This is the bridge's public retry contract, + * received by the existing outbox engine's NonRetriableError boundary. + * + * The model is an ordered dependency list: args, mutation, sync. A failure + * admits only its prefix. Generated histories choose the failing stage or no + * failure, plus arbitrary text and durable keys. Each stage records its call; + * the comparison checks the prefix and caller error after settlement. It does + * not model outbox storage or leadership, whose owners remain in that package. + * The real process-crash witness uses actual Convex mutations in e2e. + */ +import { createCollection, withAcceptedReceipt } from '@tanstack/db' +import fc from 'fast-check' +import { NonRetriableError } from '@tanstack/offline-transactions' +import { makeFunctionReference } from 'convex/server' +import { expect, it } from 'vitest' +import { convexCollectionOptions } from '../src/index' +import { convexOfflineMutation } from '../src/offline' +import { controlledClient, query, withTestCleanup } from './helpers' +import type { ConvexClient } from 'convex/browser' +import type { OfflineConfig } from '@tanstack/offline-transactions' + +const mutation = makeFunctionReference< + `mutation`, + { idempotencyKey: string; text: string }, + null +>(`tasks:save`) +const stages = [`args`, `mutation`, `sync`] as const +const property = fc.asyncProperty( + fc.record({ + failure: fc.constantFrom(`none`, ...stages), + key: fc.string({ maxLength: 15 }), + text: fc.string({ maxLength: 15 }), + }), + async ({ failure, key, text }) => { + const trace: Array = [] + const cause = new Error(`injected ${failure}`) + const enter = (stage: string) => { + trace.push(stage) + if (failure === stage) throw cause + } + const send = ( + _ref: unknown, + args: { idempotencyKey: string; text: string }, + ) => { + enter(`mutation`) + expect(args).toEqual({ idempotencyKey: key, text }) + return Promise.resolve(null) + } + const persist = convexOfflineMutation({ + client: { mutation: send } as unknown as Pick, + mutation, + getArgs: () => { + enter(`args`) + return { text } + }, + collections: [ + { + utils: { + awaitSync: () => { + enter(`sync`) + return Promise.resolve() + }, + }, + }, + ], + }) + const params = { + transaction: { mutations: [] }, + idempotencyKey: key, + } as unknown as Parameters[0] + const observed = await persist(params).then( + () => ({ status: `fulfilled` as const }), + (error: unknown) => ({ status: `rejected` as const, error }), + ) + const boundary = stages.indexOf(failure as (typeof stages)[number]) + expect(trace).toEqual( + boundary === -1 ? stages : stages.slice(0, boundary + 1), + ) + expect(observed.status).toBe(boundary === -1 ? `fulfilled` : `rejected`) + if (observed.status === `rejected`) { + if (boundary < 2) expect(observed.error).toBeInstanceOf(NonRetriableError) + else expect(observed.error).toBe(cause) + } + }, +) + +it(`preserves ordered replay and retry classification across bridge failures`, async () => { + const seed = process.env.CONVEX_OFFLINE_ORACLE_SEED + if (seed) + await fc.assert(property, { + seed: Number(seed), + path: process.env.CONVEX_OFFLINE_ORACLE_PATH, + numRuns: 60, + }) + else { + await fc.assert(property, { seed: 20261022, numRuns: 60 }) + await fc.assert(property, { numRuns: 60 }) + } +}) + +/** + * Source acceptance is owed by the bridge's documented Collection boundary, + * even on cold replay. This second driver uses real Collections, rather than + * treating an awaitSync stub as evidence of source acceptance. The independent + * model has one fact, whether a valid source snapshot has arrived. Before that + * fact, a successful mutation cannot fulfill replay. A provider failure rejects + * replay and preserves its cause. A valid empty snapshot counts as acceptance. + * + * Histories start idle, awaiting a result, or failed during startup, then cross + * empty/nonempty snapshots and failures before the first result. The provider + * holds callbacks at an explicit cut. We observe bridge settlement and public + * rows before and after that delivery. This does not model durable outbox I/O. + */ +const sourceProperty = fc.asyncProperty( + fc.record({ + state: fc.constantFrom(`idle`, `pending`, `startup-error`), + delivery: fc.constantFrom(`empty`, `rows`, `error`, `cleanup`), + text: fc.string({ maxLength: 15 }), + }), + async ({ state, delivery, text }) => { + const transport = controlledClient() + const cause = new Error(`source unavailable`) + const collection = createCollection( + convexCollectionOptions({ + client: + state === `startup-error` + ? { + onUpdate: () => { + throw cause + }, + } + : transport.client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + gcTime: 0, + }), + ) + if (state !== `idle`) collection.startSyncImmediate() + const persist = convexOfflineMutation({ + client: { mutation: () => Promise.resolve(null) } as unknown as Pick< + ConvexClient, + `mutation` + >, + mutation, + getArgs: () => ({ text }), + collections: [collection], + }) + let outcome: { status: string; error?: unknown } = { status: `pending` } + const pending = persist({ + transaction: { mutations: [] }, + idempotencyKey: `source-key`, + } as unknown as Parameters[0]).then( + () => { + outcome = { status: `fulfilled` } + }, + (error) => { + outcome = { status: `rejected`, error } + }, + ) + await withTestCleanup( + async () => { + await new Promise((resolve) => setTimeout(resolve, 0)) + if (state === `startup-error`) { + expect(outcome).toEqual({ status: `rejected`, error: cause }) + } else { + expect(outcome.status).toBe(`pending`) + expect(transport.counts().subscriptions).toBe(1) + if (delivery === `error`) transport.error(cause) + else if (delivery === `cleanup`) await collection.cleanup() + else transport.emit(delivery === `empty` ? [] : [{ _id: `a`, text }]) + await pending + expect(outcome.status).toBe( + delivery === `error` || delivery === `cleanup` + ? `rejected` + : `fulfilled`, + ) + if (delivery === `error`) { + expect(outcome.error).toBe(cause) + // Provider errors can recover in this run when a valid result arrives. + transport.emit([{ _id: `a`, text }]) + await persist({ + transaction: { mutations: [] }, + idempotencyKey: `source-key`, + } as unknown as Parameters[0]) + expect(collection.get(`a`)?.text).toBe(text) + } + if (delivery === `rows`) expect(collection.get(`a`)?.text).toBe(text) + if (delivery === `empty`) expect(collection.size).toBe(0) + } + }, + async () => { + await collection.cleanup() + }, + ) + }, +) + +it(`requires real source acceptance through cold replay histories`, async () => { + const seed = process.env.CONVEX_OFFLINE_SOURCE_ORACLE_SEED + if (seed) + await fc.assert(sourceProperty, { + seed: Number(seed), + path: process.env.CONVEX_OFFLINE_SOURCE_ORACLE_PATH, + numRuns: 60, + }) + else { + await fc.assert(sourceProperty, { seed: 20261023, numRuns: 60 }) + await fc.assert(sourceProperty, { numRuns: 60 }) + } +}) + +/** + * A durable wrapper can reject before handing a committed source proposal to + * core, as specified by whenSyncAccepted. Once that happens, later diffs cannot + * assume that proposal became the source base. This run stays failed until an + * explicit cleanup/restart establishes a fresh authoritative snapshot. + * + * The reference has two states, failed run and restarted run. Same-cache reads + * and new provider callbacks cannot distinguish failed states, all reject with + * the durable cause. Restart is the one action that changes that judgment. + * Histories vary empty/nonempty initial snapshots and one to four retries with + * unchanged or changed cached rows. Actual commit is deferred behind a rejected + * durability promise, matching the persistence adapter's acceptance boundary. + */ +const acceptanceProperty = fc.asyncProperty( + fc.boolean(), + fc.array(fc.boolean(), { minLength: 1, maxLength: 4 }), + fc.string({ maxLength: 12 }), + async (empty, retries, text) => { + const transport = controlledClient() + const cause = new Error(`durable acceptance failed`) + let rejectDurable!: (error: unknown) => void + let durable = new Promise((_resolve, reject) => { + rejectDurable = reject + }) + const options = convexCollectionOptions({ + client: transport.client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + gcTime: 0, + }) + const collection = createCollection({ + ...options, + sync: { + ...options.sync, + sync: (syncParams) => + options.sync.sync({ + ...syncParams, + commit: () => { + const visible = durable + .then(async () => { + await syncParams.commit() + }) + .then(() => undefined) + return withAcceptedReceipt(visible, durable) + }, + }), + }, + }) + const persist = convexOfflineMutation({ + client: { mutation: () => Promise.resolve(null) } as unknown as Pick< + ConvexClient, + `mutation` + >, + mutation, + getArgs: () => ({ text }), + collections: [collection], + }) + const params = { + transaction: { mutations: [] }, + idempotencyKey: `same-durable-key`, + } as unknown as Parameters[0] + await withTestCleanup( + async () => { + const pending = persist(params) + const rejected = expect(pending).rejects.toBe(cause) + await new Promise((resolve) => setTimeout(resolve, 0)) + const initial = empty ? [] : [{ _id: `a`, text }] + transport.emit(initial) + rejectDurable(cause) + await rejected + for (const changed of retries) { + if (changed) transport.emit([{ _id: `a`, text: `new:${text}` }]) + await expect(persist(params)).rejects.toBe(cause) + expect(collection.size).toBe(0) + } + await collection.cleanup() + durable = Promise.resolve() + transport.cache([{ _id: `a`, text: `restarted:${text}` }]) + collection.startSyncImmediate() + await persist(params) + expect(collection.get(`a`)?.text).toBe(`restarted:${text}`) + }, + () => collection.cleanup(), + ) + }, +) + +it(`keeps rejected source acceptance terminal until a fresh sync run`, async () => { + const seed = process.env.CONVEX_OFFLINE_ACCEPTANCE_ORACLE_SEED + if (seed) + await fc.assert(acceptanceProperty, { + seed: Number(seed), + path: process.env.CONVEX_OFFLINE_ACCEPTANCE_ORACLE_PATH, + numRuns: 60, + }) + else { + await fc.assert(acceptanceProperty, { seed: 20261023, numRuns: 60 }) + await fc.assert(acceptanceProperty, { numRuns: 60 }) + } +}) diff --git a/packages/convex-db-collection/tests/offline.test.ts b/packages/convex-db-collection/tests/offline.test.ts new file mode 100644 index 0000000000..77139e278b --- /dev/null +++ b/packages/convex-db-collection/tests/offline.test.ts @@ -0,0 +1,219 @@ +import { createCollection, withAcceptedReceipt } from '@tanstack/db' +import { NonRetriableError } from '@tanstack/offline-transactions' +import { makeFunctionReference } from 'convex/server' +import { expect, it, vi } from 'vitest' +import { convexCollectionOptions } from '../src/index' +import { convexOfflineMutation } from '../src/offline' +import { controlledClient, query } from './helpers' +import type { ConvexClient } from 'convex/browser' +import type { OfflineConfig } from '@tanstack/offline-transactions' + +const mutation = makeFunctionReference< + `mutation`, + { idempotencyKey: string; text: string }, + null +>(`tasks:save`) +const params = { + transaction: { mutations: [] }, + idempotencyKey: `durable-id`, +} as unknown as Parameters[0] + +it(`sends the durable idempotency key and waits for sync acceptance`, async () => { + const send = vi.fn(() => Promise.resolve(null)) + let accepted!: () => void + const gate = new Promise((resolve) => { + accepted = resolve + }) + const synced = vi.fn(() => gate) + const persist = convexOfflineMutation({ + client: { mutation: send } as unknown as Pick, + mutation, + getArgs: () => ({ text: `hello` }), + collections: [{ utils: { awaitSync: synced } }], + }) + let settled = false + const pending = persist(params).then(() => { + settled = true + }) + await vi.waitFor(() => expect(synced).toHaveBeenCalledOnce()) + expect(settled).toBe(false) + expect(send).toHaveBeenCalledWith(mutation, { + text: `hello`, + idempotencyKey: `durable-id`, + }) + accepted() + await pending + expect(settled).toBe(true) +}) + +it(`rolls back rejected server functions without durable retry`, async () => { + const persist = convexOfflineMutation({ + client: { + mutation: () => Promise.reject(new Error(`permission denied`)), + } as unknown as Pick, + mutation, + getArgs: () => ({ text: `hello` }), + collections: [], + }) + await expect(persist(params)).rejects.toBeInstanceOf(NonRetriableError) +}) + +it.each([`idle`, `pending`, `failed`] as const)( + `requires a real source snapshot during %s replay`, + async (state) => { + const transport = controlledClient() + const startupError = new Error(`startup failed`) + const collection = createCollection( + convexCollectionOptions({ + client: + state === `failed` + ? { + onUpdate: () => { + throw startupError + }, + } + : transport.client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + gcTime: 0, + }), + ) + if (state !== `idle`) collection.startSyncImmediate() + const persist = convexOfflineMutation({ + client: { mutation: () => Promise.resolve(null) } as unknown as Pick< + ConvexClient, + `mutation` + >, + mutation, + getArgs: () => ({ text: `hello` }), + collections: [collection], + }) + let outcome: unknown = `pending` + const pending = persist(params).then( + () => { + outcome = `fulfilled` + }, + (error) => { + outcome = error + }, + ) + await new Promise((resolve) => setTimeout(resolve, 0)) + try { + if (state === `failed`) { + expect(outcome).toBe(startupError) + } else { + expect(outcome).toBe(`pending`) + expect(transport.counts().subscriptions).toBe(1) + transport.emit([]) + await pending + expect(outcome).toBe(`fulfilled`) + } + } finally { + await collection.cleanup() + } + }, +) + +it(`rejects replay when a wrapping sync rejects source acceptance`, async () => { + const transport = controlledClient() + const cause = new Error(`durable sync failed`) + let rejectAcceptance!: (error: unknown) => void + const accepted = new Promise((_resolve, reject) => { + rejectAcceptance = reject + }) + const options = convexCollectionOptions({ + client: transport.client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + gcTime: 0, + }) + const collection = createCollection({ + ...options, + sync: { + ...options.sync, + sync: (syncParams) => + options.sync.sync({ + ...syncParams, + commit: () => { + const visible = syncParams.commit() + return withAcceptedReceipt( + Promise.resolve(visible).then(() => undefined), + accepted, + ) + }, + }), + }, + }) + const persist = convexOfflineMutation({ + client: { mutation: () => Promise.resolve(null) } as unknown as Pick< + ConvexClient, + `mutation` + >, + mutation, + getArgs: () => ({ text: `hello` }), + collections: [collection], + }) + const pending = persist(params) + const rejected = expect(pending).rejects.toBe(cause) + await new Promise((resolve) => setTimeout(resolve, 0)) + transport.emit([]) + rejectAcceptance(cause) + try { + await rejected + } finally { + await collection.cleanup() + } +}) + +it(`requires a new sync run after durable acceptance fails`, async () => { + const transport = controlledClient() + const cause = new Error(`durable source failure`) + let rejectAcceptance!: (error: unknown) => void + let durable = new Promise((_resolve, reject) => { + rejectAcceptance = reject + }) + const options = convexCollectionOptions({ + client: transport.client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + gcTime: 0, + }) + const collection = createCollection({ + ...options, + sync: { + ...options.sync, + sync: (syncParams) => + options.sync.sync({ + ...syncParams, + commit: () => { + const visible = durable + .then(async () => { + await syncParams.commit() + }) + .then(() => undefined) + return withAcceptedReceipt(visible, durable) + }, + }), + }, + }) + const waiting = collection.utils.awaitSync() + const rejected = expect(waiting).rejects.toBe(cause) + transport.emit([]) + rejectAcceptance(cause) + await rejected + try { + await expect(collection.utils.awaitSync()).rejects.toBe(cause) + transport.emit([{ _id: `a`, text: `later callback` }]) + await expect(collection.utils.awaitSync()).rejects.toBe(cause) + await collection.cleanup() + durable = Promise.resolve() + collection.startSyncImmediate() + await collection.utils.awaitSync() + expect(collection.get(`a`)?.text).toBe(`later callback`) + } finally { + await collection.cleanup() + } +}) diff --git a/packages/convex-db-collection/tests/ownership-oracle.property.test.ts b/packages/convex-db-collection/tests/ownership-oracle.property.test.ts new file mode 100644 index 0000000000..9deae66edf --- /dev/null +++ b/packages/convex-db-collection/tests/ownership-oracle.property.test.ts @@ -0,0 +1,109 @@ +/** + * Concrete adapter options own one Collection, following the existing + * LocalStorage options claim policy. Cleanup ends a sync run, not the options + * owner's identity. A descriptor materializes fresh options for each DbClient. + * This boundary prevents a second Collection from stealing cached mutation + * confirmation or error state from the first. + * + * The model judges identity alone: concrete and shallow-copy attempts reuse + * one owner, while fresh options and separate DbClients have separate owners. + * Generated histories cross reuse form with before-sync, ready, and cleanup + * checkpoints. At admission, a reused owner must reject. Accepted independent + * owners must each settle a mutation against the latest cached provider rows. + * The controlled provider holds callbacks after changing its cache, which is + * the SDK's documented cached-listener window. Real SDK timing is separately + * received in sdk.test.ts. This owner does not model arbitrary DB lifecycle. + */ +import { DbClient, collectionOptions, createCollection } from '@tanstack/db' +import { fc } from '@fast-check/vitest' +import { expect, it } from 'vitest' +import { convexCollectionOptions } from '../src/index' +import { controlledClient, query, withTestCleanup } from './helpers' + +it(`keeps concrete options ownership across generated lifecycle checkpoints`, async () => { + const property = fc.asyncProperty( + fc.constantFrom(`direct`, `shallow`, `fresh`, `descriptor`), + fc.constantFrom(`idle`, `ready`, `cleaned-up`), + fc.string({ maxLength: 16 }), + async (form, checkpoint, text) => { + const transport = controlledClient() + transport.cache([{ _id: `a`, text: `old` }]) + const makeOptions = () => + convexCollectionOptions({ + id: `review-owned-tasks`, + client: transport.client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + gcTime: 0, + onUpdate: () => Promise.resolve(), + }) + const options = makeOptions() + const descriptor = collectionOptions(options) + const first = + form === `descriptor` + ? new DbClient().collection(descriptor) + : createCollection(options) + const collections = [first] + await withTestCleanup( + async () => { + if (checkpoint !== `idle`) await first.preload() + if (checkpoint === `cleaned-up`) await first.cleanup() + const reusedOwner = form === `direct` || form === `shallow` + const construct = () => + form === `descriptor` + ? new DbClient().collection(descriptor) + : createCollection( + form === `fresh` + ? makeOptions() + : form === `shallow` + ? { ...options } + : options, + ) + if (reusedOwner) { + expect(construct).toThrow(`only one Collection`) + } else { + const second = construct() + collections.push(second) + await second.preload() + transport.cache([{ _id: `a`, text }]) + const transaction = second.update(`a`, (row) => { + row.text = `optimistic` + }) + await transaction.isPersisted.promise + expect(second.get(`a`)?.text).toBe(text) + } + // Failed reuse and sibling cleanup cannot revoke the rightful owner. + for (const collection of collections.slice(1)) + await collection.cleanup() + await first.preload() + transport.cache([{ _id: `a`, text: `owner:${text}` }]) + const transaction = first.update(`a`, (row) => { + row.text = `optimistic` + }) + await transaction.isPersisted.promise + expect(first.get(`a`)?.text).toBe(`owner:${text}`) + }, + async () => { + // The second owner can be admitted inside the body. Enumerate all + // resources at cleanup time, then attempt every release. + await withTestCleanup( + () => {}, + ...collections.map((collection) => () => collection.cleanup()), + ) + }, + ) + }, + ) + const seed = process.env.CONVEX_OWNERSHIP_ORACLE_SEED + if (seed) + await fc.assert(property, { + seed: Number(seed), + path: process.env.CONVEX_OWNERSHIP_ORACLE_PATH, + numRuns: 60, + }) + else { + await fc.assert(property, { seed: 20261022, numRuns: 60 }) + await fc.assert(property, { numRuns: 60 }) + } +}) diff --git a/packages/convex-db-collection/tests/pagination-oracle.property.test.ts b/packages/convex-db-collection/tests/pagination-oracle.property.test.ts new file mode 100644 index 0000000000..7792e69948 --- /dev/null +++ b/packages/convex-db-collection/tests/pagination-oracle.property.test.ts @@ -0,0 +1,444 @@ +/** + * Automatic pagination owes the same complete snapshot as an unpaginated + * query. Convex's documented PaginationOptions fixes each range with + * endCursor; SplitRequired means the current page is incomplete and cannot + * be published. Our model is the latest input row array in rank order. The select + * boundary observes page concatenation order; Collection rows are also compared + * by key because Collection iteration order is not query order. The model does + * not compute cursors or adapter sync operations. + * + * The provider fixture below answers mathematical rank intervals, rather than + * following the adapter's page state. Histories replace up to eight distinct + * keyed rows, moving their ranks and adding/removing keys. Page sizes 1-4, + * forward/reverse delivery, and required/recommended/no splits vary separately. + * A pump delivers every active subscription once per source revision, including + * newly acquired ranges, before the exhaustion checkpoint. It never supplies + * an endCursor unless production requested it. The real-backend receiving + * witness in e2e/backend-oracle.test.ts checks these range premises with Convex. + * + * Limits: finite exhausted queries and one Collection. Continuous unbounded + * growth and atomic publication across separate Collections are not promised. + */ +import fc from 'fast-check' +import { createCollection } from '@tanstack/db' +import { makeFunctionReference } from 'convex/server' +import { expect, it } from 'vitest' +import { convexPaginatedCollectionOptions } from '../src/index' +import { withTestCleanup } from './helpers' +import type { PaginationOptions, PaginationResult } from 'convex/server' +import type { ConvexCollectionClient } from '../src/index' + +type Row = { id: string; rank: number; text: string } +const query = makeFunctionReference< + `query`, + { paginationOpts: PaginationOptions }, + PaginationResult +>(`rows:list`) +type Slot = { + options: PaginationOptions + current?: PaginationResult + callback: (value: PaginationResult) => void + active: boolean +} +function provider( + reverse: boolean, + split: `none` | `required` | `recommended`, +) { + const slots: Array = [] + let rowVisits = 0 + const client = { + onUpdate: ( + _query: unknown, + args: { paginationOpts: PaginationOptions }, + callback: Slot[`callback`], + ) => { + const slot: Slot = { + options: args.paginationOpts, + callback, + active: true, + } + slots.push(slot) + return Object.assign( + () => { + slot.active = false + }, + { getCurrentValue: () => slot.current }, + ) + }, + } as ConvexCollectionClient + return { + client, + slots, + get rowVisits() { + return rowVisits + }, + publish: async (rows: Array, checkpoint?: () => void) => { + const served = new Set() + for (let turn = 0; turn < 100; turn++) { + const pending = slots.filter((slot) => slot.active && !served.has(slot)) + if (reverse) pending.reverse() + for (const slot of pending) { + served.add(slot) + const { cursor, endCursor, numItems } = slot.options + const from = cursor === null ? -Infinity : Number(cursor) + const to = endCursor == null ? Infinity : Number(endCursor) + const interval = rows + .filter((row) => row.rank > from && row.rank <= to) + .sort((a, b) => a.rank - b.rank) + const page = + endCursor == null ? interval.slice(0, numItems) : interval + const boundary = endCursor ?? String(page.at(-1)?.rank ?? from) + // A split belongs strictly inside the returned cursor range. For an + // unbounded query that range ends at the returned page's boundary, + // not at the end of all rows matching the query. + const needsSplit = split !== `none` && page.length > 2 + const splitCursor = needsSplit + ? String(page[Math.floor((page.length - 1) / 2)]!.rank) + : undefined + if (splitCursor !== undefined) { + expect(Number(splitCursor)).toBeGreaterThan(from) + expect(Number(splitCursor)).toBeLessThan(Number(boundary)) + } + slot.current = { + page: new Proxy(needsSplit && split === `required` ? [] : page, { + get(target, key, receiver) { + if (typeof key === `string` && /^\d+$/.test(key)) rowVisits++ + return Reflect.get(target, key, receiver) + }, + }), + continueCursor: boundary, + isDone: endCursor == null && page.length === interval.length, + ...(needsSplit + ? { + pageStatus: + split === `required` + ? (`SplitRequired` as const) + : (`SplitRecommended` as const), + splitCursor, + } + : {}), + } + slot.callback(slot.current) + } + await Promise.resolve() + if (slots.some((slot) => slot.active && slot.current === undefined)) { + // Drain the acquisition's scheduled flush before checking. Otherwise + // the fixture could fill its new subscriptions before production ever + // reaches the incomplete-range path, hiding partial publication. + await Promise.resolve() + checkpoint?.() + } + if (slots.every((slot) => !slot.active || served.has(slot))) return + } + throw new Error(`Pagination did not exhaust finite fixture`) + }, + } +} + +// Shrinking must retain the law mismatch even when disposal also fails. +it(`retains the primary comparison when cleanup also fails`, async () => { + const primary = new Error(`row mismatch`) + const cleanup = new Error(`release mismatch`) + let lastReleased = false + await expect( + withTestCleanup( + () => Promise.reject(primary), + () => Promise.reject(cleanup), + () => { + lastReleased = true + }, + ), + ).rejects.toMatchObject({ cause: primary, errors: [primary, cleanup] }) + expect(lastReleased).toBe(true) +}) + +const snapshot = fc.shuffledSubarray([0, 1, 2, 3, 4, 5, 6, 7]).chain((ids) => + fc + .array(fc.string({ maxLength: 8 }), { + minLength: ids.length, + maxLength: ids.length, + }) + .map((texts) => + ids.map((id, rank): Row => ({ + id: String(id), + rank, + text: texts[rank]!, + })), + ), +) +const history = fc.record({ + snapshots: fc.array(snapshot, { minLength: 1, maxLength: 10 }), + pageSize: fc.integer({ min: 1, max: 4 }), + reverse: fc.boolean(), + split: fc.constantFrom( + `none` as const, + `required` as const, + `recommended` as const, + ), +}) +// Calibration on the legal provider grammar: reversing page order fails the +// select-boundary comparison at seed 20261022/path 0:1:0:2:3:1:1:1. Publishing +// an available prefix before a missing range fails the intermediate checkpoint +// at seed 20261022/path 0:1:3:1:2:5:7:3:3:0:1:1. Both are assertion failures +// after reaching the named observation, not setup errors or timeouts. +const property = fc.asyncProperty( + history, + async ({ snapshots, pageSize, reverse, split }) => { + const source = provider(reverse, split) + let selected: ReadonlyArray = [] + let accepted: ReadonlyArray = [] + let pendingCheckpoints = 0 + const collection = createCollection( + convexPaginatedCollectionOptions({ + client: source.client, + query, + queryArgs: {}, + pageSize, + select: (rows) => { + selected = rows + return rows + }, + getKey: (row) => row.id, + gcTime: 0, + }), + ) + collection.startSyncImmediate() + await withTestCleanup( + async () => { + for (const rows of snapshots) { + await source.publish(rows, () => { + pendingCheckpoints++ + // A newly acquired range has no result yet. The last complete + // snapshot, including its order at select, must remain authoritative. + expect(selected).toEqual(accepted) + expect( + [...collection.values()] + .map(({ id, rank, text }) => ({ id, rank, text })) + .sort((a, b) => a.id.localeCompare(b.id)), + ).toEqual([...accepted].sort((a, b) => a.id.localeCompare(b.id))) + }) + await collection.utils.awaitSync() + expect(selected).toEqual([...rows].sort((a, b) => a.rank - b.rank)) + accepted = [...rows].sort((a, b) => a.rank - b.rank) + const actual = [...collection.values()] + .map(({ id, rank, text }) => ({ id, rank, text })) + .sort((a, b) => a.id.localeCompare(b.id)) + const expected = [...rows].sort((a, b) => a.id.localeCompare(b.id)) + expect(actual).toEqual(expected) + expect(collection.status).toBe(`ready`) + } + if (snapshots[0]!.length > pageSize) + expect(pendingCheckpoints).toBeGreaterThan(0) + }, + async () => { + await collection.cleanup() + expect(source.slots.every((slot) => !slot.active)).toBe(true) + }, + ) + }, +) + +it(`refines exhausted snapshots across reactive ranges and page splits`, async () => { + const seed = process.env.CONVEX_PAGINATION_ROWS_SEED + const path = process.env.CONVEX_PAGINATION_ROWS_PATH + if (seed) await fc.assert(property, { seed: Number(seed), path, numRuns: 60 }) + else { + await fc.assert(property, { seed: 20261022, numRuns: 60 }) + await fc.assert(property, { numRuns: 60 }) + } +}) + +// A finite initial load must materialize its rows only after all ranges are +// available. Recopying every earlier prefix on each continuation makes initial +// acquisition quadratic. This work law follows the adapter's complete-snapshot +// design: incomplete rows cannot be published, so materializing them earns no +// useful output. Count numeric element reads at the provider boundary, rather +// than copying production's loop or predicting its number of subscriptions. +// This bounds row materialization, not cache lookups, RPC count, or elapsed time. +// Calibration: copying incomplete prefixes failed the row-visit comparison at +// seed 20261022/path 0:0:0:0, three rows at pageSize 1 read four elements. +// The same direct replay passes after deferring materialization until exhaustion. +const initialWork = fc.asyncProperty( + fc.integer({ min: 3, max: 40 }), + fc.integer({ min: 1, max: 8 }), + fc.boolean(), + async (pageCount, pageSize, reverse) => { + const rows = Array.from({ length: pageCount * pageSize }, (_, rank) => ({ + id: String(rank), + rank, + text: String(rank), + })) + const source = provider(reverse, `none`) + const collection = createCollection( + convexPaginatedCollectionOptions({ + client: source.client, + query, + queryArgs: {}, + pageSize, + getKey: (row) => row.id, + gcTime: 0, + }), + ) + collection.startSyncImmediate() + await withTestCleanup( + async () => { + await source.publish(rows) + // Check before awaitSync's explicit cache refreshes. At initial exhaustion + // there is exactly one useful materialization, the complete public result. + expect(collection.size).toBe(rows.length) + expect(source.rowVisits).toBe(rows.length) + }, + async () => { + await collection.cleanup() + }, + ) + }, +) +it(`materializes complete rows once during finite initial pagination`, async () => { + const seed = process.env.CONVEX_PAGINATION_WORK_SEED + const path = process.env.CONVEX_PAGINATION_WORK_PATH + if (seed) + await fc.assert(initialWork, { seed: Number(seed), path, numRuns: 60 }) + else { + await fc.assert(initialWork, { seed: 20261022, numRuns: 60 }) + await fc.assert(initialWork, { numRuns: 60 }) + } +}) + +// A configured maxRows bounds the complete eager mirror's accepted row count. +// It is not a transport-byte or strict peak-memory bound: one provider response +// can itself be larger. The reference accepts a finite snapshot iff its row +// count fits, otherwise preserving the prior public rows. Growing a previously +// exhausted source must reject before opening any continuation beyond that cap. +// Calibration: omitting the row cap fails at the public error observation, +// seed 20261023/path 0:0, maxRows 1 followed by a two-row source snapshot. +const rowBound = fc.asyncProperty( + fc.integer({ min: 1, max: 20 }), + fc.integer({ min: 1, max: 6 }), + fc.integer({ min: 1, max: 6 }), + async (maxRows, pageSize, excess) => { + const rows = Array.from({ length: maxRows + excess }, (_, rank) => ({ + id: String(rank), + rank, + text: String(rank), + })) + const source = provider(false, `none`) + const collection = createCollection( + convexPaginatedCollectionOptions({ + client: source.client, + query, + queryArgs: {}, + pageSize, + maxRows, + getKey: (row) => row.id, + gcTime: 0, + }), + ) + collection.startSyncImmediate() + await withTestCleanup( + async () => { + const accepted = rows.slice(0, maxRows) + await source.publish(accepted) + await collection.utils.awaitSync() + expect( + [...collection.values()] + .map(({ id, rank, text }) => ({ + id, + rank, + text, + })) + .sort((a, b) => a.rank - b.rank), + ).toEqual(accepted) + await source.publish(rows) + expect(collection.utils.getSyncError()).toBeInstanceOf(RangeError) + expect( + [...collection.values()] + .map(({ id, rank, text }) => ({ + id, + rank, + text, + })) + .sort((a, b) => a.rank - b.rank), + ).toEqual(accepted) + // One continuation at the cap may be necessary to distinguish an empty + // remainder from an extra row. No cursor may advance past that first + // excess row once its response establishes that the mirror is too large. + for (const slot of source.slots) { + if (slot.options.cursor !== null) + expect(Number(slot.options.cursor)).toBeLessThan(maxRows) + } + await expect(collection.utils.awaitSync()).rejects.toBeInstanceOf( + RangeError, + ) + // The row cap is a valid source error, not a terminal run failure. + // Removing the excess rows permits this same subscription run to recover. + await source.publish(accepted) + await collection.utils.awaitSync() + expect(collection.utils.getSyncError()).toBeUndefined() + expect(collection.size).toBe(maxRows) + }, + async () => { + await collection.cleanup() + }, + ) + }, +) +it(`rejects an oversized eager mirror before continuing and preserves accepted rows`, async () => { + const seed = process.env.CONVEX_PAGINATION_BOUND_SEED + const path = process.env.CONVEX_PAGINATION_BOUND_PATH + if (seed) await fc.assert(rowBound, { seed: Number(seed), path, numRuns: 60 }) + else { + await fc.assert(rowBound, { seed: 20261023, numRuns: 60 }) + await fc.assert(rowBound, { numRuns: 60 }) + } +}) + +it.each([0, -1, 1.5, Infinity, NaN])( + `rejects invalid maxRows %s`, + async (maxRows) => { + const source = provider(false, `none`) + const collection = createCollection( + convexPaginatedCollectionOptions({ + client: source.client, + query, + queryArgs: {}, + maxRows, + getKey: (row) => row.id, + }), + ) + try { + await expect(collection.preload()).rejects.toThrow(`maxRows`) + expect(source.slots).toHaveLength(0) + } finally { + await collection.cleanup() + } + }, +) + +it(`defaults the eager mirror cap to ten thousand rows`, async () => { + const source = provider(false, `none`) + const collection = createCollection( + convexPaginatedCollectionOptions({ + client: source.client, + query, + queryArgs: {}, + pageSize: 10_001, + getKey: (row) => row.id, + }), + ) + collection.startSyncImmediate() + try { + await source.publish( + Array.from({ length: 10_001 }, (_, rank) => ({ + id: String(rank), + rank, + text: String(rank), + })), + ) + expect(collection.utils.getSyncError()).toBeInstanceOf(RangeError) + expect(collection.size).toBe(0) + expect(source.slots).toHaveLength(1) + } finally { + await collection.cleanup() + } +}) diff --git a/packages/convex-db-collection/tests/sdk.test.ts b/packages/convex-db-collection/tests/sdk.test.ts new file mode 100644 index 0000000000..d24b77a551 --- /dev/null +++ b/packages/convex-db-collection/tests/sdk.test.ts @@ -0,0 +1,364 @@ +import { Buffer } from 'node:buffer' +import { createCollection } from '@tanstack/db' +import { ConvexClient } from 'convex/browser' +import { ConvexReactClient } from 'convex/react' +import { makeFunctionReference } from 'convex/server' +import { convexToJson } from 'convex/values' +import { describe, expect, it, vi } from 'vitest' +import { convexCollectionOptions } from '../src/index' +import { mutation, query, withTestCleanup } from './helpers' +import type { Value } from 'convex/values' + +// Only the transport is controlled. Both clients, their caches, subscriptions, +// mutation timestamp barriers, decoding, and the DB Collection are real. +class Socket { + static instances: Array = [] + onopen: (() => void) | null = null + onmessage: ((event: { data: string }) => void) | null = null + onclose: ((event: { code: number; reason: string }) => void) | null = null + onerror: (() => void) | null = null + sent: Array> = [] + constructor(_url: string) { + Socket.instances.push(this) + } + send(data: string) { + this.sent.push(JSON.parse(data) as Record) + } + close() { + queueMicrotask(() => this.onclose?.({ code: 1000, reason: `` })) + } + receive(message: unknown) { + this.onmessage?.({ data: JSON.stringify(message) }) + } + open() { + this.onopen?.() + } +} + +function timestamp(value: number): string { + const bytes = Buffer.alloc(8) + bytes.writeBigUInt64LE(BigInt(value)) + return bytes.toString(`base64`) +} + +function server(socket: Socket) { + let time = 0 + let querySet = 0 + return { + snapshot(rows: Value) { + const changes = socket.sent.filter((msg) => msg.type === `ModifyQuerySet`) + const version = changes.at(-1)?.newVersion as number + const modifications = changes.flatMap( + (msg) => msg.modifications as Array<{ type: string; queryId: number }>, + ) + const id = modifications.find((mod) => mod.type === `Add`)!.queryId + socket.receive({ + type: `Transition`, + startVersion: { querySet, ts: timestamp(time), identity: 0 }, + endVersion: { querySet: version, ts: timestamp(++time), identity: 0 }, + modifications: [ + { + type: `QueryUpdated`, + queryId: id, + value: convexToJson(rows), + logLines: [], + journal: null, + }, + ], + }) + querySet = version + }, + acknowledge(success = true) { + const request = [...socket.sent] + .reverse() + .find((msg) => msg.type === `Mutation`)! + socket.receive( + success + ? { + type: `MutationResponse`, + requestId: request.requestId, + success: true, + result: null, + ts: timestamp(time + 1), + logLines: [], + } + : { + type: `MutationResponse`, + requestId: request.requestId, + success: false, + result: `permission denied`, + logLines: [], + }, + ) + }, + } +} + +describe.each([`javascript`, `react`] as const)( + `real Convex %s client`, + (kind) => { + function setup() { + const options = { + webSocketConstructor: Socket as unknown as typeof WebSocket, + unsavedChangesWarning: false, + logger: false as const, + } + const client = + kind === `javascript` + ? new ConvexClient(`https://test.convex.cloud`, options) + : new ConvexReactClient(`https://test.convex.cloud`, options) + const collection = createCollection( + convexCollectionOptions({ + client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + gcTime: 0, + onUpdate: async ({ transaction }) => { + await client.mutation(mutation, { + text: transaction.mutations[0].modified.text, + }) + }, + }), + ) + collection.startSyncImmediate() + const socket = Socket.instances.at(-1)! + socket.open() + return { client, collection, socket, remote: server(socket) } + } + + // Receiving witness for the ownership oracle: the actual SDK deduplicates + // two listeners, and releasing either owner leaves the other live. + it.each([`first`, `second`] as const)( + `keeps sibling updates after cleaning up %s listener`, + async (retired) => { + const { client, collection: first, remote } = setup() + remote.snapshot([{ _id: `a`, text: `old` }]) + await first.preload() + const second = createCollection( + convexCollectionOptions({ + client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + select: (rows) => + rows.map((row) => ({ ...row, text: row.text.toUpperCase() })), + gcTime: 0, + }), + ) + await withTestCleanup( + async () => { + await second.preload() + expect(second.get(`a`)?.text).toBe(`OLD`) + const survivor = retired === `first` ? second : first + const removed = retired === `first` ? first : second + await removed.cleanup() + remote.snapshot([{ _id: `a`, text: `updated` }]) + expect(survivor.get(`a`)?.text).toBe( + retired === `first` ? `UPDATED` : `updated`, + ) + expect(removed.status).toBe(`cleaned-up`) + }, + () => first.cleanup(), + () => second.cleanup(), + () => client.close(), + ) + }, + ) + + it(`subscribes with generated function names and typed args`, async () => { + const { client, collection, socket, remote } = setup() + await withTestCleanup( + async () => { + const subscription = socket.sent.find( + (msg) => msg.type === `ModifyQuerySet`, + )! + expect(subscription.modifications).toMatchObject([ + { type: `Add`, udfPath: `tasks:list`, args: [{ owner: `alice` }] }, + ]) + remote.snapshot([{ _id: `a`, text: `hello` }]) + await collection.preload() + expect(collection.get(`a`)?.text).toBe(`hello`) + remote.snapshot([]) + expect(collection.size).toBe(0) + await collection.cleanup() + expect(socket.sent.at(-1)).toMatchObject({ + type: `ModifyQuerySet`, + modifications: [{ type: `Remove` }], + }) + }, + () => collection.cleanup(), + () => client.close(), + ) + }) + + it(`waits for a query transition past the mutation timestamp before settling`, async () => { + const { client, collection, remote } = setup() + await withTestCleanup( + async () => { + remote.snapshot([{ _id: `a`, text: `old` }]) + await collection.preload() + const seen: Array = [] + const subscription = collection.subscribeChanges(() => { + seen.push(collection.get(`a`)?.text) + }) + const transaction = collection.update(`a`, (row) => { + row.text = `optimistic` + }) + remote.acknowledge() + await Promise.resolve() + expect(transaction.state).toBe(`persisting`) + expect(collection.get(`a`)?.text).toBe(`optimistic`) + remote.snapshot([{ _id: `a`, text: `server` }]) + await transaction.isPersisted.promise + expect(collection.get(`a`)?.text).toBe(`server`) + expect(seen).not.toContain(`old`) + subscription.unsubscribe() + }, + () => collection.cleanup(), + () => client.close(), + ) + }) + + it(`resubscribes and replays an in-flight mutation after reconnect`, async () => { + vi.useFakeTimers() + const { client, collection, socket, remote } = setup() + await withTestCleanup( + async () => { + remote.snapshot([{ _id: `a`, text: `old` }]) + await collection.preload() + const transaction = collection.update(`a`, (row) => { + row.text = `optimistic` + }) + socket.onclose?.({ code: 1006, reason: `network lost` }) + await vi.advanceTimersByTimeAsync(3000) + const replacement = Socket.instances.at(-1)! + expect(replacement).not.toBe(socket) + replacement.open() + expect( + replacement.sent.some((message) => message.type === `Mutation`), + ).toBe(true) + expect(collection.get(`a`)?.text).toBe(`optimistic`) + const nextRemote = server(replacement) + nextRemote.acknowledge() + nextRemote.snapshot([{ _id: `a`, text: `reconnected` }]) + await transaction.isPersisted.promise + expect(collection.get(`a`)?.text).toBe(`reconnected`) + }, + () => collection.cleanup(), + async () => { + // close() uses a microtask, which fake timers otherwise hold. + const closed = client.close() + await vi.runAllTimersAsync() + await closed + }, + () => { + vi.useRealTimers() + }, + ) + }) + + it(`rolls back on an actual Convex mutation failure`, async () => { + const { client, collection, remote } = setup() + await withTestCleanup( + async () => { + remote.snapshot([{ _id: `a`, text: `old` }]) + await collection.preload() + const transaction = collection.update(`a`, (row) => { + row.text = `optimistic` + }) + remote.acknowledge(false) + await expect(transaction.isPersisted.promise).rejects.toThrow( + `permission denied`, + ) + expect(collection.get(`a`)?.text).toBe(`old`) + }, + () => collection.cleanup(), + () => client.close(), + ) + }) + }, +) + +it(`preserves Convex integers, special floats, nested values, and bytes`, async () => { + const client = new ConvexClient(`https://test.convex.cloud`, { + webSocketConstructor: Socket as unknown as typeof WebSocket, + unsavedChangesWarning: false, + logger: false, + }) + const valuesQuery = makeFunctionReference< + `query`, + {}, + Array<{ _id: string; payload: Value }> + >(`values:list`) + const collection = createCollection( + convexCollectionOptions({ + client, + query: valuesQuery, + queryArgs: {}, + getKey: (row) => row._id, + gcTime: 0, + }), + ) + collection.startSyncImmediate() + const socket = Socket.instances.at(-1)! + socket.open() + const remote = server(socket) + await withTestCleanup( + async () => { + const values: Array = [ + 9223372036854775807n, + NaN, + Infinity, + -Infinity, + -0, + { nested: [null, true, `hello`, { key: `value` }] }, + new Uint8Array([1, 2]).buffer, + ] + for (const payload of values) { + remote.snapshot([{ _id: `a`, payload }]) + await collection.preload() + const actual = collection.get(`a`)!.payload + if (payload instanceof ArrayBuffer) + expect([...new Uint8Array(actual as ArrayBuffer)]).toEqual([ + ...new Uint8Array(payload), + ]) + else expect(actual).toEqual(payload) + } + const changed = vi.fn() + const listener = collection.subscribeChanges(changed) + remote.snapshot([{ _id: `a`, payload: new Uint8Array([3, 4]).buffer }]) + expect(changed).toHaveBeenCalledOnce() + listener.unsubscribe() + expect([ + ...new Uint8Array(collection.get(`a`)!.payload as ArrayBuffer), + ]).toEqual([3, 4]) + }, + () => collection.cleanup(), + () => client.close(), + ) +}) + +it(`rejects preload for a disabled JavaScript client`, async () => { + const client = new ConvexClient(`https://test.convex.cloud`, { + disabled: true, + }) + const collection = createCollection( + convexCollectionOptions({ + client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + gcTime: 0, + }), + ) + await withTestCleanup( + async () => { + await expect(collection.preload()).rejects.toThrow( + `disabled Convex client`, + ) + }, + () => collection.cleanup(), + () => client.close(), + ) +}) diff --git a/packages/convex-db-collection/tests/snapshot-oracle.property.test.ts b/packages/convex-db-collection/tests/snapshot-oracle.property.test.ts new file mode 100644 index 0000000000..dd6d92d1c7 --- /dev/null +++ b/packages/convex-db-collection/tests/snapshot-oracle.property.test.ts @@ -0,0 +1,113 @@ +/** + * A Collection mirrors the complete keyed rows of its latest valid Convex + * query result, as promised by convexCollectionOptions. An omitted key is a + * deletion, and an omitted field is absent, rather than retained from an older + * row. Duplicate keys reject the whole result and leave the last valid result. + * + * The reference is simply the last valid input array. It never computes sync + * operations or uses the adapter's key map. Generated histories contain empty, + * reordered, changed, removed, reintroduced, and duplicate-key snapshots. + * Every history runs both raw rows and a successful text projection. The + * reference labels selected text with an independently specified uppercase + * operation, so omitting select cannot pass merely by preserving row keys. + * The driver invokes the real Collection through a controlled Convex callback. + * After each synchronous delivery, public rows and the error flag are compared. + * This owner covers snapshot refinement only. SDK transport, mutation timing, + * errors, and cleanup have separate focused receiving tests in this package. + */ +import { isDeepStrictEqual } from 'node:util' +import { createCollection } from '@tanstack/db' +import { fc } from '@fast-check/vitest' +import { expect, it } from 'vitest' +import { convexCollectionOptions } from '../src/index' +import { controlledClient, query, withTestCleanup } from './helpers' +import type { Row } from './helpers' + +const row = fc + .record({ + _id: fc.constantFrom(`a`, `b`, `c`), + text: fc.string({ maxLength: 12 }), + optional: fc.option(fc.string({ maxLength: 6 }), { nil: undefined }), + }) + .map((value): Row => + value.optional === undefined ? { _id: value._id, text: value.text } : value, + ) + +it(`refines complete snapshots throughout generated histories`, async () => { + const property = fc.asyncProperty( + fc.array(fc.array(row, { maxLength: 5 }), { + minLength: 1, + maxLength: 30, + }), + async (history) => { + for (const projected of [false, true]) { + const transport = controlledClient() + const collection = createCollection( + convexCollectionOptions({ + client: transport.client, + query, + queryArgs: { owner: `alice` }, + getKey: (value) => value._id, + ...(projected + ? { + select: (rows: Array) => + rows.map((value) => ({ + ...value, + text: value.text.toUpperCase(), + })), + } + : {}), + gcTime: 0, + }), + ) + collection.startSyncImmediate() + let expected: Array = [] + await withTestCleanup( + () => { + for (const snapshot of history) { + // Independent validity judgment: compare every pair of input keys. + const valid = snapshot.every((value, index) => + snapshot + .slice(index + 1) + .every((other) => other._id !== value._id), + ) + if (valid) + expected = snapshot.map((value) => ({ + ...value, + text: projected ? value.text.toUpperCase() : value.text, + })) + transport.emit(snapshot) + expect(collection.utils.getSyncError() !== undefined).toBe(!valid) + const actual = [...collection.values()] + .map((value): Row => { + const result: Row = { _id: value._id, text: value.text } + if (Object.hasOwn(value, `optional`)) + result.optional = value.optional + return result + }) + .sort((a, b) => a._id.localeCompare(b._id)) + const wanted = [...expected].sort((a, b) => + a._id.localeCompare(b._id), + ) + expect(isDeepStrictEqual(actual, wanted)).toBe(true) + } + }, + async () => { + await collection.cleanup() + }, + ) + } + }, + ) + const seed = process.env.CONVEX_SNAPSHOT_ORACLE_SEED + if (seed) + await fc.assert(property, { + seed: Number(seed), + path: process.env.CONVEX_SNAPSHOT_ORACLE_PATH, + numRuns: 150, + }) + else { + await fc.assert(property, { numRuns: 150, seed: 20261022 }) + await fc.assert(property, { numRuns: 150 }) + } +}) diff --git a/packages/convex-db-collection/tests/types.test-d.ts b/packages/convex-db-collection/tests/types.test-d.ts new file mode 100644 index 0000000000..98c7258abd --- /dev/null +++ b/packages/convex-db-collection/tests/types.test-d.ts @@ -0,0 +1,154 @@ +import { createCollection } from '@tanstack/db' +import { makeFunctionReference } from 'convex/server' +import { expectTypeOf, it } from 'vitest' +import { z } from 'zod' +import { convexOfflineMutation } from '../src/offline' +import { + convexCollectionOptions, + convexPaginatedCollectionOptions, +} from '../src/index' +import { query } from './helpers' +import type { PaginationOptions, PaginationResult } from 'convex/server' +import type { ConvexReactClient } from 'convex/react' +import type { ConvexClient } from 'convex/browser' +import type { Row } from './helpers' + +it(`infers rows, arguments, keys, projections, and schemas`, () => { + const client = {} as ConvexClient + const reactClient = {} as ConvexReactClient + const options = convexCollectionOptions({ + client, + query, + queryArgs: { owner: `alice` }, + getKey: (row) => row._id, + onUpdate: ({ transaction, collection }) => { + expectTypeOf(transaction.mutations[0].modified).toEqualTypeOf() + expectTypeOf(collection.utils.getSyncError()).toEqualTypeOf() + return Promise.resolve() + }, + }) + const collection = createCollection(options) + expectTypeOf(collection.get(`id`)!.text).toEqualTypeOf() + convexCollectionOptions({ + client, + query, + // @ts-expect-error Query arguments are inferred from the function reference. + queryArgs: { owner: 1 }, + getKey: (row: Row) => row._id, + }) + // @ts-expect-error Required query arguments cannot be omitted. + convexCollectionOptions({ client, query, getKey: (row: Row) => row._id }) + convexCollectionOptions({ + client, + query, + queryArgs: { owner: `alice` }, + // @ts-expect-error A misspelled document field cannot be used as a key. + getKey: (row) => row.missing, + }) + + const envelope = makeFunctionReference<`query`, {}, { rows: Array }>( + `tasks:envelope`, + ) + const selected = convexCollectionOptions({ + client: reactClient, + query: envelope, + queryArgs: {}, + select: (result) => + result.rows.map((row) => ({ id: row._id, length: row.text.length })), + getKey: (row) => row.id, + }) + expectTypeOf( + createCollection(selected).get(`id`)!.length, + ).toEqualTypeOf() + // @ts-expect-error Non-array query results need a projection. + convexCollectionOptions({ + client, + query: envelope, + queryArgs: {}, + getKey: (row: Row) => row._id, + }) + + const schema = z.object({ + _id: z.string(), + text: z.string(), + optional: z.string().optional(), + }) + const validated = createCollection( + convexCollectionOptions({ + client, + query, + queryArgs: { owner: `alice` }, + schema, + getKey: (row) => row._id, + }), + ) + expectTypeOf(validated.get(`id`)!.text).toEqualTypeOf() +}) + +it(`infers paginated arguments and projected row types`, () => { + const paginatedQuery = makeFunctionReference< + `query`, + { owner: string; paginationOpts: PaginationOptions }, + PaginationResult + >(`tasks:pages`) + const client = {} as ConvexClient + const options = convexPaginatedCollectionOptions({ + client, + query: paginatedQuery, + queryArgs: { owner: `alice` }, + select: (rows) => + rows.map((row) => ({ id: row._id, length: row.text.length })), + getKey: (row) => row.id, + }) + expectTypeOf( + createCollection(options).get(`a`)!.length, + ).toEqualTypeOf() + convexPaginatedCollectionOptions({ + client, + query: paginatedQuery, + queryArgs: { + owner: `alice`, + // @ts-expect-error The adapter supplies paginationOpts, callers do not supply a cursor. + paginationOpts: { cursor: null, numItems: 4 }, + }, + getKey: (row: Row) => row._id, + }) + const invalid = makeFunctionReference<`mutation`, { text: string }, null>( + `tasks:invalid`, + ) + // @ts-expect-error Durable mutations must declare an idempotencyKey argument. + convexOfflineMutation({ + client, + mutation: invalid, + getArgs: () => ({ text: `hello` }), + collections: [], + }) +}) + +it(`requires a projection for changed paginated row types and server pagination arguments`, () => { + const client = {} as ConvexClient + const paginated = makeFunctionReference< + `query`, + { paginationOpts: PaginationOptions }, + PaginationResult + >(`tasks:pages`) + // @ts-expect-error A changed row type needs a select projection. + convexPaginatedCollectionOptions({ + client, + query: paginated, + queryArgs: {}, + getKey: (row: { id: number }) => row.id, + }) + const missingArguments = makeFunctionReference< + `query`, + {}, + PaginationResult + >(`tasks:invalidPages`) + // @ts-expect-error A paginated server query must accept paginationOpts. + convexPaginatedCollectionOptions({ + client, + query: missingArguments, + queryArgs: {}, + getKey: (row: Row) => row._id, + }) +}) diff --git a/packages/convex-db-collection/tests/values-oracle.property.test.ts b/packages/convex-db-collection/tests/values-oracle.property.test.ts new file mode 100644 index 0000000000..08f9ab4885 --- /dev/null +++ b/packages/convex-db-collection/tests/values-oracle.property.test.ts @@ -0,0 +1,165 @@ +/** + * Convex values must keep their meaning through the durable outbox, including + * integers, binary data, special floats, and removed (undefined) fields. The + * reference is the authored native tree. An observation projection labels + * scalar types and bytes, so JSON normalization cannot pass as equality. + * + * Bounded trees combine native leaves, arrays, ordinary fields, and marker-like + * user data. A real OutboxManager writes each transaction, then a fresh manager + * and Collection restore it with convexMutationCodec. Comparison happens on all + * three restored mutation payloads, before any provider call. The core serializer + * owner covers wire dispatch and unsupported codecs; this receives that contract + * with Convex's actual public scalar encoding. The backend process oracle then + * receives these values on disk and across killed writer/replay processes. + */ +import fc from 'fast-check' +import { createCollection } from '@tanstack/db' +import { OutboxManager } from '@tanstack/offline-transactions' +import { expect, it } from 'vitest' +import { convexMutationCodec } from '../src/offline' +import { cleanupOfflineOracle } from '../../offline-transactions/tests/oracle-lifecycle' +import type { PendingMutation } from '@tanstack/db' +import type { + OfflineTransaction, + StorageAdapter, +} from '@tanstack/offline-transactions' + +type Tree = + | undefined + | null + | boolean + | number + | bigint + | string + | ArrayBuffer + | Array + | { [key: string]: Tree } +const leaf: fc.Arbitrary = fc.oneof( + fc.constant(undefined), + fc.constant(null), + fc.boolean(), + fc.string({ maxLength: 8 }), + fc.double(), + fc.bigInt({ min: -(2n ** 63n), max: 2n ** 63n - 1n }), + fc.uint8Array({ maxLength: 8 }).map((value) => value.buffer as ArrayBuffer), +) +function tree(depth: number): fc.Arbitrary { + if (!depth) return leaf + const child = tree(depth - 1) + return fc.oneof( + leaf, + fc.array(child, { maxLength: 3 }), + fc.dictionary( + fc.constantFrom( + `__proto__`, + `__type`, + `value`, + `array`, + `scalar`, + `removed`, + ), + child, + { maxKeys: 3 }, + ), + ) +} +function observe(value: unknown): unknown { + if (value instanceof ArrayBuffer) return [`bytes`, [...new Uint8Array(value)]] + if (Array.isArray(value)) return [`array`, value.map(observe)] + if (value && typeof value === `object`) + return [ + `object`, + Object.entries(value) + .map(([key, field]) => [key, observe(field)]) + .sort((a, b) => String(a[0]).localeCompare(String(b[0]))), + ] + if (typeof value === `number`) + return [`number`, Object.is(value, -0) ? `-0` : String(value)] + return [typeof value, String(value)] +} +const property = fc.asyncProperty(tree(2), async (payload) => { + type Row = { id: string; payload: Tree } + const make = (id: string) => + createCollection({ + id, + getKey: (row) => row.id, + sync: { sync: ({ markReady }) => markReady() }, + }) + const writer = make(`writer`) + const reader = make(`reader`) + const data = new Map() + const storage: StorageAdapter = { + get: (key) => Promise.resolve(data.get(key) ?? null), + set: (key, value) => { + data.set(key, value) + return Promise.resolve() + }, + delete: (key) => { + data.delete(key) + return Promise.resolve() + }, + keys: () => Promise.resolve([...data.keys()]), + clear: () => { + data.clear() + return Promise.resolve() + }, + } + const row: Row = { id: `one`, payload } + const transaction: OfflineTransaction = { + id: `durable`, + mutationFnName: `save`, + idempotencyKey: `same`, + createdAt: new Date(0), + retryCount: 0, + nextAttemptAt: 0, + version: 1, + keys: [`one`], + mutations: [ + { + globalKey: `one`, + type: `update`, + modified: row, + original: row, + changes: { payload }, + collection: writer, + } as unknown as PendingMutation, + ], + } + let hasPrimaryFailure = false + try { + await new OutboxManager(storage, { rows: writer }, convexMutationCodec).add( + transaction, + ) + const restored = await new OutboxManager( + storage, + { rows: reader }, + convexMutationCodec, + ).get(`durable`) + expect(observe(restored!.mutations[0]!.modified)).toEqual(observe(row)) + expect(observe(restored!.mutations[0]!.original)).toEqual(observe(row)) + expect(observe(restored!.mutations[0]!.changes)).toEqual( + observe({ payload }), + ) + } catch (error) { + hasPrimaryFailure = true + throw error + } finally { + await cleanupOfflineOracle( + [() => writer.cleanup(), () => reader.cleanup()], + hasPrimaryFailure, + ) + } +}) +it(`preserves the Convex mutation value domain across durable serialization`, async () => { + const seed = process.env.CONVEX_VALUES_ORACLE_SEED + if (seed) + await fc.assert(property, { + seed: Number(seed), + path: process.env.CONVEX_VALUES_ORACLE_PATH, + numRuns: 100, + }) + else { + await fc.assert(property, { seed: 20261022, numRuns: 100 }) + await fc.assert(property, { numRuns: 100 }) + } +}) diff --git a/packages/convex-db-collection/tsconfig.docs.json b/packages/convex-db-collection/tsconfig.docs.json new file mode 100644 index 0000000000..b90fc83e04 --- /dev/null +++ b/packages/convex-db-collection/tsconfig.docs.json @@ -0,0 +1,4 @@ +{ + "extends": "./tsconfig.json", + "include": ["src"] +} diff --git a/packages/convex-db-collection/tsconfig.json b/packages/convex-db-collection/tsconfig.json new file mode 100644 index 0000000000..02471eaff8 --- /dev/null +++ b/packages/convex-db-collection/tsconfig.json @@ -0,0 +1,29 @@ +{ + "extends": "../../tsconfig.json", + "compilerOptions": { + "target": "ES2020", + "module": "ESNext", + "moduleResolution": "Bundler", + "declaration": true, + "outDir": "dist", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "jsx": "react", + "paths": { + "@tanstack/db": ["../db/src"], + "@tanstack/db-ivm": ["../db-ivm/src"], + "@tanstack/offline-transactions": ["../offline-transactions/src"] + } + }, + "include": [ + "src", + "tests", + "e2e/*.ts", + "e2e/auth-app/*.ts", + "vite.config.ts", + "vitest.e2e.config.ts" + ], + "exclude": ["node_modules", "dist"] +} diff --git a/packages/convex-db-collection/vite.config.ts b/packages/convex-db-collection/vite.config.ts new file mode 100644 index 0000000000..abd0709e33 --- /dev/null +++ b/packages/convex-db-collection/vite.config.ts @@ -0,0 +1,23 @@ +import { defineConfig, mergeConfig } from 'vitest/config' +import { tanstackViteConfig } from '@tanstack/vite-config' +import { commonJsDeclarations } from '../../scripts/commonjs-declarations.mjs' +import packageJson from './package.json' + +const config = defineConfig({ + test: { + name: packageJson.name, + include: [`tests/**/*.test.ts`], + environment: `jsdom`, + coverage: { enabled: true, provider: `istanbul`, include: [`src/**/*`] }, + typecheck: { enabled: true, include: [`tests/**/*.test-d.ts`] }, + }, +}) + +export default mergeConfig( + config, + tanstackViteConfig({ + entry: [`./src/index.ts`, `./src/offline.ts`], + srcDir: `./src`, + beforeWriteDeclarationFile: commonJsDeclarations, + }), +) diff --git a/packages/convex-db-collection/vitest.e2e.config.ts b/packages/convex-db-collection/vitest.e2e.config.ts new file mode 100644 index 0000000000..79b27b1e1d --- /dev/null +++ b/packages/convex-db-collection/vitest.e2e.config.ts @@ -0,0 +1,16 @@ +import { defineConfig } from 'vitest/config' + +export default defineConfig({ + test: { + globalSetup: [ + `./e2e/setup.ts`, + `./e2e/auth-app/setup.ts`, + `./e2e/public-pagination-setup.ts`, + ], + include: [`e2e/**/*.test.ts`], + environment: `node`, + testTimeout: 30000, + hookTimeout: 30000, + fileParallelism: false, + }, +}) diff --git a/packages/offline-transactions/src/OfflineExecutor.ts b/packages/offline-transactions/src/OfflineExecutor.ts index 5690d257a3..a2d6d712e3 100644 --- a/packages/offline-transactions/src/OfflineExecutor.ts +++ b/packages/offline-transactions/src/OfflineExecutor.ts @@ -299,7 +299,11 @@ export class OfflineExecutor { } // Storage available - set up offline components - this.outbox = new OutboxManager(storage, this.config.collections) + this.outbox = new OutboxManager( + storage, + this.config.collections, + this.config.mutationCodec, + ) this.executor = new TransactionExecutor( this.scheduler, this.outbox, diff --git a/packages/offline-transactions/src/index.ts b/packages/offline-transactions/src/index.ts index f43f4633a1..0d1f114498 100644 --- a/packages/offline-transactions/src/index.ts +++ b/packages/offline-transactions/src/index.ts @@ -5,6 +5,7 @@ export { OfflineExecutor, startOfflineExecutor } from './OfflineExecutor' export type { OfflineTransaction, OfflineConfig, + MutationValueCodec, OfflineMode, StorageAdapter, StorageDiagnostic, diff --git a/packages/offline-transactions/src/outbox/OutboxManager.ts b/packages/offline-transactions/src/outbox/OutboxManager.ts index 9b862a7785..ade147717a 100644 --- a/packages/offline-transactions/src/outbox/OutboxManager.ts +++ b/packages/offline-transactions/src/outbox/OutboxManager.ts @@ -1,8 +1,13 @@ import { MissingTemporalConstructorError, + MutationCodecError, TransactionSerializer, } from './TransactionSerializer' -import type { OfflineTransaction, StorageAdapter } from '../types' +import type { + MutationValueCodec, + OfflineTransaction, + StorageAdapter, +} from '../types' import type { Collection } from '@tanstack/db' export class OutboxTransactionNotFoundError extends Error {} @@ -19,9 +24,10 @@ export class OutboxManager { storage: StorageAdapter, collections: Record>, + mutationCodec?: MutationValueCodec, ) { this.storage = storage - this.serializer = new TransactionSerializer(collections) + this.serializer = new TransactionSerializer(collections, mutationCodec) } private getStorageKey(id: string): string { @@ -61,7 +67,10 @@ export class OutboxManager { try { return this.serializer.deserialize(data) } catch (error) { - if (error instanceof MissingTemporalConstructorError) { + if ( + error instanceof MissingTemporalConstructorError || + error instanceof MutationCodecError + ) { error.message = `transaction ${id}: ${error.message}` throw error } @@ -94,7 +103,10 @@ export class OutboxManager { const transaction = this.serializer.deserialize(data) transactions.push(transaction) } catch (error) { - if (error instanceof MissingTemporalConstructorError) { + if ( + error instanceof MissingTemporalConstructorError || + error instanceof MutationCodecError + ) { error.message = `transaction ${key.slice(this.keyPrefix.length)}: ${error.message}` throw error } diff --git a/packages/offline-transactions/src/outbox/TransactionSerializer.ts b/packages/offline-transactions/src/outbox/TransactionSerializer.ts index bb9fbd3c16..3017ec2578 100644 --- a/packages/offline-transactions/src/outbox/TransactionSerializer.ts +++ b/packages/offline-transactions/src/outbox/TransactionSerializer.ts @@ -1,4 +1,5 @@ import type { + MutationValueCodec, OfflineTransaction, SerializedError, SerializedMutation, @@ -79,6 +80,8 @@ function mayBeNativeScalar(value: object): boolean { export class MissingTemporalConstructorError extends Error {} +export class MutationCodecError extends Error {} + function setDataProperty( object: Record, key: string, @@ -111,6 +114,7 @@ export class TransactionSerializer { constructor( collections: Record>, + private readonly mutationCodec?: MutationValueCodec, ) { this.collections = collections // Create reverse lookup from collection.id to registry key @@ -124,7 +128,10 @@ export class TransactionSerializer { assertOutboxPhase(transaction.outboxPhase) const serialized: SerializedOfflineTransaction = { ...transaction, - valueEncoding: 3, + valueEncoding: this.mutationCodec ? 4 : 3, + ...(this.mutationCodec + ? { mutationEncoding: this.mutationCodec.id } + : {}), createdAt: transaction.createdAt.toISOString(), metadata: this.serializeValue(transaction.metadata, `metadata`), mutations: transaction.mutations.map((mutation) => @@ -139,21 +146,33 @@ export class TransactionSerializer { // marker-shaped user objects; the encoding tag is not application metadata. const { valueEncoding, + mutationEncoding, outboxPhase, ...parsed - }: Omit & { + }: Omit< + SerializedOfflineTransaction, + `valueEncoding` | `mutationEncoding` | `outboxPhase` + > & { valueEncoding?: unknown + mutationEncoding?: unknown outboxPhase?: unknown } = JSON.parse(data) if ( valueEncoding !== undefined && valueEncoding !== 2 && - valueEncoding !== 3 + valueEncoding !== 3 && + valueEncoding !== 4 ) { throw new Error( `Unsupported transaction value encoding: ${valueEncoding}`, ) } + if ( + valueEncoding === 4 && + (!this.mutationCodec || mutationEncoding !== this.mutationCodec.id) + ) { + throw new MutationCodecError(`Unsupported mutation codec`) + } assertOutboxPhase(outboxPhase) const createdAt = new Date(parsed.createdAt) @@ -168,8 +187,8 @@ export class TransactionSerializer { ...(outboxPhase === undefined ? {} : { outboxPhase }), createdAt, metadata: - valueEncoding === 3 - ? this.deserializeValue(parsed.metadata, valueEncoding) + valueEncoding === 3 || valueEncoding === 4 + ? this.deserializeValue(parsed.metadata, 3) : parsed.metadata, mutations: parsed.mutations.map((mutationData) => this.deserializeMutation(mutationData, valueEncoding), @@ -188,23 +207,44 @@ export class TransactionSerializer { return { globalKey: mutation.globalKey, type: mutation.type, - modified: this.serializeValue(mutation.modified), - original: this.serializeValue(mutation.original), - changes: this.serializeValue(mutation.changes), + modified: this.encodeMutationValue(mutation.modified), + original: this.encodeMutationValue(mutation.original), + changes: this.encodeMutationValue(mutation.changes), collectionId: registryKey, // Store registry key instead of collection.id } } + private encodeMutationValue(value: unknown): unknown { + if (!this.mutationCodec) return this.serializeValue(value) + const encoded = this.mutationCodec.encode(value) + if (typeof encoded !== `string`) + throw new MutationCodecError(`Mutation codec encode must return a string`) + return encoded + } + private deserializeMutation( data: SerializedMutation, - valueEncoding: 2 | 3 | undefined, + valueEncoding: 2 | 3 | 4 | undefined, ): PendingMutation { const collection = this.collections[data.collectionId] if (!collection) { throw new Error(`Collection with id ${data.collectionId} not found`) } - const modified = this.deserializeValue(data.modified, valueEncoding) + const decode = (value: unknown): unknown => { + if (valueEncoding !== 4) + return this.deserializeValue(value, valueEncoding) + try { + if (typeof value !== `string`) + throw new TypeError(`Expected a mutation codec string`) + return this.mutationCodec!.decode(value) + } catch (cause) { + throw new MutationCodecError(`Failed to decode mutation codec`, { + cause, + }) + } + } + const modified = decode(data.modified) // Extract the key from the modified data using the collection's getKey function // This is needed for optimistic state restoration to work correctly @@ -216,8 +256,8 @@ export class TransactionSerializer { globalKey: data.globalKey, type: data.type as any, modified, - original: this.deserializeValue(data.original, valueEncoding), - changes: this.deserializeValue(data.changes, valueEncoding) ?? {}, + original: decode(data.original), + changes: decode(data.changes) ?? {}, collection, // These fields would need to be reconstructed by the executor mutationId: ``, // Will be regenerated diff --git a/packages/offline-transactions/src/react-native/index.ts b/packages/offline-transactions/src/react-native/index.ts index 977b1e93f5..ed63ca370d 100644 --- a/packages/offline-transactions/src/react-native/index.ts +++ b/packages/offline-transactions/src/react-native/index.ts @@ -3,6 +3,7 @@ export { // Types type OfflineTransaction, type OfflineConfig, + type MutationValueCodec, type OfflineMode, type StorageAdapter, type StorageDiagnostic, diff --git a/packages/offline-transactions/src/types.ts b/packages/offline-transactions/src/types.ts index 3101328496..15f625ae83 100644 --- a/packages/offline-transactions/src/types.ts +++ b/packages/offline-transactions/src/types.ts @@ -60,7 +60,9 @@ export interface OfflineTransaction { // Serialized representation for storage export interface SerializedOfflineTransaction { /** Absent for the original Date-marker format. */ - valueEncoding?: 2 | 3 + valueEncoding?: 2 | 3 | 4 + /** Required for opt-in mutation encoding v4. */ + mutationEncoding?: string id: string mutationFnName: string mutations: Array @@ -95,7 +97,16 @@ export interface StorageDiagnostic { error?: Error } +/** Encodes mutation payloads into JSON-safe values, and restores them on replay. */ +export interface MutationValueCodec { + /** Stable format/version identifier, retained in every encoded transaction. */ + id: string + encode: (value: unknown) => string + decode: (value: string) => unknown +} + export interface OfflineConfig { + mutationCodec?: MutationValueCodec collections: Record> mutationFns: Record storage?: StorageAdapter diff --git a/packages/offline-transactions/tests/oracle-config.ts b/packages/offline-transactions/tests/oracle-config.ts index 8384b6da64..c2ea829814 100644 --- a/packages/offline-transactions/tests/oracle-config.ts +++ b/packages/offline-transactions/tests/oracle-config.ts @@ -1,7 +1,8 @@ type OracleEnvironment = Record type OfflineOracleConfigOptions = { - prefix: `OFFLINE_ORACLE` | `TANSTACK_DB_OFFLINE_ORACLE` + prefix: + `OFFLINE_ORACLE` | `TANSTACK_DB_OFFLINE_ORACLE` | `OFFLINE_CODEC_ORACLE` defaultRuns: number defaultSeed?: number environment?: OracleEnvironment diff --git a/packages/offline-transactions/tests/transaction-serializer-oracle.property.test.ts b/packages/offline-transactions/tests/transaction-serializer-oracle.property.test.ts index 08e12dc5ed..26655204cb 100644 --- a/packages/offline-transactions/tests/transaction-serializer-oracle.property.test.ts +++ b/packages/offline-transactions/tests/transaction-serializer-oracle.property.test.ts @@ -18,6 +18,8 @@ import type { PendingMutation } from '@tanstack/db' * Contract and source: SerializedOfflineTransaction and the offline persistence * contract preserve mutation meaning across process boundaries. Encoding v3 * supports JSON trees, Date, and the eight restorable Temporal scalar types. + * Opt-in v4 records delegate mutation payloads to their identified codec. + * Default v3 metadata and mutation behavior stays unchanged. * Marker-shaped user objects remain data. Unversioned and v2 records retain * their prior meanings. Unsupported or malformed encodings fail visibly. * @@ -47,8 +49,8 @@ import type { PendingMutation } from '@tanstack/db' * versions. Malformed markers and missing Temporal constructors test failure * paths before durable data can be replaced. * - * Limits: undefined, non-finite numbers, arbitrary native objects, and - * cross-realm boxed values are outside current evidence. Cycles must fail + * Limits for default v3: undefined, non-finite numbers, arbitrary native + * objects, and cross-realm boxed values are outside that domain's evidence. Cycles must fail * visibly while repeated non-cyclic references retain their values. This * oracle does not claim byte stability for object key order beyond JSON's * established rules. Temporal witnesses install controlled constructors; @@ -291,7 +293,7 @@ async function checkRoundtrip( if (fault === `omit-changes`) encoded = encoded.replaceAll(`"changes":`, `"lostChanges":`) if (fault === `unknown-encoding`) - encoded = encoded.replace(`"valueEncoding":3`, `"valueEncoding":4`) + encoded = encoded.replace(`"valueEncoding":3`, `"valueEncoding":5`) return encoded } const serialized = serializer.serialize(offline) @@ -1580,7 +1582,7 @@ it.each( message: fault === `wrong-registry` ? `Collection with id writer:0 not found` - : `Unsupported transaction value encoding: 4`, + : `Unsupported transaction value encoding: 5`, } await expect( checkRoundtrip(pinned, 0, fault, boundary), @@ -1591,3 +1593,164 @@ it.each( } }, ) + +/** + * Opt-in mutation codecs own their mutation-value domain independently of the + * default JSON/Date/Temporal encoding above. The wire records the codec ID and + * a new version so a reader cannot mistake opaque data for ordinary rows. + * Missing, mismatched, or failing codecs reject reads without changing storage. + * + * The supplied fixture codec uses Node's V8 structured serialization. Expected + * runtime values come directly from generated native leaves, not a reimplementation + * of production dispatch or codec output. Histories cross insert/update/delete, + * native values nested in all three mutation payloads, fresh readers, and + * unsupported reader configurations. Existing v3 histories remain unchanged. + * The Convex receiving witness supplies its real public value codec and tests + * disk/process replay against the actual backend. + */ +it(`preserves opt-in mutation values and rejects unreadable codec records without losing work`, async () => { + const { serialize, deserialize } = await import('node:v8') + const codec = { + id: `v8-test/1`, + encode: (value: unknown) => serialize(value).toString(`base64`), + decode: (value: unknown) => + deserialize(Buffer.from(value as string, `base64`)) as unknown, + } + const native = fc.oneof( + fc.bigInt({ min: -(2n ** 63n), max: 2n ** 63n - 1n }), + fc.constantFrom(NaN, Infinity, -Infinity, -0), + fc.uint8Array({ maxLength: 12 }).map((bytes) => bytes.buffer), + ) + const property = fc.asyncProperty( + fc.array(native, { minLength: 1, maxLength: 5 }), + fc.constantFrom(`insert`, `update`, `delete`), + async (values, kind) => { + type CodecRow = { id: string; values: Array } + const makeCollection = (id: string) => + createCollection({ + id, + getKey: (row) => row.id, + sync: { sync: ({ markReady }) => markReady() }, + }) + const writer = makeCollection(`codec-writer`) + const reader = makeCollection(`codec-reader`) + const row: CodecRow = { id: `one`, values } + const transaction: OfflineTransaction = { + id: `codec-transaction`, + mutationFnName: `save`, + keys: [`one`], + idempotencyKey: `stable`, + createdAt: new Date(0), + retryCount: 0, + nextAttemptAt: 0, + version: 1, + metadata: { when: new Date(0) }, + mutations: [ + { + globalKey: `one`, + type: kind, + modified: row, + original: row, + changes: { values }, + collection: writer, + } as unknown as PendingMutation, + ], + } + const storage = new FakeStorageAdapter() + let hasPrimaryFailure = false + try { + await new OutboxManager(storage, { rows: writer }, codec).add( + transaction, + ) + const before = storage.snapshot() + const restored = await new OutboxManager( + storage, + { rows: reader }, + codec, + ).get(`codec-transaction`) + expect(restored?.mutations[0]?.modified).toEqual(row) + expect(restored?.mutations[0]?.original).toEqual(row) + expect(restored?.mutations[0]?.changes).toEqual({ values }) + expect(restored?.mutations[0]?.collection).toBe(reader) + expect(restored?.metadata).toEqual({ when: new Date(0) }) + const invalidEncoder = { + ...codec, + encode: () => null, + } as unknown as typeof codec + await expect( + new OutboxManager(storage, { rows: writer }, invalidEncoder).add( + transaction, + ), + ).rejects.toThrow(`codec`) + expect(storage.snapshot()).toEqual(before) + const wire = JSON.parse(Object.values(before)[0]!) + expect(wire.valueEncoding).toBe(4) + expect(wire.mutationEncoding).toBe(codec.id) + for (const unsupported of [ + undefined, + { ...codec, id: `other` }, + { + ...codec, + decode: () => { + throw new Error(`broken codec`) + }, + }, + ]) { + const outbox = new OutboxManager( + storage, + { rows: reader }, + unsupported, + ) + await expect(outbox.get(`codec-transaction`)).rejects.toThrow(`codec`) + await expect(outbox.getAll()).rejects.toThrow(`codec`) + expect(storage.snapshot()).toEqual(before) + } + const plain = { id: `one`, values: [] } + const legacy = { + ...transaction, + id: `legacy-transaction`, + mutations: [ + { + ...transaction.mutations[0]!, + modified: plain, + original: plain, + changes: { values: [] }, + }, + ], + } + await new OutboxManager(storage, { rows: writer }).add(legacy) + const mixed = await new OutboxManager( + storage, + { rows: reader }, + codec, + ).getAll() + expect(mixed).toHaveLength(2) + expect( + mixed.find((entry) => entry.id === legacy.id)?.mutations[0]?.modified, + ).toEqual(plain) + } catch (error) { + hasPrimaryFailure = true + throw error + } finally { + await cleanupOfflineOracle( + [() => writer.cleanup(), () => reader.cleanup()], + hasPrimaryFailure, + ) + } + }, + ) + const config = readOfflineOracleConfig({ + prefix: `OFFLINE_CODEC_ORACLE`, + defaultRuns: 60, + }) + if (config.seed !== undefined) + await fc.assert(property, { + numRuns: config.runs, + seed: config.seed, + path: config.path, + }) + else { + await fc.assert(property, { numRuns: config.runs, seed: 20261022 }) + await fc.assert(property, { numRuns: config.runs }) + } +}) diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 3453d43a2c..4cfcdae735 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -1205,6 +1205,49 @@ importers: specifier: ^4.64.0 version: 4.76.0 + packages/convex-db-collection: + dependencies: + '@standard-schema/spec': + specifier: ^1.1.0 + version: 1.1.0 + '@tanstack/db': + specifier: workspace:* + version: link:../db + '@tanstack/offline-transactions': + specifier: workspace:* + version: link:../offline-transactions + typescript: + specifier: '>=4.7' + version: 5.9.3 + devDependencies: + '@vitest/coverage-istanbul': + specifier: ^3.2.4 + version: 3.2.4(vitest@3.2.4) + convex: + specifier: ^1.47.0 + version: 1.47.0(react@19.2.4) + + packages/convex-db-collection/e2e/auth-app: + dependencies: + '@auth/core': + specifier: 0.41.3 + version: 0.41.3 + '@convex-dev/auth': + specifier: 0.0.96 + version: 0.0.96(@auth/core@0.41.3)(convex@1.47.0(react@19.2.4))(react@19.2.4) + convex: + specifier: ^1.47.0 + version: 1.47.0(react@19.2.4) + jose: + specifier: 5.10.0 + version: 5.10.0 + + packages/convex-db-collection/e2e/backend: + dependencies: + convex: + specifier: ^1.47.0 + version: 1.47.0(react@19.2.4) + packages/db: dependencies: '@standard-schema/spec': @@ -2007,6 +2050,20 @@ packages: '@asamuzakjp/nwsapi@2.3.9': resolution: {integrity: sha512-n8GuYSrI9bF7FFZ/SjhwevlHc8xaVlb/7HmHelnc/PZXBD2ZR49NnN9sMMuDdEGPeeRQ5d0hqlSlEpgCX3Wl0Q==} + '@auth/core@0.41.3': + resolution: {integrity: sha512-sJ3JMHHkXMD3aOjopv7mOBTO1Ocw4b0fAEXJBz6k7YHLpYQI6C40jCUPc5fNvUKxXRXNE1/sRISA15UrwWJBTw==} + peerDependencies: + '@simplewebauthn/browser': ^9.0.1 + '@simplewebauthn/server': ^9.0.2 + nodemailer: ^7.0.7 || ^8.0.5 + peerDependenciesMeta: + '@simplewebauthn/browser': + optional: true + '@simplewebauthn/server': + optional: true + nodemailer: + optional: true + '@babel/code-frame@7.10.4': resolution: {integrity: sha512-vG6SvB6oYEhvgisZNFRmRCUkLz11c7rp+tbNTynGqc6mS1d5ATd/sGyV6W0KZZnXRKMTzZDRgQT3Ou9jhpAfUg==} @@ -2728,6 +2785,17 @@ packages: resolution: {integrity: sha512-KTy0OqRDLR5y/zZMnizyx09z/rPlPC/zKhYgH8o/q6PuAjoQAKlRfY4zzv0M64yybQ//6//4H1n14pxaLZfUnA==} engines: {node: '>=v18'} + '@convex-dev/auth@0.0.96': + resolution: {integrity: sha512-4vcoQmmlbInLpR6BK43iIH21JwVPG5U7kRIjLaUWkeEr+hVaS/DuYsQRrDsDLu9rbHpq9PT2ZkUrtPDLkxYx0A==} + hasBin: true + peerDependencies: + '@auth/core': ^0.41.1 + convex: ^1.17.0 + react: ^18.2.0 || ^19.0.0-0 + peerDependenciesMeta: + react: + optional: true + '@cspotcode/source-map-support@0.8.1': resolution: {integrity: sha512-IchNf6dN4tHoMFIn/7OE8LWZ19Y6q/67Bmf6vnGREv8RSbBVb9LPJxEcnwrcwX6ixSvaiGoomAUvu4YSxXrVgw==} engines: {node: '>=12'} @@ -2807,6 +2875,12 @@ packages: cpu: [ppc64] os: [aix] + '@esbuild/aix-ppc64@0.27.0': + resolution: {integrity: sha512-KuZrd2hRjz01y5JK9mEBSD3Vj3mbCvemhT466rSuJYeE/hjuBrHfjjcjMdTm/sz7au+++sdbJZJmuBwQLuw68A==} + engines: {node: '>=18'} + cpu: [ppc64] + os: [aix] + '@esbuild/aix-ppc64@0.27.3': resolution: {integrity: sha512-9fJMTNFTWZMh5qwrBItuziu834eOCUcEqymSH7pY+zoMVEZg3gcPuBNxH1EvfVYe9h0x/Ptw8KBzv7qxb7l8dg==} engines: {node: '>=18'} @@ -2837,6 +2911,12 @@ packages: cpu: [arm64] os: [android] + '@esbuild/android-arm64@0.27.0': + resolution: {integrity: sha512-CC3vt4+1xZrs97/PKDkl0yN7w8edvU2vZvAFGD16n9F0Cvniy5qvzRXjfO1l94efczkkQE6g1x0i73Qf5uthOQ==} + engines: {node: '>=18'} + cpu: [arm64] + os: [android] + '@esbuild/android-arm64@0.27.3': resolution: {integrity: sha512-YdghPYUmj/FX2SYKJ0OZxf+iaKgMsKHVPF1MAq/P8WirnSpCStzKJFjOjzsW0QQ7oIAiccHdcqjbHmJxRb/dmg==} engines: {node: '>=18'} @@ -2867,6 +2947,12 @@ packages: cpu: [arm] os: [android] + '@esbuild/android-arm@0.27.0': + resolution: {integrity: sha512-j67aezrPNYWJEOHUNLPj9maeJte7uSMM6gMoxfPC9hOg8N02JuQi/T7ewumf4tNvJadFkvLZMlAq73b9uwdMyQ==} + engines: {node: '>=18'} + cpu: [arm] + os: [android] + '@esbuild/android-arm@0.27.3': resolution: {integrity: sha512-i5D1hPY7GIQmXlXhs2w8AWHhenb00+GxjxRncS2ZM7YNVGNfaMxgzSGuO8o8SJzRc/oZwU2bcScvVERk03QhzA==} engines: {node: '>=18'} @@ -2897,6 +2983,12 @@ packages: cpu: [x64] os: [android] + '@esbuild/android-x64@0.27.0': + resolution: {integrity: sha512-wurMkF1nmQajBO1+0CJmcN17U4BP6GqNSROP8t0X/Jiw2ltYGLHpEksp9MpoBqkrFR3kv2/te6Sha26k3+yZ9Q==} + engines: {node: '>=18'} + cpu: [x64] + os: [android] + '@esbuild/android-x64@0.27.3': resolution: {integrity: sha512-IN/0BNTkHtk8lkOM8JWAYFg4ORxBkZQf9zXiEOfERX/CzxW3Vg1ewAhU7QSWQpVIzTW+b8Xy+lGzdYXV6UZObQ==} engines: {node: '>=18'} @@ -2927,6 +3019,12 @@ packages: cpu: [arm64] os: [darwin] + '@esbuild/darwin-arm64@0.27.0': + resolution: {integrity: sha512-uJOQKYCcHhg07DL7i8MzjvS2LaP7W7Pn/7uA0B5S1EnqAirJtbyw4yC5jQ5qcFjHK9l6o/MX9QisBg12kNkdHg==} + engines: {node: '>=18'} + cpu: [arm64] + os: [darwin] + '@esbuild/darwin-arm64@0.27.3': resolution: {integrity: sha512-Re491k7ByTVRy0t3EKWajdLIr0gz2kKKfzafkth4Q8A5n1xTHrkqZgLLjFEHVD+AXdUGgQMq+Godfq45mGpCKg==} engines: {node: '>=18'} @@ -2957,6 +3055,12 @@ packages: cpu: [x64] os: [darwin] + '@esbuild/darwin-x64@0.27.0': + resolution: {integrity: sha512-8mG6arH3yB/4ZXiEnXof5MK72dE6zM9cDvUcPtxhUZsDjESl9JipZYW60C3JGreKCEP+p8P/72r69m4AZGJd5g==} + engines: {node: '>=18'} + cpu: [x64] + os: [darwin] + '@esbuild/darwin-x64@0.27.3': resolution: {integrity: sha512-vHk/hA7/1AckjGzRqi6wbo+jaShzRowYip6rt6q7VYEDX4LEy1pZfDpdxCBnGtl+A5zq8iXDcyuxwtv3hNtHFg==} engines: {node: '>=18'} @@ -2987,6 +3091,12 @@ packages: cpu: [arm64] os: [freebsd] + '@esbuild/freebsd-arm64@0.27.0': + resolution: {integrity: sha512-9FHtyO988CwNMMOE3YIeci+UV+x5Zy8fI2qHNpsEtSF83YPBmE8UWmfYAQg6Ux7Gsmd4FejZqnEUZCMGaNQHQw==} + engines: {node: '>=18'} + cpu: [arm64] + os: [freebsd] + '@esbuild/freebsd-arm64@0.27.3': resolution: {integrity: sha512-ipTYM2fjt3kQAYOvo6vcxJx3nBYAzPjgTCk7QEgZG8AUO3ydUhvelmhrbOheMnGOlaSFUoHXB6un+A7q4ygY9w==} engines: {node: '>=18'} @@ -3017,6 +3127,12 @@ packages: cpu: [x64] os: [freebsd] + '@esbuild/freebsd-x64@0.27.0': + resolution: {integrity: sha512-zCMeMXI4HS/tXvJz8vWGexpZj2YVtRAihHLk1imZj4efx1BQzN76YFeKqlDr3bUWI26wHwLWPd3rwh6pe4EV7g==} + engines: {node: '>=18'} + cpu: [x64] + os: [freebsd] + '@esbuild/freebsd-x64@0.27.3': resolution: {integrity: sha512-dDk0X87T7mI6U3K9VjWtHOXqwAMJBNN2r7bejDsc+j03SEjtD9HrOl8gVFByeM0aJksoUuUVU9TBaZa2rgj0oA==} engines: {node: '>=18'} @@ -3047,6 +3163,12 @@ packages: cpu: [arm64] os: [linux] + '@esbuild/linux-arm64@0.27.0': + resolution: {integrity: sha512-AS18v0V+vZiLJyi/4LphvBE+OIX682Pu7ZYNsdUHyUKSoRwdnOsMf6FDekwoAFKej14WAkOef3zAORJgAtXnlQ==} + engines: {node: '>=18'} + cpu: [arm64] + os: [linux] + '@esbuild/linux-arm64@0.27.3': resolution: {integrity: sha512-sZOuFz/xWnZ4KH3YfFrKCf1WyPZHakVzTiqji3WDc0BCl2kBwiJLCXpzLzUBLgmp4veFZdvN5ChW4Eq/8Fc2Fg==} engines: {node: '>=18'} @@ -3077,6 +3199,12 @@ packages: cpu: [arm] os: [linux] + '@esbuild/linux-arm@0.27.0': + resolution: {integrity: sha512-t76XLQDpxgmq2cNXKTVEB7O7YMb42atj2Re2Haf45HkaUpjM2J0UuJZDuaGbPbamzZ7bawyGFUkodL+zcE+jvQ==} + engines: {node: '>=18'} + cpu: [arm] + os: [linux] + '@esbuild/linux-arm@0.27.3': resolution: {integrity: sha512-s6nPv2QkSupJwLYyfS+gwdirm0ukyTFNl3KTgZEAiJDd+iHZcbTPPcWCcRYH+WlNbwChgH2QkE9NSlNrMT8Gfw==} engines: {node: '>=18'} @@ -3107,6 +3235,12 @@ packages: cpu: [ia32] os: [linux] + '@esbuild/linux-ia32@0.27.0': + resolution: {integrity: sha512-Mz1jxqm/kfgKkc/KLHC5qIujMvnnarD9ra1cEcrs7qshTUSksPihGrWHVG5+osAIQ68577Zpww7SGapmzSt4Nw==} + engines: {node: '>=18'} + cpu: [ia32] + os: [linux] + '@esbuild/linux-ia32@0.27.3': resolution: {integrity: sha512-yGlQYjdxtLdh0a3jHjuwOrxQjOZYD/C9PfdbgJJF3TIZWnm/tMd/RcNiLngiu4iwcBAOezdnSLAwQDPqTmtTYg==} engines: {node: '>=18'} @@ -3137,6 +3271,12 @@ packages: cpu: [loong64] os: [linux] + '@esbuild/linux-loong64@0.27.0': + resolution: {integrity: sha512-QbEREjdJeIreIAbdG2hLU1yXm1uu+LTdzoq1KCo4G4pFOLlvIspBm36QrQOar9LFduavoWX2msNFAAAY9j4BDg==} + engines: {node: '>=18'} + cpu: [loong64] + os: [linux] + '@esbuild/linux-loong64@0.27.3': resolution: {integrity: sha512-WO60Sn8ly3gtzhyjATDgieJNet/KqsDlX5nRC5Y3oTFcS1l0KWba+SEa9Ja1GfDqSF1z6hif/SkpQJbL63cgOA==} engines: {node: '>=18'} @@ -3167,6 +3307,12 @@ packages: cpu: [mips64el] os: [linux] + '@esbuild/linux-mips64el@0.27.0': + resolution: {integrity: sha512-sJz3zRNe4tO2wxvDpH/HYJilb6+2YJxo/ZNbVdtFiKDufzWq4JmKAiHy9iGoLjAV7r/W32VgaHGkk35cUXlNOg==} + engines: {node: '>=18'} + cpu: [mips64el] + os: [linux] + '@esbuild/linux-mips64el@0.27.3': resolution: {integrity: sha512-APsymYA6sGcZ4pD6k+UxbDjOFSvPWyZhjaiPyl/f79xKxwTnrn5QUnXR5prvetuaSMsb4jgeHewIDCIWljrSxw==} engines: {node: '>=18'} @@ -3197,6 +3343,12 @@ packages: cpu: [ppc64] os: [linux] + '@esbuild/linux-ppc64@0.27.0': + resolution: {integrity: sha512-z9N10FBD0DCS2dmSABDBb5TLAyF1/ydVb+N4pi88T45efQ/w4ohr/F/QYCkxDPnkhkp6AIpIcQKQ8F0ANoA2JA==} + engines: {node: '>=18'} + cpu: [ppc64] + os: [linux] + '@esbuild/linux-ppc64@0.27.3': resolution: {integrity: sha512-eizBnTeBefojtDb9nSh4vvVQ3V9Qf9Df01PfawPcRzJH4gFSgrObw+LveUyDoKU3kxi5+9RJTCWlj4FjYXVPEA==} engines: {node: '>=18'} @@ -3227,6 +3379,12 @@ packages: cpu: [riscv64] os: [linux] + '@esbuild/linux-riscv64@0.27.0': + resolution: {integrity: sha512-pQdyAIZ0BWIC5GyvVFn5awDiO14TkT/19FTmFcPdDec94KJ1uZcmFs21Fo8auMXzD4Tt+diXu1LW1gHus9fhFQ==} + engines: {node: '>=18'} + cpu: [riscv64] + os: [linux] + '@esbuild/linux-riscv64@0.27.3': resolution: {integrity: sha512-3Emwh0r5wmfm3ssTWRQSyVhbOHvqegUDRd0WhmXKX2mkHJe1SFCMJhagUleMq+Uci34wLSipf8Lagt4LlpRFWQ==} engines: {node: '>=18'} @@ -3257,6 +3415,12 @@ packages: cpu: [s390x] os: [linux] + '@esbuild/linux-s390x@0.27.0': + resolution: {integrity: sha512-hPlRWR4eIDDEci953RI1BLZitgi5uqcsjKMxwYfmi4LcwyWo2IcRP+lThVnKjNtk90pLS8nKdroXYOqW+QQH+w==} + engines: {node: '>=18'} + cpu: [s390x] + os: [linux] + '@esbuild/linux-s390x@0.27.3': resolution: {integrity: sha512-pBHUx9LzXWBc7MFIEEL0yD/ZVtNgLytvx60gES28GcWMqil8ElCYR4kvbV2BDqsHOvVDRrOxGySBM9Fcv744hw==} engines: {node: '>=18'} @@ -3287,6 +3451,12 @@ packages: cpu: [x64] os: [linux] + '@esbuild/linux-x64@0.27.0': + resolution: {integrity: sha512-1hBWx4OUJE2cab++aVZ7pObD6s+DK4mPGpemtnAORBvb5l/g5xFGk0vc0PjSkrDs0XaXj9yyob3d14XqvnQ4gw==} + engines: {node: '>=18'} + cpu: [x64] + os: [linux] + '@esbuild/linux-x64@0.27.3': resolution: {integrity: sha512-Czi8yzXUWIQYAtL/2y6vogER8pvcsOsk5cpwL4Gk5nJqH5UZiVByIY8Eorm5R13gq+DQKYg0+JyQoytLQas4dA==} engines: {node: '>=18'} @@ -3311,6 +3481,12 @@ packages: cpu: [arm64] os: [netbsd] + '@esbuild/netbsd-arm64@0.27.0': + resolution: {integrity: sha512-6m0sfQfxfQfy1qRuecMkJlf1cIzTOgyaeXaiVaaki8/v+WB+U4hc6ik15ZW6TAllRlg/WuQXxWj1jx6C+dfy3w==} + engines: {node: '>=18'} + cpu: [arm64] + os: [netbsd] + '@esbuild/netbsd-arm64@0.27.3': resolution: {integrity: sha512-sDpk0RgmTCR/5HguIZa9n9u+HVKf40fbEUt+iTzSnCaGvY9kFP0YKBWZtJaraonFnqef5SlJ8/TiPAxzyS+UoA==} engines: {node: '>=18'} @@ -3341,6 +3517,12 @@ packages: cpu: [x64] os: [netbsd] + '@esbuild/netbsd-x64@0.27.0': + resolution: {integrity: sha512-xbbOdfn06FtcJ9d0ShxxvSn2iUsGd/lgPIO2V3VZIPDbEaIj1/3nBBe1AwuEZKXVXkMmpr6LUAgMkLD/4D2PPA==} + engines: {node: '>=18'} + cpu: [x64] + os: [netbsd] + '@esbuild/netbsd-x64@0.27.3': resolution: {integrity: sha512-P14lFKJl/DdaE00LItAukUdZO5iqNH7+PjoBm+fLQjtxfcfFE20Xf5CrLsmZdq5LFFZzb5JMZ9grUwvtVYzjiA==} engines: {node: '>=18'} @@ -3365,6 +3547,12 @@ packages: cpu: [arm64] os: [openbsd] + '@esbuild/openbsd-arm64@0.27.0': + resolution: {integrity: sha512-fWgqR8uNbCQ/GGv0yhzttj6sU/9Z5/Sv/VGU3F5OuXK6J6SlriONKrQ7tNlwBrJZXRYk5jUhuWvF7GYzGguBZQ==} + engines: {node: '>=18'} + cpu: [arm64] + os: [openbsd] + '@esbuild/openbsd-arm64@0.27.3': resolution: {integrity: sha512-AIcMP77AvirGbRl/UZFTq5hjXK+2wC7qFRGoHSDrZ5v5b8DK/GYpXW3CPRL53NkvDqb9D+alBiC/dV0Fb7eJcw==} engines: {node: '>=18'} @@ -3395,6 +3583,12 @@ packages: cpu: [x64] os: [openbsd] + '@esbuild/openbsd-x64@0.27.0': + resolution: {integrity: sha512-aCwlRdSNMNxkGGqQajMUza6uXzR/U0dIl1QmLjPtRbLOx3Gy3otfFu/VjATy4yQzo9yFDGTxYDo1FfAD9oRD2A==} + engines: {node: '>=18'} + cpu: [x64] + os: [openbsd] + '@esbuild/openbsd-x64@0.27.3': resolution: {integrity: sha512-DnW2sRrBzA+YnE70LKqnM3P+z8vehfJWHXECbwBmH/CU51z6FiqTQTHFenPlHmo3a8UgpLyH3PT+87OViOh1AQ==} engines: {node: '>=18'} @@ -3419,6 +3613,12 @@ packages: cpu: [arm64] os: [openharmony] + '@esbuild/openharmony-arm64@0.27.0': + resolution: {integrity: sha512-nyvsBccxNAsNYz2jVFYwEGuRRomqZ149A39SHWk4hV0jWxKM0hjBPm3AmdxcbHiFLbBSwG6SbpIcUbXjgyECfA==} + engines: {node: '>=18'} + cpu: [arm64] + os: [openharmony] + '@esbuild/openharmony-arm64@0.27.3': resolution: {integrity: sha512-NinAEgr/etERPTsZJ7aEZQvvg/A6IsZG/LgZy+81wON2huV7SrK3e63dU0XhyZP4RKGyTm7aOgmQk0bGp0fy2g==} engines: {node: '>=18'} @@ -3449,6 +3649,12 @@ packages: cpu: [x64] os: [sunos] + '@esbuild/sunos-x64@0.27.0': + resolution: {integrity: sha512-Q1KY1iJafM+UX6CFEL+F4HRTgygmEW568YMqDA5UV97AuZSm21b7SXIrRJDwXWPzr8MGr75fUZPV67FdtMHlHA==} + engines: {node: '>=18'} + cpu: [x64] + os: [sunos] + '@esbuild/sunos-x64@0.27.3': resolution: {integrity: sha512-PanZ+nEz+eWoBJ8/f8HKxTTD172SKwdXebZ0ndd953gt1HRBbhMsaNqjTyYLGLPdoWHy4zLU7bDVJztF5f3BHA==} engines: {node: '>=18'} @@ -3479,6 +3685,12 @@ packages: cpu: [arm64] os: [win32] + '@esbuild/win32-arm64@0.27.0': + resolution: {integrity: sha512-W1eyGNi6d+8kOmZIwi/EDjrL9nxQIQ0MiGqe/AWc6+IaHloxHSGoeRgDRKHFISThLmsewZ5nHFvGFWdBYlgKPg==} + engines: {node: '>=18'} + cpu: [arm64] + os: [win32] + '@esbuild/win32-arm64@0.27.3': resolution: {integrity: sha512-B2t59lWWYrbRDw/tjiWOuzSsFh1Y/E95ofKz7rIVYSQkUYBjfSgf6oeYPNWHToFRr2zx52JKApIcAS/D5TUBnA==} engines: {node: '>=18'} @@ -3509,6 +3721,12 @@ packages: cpu: [ia32] os: [win32] + '@esbuild/win32-ia32@0.27.0': + resolution: {integrity: sha512-30z1aKL9h22kQhilnYkORFYt+3wp7yZsHWus+wSKAJR8JtdfI76LJ4SBdMsCopTR3z/ORqVu5L1vtnHZWVj4cQ==} + engines: {node: '>=18'} + cpu: [ia32] + os: [win32] + '@esbuild/win32-ia32@0.27.3': resolution: {integrity: sha512-QLKSFeXNS8+tHW7tZpMtjlNb7HKau0QDpwm49u0vUp9y1WOF+PEzkU84y9GqYaAVW8aH8f3GcBck26jh54cX4Q==} engines: {node: '>=18'} @@ -3539,6 +3757,12 @@ packages: cpu: [x64] os: [win32] + '@esbuild/win32-x64@0.27.0': + resolution: {integrity: sha512-aIitBcjQeyOhMTImhLZmtxfdOcuNRpwlPNmlFKPcHQYPhEssw75Cl1TSXJXpMkzaua9FUetx/4OQKq7eJul5Cg==} + engines: {node: '>=18'} + cpu: [x64] + os: [win32] + '@esbuild/win32-x64@0.27.3': resolution: {integrity: sha512-4uJGhsxuptu3OcpVAzli+/gWusVGwZZHTlS63hh++ehExkVT8SgiEf7/uC/PclrPPkLhZqGgCTjd0VWLo6xMqA==} engines: {node: '>=18'} @@ -4803,6 +5027,21 @@ packages: resolution: {integrity: sha512-eSYWTm620tTk45EKSedaUL8MFYI8hW164hIXsgIHyxu3VobUB3fFCu5t0hQby6OoWRPsG1KkKUG2M5UadiLiVg==} engines: {node: '>=14'} + '@oslojs/asn1@1.0.0': + resolution: {integrity: sha512-zw/wn0sj0j0QKbIXfIlnEcTviaCzYOY3V5rAyjR6YtOByFtJiT574+8p9Wlach0lZH9fddD4yb9laEAIl4vXQA==} + deprecated: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info. + + '@oslojs/binary@1.0.0': + resolution: {integrity: sha512-9RCU6OwXU6p67H4NODbuxv2S3eenuQ4/WFLrsq+K/k682xrznH5EVWA7N4VFk9VYVcbFtKqur5YQQZc0ySGhsQ==} + deprecated: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info. + + '@oslojs/crypto@1.0.1': + resolution: {integrity: sha512-7n08G8nWjAr/Yu3vu9zzrd0L9XnrJfpMioQcvCMxBIiF5orECHe5/3J0jmXRVvgfqMm/+4oxlQ+Sq39COYLcNQ==} + deprecated: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info. + + '@oslojs/encoding@1.1.0': + resolution: {integrity: sha512-70wQhgYmndg4GCPxPPxPGevRKqTIJ2Nh4OkiMWmDAVYsTQ+Ta7Sq+rPevXyXGdzr30/qZBnyOalCszoMxlyldQ==} + '@oxc-resolver/binding-android-arm-eabi@11.16.2': resolution: {integrity: sha512-lVJbvydLQIDZHKUb6Zs9Rq80QVTQ9xdCQE30eC9/cjg4wsMoEOg65QZPymUAIVJotpUAWJD0XYcwE7ugfxx5kQ==} cpu: [arm] @@ -4911,6 +5150,9 @@ packages: cpu: [x64] os: [win32] + '@panva/hkdf@1.2.1': + resolution: {integrity: sha512-6oclG6Y3PiDFcoyk8srjLfVKyMfVCKJ27JwNPViuXziFpmdz+MZnZN/aKY0JGXgYuO/VghU0jcOAZgWXZ1Dmrw==} + '@parcel/watcher-android-arm64@2.5.1': resolution: {integrity: sha512-KF8+j9nNbUN8vzOFDpRMsaKBHZ/mcjEjMToVMJOhTozkDonQFFrRcfdLWn6yWKCmJKmdVxSgHiYvTCef4/qcBA==} engines: {node: '>= 10.0.0'} @@ -7491,6 +7733,25 @@ packages: convert-source-map@2.0.0: resolution: {integrity: sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==} + convex@1.47.0: + resolution: {integrity: sha512-B1z/lOmex7i09NASq8+2WG9rZLzGXXOSkpwGOpaNIvOs4awyW4RMgpVBjt8r2wvvHwhEBEsOr2pGYpulnmAxYQ==} + engines: {node: '>=20.0.0', npm: '>=7.0.0'} + hasBin: true + peerDependencies: + '@auth0/auth0-react': ^2.0.1 + '@clerk/clerk-react': ^4.12.8 || ^5.0.0 + '@clerk/react': ^6.4.3 + react: ^18.0.0 || ^19.0.0-0 || ^19.0.0 + peerDependenciesMeta: + '@auth0/auth0-react': + optional: true + '@clerk/clerk-react': + optional: true + '@clerk/react': + optional: true + react: + optional: true + cookie-es@3.1.1: resolution: {integrity: sha512-UaXxwISYJPTr9hwQxMFYZ7kNhSXboMXP+Z3TRX6f1/NyaGPfuNUZOWP1pUEb75B2HjfklIYLVRfWiFZJyC6Npg==} @@ -8002,6 +8263,11 @@ packages: engines: {node: '>=18'} hasBin: true + esbuild@0.27.0: + resolution: {integrity: sha512-jd0f4NHbD6cALCyGElNpGAOtWxSq46l9X/sWB0Nzd5er4Kz2YTm+Vl0qKFT9KUJvD8+fiO8AvoHhFvEatfVixA==} + engines: {node: '>=18'} + hasBin: true + esbuild@0.27.3: resolution: {integrity: sha512-8VwMnyGCONIs6cWue2IdpHxHnAjzxnw2Zr7MkVxB2vjmQ2ivqGFb4LEG3SMnv0Gb2F/G/2yA8zUaiL1gywDCCg==} engines: {node: '>=18'} @@ -9088,6 +9354,10 @@ packages: resolution: {integrity: sha512-5KoIu2Ngpyek75jXodFvnafB6DJgr3u8uuK0LEZJjrU19DrMD3EVERaR8sjz8CCGgpZvxPl9SuE1GMVPFHx1mw==} engines: {node: '>= 0.4'} + is-network-error@1.3.2: + resolution: {integrity: sha512-PhBY86zaxNZUuWP6h13Vu5oFe0XY6/UlKzQnYFELzGVHygP3MxmvTfYSG7GN3aIab/iWudSMgjSnG9Dq+nHrgA==} + engines: {node: '>=16'} + is-number-object@1.1.1: resolution: {integrity: sha512-lZhclumE1G6VYD8VHe35wFaIif+CTy5SJIi5+3y4psDgWu4wPDoBhF8NxUOinEc7pHgiTsT6MaBb92rKhhD+Xw==} engines: {node: '>= 0.4'} @@ -9289,6 +9559,9 @@ packages: resolution: {integrity: sha512-RuCOQMIt78LWnktPoeBL0GErkNaJPTBGcYuyaBvUOQSpcpcLfWrHPPihYdOGbV5pam9VTWbeoF7TsGiHugcjGA==} engines: {node: '>= 20'} + jose@5.10.0: + resolution: {integrity: sha512-s+3Al/p9g32Iq+oqXxkW//7jk2Vig6FF1CFqzVXoTUXt2qz89YWbL+OwS17NFYEvxC35n0FKeGO2LGYSxeM2Gg==} + jose@6.2.12: resolution: {integrity: sha512-9NiFmJEex0sy2Dk58j2UGBSHgUs2ypF9eZSu4L6vjOX3Dp96Sw1F3uL+H+D1sx02jZZdzUT0HgvCy59CuvXcWw==} @@ -9753,6 +10026,10 @@ packages: resolution: {integrity: sha512-Jo6dJ04CmSjuznwJSS3pUeWmd/H0ffTlkXXgwZi+eq1UCmqQwCh+eLsYOYCwY991i2Fah4h1BEMCx4qThGbsiA==} engines: {node: '>=10'} + lucia@3.2.2: + resolution: {integrity: sha512-P1FlFBGCMPMXu+EGdVD9W4Mjm0DqsusmKgO7Xc33mI5X1bklmsQb0hfzPhXomQr9waWIBDsiOjvr1e6BTaUqpA==} + deprecated: This package has been deprecated. Please see https://lucia-auth.com/lucia-v3/migrate. + lunr@2.3.9: resolution: {integrity: sha512-zTU3DaZaF3Rt9rhN3uBMGQD3dD2/vFQqnvZCDv4dl5iOzq2IZQqTxu90r4E5J+nP70J3ilqVCrbho2eWaeW8Ow==} @@ -10203,6 +10480,9 @@ packages: nullthrows@1.1.1: resolution: {integrity: sha512-2vPPEi+Z7WqML2jZYddDIfy5Dqb0r2fze2zTxNNknZaFpVHU3mFB3R+DWeJWGVx0ecvttSGlJTI+WG+8Z4cDWw==} + oauth4webapi@3.8.8: + resolution: {integrity: sha512-8N28E+a/oxfXWBgOMt+ZP/JUf/XR+IFbvkAEPP3gznXOMv9BpAAwiIj0TFNz3tGTPc0ZQ8zmWBNgN1nAys0gng==} + ob1@0.82.5: resolution: {integrity: sha512-QyQQ6e66f+Ut/qUVjEce0E/wux5nAGLXYZDn1jr15JWstHsCH3l6VVrg8NKDptW9NEiBXKOJeGF/ydxeSDF3IQ==} engines: {node: '>=18.18'} @@ -10588,6 +10868,14 @@ packages: resolution: {integrity: sha512-d+JFcLM17njZaOLkv6SCev7uoLaBtfK86vMUXhW1Z4glPWh4jozno9APvW/XKFJ3CCxVoC7OL38BqRydtu5nGg==} engines: {node: '>=12'} + preact-render-to-string@6.5.11: + resolution: {integrity: sha512-ubnauqoGczeGISiOh6RjX0/cdaF8v/oDXIjO85XALCQjwQP+SB4RDXXtvZ6yTYSjG+PC1QRP2AhPgCEsM2EvUw==} + peerDependencies: + preact: '>=10' + + preact@10.24.3: + resolution: {integrity: sha512-Z2dPnBnMUfyQfSQ+GBdsGa16hz35YmLmtTLhM169uW944hYL6xzTYkJjC07j+Wosz733pMWx0fgON3JNw1jJQA==} + prebuild-install@7.1.3: resolution: {integrity: sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==} engines: {node: '>=10'} @@ -12362,6 +12650,18 @@ packages: utf-8-validate: optional: true + ws@8.21.0: + resolution: {integrity: sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==} + engines: {node: '>=10.0.0'} + peerDependencies: + bufferutil: ^4.0.1 + utf-8-validate: '>=5.0.2' + peerDependenciesMeta: + bufferutil: + optional: true + utf-8-validate: + optional: true + xcode@3.0.1: resolution: {integrity: sha512-kCz5k7J7XbJtjABOvkc5lJmkiDh8VhjVCGNiqdKCscmVpdVUpEAyXv1xmCLkQJ5dsHqx3IPO4XW+NTDhU/fatA==} engines: {node: '>=10.0.0'} @@ -12769,6 +13069,14 @@ snapshots: '@asamuzakjp/nwsapi@2.3.9': {} + '@auth/core@0.41.3': + dependencies: + '@panva/hkdf': 1.2.1 + jose: 6.2.12 + oauth4webapi: 3.8.8 + preact: 10.24.3 + preact-render-to-string: 6.5.11(preact@10.24.3) + '@babel/code-frame@7.10.4': dependencies: '@babel/highlight': 7.25.9 @@ -13661,6 +13969,23 @@ snapshots: '@types/conventional-commits-parser': 5.0.2 chalk: 5.6.2 + '@convex-dev/auth@0.0.96(@auth/core@0.41.3)(convex@1.47.0(react@19.2.4))(react@19.2.4)': + dependencies: + '@auth/core': 0.41.3 + '@oslojs/crypto': 1.0.1 + '@oslojs/encoding': 1.1.0 + convex: 1.47.0(react@19.2.4) + cookie: 1.1.1 + is-network-error: 1.3.2 + jose: 5.10.0 + jwt-decode: 4.0.0 + lucia: 3.2.2 + oauth4webapi: 3.8.8 + path-to-regexp: 6.3.0 + server-only: 0.0.1 + optionalDependencies: + react: 19.2.4 + '@cspotcode/source-map-support@0.8.1': dependencies: '@jridgewell/trace-mapping': 0.3.9 @@ -13750,6 +14075,9 @@ snapshots: '@esbuild/aix-ppc64@0.25.11': optional: true + '@esbuild/aix-ppc64@0.27.0': + optional: true + '@esbuild/aix-ppc64@0.27.3': optional: true @@ -13765,6 +14093,9 @@ snapshots: '@esbuild/android-arm64@0.25.11': optional: true + '@esbuild/android-arm64@0.27.0': + optional: true + '@esbuild/android-arm64@0.27.3': optional: true @@ -13780,6 +14111,9 @@ snapshots: '@esbuild/android-arm@0.25.11': optional: true + '@esbuild/android-arm@0.27.0': + optional: true + '@esbuild/android-arm@0.27.3': optional: true @@ -13795,6 +14129,9 @@ snapshots: '@esbuild/android-x64@0.25.11': optional: true + '@esbuild/android-x64@0.27.0': + optional: true + '@esbuild/android-x64@0.27.3': optional: true @@ -13810,6 +14147,9 @@ snapshots: '@esbuild/darwin-arm64@0.25.11': optional: true + '@esbuild/darwin-arm64@0.27.0': + optional: true + '@esbuild/darwin-arm64@0.27.3': optional: true @@ -13825,6 +14165,9 @@ snapshots: '@esbuild/darwin-x64@0.25.11': optional: true + '@esbuild/darwin-x64@0.27.0': + optional: true + '@esbuild/darwin-x64@0.27.3': optional: true @@ -13840,6 +14183,9 @@ snapshots: '@esbuild/freebsd-arm64@0.25.11': optional: true + '@esbuild/freebsd-arm64@0.27.0': + optional: true + '@esbuild/freebsd-arm64@0.27.3': optional: true @@ -13855,6 +14201,9 @@ snapshots: '@esbuild/freebsd-x64@0.25.11': optional: true + '@esbuild/freebsd-x64@0.27.0': + optional: true + '@esbuild/freebsd-x64@0.27.3': optional: true @@ -13870,6 +14219,9 @@ snapshots: '@esbuild/linux-arm64@0.25.11': optional: true + '@esbuild/linux-arm64@0.27.0': + optional: true + '@esbuild/linux-arm64@0.27.3': optional: true @@ -13885,6 +14237,9 @@ snapshots: '@esbuild/linux-arm@0.25.11': optional: true + '@esbuild/linux-arm@0.27.0': + optional: true + '@esbuild/linux-arm@0.27.3': optional: true @@ -13900,6 +14255,9 @@ snapshots: '@esbuild/linux-ia32@0.25.11': optional: true + '@esbuild/linux-ia32@0.27.0': + optional: true + '@esbuild/linux-ia32@0.27.3': optional: true @@ -13915,6 +14273,9 @@ snapshots: '@esbuild/linux-loong64@0.25.11': optional: true + '@esbuild/linux-loong64@0.27.0': + optional: true + '@esbuild/linux-loong64@0.27.3': optional: true @@ -13930,6 +14291,9 @@ snapshots: '@esbuild/linux-mips64el@0.25.11': optional: true + '@esbuild/linux-mips64el@0.27.0': + optional: true + '@esbuild/linux-mips64el@0.27.3': optional: true @@ -13945,6 +14309,9 @@ snapshots: '@esbuild/linux-ppc64@0.25.11': optional: true + '@esbuild/linux-ppc64@0.27.0': + optional: true + '@esbuild/linux-ppc64@0.27.3': optional: true @@ -13960,6 +14327,9 @@ snapshots: '@esbuild/linux-riscv64@0.25.11': optional: true + '@esbuild/linux-riscv64@0.27.0': + optional: true + '@esbuild/linux-riscv64@0.27.3': optional: true @@ -13975,6 +14345,9 @@ snapshots: '@esbuild/linux-s390x@0.25.11': optional: true + '@esbuild/linux-s390x@0.27.0': + optional: true + '@esbuild/linux-s390x@0.27.3': optional: true @@ -13990,6 +14363,9 @@ snapshots: '@esbuild/linux-x64@0.25.11': optional: true + '@esbuild/linux-x64@0.27.0': + optional: true + '@esbuild/linux-x64@0.27.3': optional: true @@ -14002,6 +14378,9 @@ snapshots: '@esbuild/netbsd-arm64@0.25.11': optional: true + '@esbuild/netbsd-arm64@0.27.0': + optional: true + '@esbuild/netbsd-arm64@0.27.3': optional: true @@ -14017,6 +14396,9 @@ snapshots: '@esbuild/netbsd-x64@0.25.11': optional: true + '@esbuild/netbsd-x64@0.27.0': + optional: true + '@esbuild/netbsd-x64@0.27.3': optional: true @@ -14029,6 +14411,9 @@ snapshots: '@esbuild/openbsd-arm64@0.25.11': optional: true + '@esbuild/openbsd-arm64@0.27.0': + optional: true + '@esbuild/openbsd-arm64@0.27.3': optional: true @@ -14044,6 +14429,9 @@ snapshots: '@esbuild/openbsd-x64@0.25.11': optional: true + '@esbuild/openbsd-x64@0.27.0': + optional: true + '@esbuild/openbsd-x64@0.27.3': optional: true @@ -14056,6 +14444,9 @@ snapshots: '@esbuild/openharmony-arm64@0.25.11': optional: true + '@esbuild/openharmony-arm64@0.27.0': + optional: true + '@esbuild/openharmony-arm64@0.27.3': optional: true @@ -14071,6 +14462,9 @@ snapshots: '@esbuild/sunos-x64@0.25.11': optional: true + '@esbuild/sunos-x64@0.27.0': + optional: true + '@esbuild/sunos-x64@0.27.3': optional: true @@ -14086,6 +14480,9 @@ snapshots: '@esbuild/win32-arm64@0.25.11': optional: true + '@esbuild/win32-arm64@0.27.0': + optional: true + '@esbuild/win32-arm64@0.27.3': optional: true @@ -14101,6 +14498,9 @@ snapshots: '@esbuild/win32-ia32@0.25.11': optional: true + '@esbuild/win32-ia32@0.27.0': + optional: true + '@esbuild/win32-ia32@0.27.3': optional: true @@ -14116,6 +14516,9 @@ snapshots: '@esbuild/win32-x64@0.25.11': optional: true + '@esbuild/win32-x64@0.27.0': + optional: true + '@esbuild/win32-x64@0.27.3': optional: true @@ -14307,7 +14710,7 @@ snapshots: terminal-link: 2.1.1 toqr: 0.1.1 wrap-ansi: 7.0.0 - ws: 8.20.0 + ws: 8.21.0 zod: 3.25.76 optionalDependencies: expo-router: 5.1.11(@types/react@19.2.13)(expo-constants@55.0.9)(expo-linking@7.1.7)(expo@55.0.8)(react-native-safe-area-context@5.4.0(react-native@0.83.2(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.2.0))(react@19.2.0))(react-native-screens@4.11.1(react-native@0.83.2(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.2.0))(react@19.2.0))(react-native@0.83.2(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.2.0))(react@19.2.0) @@ -14383,7 +14786,7 @@ snapshots: terminal-link: 2.1.1 toqr: 0.1.1 wrap-ansi: 7.0.0 - ws: 8.20.0 + ws: 8.21.0 zod: 3.25.76 optionalDependencies: expo-router: 5.1.11(@types/react@19.2.13)(expo-constants@55.0.9)(expo-linking@7.1.7)(expo@55.0.8)(react-native-safe-area-context@5.4.0(react-native@0.83.2(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.2.4))(react@19.2.4))(react-native-screens@4.11.1(react-native@0.83.2(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.2.4))(react@19.2.4))(react-native@0.83.2(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.2.4))(react@19.2.4) @@ -15774,6 +16177,19 @@ snapshots: '@opentelemetry/semantic-conventions@1.43.0': {} + '@oslojs/asn1@1.0.0': + dependencies: + '@oslojs/binary': 1.0.0 + + '@oslojs/binary@1.0.0': {} + + '@oslojs/crypto@1.0.1': + dependencies: + '@oslojs/asn1': 1.0.0 + '@oslojs/binary': 1.0.0 + + '@oslojs/encoding@1.1.0': {} + '@oxc-resolver/binding-android-arm-eabi@11.16.2': optional: true @@ -15836,6 +16252,8 @@ snapshots: '@oxc-resolver/binding-win32-x64-msvc@11.16.2': optional: true + '@panva/hkdf@1.2.1': {} + '@parcel/watcher-android-arm64@2.5.1': optional: true @@ -19087,6 +19505,17 @@ snapshots: convert-source-map@2.0.0: {} + convex@1.47.0(react@19.2.4): + dependencies: + esbuild: 0.27.0 + prettier: 3.9.9 + ws: 8.21.0 + optionalDependencies: + react: 19.2.4 + transitivePeerDependencies: + - bufferutil + - utf-8-validate + cookie-es@3.1.1: {} cookie-signature@1.2.2: {} @@ -19585,6 +20014,35 @@ snapshots: '@esbuild/win32-ia32': 0.25.11 '@esbuild/win32-x64': 0.25.11 + esbuild@0.27.0: + optionalDependencies: + '@esbuild/aix-ppc64': 0.27.0 + '@esbuild/android-arm': 0.27.0 + '@esbuild/android-arm64': 0.27.0 + '@esbuild/android-x64': 0.27.0 + '@esbuild/darwin-arm64': 0.27.0 + '@esbuild/darwin-x64': 0.27.0 + '@esbuild/freebsd-arm64': 0.27.0 + '@esbuild/freebsd-x64': 0.27.0 + '@esbuild/linux-arm': 0.27.0 + '@esbuild/linux-arm64': 0.27.0 + '@esbuild/linux-ia32': 0.27.0 + '@esbuild/linux-loong64': 0.27.0 + '@esbuild/linux-mips64el': 0.27.0 + '@esbuild/linux-ppc64': 0.27.0 + '@esbuild/linux-riscv64': 0.27.0 + '@esbuild/linux-s390x': 0.27.0 + '@esbuild/linux-x64': 0.27.0 + '@esbuild/netbsd-arm64': 0.27.0 + '@esbuild/netbsd-x64': 0.27.0 + '@esbuild/openbsd-arm64': 0.27.0 + '@esbuild/openbsd-x64': 0.27.0 + '@esbuild/openharmony-arm64': 0.27.0 + '@esbuild/sunos-x64': 0.27.0 + '@esbuild/win32-arm64': 0.27.0 + '@esbuild/win32-ia32': 0.27.0 + '@esbuild/win32-x64': 0.27.0 + esbuild@0.27.3: optionalDependencies: '@esbuild/aix-ppc64': 0.27.3 @@ -21025,6 +21483,8 @@ snapshots: is-negative-zero@2.0.3: {} + is-network-error@1.3.2: {} + is-number-object@1.1.1: dependencies: call-bound: 1.0.4 @@ -21271,6 +21731,8 @@ snapshots: '@hapi/topo': 6.0.2 '@standard-schema/spec': 1.1.0 + jose@5.10.0: {} + jose@6.2.12: {} js-base64@3.7.8: {} @@ -21698,6 +22160,11 @@ snapshots: dependencies: yallist: 4.0.0 + lucia@3.2.2: + dependencies: + '@oslojs/crypto': 1.0.1 + '@oslojs/encoding': 1.1.0 + lunr@2.3.9: {} lz-string@1.5.0: {} @@ -22287,6 +22754,8 @@ snapshots: nullthrows@1.1.1: {} + oauth4webapi@3.8.8: {} + ob1@0.82.5: dependencies: flow-enums-runtime: 0.0.6 @@ -22697,6 +23166,12 @@ snapshots: postgres@3.4.8: {} + preact-render-to-string@6.5.11(preact@10.24.3): + dependencies: + preact: 10.24.3 + + preact@10.24.3: {} + prebuild-install@7.1.3: dependencies: detect-libc: 2.1.2 @@ -24926,6 +25401,8 @@ snapshots: ws@8.20.0: {} + ws@8.21.0: {} + xcode@3.0.1: dependencies: simple-plist: 1.3.1 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 3f51a10356..dfb706b17c 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -9,6 +9,7 @@ trustPolicyExclude: packages: - packages/** + - '!packages/convex-db-collection/e2e/auth-app/runtime' - examples/**/* overrides: diff --git a/scripts/ci-tests.mjs b/scripts/ci-tests.mjs index 6c43e06889..d49df1f31d 100644 --- a/scripts/ci-tests.mjs +++ b/scripts/ci-tests.mjs @@ -28,6 +28,7 @@ const groups = { `@tanstack/angular-db`, ], collections: [ + `@tanstack/convex-db-collection`, `@tanstack/electric-db-collection`, `@tanstack/query-db-collection`, `@tanstack/powersync-db-collection`, diff --git a/scripts/generate-docs.ts b/scripts/generate-docs.ts index cbedd2a3c1..2a5ab8fdf3 100644 --- a/scripts/generate-docs.ts +++ b/scripts/generate-docs.ts @@ -12,6 +12,33 @@ await generateReferenceDocs({ tsconfig: resolve(__dirname, `../packages/db/tsconfig.docs.json`), outputDir: resolve(__dirname, `../docs/reference`), }, + { + name: `convex-db-collection`, + entryPoints: [ + resolve(__dirname, `../packages/convex-db-collection/src/index.ts`), + ], + tsconfig: resolve( + __dirname, + `../packages/convex-db-collection/tsconfig.docs.json`, + ), + outputDir: resolve(__dirname, `../docs/reference/convex-db-collection`), + exclude: [`packages/db/**/*`], + }, + { + name: `convex-db-collection/offline`, + entryPoints: [ + resolve(__dirname, `../packages/convex-db-collection/src/offline.ts`), + ], + tsconfig: resolve( + __dirname, + `../packages/convex-db-collection/tsconfig.docs.json`, + ), + outputDir: resolve( + __dirname, + `../docs/reference/convex-db-collection/offline`, + ), + exclude: [`packages/db/**/*`, `packages/offline-transactions/**/*`], + }, { name: `electric-db-collection`, entryPoints: [ diff --git a/scripts/size-compare.mjs b/scripts/size-compare.mjs index 838c79eca0..b7e7876c59 100644 --- a/scripts/size-compare.mjs +++ b/scripts/size-compare.mjs @@ -56,6 +56,7 @@ const flag = (name) => args.includes(`--${name}`) const DEFAULT_PACKAGES = [ `db`, + `convex-db-collection`, `react-db`, `query-db-collection`, `electric-db-collection`, @@ -216,6 +217,8 @@ function exportedEntries(pkgDir) { // is the compressed bundle; each file's entry is its minified bytes in it. async function measureProduction(dir, pkg) { const pkgDir = join(dir, `packages`, pkg) + // New or removed packages have no shipped bytes on the absent side. + if (!existsSync(pkgDir)) return new Map() const entries = exportedEntries(pkgDir).filter(existsSync) if (entries.length === 0) return new Map() const result = await esbuild({