From 91ee6db4f1e37fffb44ca4432d01e27c63bed45c Mon Sep 17 00:00:00 2001 From: Test Date: Fri, 23 Oct 2026 08:53:32 -0600 Subject: [PATCH 1/4] docs(oracles): score oracle-first work and add declared reach Agents kept taking the shortcut path: change production first, then fit the oracle with a relaxed comparison, an exclusion, a model rule that matches the new output, or one pinned example. The guidance already said not to. This change makes the economics explicit and adds one mechanical check. - AGENTS.md "How oracle work is scored": the oracle-first and shortcut paths and their costs, extra credit for proven oracle improvements and light production code, heavy penalties for model hacking and the other audited shortcuts, and a required evidence block (law, reach, RED, GREEN, mutant, limits). - ORC-015 declared reach, and `createOracleReach` in packages/db/tests/oracle-reach.ts: a generated property declares the states its laws need and fails when too few histories reach one. - oracle-authoring: the oracle-first order of work. oracle-review: an audit list for the penalized shortcuts. Co-authored-by: Isaac --- .agents/skills/oracle-authoring/SKILL.md | 21 +++++- .agents/skills/oracle-review/SKILL.md | 15 ++++ AGENTS.md | 88 ++++++++++++++++++++++++ docs/contributing/oracle-rubric.md | 17 +++++ packages/db/tests/oracle-reach.test.ts | 32 +++++++++ packages/db/tests/oracle-reach.ts | 62 +++++++++++++++++ 6 files changed, 234 insertions(+), 1 deletion(-) create mode 100644 packages/db/tests/oracle-reach.test.ts create mode 100644 packages/db/tests/oracle-reach.ts diff --git a/.agents/skills/oracle-authoring/SKILL.md b/.agents/skills/oracle-authoring/SKILL.md index bbbce6fc95..e40686211f 100644 --- a/.agents/skills/oracle-authoring/SKILL.md +++ b/.agents/skills/oracle-authoring/SKILL.md @@ -13,6 +13,24 @@ Use this skill whenever an oracle changes, or the production driver, history grammar, or comparison that it uses. This includes a change made while you fix a bug, fix a review finding, or evaluate review findings. +Read [how oracle work is scored](../../../AGENTS.md#how-oracle-work-is-scored) +first. Auditors check the order of your work and penalize shortcuts. + +Work in this order when a production change is part of the task: + +1. State the law and its authority. +2. Extend the history grammar, driver, or observation until generated + histories reach the state where the law matters. Declare that state with + `createOracleReach` (ORC-015). +3. Run the extended oracle on the unchanged production code and record the RED + failure. +4. Change production, with the oracle running after each attempt. +5. Record the GREEN result and a hostile mutant that fails. + +Do not change production before step 3. If a model rule disagrees with +production, find its authority before you edit either one. A rule that you +change only so that it matches production output is model hacking. + Read [the oracle guide](../../../docs/contributing/oracle-rubric.md), [the glossary](../../../docs/contributing/glossary.md), and the relevant owner and limits in [the coverage map](../../../docs/contributing/oracle-coverage.md). @@ -50,7 +68,8 @@ predictions, and the experiment that could discriminate them. Implement tests when requested by the task. A proposal with unresolved authority can be explored in a scratch model; do not silently make it a product expectation. -At closeout, connect the law to the evidence actually obtained: which histories +At closeout, write the evidence block from AGENTS.md (law, reach, RED, GREEN, +mutant, limits). Connect the law to the evidence actually obtained: which histories and paths ran, which wrong behavior was rejected, and what remains untested. Use concise prose suited to the change; no particular report template is required. When changing repository coverage, update the existing owner and diff --git a/.agents/skills/oracle-review/SKILL.md b/.agents/skills/oracle-review/SKILL.md index 1c3fd118ff..8e0acb7385 100644 --- a/.agents/skills/oracle-review/SKILL.md +++ b/.agents/skills/oracle-review/SKILL.md @@ -53,6 +53,21 @@ behavior, but must name the external standard or engineering reason, its applicability, and migration cost. Existing behavior does not establish its own correctness; a preference does not establish a contract violation. +Audit the change for the shortcuts that +[the scoring section](../../../AGENTS.md#how-oracle-work-is-scored) penalizes. +Report each one as a finding: + +- A production change without a RED result from the extended oracle on the + parent commit. +- A model rule that changed to match production output without an authority. +- A claim about a state that no declared reach shows generated histories enter. +- A skipped, narrowed, or relaxed comparison without a coverage-map entry and a + maintainer decision. +- A pinned example where the law needs a generator dimension. +- A counterexample labeled as older than the change and then left without a + fix, a coverage-map record, or a maintainer decision. +- A closure claim without a hostile mutant that fails the oracle. + Report supported findings and remaining uncertainty. Do not invent findings, retire laws, or rewrite expectations to make a review come out decisive. diff --git a/AGENTS.md b/AGENTS.md index e6b1273d14..8238e95815 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -37,6 +37,94 @@ oracle guide below and a small [instrument library](docs/contributing/instrument that is also available for other design work. Select instruments as needed; routine work does not require a full instrument sequence. +## How oracle work is scored + +Every behavior change in an area that an oracle owns has two possible paths. +The maintainer runs hostile auditors on every pull request. The auditors look +for the shortcuts listed below and report each one. Pick the path with this in +mind: a shortcut does not finish the task sooner. It moves the work into +review, where it costs more. + +### The oracle-first path + +1. State the law and its authority. +2. Extend the oracle so that generated histories reach the state where the law + matters. Declare that state with + [`createOracleReach`](packages/db/tests/oracle-reach.ts), and show that the + generated histories reach it. +3. Run the oracle against the unchanged production code. It fails: this is the + RED result, and it proves that the oracle sees the defect. +4. Change production. The oracle gives immediate feedback on each attempt, + including attempts that break a neighboring law. +5. Show that a hostile mutant, such as the original defect, fails the oracle. + +On this path, you know the fix is correct before you push it. Reviewers find +little that is new, so the total work and token spend stay low. Later agents +inherit a clean model of the subsystem that they can extend in place. + +### The shortcut path + +Change production first, then adjust the oracle until it passes: relax a +comparison, add an exclusion, change a model rule to match the new output, or +add one pinned example. The tests pass, but nothing shows that the oracle can +see the defect class. Auditors find the gap. Each finding costs a fix, a new +witness, and another review round, and the next fix often breaks a +neighboring law that the oracle still cannot see. + +In one recorded case, a production-first rewrite of reload handling needed four +more review rounds and about ten follow-up commits. Each round found a defect +in the previous round's fix. A grammar-first pass then found the last defect on +its first run, and the fix that followed made the rule smaller. The result is +usually the same code, after much more work, rework, and maintainer +frustration. + +### Extra credit + +Work earns extra credit for: + +- an oracle improvement that has proof: a declared reach, a RED result on the + parent commit, a GREEN result, and a hostile mutant that fails. +- a model that stays independent of production and short, with each rule tied + to its authority. +- a production change that is clean and light in shipped bytes, especially one + that removes states, branches, or recovery paths. + +### Penalties + +Auditors penalize these shortcuts heavily, and a pull request that contains one +is not ready: + +- **Production first:** a production change in an oracle-owned area without a + RED result from the extended oracle on the code before the change. +- **Model hacking:** a model rule changed to match production output without a + stated authority (a contract document and section, or a maintainer + decision). Production output is evidence. It is not authority. +- **Unreachable claims:** a closure claim for a state that generated histories + do not reach, or a reach that nobody measured. +- **Quiet exclusions:** a skipped, narrowed, or relaxed comparison without an + entry in the [coverage map](docs/contributing/oracle-coverage.md) that names + the owner and the maintainer decision. +- **Pinned-only coverage:** a named example where the law needs a generator + dimension. +- **"Pre-existing" as a dismissal:** a reachable counterexample that an agent + labels as older than the change, but does not fix, record in the coverage + map with its trace, or bring to the maintainer for a decision. +- **No mutant:** a closure claim without a hostile mutant that fails the + oracle. + +### The evidence block + +Every pull request or commit that changes behavior in an oracle-owned area +includes this block: + +- **Law:** the law and its authority. +- **Reach:** the declared state and how many generated histories reached it. +- **RED:** the command and the failure on the parent commit. +- **GREEN:** the command and the result after the change. +- **Mutant:** the hostile mutant and how the oracle rejected it. +- **Limits:** the states, paths, and observations that the evidence does not + cover. + ## Required reading: live-query materialization Before reading, analyzing, or modifying correlated live-query materialization diff --git a/docs/contributing/oracle-rubric.md b/docs/contributing/oracle-rubric.md index a6fda20f17..4b3fc0f6dc 100644 --- a/docs/contributing/oracle-rubric.md +++ b/docs/contributing/oracle-rubric.md @@ -291,6 +291,23 @@ presence of a preferred heading, class, comment template, or helper. whose claim is explicitly limited to the controlled boundary does not need a real-provider witness to satisfy this requirement. +### ORC-015: Declared reach + +- **Trigger:** A pull request adds or changes an important generated property, + its history grammar, or its production driver, or cites one as evidence for a + bug-class repair. +- **Obligation:** The property MUST declare the states that its changed or + cited laws depend on, and MUST fail when generated histories reach a declared + state fewer times than its stated minimum. Use + [`createOracleReach`](../../packages/db/tests/oracle-reach.ts) or an + equivalent counter that fails the property. A passing run count does not + satisfy this requirement. +- **Acceptance evidence:** The pull request reports each declared state and how + many histories reached it in the normal test run. A state that only a larger + run budget reaches is reported as such. +- **Not required:** A declared state for every modeled transition. Declare the + states that the cited laws need, so that reach stays honest about scope. + ## A quick start One way to supply that judgment is a small reference model. For a simple stateful test, choose a legal action, apply it to production and the reference, and compare the promised result at the agreed point. “After the action returns” may be the right point for one API; a callback during that action may matter for another. diff --git a/packages/db/tests/oracle-reach.test.ts b/packages/db/tests/oracle-reach.test.ts new file mode 100644 index 0000000000..51f6f08a35 --- /dev/null +++ b/packages/db/tests/oracle-reach.test.ts @@ -0,0 +1,32 @@ +import { describe, expect, it } from 'vitest' +import { createOracleReach } from './oracle-reach.js' + +describe(`createOracleReach`, () => { + it(`fails for a declared state that no history entered`, () => { + const reach = createOracleReach(`example`, [`seen`, `unseen`] as const) + reach.beginHistory() + reach.reach(`seen`) + expect(() => reach.assertReached()).toThrow(/unseen \(0\)/) + }) + + it(`counts a history once per state`, () => { + const reach = createOracleReach(`example`, [`state`] as const, { + minimum: 2, + }) + reach.beginHistory() + reach.reach(`state`) + reach.reach(`state`) + expect(reach.counts().get(`state`)).toBe(1) + expect(() => reach.assertReached()).toThrow(/state \(1\)/) + reach.beginHistory() + reach.reach(`state`) + expect(() => reach.assertReached()).not.toThrow() + }) + + it(`rejects an undeclared state`, () => { + const reach = createOracleReach(`example`, [`state`] as const) + expect(() => reach.reach(`other` as unknown as `state`)).toThrow( + /undeclared reach state other/, + ) + }) +}) diff --git a/packages/db/tests/oracle-reach.ts b/packages/db/tests/oracle-reach.ts new file mode 100644 index 0000000000..9fa10c0cf3 --- /dev/null +++ b/packages/db/tests/oracle-reach.ts @@ -0,0 +1,62 @@ +import { readOracleRunConfig } from './oracle-config.js' + +/** + * Declared reach for a generated-history property. + * + * A law is enforced only in the states that generated histories reach. A + * property can pass any number of runs and still check nothing in the state + * its law is about, because the grammar, the driver, or the run budget never + * produces that state. Counting passing runs does not show reach. + * + * A property declares the states its laws depend on. Its driver calls + * `reach(state)` when a history enters one. After the property ran, + * `assertReached()` fails for every declared state that fewer than `minimum` + * histories entered. A history counts once per state, however often it + * enters it. + * + * A replay of one seed or path checks one history, not reach, so the check + * does nothing then. + */ +export function createOracleReach( + property: string, + states: ReadonlyArray, + { minimum = 1 }: { minimum?: number } = {}, +) { + const counts = new Map(states.map((state) => [state, 0])) + let current = new Set() + const { replaySeed, replayPath } = readOracleRunConfig() + const replaying = replaySeed !== undefined || replayPath !== undefined + return { + /** Starts one generated history. */ + beginHistory(): void { + current = new Set() + }, + /** The current history entered a declared state. */ + reach(state: TState): void { + if (!counts.has(state)) { + throw new Error(`${property}: undeclared reach state ${state}`) + } + if (current.has(state)) return + current.add(state) + counts.set(state, counts.get(state)! + 1) + }, + /** How many histories entered each declared state. */ + counts(): ReadonlyMap { + return counts + }, + /** Fails for each declared state that too few histories entered. */ + assertReached(): void { + if (replaying) return + const missed = [...counts] + .filter(([, count]) => count < minimum) + .map(([state, count]) => `${state} (${count})`) + if (missed.length > 0) { + throw new Error( + `${property}: generated histories reached these states fewer than ` + + `${minimum} times, so their laws were not enforced there: ` + + missed.join(`, `), + ) + } + }, + } +} From 098a06ae061dabbf44a54668d979d49e3cfcbec7 Mon Sep 17 00:00:00 2001 From: Test Date: Fri, 23 Oct 2026 09:11:10 -0600 Subject: [PATCH 2/4] docs(oracles): plan the oracle lint rules, CI checks, and agent feedback hooks Co-authored-by: Isaac --- .../designs/oracle-enforcement/PLAN.md | 138 ++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100644 docs/contributing/designs/oracle-enforcement/PLAN.md diff --git a/docs/contributing/designs/oracle-enforcement/PLAN.md b/docs/contributing/designs/oracle-enforcement/PLAN.md new file mode 100644 index 0000000000..33153111c6 --- /dev/null +++ b/docs/contributing/designs/oracle-enforcement/PLAN.md @@ -0,0 +1,138 @@ +# Oracle enforcement: plan and handoff + +Status: plan. A later session implements it. This document carries the context +that session needs. + +## Why + +Agents keep taking the shortcut path when they change behavior in an area that +an oracle owns. They change production first, then make the oracle pass with a +relaxed comparison, an exclusion, a model rule that matches the new output, or +one pinned example. `AGENTS.md` and the oracle guide already forbid this. The +prose did not stop it, because nothing made the shortcut visible or slow. + +PR #2233 adds the scoring section in `AGENTS.md` ("How oracle work is scored"), +ORC-015 declared reach, and `createOracleReach` in +`packages/db/tests/oracle-reach.ts`. This plan adds the mechanical checks. + +The evidence comes from PR #2168 (source-health channel). A production-first +rewrite of reload handling there needed four more review rounds and about ten +follow-up commits. Each round found a defect in the previous fix. These +shortcuts appeared in that work: + +- Production changed before the oracle could see the defect. +- A truncate step could never land while a reload was in flight, because the + driver's delayed `markReady()` always ran within the same step. The oracle + was green in a state that it never reached. +- A direct-subscriber observation existed only on eager sources. The + subscription lifecycle oracle subscribed without initial state, so its + replay flush published nothing. +- Model rules changed to match production traces. Two of them (an automatic + query restart, and "a key requested again drops a replay failure") were later + found wrong and removed. +- Comparison exclusions (`replayOpen`, a narrowed identity check) widened + without a fresh maintainer decision. +- Most edits went through shell commands (Python and `sed`), so an edit hook + that matches only the harness's own edit tool would not have fired. + +## Order of work + +1. Install the agent feedback hooks with a small starter rule set, so that the + session that does the cleanup gets feedback as it works. +2. Add the custom ESLint oracle rules. Run them over the repository and record + every existing violation. +3. Clean up the violations owner by owner, each with the oracle-first order + from `AGENTS.md`. Expect to find real gaps: a conditional `expect` often + hides an unreached state. +4. Add the CI jobs (RED replay, owner coupling, model tripwire). +5. Adopt `createOracleReach` in `packages/db/tests/source-health-oracle.test.ts` + (PR #2168, or a follow-up after it merges). + +## Static checks: custom ESLint rules + +Apply these rules to files whose names contain `oracle`. Keep them syntactic +and fast, so the edit hook can run them on one file. + +| Rule | Reports | Notes | +| --- | --- | --- | +| `oracle/declared-reach` | A generated property (`fcTest.prop`, `fc.assert`) in a file that has no `createOracleReach` call and no `assertReached()` call. | Few false positives. Several oracles count reach in their own way today, for example `collection-subscription-lifecycle-history-oracle.property.test.ts`. Convert them or allow a documented equivalent. | +| `oracle/no-conditional-expect` | An `expect` under an `if`, a ternary, `&&`, `\|\|`, or after an early `return` or `continue` inside a comparison. | This is the main exclusion pattern. Allow it only with `// oracle-exclusion: `. A script checks that `` exists in `docs/contributing/oracle-coverage.md`. Some conditional checks are legitimate (a check that applies only in a modeled state). The comment covers those, and the auditor judges the reason. | +| `oracle/model-independence` | A model function that imports from, or calls into, production `src/`. | Needs a convention first: models in `*-model.ts` files, or a marked region. Drivers may import production. Models may not, except public types. | +| `oracle/no-disabled` | `.only`, `.skip`, or `.todo` on a generated property, and a literal run count below the base, such as `numRuns: 1`. | Trivial. | +| `oracle/registered-property` | A property name that `oracle-config.ts` does not register. | The registry already checks this at run time. Lint reports it earlier. | + +## Diff checks in CI + +- **RED replay.** The PR's evidence block names the oracle tests. CI runs those + tests from the PR head against the merge-base production `src/`. They must + fail there and pass on the head. This checks the oracle-first order + mechanically: an oracle that cannot see the defect cannot produce RED. +- **Owner coupling.** A production change in an area that the coverage map + assigns to an owner, without a change to that owner and without an evidence + block, fails. This needs a machine-readable table of owners to production + paths, which the coverage map does not have yet. +- **Model tripwire.** A PR that changes both production and a model region gets + a flag for the auditor. It does not fail the build. It makes the pair hard to + miss. + +## Run-time checks + +- Declared reach (`createOracleReach`, ORC-015). +- Mutation testing (for example Stryker) on the production files that each + oracle owns, as a nightly job. A low kill rate for an owner shows an oracle + that accepts wrong implementations. It is too slow for each PR. + +## Immediate agent feedback across harnesses + +Contributors use several agent tools. Most now have a post-edit hook that can +return lint output to the model. Each has its own configuration path, tool +names, and feedback protocol. Research from 2026-10-23: + +| Harness | Post-edit hook | Configuration | Feedback to the model | End-of-turn hook that can force a retry | +| --- | --- | --- | --- | --- | +| Claude Code | `PostToolUse`, matcher `Edit\|Write` | `.claude/settings.json` | exit `2` with stderr | `Stop` | +| Codex CLI | `PostToolUse`, matcher `apply_patch` | `.codex/hooks.json`; the project must be trusted, and the hook approved in `/hooks` | exit `2` with stderr | not found | +| Cursor | `afterFileEdit` (stdin has `file_path`) | `.cursor/hooks.json` | not confirmed: the documentation shows a logging-style hook | not confirmed | +| Gemini CLI | `AfterTool`, matcher `write_file\|replace` | `.gemini/settings.json` | JSON `hookSpecificOutput.additionalContext` | `AfterAgent` with `decision: "deny"` | +| GitHub Copilot coding agent | `postToolUse` | `.github/hooks/*.json`; read only from the default branch | JSON `additionalContext`; a nonzero exit does not block | `agentStop` with `decision: "block"` | +| Windsurf / Devin Desktop | `post_write_code` (stdin has `tool_info.file_path`) | `.windsurf/hooks.json` or `.devin/hooks.json` | exit `2` with stderr | not found | +| Cline | executable `PostToolUse` | `.clinerules/hooks/PostToolUse`; turn on "Enable Hooks" | JSON `contextModification` | not found | +| OpenCode | plugin `tool.execute.after` | `.opencode/plugins/*.ts` | append to the tool's `output.output`; `"lsp": true` also reports LSP diagnostics on edits | not found | +| Aider | built-in `--auto-lint` with `--lint-cmd` | `.aider.conf.yml` | built in: a nonzero exit makes Aider offer a fix | not applicable | + +Sources: code.claude.com/docs/en/hooks, developers.openai.com/codex/hooks, +cursor.com/docs/hooks, geminicli.com/docs/hooks/reference, +docs.github.com/en/copilot/reference/hooks-reference, +docs.windsurf.com/windsurf/cascade/hooks, the Cline repository's +`.clinerules/hooks/README.md`, opencode.ai/docs/plugins, and +aider.chat/docs/usage/lint-test.html. Check each one again before you build: +these APIs change often. + +### Design + +1. **One shared script with thin adapters.** `scripts/agent-lint-feedback.mjs` + reads the hook's JSON from stdin, finds the edited paths (each harness uses + a different field, and Codex sends the patch text of `apply_patch`), runs + ESLint on those files only, and writes the response in the format the + harness expects (`--format=claude|codex|gemini|copilot|cline|windsurf`). + Each harness gets a small committed configuration that calls the script. +2. **An end-of-turn hook for shell edits.** A post-edit hook matches only the + harness's own edit tool. Where a harness has an end-of-turn hook, run lint on + `git diff --name-only` there. That catches shell edits too, and it can force + another turn. For the other harnesses, the pre-commit hook is the fallback. +3. **Speed.** A hook that takes seconds on each edit gets turned off. Use + `eslint --cache` on single files, or a daemon such as `eslint_d`. Keep the + oracle rules syntactic, and keep type-aware rules out of the edit hook. +4. **Fallbacks.** Some harnesses need setup by each contributor (Codex trust, + the Cline toggle, the Copilot default branch). Keep these layers: + - an `AGENTS.md` instruction to run `pnpm lint:changed` after edits; + - editor diagnostics, which IDE-based agents read; + - the pre-commit hook; + - CI. + +### Pre-commit + +`lint-staged` already runs `eslint --fix` on staged files. The oracle rules run +there once they are in the ESLint configuration. On a commit of more than about +100 files, ESLint ran out of memory in PR #2168. Lint in chunks, or raise the +Node heap in the hook. From f41f4c99ac696712828cc325bb90338e85748e55 Mon Sep 17 00:00:00 2001 From: Test Date: Fri, 23 Oct 2026 09:11:59 -0600 Subject: [PATCH 3/4] docs(oracles): split long plan sentences Co-authored-by: Isaac --- .../designs/oracle-enforcement/PLAN.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/contributing/designs/oracle-enforcement/PLAN.md b/docs/contributing/designs/oracle-enforcement/PLAN.md index 33153111c6..b76c2c0731 100644 --- a/docs/contributing/designs/oracle-enforcement/PLAN.md +++ b/docs/contributing/designs/oracle-enforcement/PLAN.md @@ -33,7 +33,7 @@ shortcuts appeared in that work: - Comparison exclusions (`replayOpen`, a narrowed identity check) widened without a fresh maintainer decision. - Most edits went through shell commands (Python and `sed`), so an edit hook - that matches only the harness's own edit tool would not have fired. + that matches only the harness's own edit tool would not fire. ## Order of work @@ -91,13 +91,13 @@ names, and feedback protocol. Research from 2026-10-23: | Harness | Post-edit hook | Configuration | Feedback to the model | End-of-turn hook that can force a retry | | --- | --- | --- | --- | --- | | Claude Code | `PostToolUse`, matcher `Edit\|Write` | `.claude/settings.json` | exit `2` with stderr | `Stop` | -| Codex CLI | `PostToolUse`, matcher `apply_patch` | `.codex/hooks.json`; the project must be trusted, and the hook approved in `/hooks` | exit `2` with stderr | not found | -| Cursor | `afterFileEdit` (stdin has `file_path`) | `.cursor/hooks.json` | not confirmed: the documentation shows a logging-style hook | not confirmed | +| Codex CLI | `PostToolUse`, matcher `apply_patch` | `.codex/hooks.json`. The project must be trusted, and the hook approved in `/hooks` | exit `2` with stderr | not found | +| Cursor | `afterFileEdit` (stdin has `file_path`) | `.cursor/hooks.json` | not confirmed. The documentation shows a logging-style hook | not confirmed | | Gemini CLI | `AfterTool`, matcher `write_file\|replace` | `.gemini/settings.json` | JSON `hookSpecificOutput.additionalContext` | `AfterAgent` with `decision: "deny"` | -| GitHub Copilot coding agent | `postToolUse` | `.github/hooks/*.json`; read only from the default branch | JSON `additionalContext`; a nonzero exit does not block | `agentStop` with `decision: "block"` | +| GitHub Copilot coding agent | `postToolUse` | `.github/hooks/*.json`, read only from the default branch | JSON `additionalContext`. A nonzero exit does not block | `agentStop` with `decision: "block"` | | Windsurf / Devin Desktop | `post_write_code` (stdin has `tool_info.file_path`) | `.windsurf/hooks.json` or `.devin/hooks.json` | exit `2` with stderr | not found | -| Cline | executable `PostToolUse` | `.clinerules/hooks/PostToolUse`; turn on "Enable Hooks" | JSON `contextModification` | not found | -| OpenCode | plugin `tool.execute.after` | `.opencode/plugins/*.ts` | append to the tool's `output.output`; `"lsp": true` also reports LSP diagnostics on edits | not found | +| Cline | executable `PostToolUse` | `.clinerules/hooks/PostToolUse`, with "Enable Hooks" turned on | JSON `contextModification` | not found | +| OpenCode | plugin `tool.execute.after` | `.opencode/plugins/*.ts` | append to the tool's `output.output`. `"lsp": true` also reports LSP diagnostics on edits | not found | | Aider | built-in `--auto-lint` with `--lint-cmd` | `.aider.conf.yml` | built in: a nonzero exit makes Aider offer a fix | not applicable | Sources: code.claude.com/docs/en/hooks, developers.openai.com/codex/hooks, @@ -125,9 +125,9 @@ these APIs change often. oracle rules syntactic, and keep type-aware rules out of the edit hook. 4. **Fallbacks.** Some harnesses need setup by each contributor (Codex trust, the Cline toggle, the Copilot default branch). Keep these layers: - - an `AGENTS.md` instruction to run `pnpm lint:changed` after edits; - - editor diagnostics, which IDE-based agents read; - - the pre-commit hook; + - an `AGENTS.md` instruction to run `pnpm lint:changed` after edits. + - editor diagnostics, which IDE-based agents read. + - the pre-commit hook. - CI. ### Pre-commit From be0631ad15ee793bbfa3bbe2f9a65edcc40949fe Mon Sep 17 00:00:00 2001 From: Test Date: Fri, 23 Oct 2026 09:14:22 -0600 Subject: [PATCH 4/4] docs(oracles): move the enforcement plan to issue #2234 Co-authored-by: Isaac --- .../designs/oracle-enforcement/PLAN.md | 138 ------------------ 1 file changed, 138 deletions(-) delete mode 100644 docs/contributing/designs/oracle-enforcement/PLAN.md diff --git a/docs/contributing/designs/oracle-enforcement/PLAN.md b/docs/contributing/designs/oracle-enforcement/PLAN.md deleted file mode 100644 index b76c2c0731..0000000000 --- a/docs/contributing/designs/oracle-enforcement/PLAN.md +++ /dev/null @@ -1,138 +0,0 @@ -# Oracle enforcement: plan and handoff - -Status: plan. A later session implements it. This document carries the context -that session needs. - -## Why - -Agents keep taking the shortcut path when they change behavior in an area that -an oracle owns. They change production first, then make the oracle pass with a -relaxed comparison, an exclusion, a model rule that matches the new output, or -one pinned example. `AGENTS.md` and the oracle guide already forbid this. The -prose did not stop it, because nothing made the shortcut visible or slow. - -PR #2233 adds the scoring section in `AGENTS.md` ("How oracle work is scored"), -ORC-015 declared reach, and `createOracleReach` in -`packages/db/tests/oracle-reach.ts`. This plan adds the mechanical checks. - -The evidence comes from PR #2168 (source-health channel). A production-first -rewrite of reload handling there needed four more review rounds and about ten -follow-up commits. Each round found a defect in the previous fix. These -shortcuts appeared in that work: - -- Production changed before the oracle could see the defect. -- A truncate step could never land while a reload was in flight, because the - driver's delayed `markReady()` always ran within the same step. The oracle - was green in a state that it never reached. -- A direct-subscriber observation existed only on eager sources. The - subscription lifecycle oracle subscribed without initial state, so its - replay flush published nothing. -- Model rules changed to match production traces. Two of them (an automatic - query restart, and "a key requested again drops a replay failure") were later - found wrong and removed. -- Comparison exclusions (`replayOpen`, a narrowed identity check) widened - without a fresh maintainer decision. -- Most edits went through shell commands (Python and `sed`), so an edit hook - that matches only the harness's own edit tool would not fire. - -## Order of work - -1. Install the agent feedback hooks with a small starter rule set, so that the - session that does the cleanup gets feedback as it works. -2. Add the custom ESLint oracle rules. Run them over the repository and record - every existing violation. -3. Clean up the violations owner by owner, each with the oracle-first order - from `AGENTS.md`. Expect to find real gaps: a conditional `expect` often - hides an unreached state. -4. Add the CI jobs (RED replay, owner coupling, model tripwire). -5. Adopt `createOracleReach` in `packages/db/tests/source-health-oracle.test.ts` - (PR #2168, or a follow-up after it merges). - -## Static checks: custom ESLint rules - -Apply these rules to files whose names contain `oracle`. Keep them syntactic -and fast, so the edit hook can run them on one file. - -| Rule | Reports | Notes | -| --- | --- | --- | -| `oracle/declared-reach` | A generated property (`fcTest.prop`, `fc.assert`) in a file that has no `createOracleReach` call and no `assertReached()` call. | Few false positives. Several oracles count reach in their own way today, for example `collection-subscription-lifecycle-history-oracle.property.test.ts`. Convert them or allow a documented equivalent. | -| `oracle/no-conditional-expect` | An `expect` under an `if`, a ternary, `&&`, `\|\|`, or after an early `return` or `continue` inside a comparison. | This is the main exclusion pattern. Allow it only with `// oracle-exclusion: `. A script checks that `` exists in `docs/contributing/oracle-coverage.md`. Some conditional checks are legitimate (a check that applies only in a modeled state). The comment covers those, and the auditor judges the reason. | -| `oracle/model-independence` | A model function that imports from, or calls into, production `src/`. | Needs a convention first: models in `*-model.ts` files, or a marked region. Drivers may import production. Models may not, except public types. | -| `oracle/no-disabled` | `.only`, `.skip`, or `.todo` on a generated property, and a literal run count below the base, such as `numRuns: 1`. | Trivial. | -| `oracle/registered-property` | A property name that `oracle-config.ts` does not register. | The registry already checks this at run time. Lint reports it earlier. | - -## Diff checks in CI - -- **RED replay.** The PR's evidence block names the oracle tests. CI runs those - tests from the PR head against the merge-base production `src/`. They must - fail there and pass on the head. This checks the oracle-first order - mechanically: an oracle that cannot see the defect cannot produce RED. -- **Owner coupling.** A production change in an area that the coverage map - assigns to an owner, without a change to that owner and without an evidence - block, fails. This needs a machine-readable table of owners to production - paths, which the coverage map does not have yet. -- **Model tripwire.** A PR that changes both production and a model region gets - a flag for the auditor. It does not fail the build. It makes the pair hard to - miss. - -## Run-time checks - -- Declared reach (`createOracleReach`, ORC-015). -- Mutation testing (for example Stryker) on the production files that each - oracle owns, as a nightly job. A low kill rate for an owner shows an oracle - that accepts wrong implementations. It is too slow for each PR. - -## Immediate agent feedback across harnesses - -Contributors use several agent tools. Most now have a post-edit hook that can -return lint output to the model. Each has its own configuration path, tool -names, and feedback protocol. Research from 2026-10-23: - -| Harness | Post-edit hook | Configuration | Feedback to the model | End-of-turn hook that can force a retry | -| --- | --- | --- | --- | --- | -| Claude Code | `PostToolUse`, matcher `Edit\|Write` | `.claude/settings.json` | exit `2` with stderr | `Stop` | -| Codex CLI | `PostToolUse`, matcher `apply_patch` | `.codex/hooks.json`. The project must be trusted, and the hook approved in `/hooks` | exit `2` with stderr | not found | -| Cursor | `afterFileEdit` (stdin has `file_path`) | `.cursor/hooks.json` | not confirmed. The documentation shows a logging-style hook | not confirmed | -| Gemini CLI | `AfterTool`, matcher `write_file\|replace` | `.gemini/settings.json` | JSON `hookSpecificOutput.additionalContext` | `AfterAgent` with `decision: "deny"` | -| GitHub Copilot coding agent | `postToolUse` | `.github/hooks/*.json`, read only from the default branch | JSON `additionalContext`. A nonzero exit does not block | `agentStop` with `decision: "block"` | -| Windsurf / Devin Desktop | `post_write_code` (stdin has `tool_info.file_path`) | `.windsurf/hooks.json` or `.devin/hooks.json` | exit `2` with stderr | not found | -| Cline | executable `PostToolUse` | `.clinerules/hooks/PostToolUse`, with "Enable Hooks" turned on | JSON `contextModification` | not found | -| OpenCode | plugin `tool.execute.after` | `.opencode/plugins/*.ts` | append to the tool's `output.output`. `"lsp": true` also reports LSP diagnostics on edits | not found | -| Aider | built-in `--auto-lint` with `--lint-cmd` | `.aider.conf.yml` | built in: a nonzero exit makes Aider offer a fix | not applicable | - -Sources: code.claude.com/docs/en/hooks, developers.openai.com/codex/hooks, -cursor.com/docs/hooks, geminicli.com/docs/hooks/reference, -docs.github.com/en/copilot/reference/hooks-reference, -docs.windsurf.com/windsurf/cascade/hooks, the Cline repository's -`.clinerules/hooks/README.md`, opencode.ai/docs/plugins, and -aider.chat/docs/usage/lint-test.html. Check each one again before you build: -these APIs change often. - -### Design - -1. **One shared script with thin adapters.** `scripts/agent-lint-feedback.mjs` - reads the hook's JSON from stdin, finds the edited paths (each harness uses - a different field, and Codex sends the patch text of `apply_patch`), runs - ESLint on those files only, and writes the response in the format the - harness expects (`--format=claude|codex|gemini|copilot|cline|windsurf`). - Each harness gets a small committed configuration that calls the script. -2. **An end-of-turn hook for shell edits.** A post-edit hook matches only the - harness's own edit tool. Where a harness has an end-of-turn hook, run lint on - `git diff --name-only` there. That catches shell edits too, and it can force - another turn. For the other harnesses, the pre-commit hook is the fallback. -3. **Speed.** A hook that takes seconds on each edit gets turned off. Use - `eslint --cache` on single files, or a daemon such as `eslint_d`. Keep the - oracle rules syntactic, and keep type-aware rules out of the edit hook. -4. **Fallbacks.** Some harnesses need setup by each contributor (Codex trust, - the Cline toggle, the Copilot default branch). Keep these layers: - - an `AGENTS.md` instruction to run `pnpm lint:changed` after edits. - - editor diagnostics, which IDE-based agents read. - - the pre-commit hook. - - CI. - -### Pre-commit - -`lint-staged` already runs `eslint --fix` on staged files. The oracle rules run -there once they are in the ESLint configuration. On a commit of more than about -100 files, ESLint ran out of memory in PR #2168. Lint in chunks, or raise the -Node heap in the hook.