Add advisory review and risk checks for the shadow week by gabe-bots[bot] · Pull Request #94 · gabe4coding/plain

gabe-bots · 2026-10-18T22:32:26Z

Intent

Start the shadow week of the validation loop on this repo, the same setup as gabe4coding/puredori#10 and gabe4coding/retroagent#58. Two new checks run on every PR. Both are advisory: they are not required, they never fail a PR, and nothing here merges, approves or changes repo settings.

review (.github/workflows/review.yml): two passes in one job.
1. shall: the diff against the engineering standards, adapted from shall's ci/github/shall-review.yml.
2. General pass (.github/scripts/general_review.py): one tool-less, isolated claude -p call that checks for bugs, missing or weaker tests, silent behavior changes, PR body vs diff drift and irreversible changes. The PR text is passed as untrusted data. It also records the risk-judgment JSON from the design, which does not change the tier yet.
- Findings go in PR comments and the job summary. This week the job always ends green.
- If secrets are missing, each pass skips with a neutral ::notice that names the secret.
risk (.github/workflows/risk.yml, .github/scripts/risk_tier.py, .github/risk.yml): the deterministic tiers HOLD / T2-hard / T2-soft / T1 / T0.
- The rules and the script are read from the base branch, so a PR cannot loosen its own tier. This PR is the bootstrap and uses its own copy.
- The tier is posted as a risk tier check run (success for T0/T1, neutral for T2/HOLD, never failure) and as a risk:* label.
- Kill switch: if the repo variable AUTOMERGE is off, every PR becomes T2-hard. It is read only.
- There is no LLM layer yet. The vendor names that HOLD a PR are stored as sha256 hashes, so this public repo never spells them out.
T2-hard: any version change in package.json (merging it publishes to npm and tags v<version>), plus publish.yml.
T2-soft paths: bin/**, the CLIs and their flags (src/cli.ts, src/computer/cli.ts, src/mobile/cli.ts, src/suite/native-cli.ts, src/suite/options.ts), env loading (src/core/env.ts), the Jev key and model pin (src/jev/provider.ts), the gate floors (scripts/gates.test.mjs), the source rules (scripts/source-rules/**), the agent PR gate (scripts/pr-gate.mjs, scripts/validation-stamp.mjs, .claude/settings.json, .codex/hooks.json) and the public action.yml.
T2-soft code: a changed line under src/** that matches secret|redact|mask|password|process.env|api_key. This catches maskPasswords in src/browser/page.ts and the secret checks in src/core/spec.ts.
plugins/** is generated, so it doesn't count toward the size limits, but it is not docs. scripts/gates.test.mjs is a gate, not a test.
This PR does not touch PR Prepare 3.1.0 for a paid web pilot #93 or its branch. live skips on this PR because no runtime files change.
Tests: .github/scripts/test_risk_tier.py and test_general_review.py, run by the risk (classifier tests) job.

Secrets to add (Settings > Secrets and variables > Actions):

TYPESAFE_API_KEY: needed for shall's Jev rule selection.
ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN: needed for claude -p in both passes.
TYPESAFE_API_KEY is probably already set, because live.yml uses it. If so, only the Claude secret is missing.
Data note: diffs and PR text go to TypeSafe (shall) and Anthropic (both passes).

This PR is T2-hard by its own rules, because it adds .github/risk.yml and the review and risk workflows. Gabe reviews and merges it.

Evidence

uv run --no-project --python 3.12 --with pyyaml==6.0.2 --with pytest -- python -m pytest -q .github/scripts passes. actionlint is clean on both workflows.
Repo checks: check:docs, check:agent-notes, check:source and check:examples pass. npm test and check:deps need Node 22+ (this box has Node 20), so test in CI is the proof. scripts/validate.mjs' live steps don't apply: no runtime files change.
The classifier on this branch gives T2-hard (hard paths: .github/risk.yml, the scripts and both workflows).
Backtest on the last 20 merged PRs (docs: Codex agent evals run with an empty HOME #73–Share the test fakes and run the lock tests on mobile #92), using these rules (PR base and head from GitHub, diff from the merge base):

PR	Title	Size	Tier	Why
#92	Share the test fakes and run the lock tests on mobile	+444/-392, 16 files	T0	test-only: 16 files
#91	Close known bypasses in the env, cast and gate-floor checks	+159/-29, 4 files	T2-soft	sensitive-path: touches `scripts/gates.test.mjs`; sensitive-path: touches `scripts/source-rules/casts.mjs`
#90	Read the env switches on each call and gate env reads	+163/-25, 12 files	T2-soft	sensitive-path: touches `scripts/source-rules/env.mjs`; sensitive-code: `src/core/aria-changes.ts` changes a line with `process.env`
#89	Check the module boundaries of src/ with dependency-cruiser	+701/-13, 11 files	T1	source-change: 8 code files, 166 changed lines
#88	Add a cast ratchet and fix four gaps in the source rule runner	+268/-56, 6 files	T2-soft	sensitive-path: touches `scripts/source-rules/baseline.json`; sensitive-path: touches `scripts/source-rules/casts.mjs`
#87	Send hooks stdout to stderr and stop stdout writes in MCP code	+142/-4, 4 files	T2-soft	sensitive-path: touches `scripts/source-rules/stdout.mjs`
#86	Add a gate floors test that fails when a gate gets weaker	+177/-0, 3 files	T2-soft	sensitive-path: touches `scripts/gates.test.mjs`
#85	Warn on literal secrets in specs and fail check:examples on them	+249/-25, 18 files	T2-soft	sensitive-code: `src/core/spec.ts` changes a line with `secret`; sensitive-code: `src/suite/validate.ts` changes a line with `secret`
#84	Scroll left and right on the desktop, with one direction list for all	+197/-37, 18 files	T1	source-change: 10 code files, 155 changed lines
#83	Give hooks teardown the first failed status on all engines	+96/-24, 14 files	T1	source-change: 10 code files, 53 changed lines
#82	Move shared types to leaf modules and the run contract to core	+204/-170, 40 files	T2-soft	sensitive-path: touches `src/cli.ts`; sensitive-path: touches `src/computer/cli.ts`
#81	Add a source rule gate that later rules plug into	+458/-5, 6 files	T2-soft	sensitive-path: touches `scripts/source-rules/baseline.json`
#80	Test that parseIdentity skips the resource id and keeps the iOS name f	+18/-1, 1 files	T0	test-only: 1 files
#79	Daily: say benchmarks compare on local pages only	+11/-0, 2 files	T1	source-change: 2 code files, 11 changed lines
#78	Test hover candidates: images, svg, figure, and the same extras as cli	+20/-0, 1 files	T0	test-only: 1 files
#77	Daily: say how to wait for CI in the validating-changes skill	+11/-0, 2 files	T1	source-change: 2 code files, 11 changed lines
#76	Test the click-candidate order a short cap keeps	+17/-0, 1 files	T0	test-only: 1 files
#75	Daily: tell desktop and mobile agents to use full prefixed tool names	+6/-0, 2 files	T1	source-change: 0 code files, 0 changed lines
#74	Add an agent setup prompt to the README	+49/-3, 3 files	T0	docs-only: 3 files
#73	docs: Codex agent evals run with an empty HOME	+4/-3, 1 files	T0	docs-only: 1 files

Totals: T0 6, T1 6, T2-soft 8.

Things to compare by hand this week: whether each T2-soft really needed you, and whether any T1 should have been T2.

Intent

Shadow week, updated. The hosted reviewer is Greptile. This removes the claude -p review job (it ran PR-controlled code with API secrets) and hardens the risk tier so a PR cannot set or lower its own tier.

Review workflow deleted (.github/workflows/review.yml). Nothing in a pull_request_target job reads secrets.*. general_review.py is still in the tree and still tested; no workflow runs it.
risk (.github/workflows/risk.yml, .github/scripts/risk_tier.py, .github/risk.yml): tiers HOLD / T2-hard / T2-soft / T1 / T0.
- The tier job runs on pull_request_target. The workflow, rules, and classifier are the base branch's. The PR head is fetched as data (git diff / git show) and no PR file is executed. Python runs with -I, uv with --no-config.
- Classifier tests run on pull_request with contents: read only and no stored git credentials.
- Paths come from git diff --name-status -z. The diff uses --text --no-textconv and lines split on \n only. A test-only PR that changes an assertion or adds skip/only/todo is T1. If the file list and the diff disagree, the tier is T2-hard.
- The reviewer signal is only the Greptile Review check run from app slug greptile-apps on the PR's current head SHA. Labels and comments are not read. success and neutral count as a pass and never lower the tier. A completed failure, stale, cancelled, timed_out, action_required, or startup_failure raises the tier one step and stops at T2-hard (it does not create HOLD). A missing or pending check is not a pass and does not change the tier. When Greptile's check completes, the tier job runs again.
- The tier is posted as a risk tier check run (success for T0/T1, neutral for T2/HOLD, never failure) and as a risk:* label.
- Kill switch: repo variable AUTOMERGE=off makes every PR T2-hard.
- Reviewer config is T2-hard: greptile.json, .greptile/**, .coderabbit.yaml, .coderabbit.yml.
T2-hard also still includes any version change in package.json (merging it publishes to npm) and publish.yml.
This does not touch plain hardening waves 2-5 or comment-deletion work.

Until this workflow is on the default branch, pull_request_target will not run the tier job for this PR. The classifier tests run from the PR.

Evidence

uv run --no-config --no-project --python 3.12 --with pyyaml==6.0.2 --with pytest -- python -m pytest -q .github/scripts: 95 passed, 7 skipped.
actionlint is clean on .github/workflows.
No secrets. reference remains in review.yml (the file is gone) or in any job that runs on pull_request_target.
Backtest on merged PRs docs: Codex agent evals run with an empty HOME #73–Share the test fakes and run the lock tests on mobile #92 is unchanged by the path hardening, because those runs do not pass a reviewer check. The live tier can be one step higher when Greptile's check on that head has a failed or stale conclusion.

PR	Title	Tier
#92	Share the test fakes and run the lock tests on mobile	T0
#91	Close known bypasses in the env, cast and gate-floor checks	T2-soft
#90	Read the env switches on each call and gate env reads	T2-soft
#89	Check the module boundaries of src/ with dependency-cruiser	T1
#88	Add a cast ratchet and fix four gaps in the source rule runner	T2-soft
#87	Send hooks stdout to stderr and stop stdout writes in MCP code	T2-soft
#86	Add a gate floors test that fails when a gate gets weaker	T2-soft
#85	Warn on literal secrets in specs and fail check:examples on them	T2-soft
#84	Scroll left and right on the desktop	T1
#83	Give hooks teardown the first failed status on all engines	T1
#82	Move shared types to leaf modules and the run contract to core	T2-soft
#81	Add a source rule gate that later rules plug into	T2-soft
#80	Test that parseIdentity skips the resource id	T0
#79	Daily: say benchmarks compare on local pages only	T1
#78	Test hover candidates	T0
#77	Daily: say how to wait for CI	T1
#76	Test the click-candidate order a short cap keeps	T0
#75	Daily: tell desktop and mobile agents to use full prefixed tool names	T1
#74	Add an agent setup prompt to the README	T0
#73	docs: Codex agent evals run with an empty HOME	T0

Totals: T0 6, T1 6, T2-soft 8.

gabe-bots · 2026-10-19T11:05:50Z

@greptileai review

…ets. Closes gabe4coding/puredori#12 Co-authored-by: Gabe <gpavanello@thefork.com>

The tier job runs on pull_request_target with the base workflow, rules, and classifier. The PR head is fetched as data and Python runs isolated. A Greptile Review check on the current head can raise the tier and cannot lower it. Reviewer config files are T2-hard. Co-authored-by: Gabe <gpavanello@thefork.com>

Only a completed Greptile Review check from greptile-apps with conclusion success is a pass. Every other case, including a missing, pending, neutral, or stale check, raises the deterministic tier one step and stops at T2-hard. Co-authored-by: Gabe <gpavanello@thefork.com>

greptile-apps · 2026-10-19T11:12:29Z

Confidence Score: 4/5

[Critical impact] Adds automated code review and risk classification to CI.

Fix the workflow-move classification before merging; a PR can disable CI without receiving the required hard tier.

Findings

Removed workflows get lower tiers ▶
Quoted filenames miss risk rules ▶
New assertions hide deleted checks ▶
Description reviews stay stale ▶

Summary

Adds two advisory PR workflows:

Pull requests now get two advisory review passes.
Pull requests now receive a deterministic risk tier.
Each pull request shows its risk tier as a check and label.

The classifier misses workflow removals made through renames. Smaller issues affect quoted filenames, removed assertions, and reviews after PR-description edits.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  PR[PR event] --> Review[Standards and general review]
  PR --> Base[Read base-branch rules and classifier]
  Base --> Tier[Classify changed files and PR text]
  Tier --> Label[Update risk label]
  Tier --> Check[Post advisory risk check]
  Review --> Findings[Post findings and job summary]

_{Reviews (1) · Last reviewed commit: "Add advisory review and risk checks for ..." · Reviewed by Greptile}

greptile-apps · 2026-10-19T11:12:32Z

+    if f.status == "D":
+        return Reason("T2-hard", "workflow-weakened", f"deletes workflow `{f.path}`")


Removed workflows get lower tiers

Moving .github/workflows/test.yml outside .github/workflows/ disables it. Git reports an unchanged move as R with no added or removed lines, but workflow_reason only treats D as deletion. The move therefore receives T2-soft instead of T2-hard, hiding the removal of a running CI check. Treat moves out of the workflow directory as workflow deletions.

greptile-apps · 2026-10-19T11:12:32Z

+            m = re.match(r"diff --git a/(.*) b/(.*)$", line)
+            cur = FileChange(path=m.group(2) if m else line.split(" b/", 1)[-1])


Quoted filenames miss risk rules

parse_diff does not decode Git's quoted filenames. For a non-ASCII path such as src/café.ts, Git can quote both header paths. Neither the regex nor its fallback extracts the filename, so cur.path becomes the whole header. A changed line containing process.env then misses the src/** rule and receives T1 instead of T2-soft. Read filenames from NUL-separated Git output or decode the quoted paths.

greptile-apps · 2026-10-19T11:12:32Z

+        lost = sum(1 for f in tests for l in f.removed if ASSERTION.search(l))
+        gained = sum(1 for f in tests for l in f.added if ASSERTION.search(l))
+        deleted_tests = [f.path for f in tests if f.status == "D"]
+        if deleted_tests:
+            reasons.append(Reason("T1", "test-deletion", f"deletes {', '.join(deleted_tests[:3])}"))
+        elif lost > gained:


New assertions hide deleted checks

Comparing total removed and added assertion counts lets unrelated new assertions cancel deleted checks. Removing assert.match(warn.warnings[0], /env var is not set/) and adding assert.ok(true) elsewhere leaves lost == gained, so the PR becomes T0 despite losing coverage. The configured T0 rule says no assertion may be removed. Check removals conservatively rather than treating any new assertion as a replacement.

greptile-apps · 2026-10-19T11:12:32Z

+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, ready_for_review]


Description reviews stay stale

The general pass reviews the PR title and body, but this trigger omits edited. If an author corrects a body-versus-diff finding without pushing another commit, the review comment remains stale. Adding unsupported claims after a clean review also goes unchecked. Include title and body edits in the review trigger.

Suggested change

types: [opened, synchronize, reopened, ready_for_review]

types: [opened, synchronize, reopened, ready_for_review, edited]

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

A successful Greptile Review check is no longer enough. The latest greptile-apps[bot] summary must review the current head at 4/5 or 5/5, and that bot must have no P0 or P1 inline comment on that SHA. The claude-code-mode#47 review (success check, confidence 0/5, three P1s) stays a non-pass and raises the tier one step. Co-authored-by: Gabe <gpavanello@thefork.com>

Add advisory review and risk checks for the shadow week

de33345

github-actions Bot added the risk:t2-hard label Oct 18, 2026

Harness and others added 3 commits October 19, 2026 11:07

Drop the claude review workflow so pull request jobs hold no API secr…

bd1628e

…ets. Closes gabe4coding/puredori#12 Co-authored-by: Gabe <gpavanello@thefork.com>

greptile-apps Bot reviewed Oct 19, 2026

View reviewed changes

gabe4coding merged commit 1644b9b into main Oct 19, 2026
4 checks passed

Navigation Menu

Repository navigation

Add advisory review and risk checks for the shadow week - #94

Add advisory review and risk checks for the shadow week#94
gabe4coding merged 5 commits into
mainfrom
bots/validation-shadow-week

gabe-bots Bot commented Oct 18, 2026 •

edited by cursor Bot

Loading

Uh oh!

gabe-bots Bot commented Oct 19, 2026

Uh oh!

greptile-apps Bot commented Oct 19, 2026 •

edited

Loading

Summary

Diagram

Uh oh!

greptile-apps Bot Oct 19, 2026

Uh oh!

greptile-apps Bot Oct 19, 2026

Uh oh!

greptile-apps Bot Oct 19, 2026

Uh oh!

greptile-apps Bot Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		if f.status == "D":
		return Reason("T2-hard", "workflow-weakened", f"deletes workflow `{f.path}`")

		m = re.match(r"diff --git a/(.) b/(.)$", line)
		cur = FileChange(path=m.group(2) if m else line.split(" b/", 1)[-1])

	types: [opened, synchronize, reopened, ready_for_review]
	types: [opened, synchronize, reopened, ready_for_review, edited]

Conversation

gabe-bots Bot commented Oct 18, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Intent

Evidence

Intent

Evidence

Uh oh!

gabe-bots Bot commented Oct 19, 2026

Uh oh!

greptile-apps Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Diagram

Uh oh!

greptile-apps Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

gabe-bots Bot commented Oct 18, 2026 •

edited by cursor Bot

Loading

greptile-apps Bot commented Oct 19, 2026 •

edited

Loading