Add visual regression testing for RNTester by robhogan · Pull Request #59115 · react/react-native

robhogan · 2026-10-15T14:52:51Z

RNTester's rendering coverage, particularly text, currently lives in Meta's internal jest-e2e suite. To move it into public CI we need screenshot comparison that works for fork PRs and doesn't add PNGs to the repo. This adds that, as a separate set of visual flows alongside the existing Maestro E2E.

How it works

Visual tests are Maestro flows in packages/rn-tester/visual-tests/flows/. Each one deep-links to an RNTester example, waits for it to be ready and captures a screenshot cropped to a testID. Maestro only captures - it never decides pass or fail.

The work is split across a trust boundary, so a fork PR never runs with credentials:

Visual capture runs as two jobs in Test All, reusing its RNTester Release builds: test_visual_android_rntester after build_android and test_visual_ios_rntester after test_ios_rntester. Each runs every flow in one Maestro invocation, on a Pixel 7 API 35 emulator (1080x2400, 420dpi) or an iPhone 17 Pro on iOS 26.2, and uploads the PNGs and a manifest recording the commit and device environment as an artifact. They have no secrets.
Visual Compare RNTester (workflow_run on Test All, from main) treats that artifact as data - it's validated before any image is decoded and nothing in it is executed. It finds the baseline, compares with pixelmatch, posts a check run per platform and a single PR comment with expected, captured and diff images, and on pushes to main publishes the captures as a new baseline. It runs once Test All completes, including when unrelated jobs fail, so the report arrives with the rest of CI rather than as soon as the captures are ready.
Visual Approve RNTester (pull_request_target, on the visual-change-approved label) records a maintainer's approval of an intentional change. The approval is tied to the exact set of differences, so a later push that changes them removes the label and needs a new review.

Comparison is strict by default: any changed pixel is a difference. A test can be loosened in visual-tests/config.json, with evidence. Captures have been byte-identical across separate builds and runners so far, so I'd expect that to be rare.

What defines the baseline

A baseline is the full set of captures from a commit on main, keyed by commit and by device environment. Every push to main publishes one, and nothing is ever overwritten.

A PR is compared with the baseline for the commit it's based on (the first parent of the merge commit CI tests), or the nearest earlier commit on main that has one, and the report says which. So a PR based on last week's main is compared with last week's rendering, not with whatever has landed since. Approving a PR doesn't promote its images - main is captured again after merge and that becomes the baseline.

The environment key covers everything that affects rendering (OS runtime, device, screen size, density, font scale, locale, appearance, animations, system image and, on Android, the emulator version, which is pinned). If the CI image drifts, the key changes and a fresh baseline is published rather than every PR showing a spurious diff.

Settings

Storage is any GCS bucket, configured through repository variables. Writes use GitHub's OIDC token, so there's no stored secret, and only the two trusted workflows on main can authenticate. They need create-only access (roles/storage.objectCreator). Reads are public.

Variable
`VISUAL_BUCKET`	Bucket to upload to
`VISUAL_PUBLIC_BASE_URL`	Public URL for reads and report images - the bucket, or a CDN in front of it
`VISUAL_GCP_PROJECT`	GCP project for authentication
`VISUAL_WORKLOAD_IDENTITY_PROVIDER`	Workload identity provider trusting this repo's `visual-compare-rntester.yml` and `visual-approve-rntester.yml` on `main`

Expo can host the bucket and a CDN for this. Reports and approvals expire after 30 days by lifecycle rule. Baselines are small (8-175 KB per image), and at the current rate of pushes to main the cost is cents a month.

packages/rn-tester/visual-tests/README.md covers adding a test and handling a failure. This starts with two text tests and runs non-blocking. The plan is to port high-value jest-e2e text and rendering tests, then make the checks required once they've proven reliable.

Changelog:

[INTERNAL] [ADDED] - Visual regression testing for RNTester in CI

Test Plan:

Ran the full flow in a fork (https://github.com/robhogan/react-native), with Test All trimmed to the jobs this depends on, against a personal GCS bucket:

A PR with no visual change passes on both platforms (Visual spike: chain-no-change robhogan/react-native#3).
A deliberate text layout change fails both platforms, with expected, captured and diff images on the checks page and in the PR comment (Visual spike: chain-text-regression robhogan/react-native#4).
Applying visual-change-approved passes the approval check. Pushing a further, different change removed the label with a comment and failed the check again.
Captures were byte-identical across 5 Android and 4 iOS runs, from 3 separate builds on fresh runners.
When an unrelated Test All job fails, the comparison still runs. Run against a Test All run whose Android capture had failed, it reports Android as having no capture and compares iOS as usual.
Pixel Launcher raised an ANR dialog during emulator boot in 3 of 8 probe runs. After the capture's setup step, none of the 8 had a dialog showing.
Running all flows in one Maestro invocation brought the iOS capture step to 2m20s for two tests, from 3m49s for one test when each flow started Maestro separately. Failed flows are retried individually.
New unit tests for artifact validation (path traversal, symlinks, hash and size checks), comparison, storage keys, approval and report rendering.

Not yet tested: a PR from a third-party fork, to confirm it can't reach storage credentials.

cortinico

Thats' great - can we add some markdown in the repos to explain how to use this?

cortinico · 2026-10-15T17:03:06Z

+  VISUAL_HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
+
+jobs:
+  build_android:


We already build rn_tester in previous workflows, no? Can't we reuse the APK from that jobs?

cortinico · 2026-10-15T17:03:56Z

+          api-level: 35
+          target: google_apis
+          arch: x86_64
+          ram-size: '8192M'
+          heap-size: '4096M'
+          disk-size: '10G'
+          cores: '4'
+          disable-animations: true
+          avd-name: visual_emulator


I'm a little concerned that we don't have an explicit resolution configured here and this could make the screenshots drift quite a lot once we bump this rule. Potentially we should add like profile: 'pixel' or so?

cortinico · 2026-10-16T16:08:34Z

@robhogan I'm having a harder time in reviewing this. Can you ask your agent to split it in smaller meaningful commits (doens't need to ne separate PRs), so it would be easier for me to do a pass?

Also my browser is unhappy with the diff size 😅

Adds the first two visual tests as Maestro flows under `packages/rn-tester/visual-tests/flows/`. Each flow deep-links to an RNTester example through `helpers/open-example.yml`, waits for it to render and takes a screenshot cropped to the example's `testID`. The flows only capture, and never decide whether a test passed. The last line clipping example gains a `testID` to crop to. Changelog: [Internal]

Adds two Test All jobs that run the visual flows against the RNTester Release builds Test All already makes: `test_visual_android_rntester` after `build_android` and `test_visual_ios_rntester` after `test_ios_rntester`. They call the reusable `visual-capture-rntester.yml`, which runs every flow in one Maestro invocation on one canonical device per platform (a Pixel 7 API 35 emulator and an iPhone 17 Pro on iOS 26.2), retrying only failed flows. Each job uploads the screenshots plus a manifest as a `visual-capture-<platform>` artifact. The manifest records the commit, the base commit and the device environment, hashed into an environment key so that baselines from a different device setup are never compared. The emulator version is pinned and part of the key, and the capture dismisses a Pixel Launcher ANR dialog that can appear at boot. These jobs run PR code and have no secrets. Nothing consumes the artifacts yet. Changelog: [Internal]

Adds the comparison core, as pure functions over a capture directory and a baseline. `validate.js` treats the capture artifact as untrusted: it rejects path traversal, symlinks, unlisted or duplicate files, hash mismatches and oversized images, reading PNG dimensions from the header before any image library decodes them. `compare.js` compares each image with pixelmatch under the per-test policy in `visual-tests/config.json`, which is strict by default. Adds `pixelmatch` and `pngjs` as dev dependencies. Changelog: [Internal]

Adds the remaining building blocks for the trusted workflow. `storage.js` defines the bucket layout (write-once baselines keyed by environment and commit, reports per PR head) and finds the nearest baseline in a commit's ancestry. `report.js` renders the check run output and the PR comment. `github.js` is a minimal REST client. `approval.js` fingerprints the set of differences, so an approval label only covers the differences it was applied to. Changelog: [Internal]

Adds Visual Compare RNTester, which runs from the default branch on `workflow_run` when Test All completes, including when unrelated jobs fail. It downloads each platform's capture, validates it, compares it with the baseline for the PR's base commit and uploads the report through GitHub OIDC, so no secret is stored. It posts a check run per platform, the approval gate check and a single PR comment. On pushes to `main`, it publishes the captures as a new baseline. The bucket and identity provider are configured through the `VISUAL_*` repository variables. Changelog: [Internal]

Adds Visual Approve RNTester, which runs on `pull_request_target` when a maintainer applies `visual-change-approved`. It records an approval for the current fingerprint of differences and re-evaluates the gate check. A later push with different differences removes the label and needs a new review. Changelog: [Internal]

Adds `packages/rn-tester/visual-tests/README.md` on how the tests run in CI, adding a test, running flows locally, handling a failed check and tolerances, and links it from RNTester's README. Changelog: [Internal]

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 15, 2026

robhogan marked this pull request as ready for review October 15, 2026 15:15

facebook-github-tools Bot added the Shared with Meta Applied via automation to indicate that an Issue or Pull Request has been shared with the team. label Oct 15, 2026

cortinico reviewed Oct 15, 2026

View reviewed changes

This was referenced Oct 16, 2026

Visual spike: chain-no-change robhogan/react-native#3

Open

Visual spike: chain-text-regression robhogan/react-native#4

Open

robhogan added 7 commits October 16, 2026 19:17

Document RNTester visual tests

8564417

Adds `packages/rn-tester/visual-tests/README.md` on how the tests run in CI, adding a test, running flows locally, handling a failed check and tolerances, and links it from RNTester's README. Changelog: [Internal]

robhogan force-pushed the robhogan/visual-regression-testing branch from 6fe7eed to 8564417 Compare October 17, 2026 07:10

Navigation Menu

Repository navigation

Add visual regression testing for RNTester - #59115

Add visual regression testing for RNTester#59115
robhogan wants to merge 7 commits into
mainfrom
robhogan/visual-regression-testing

robhogan commented Oct 15, 2026 •

edited

Loading

Uh oh!

cortinico left a comment

Uh oh!

cortinico Oct 15, 2026

Uh oh!

cortinico Oct 15, 2026

Uh oh!

cortinico commented Oct 16, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

robhogan commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How it works

What defines the baseline

Settings

Changelog:

Test Plan:

Uh oh!

cortinico left a comment

Choose a reason for hiding this comment

Uh oh!

cortinico Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

cortinico Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

cortinico commented Oct 16, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

robhogan commented Oct 15, 2026 •

edited

Loading