Repository navigation
Conversation
Each case waits up to 10s for the WebView to load and then sleeps another 10s before screenshotting, so it needs more than 20s. No testTimeout was configured anywhere, so Jest's 5s default applied and the first test was killed mid-sleep every run - the suite has been red on master since at least 2026-08-16. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit put testTimeout in jest.harness.config.mjs, which has no effect. @react-native-harness/jest resolves it as session.config.testTimeout ?? projectConfig.testTimeout ?? globalConfig.testTimeout and session.config is the rn-harness config, whose schema declares testTimeout with .default(5000). Being a Zod default it is always populated, so the first branch always wins and the Jest value is never consulted - CI still timed out at exactly 5000ms. Move it to rn-harness.config.mjs next to bridgeTimeout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The host app is regenerated from scratch on every CI run, and the install was unpinned, so each run resolved react-native-harness to whatever was newest at that moment. That is how the suite went red without a commit here: 1.0.0 had no per-test timeout at all, and a later release added `testTimeout` with a 5000ms default, which the 10s sleep in darkTheme.harness.js cannot meet. Pinned to 1.6.0 - the version CI was already picking up - so upstream default changes arrive as a deliberate bump instead of a surprise failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`failureThresholdType: 'percent'` compares a 0-1 ratio, not a percentage:
const percentage = differences / totalPixels;
pass = percentage <= failureThreshold;
- @react-native-harness/bridge/src/image-snapshot.ts:142-148
So `failureThreshold: 5` meant 500%, and since the ratio cannot exceed 1, both
assertions passed for every possible pair of equally sized images. The only
failures they could ever report were a dimension mismatch, a null screenshot or
a timeout - the rendering itself was never checked.
Set to 0.01, i.e. 1% of pixels. The widest signal these full-screen shots carry
is the light baseline against the dark one, measured at 5.6% of the screen on
Android and 6.3% on iOS, so 1% leaves room for antialiasing while still catching
a widget rendered in the wrong theme.
This is the first run in which these baselines are actually enforced, so CI is
where we find out whether they still hold.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each case slept a flat 10s and then screenshotted, which is where almost all of the runtime went: on the last green run the two cases took 11074ms and 10652ms, so roughly 94% of each was the sleep. The sleep was also the only thing standing in for readiness - `render()` resolves after the React commit and a double rAF, not after the WebView fetches anything - so the number had to be large enough to cover a network load nobody had measured. Waits on a real signal instead. The SDK tears down its loading overlay when the page posts its first message, so the overlay now carries a testID and the test polls until it is gone. The budget is 20s, chosen to sit above the SDK's own 15s loading timeout so a page that never comes up is reported as `loading timeout` through onMessage rather than as an unexplained harness kill; the collected messages are checked on every poll so a failure fails the test with the SDK's own description. Readiness only means `hcaptcha.render()` returned, so the widget's images may still be painting. Rather than swap one guessed delay for a smaller one, the test polls until two consecutive frames are byte-identical, and screenshots anyway if they never settle. A healthy run should now cost readiness plus a couple of hundred ms rather than a flat 10s, and a broken one should say why. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CAMOBAP
marked this pull request as ready for review
October 18, 2026 09:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The e2e harness suite had been failing on
masteron every run since at least 2026-08-16.Fixing it turned up three separate problems, one of which means the suite has never actually
verified anything.
1. The harness was installed unpinned
__scripts__/setup-e2e-host.jsregenerates the host app from scratch on every CI run andinstalled
react-native-harnesswith no version constraint, so each run resolved to whateverwas newest at that moment. Upstream added a per-test
testTimeoutdefaulting to 5000ms(1.0.0 had no per-test timeout at all), and the suite went red with no commit in this repo.
Pinned to
1.6.0. This is the root cause — without it, any timeout fix is temporary.2. The image assertions could not fail
failureThresholdType: 'percent'comparesdifferences / totalPixels— a 0–1 ratio — so5meant 500%, which no image pair can exceed. Every green e2e run in this repo's history
passed vacuously.
Now
0.01(1%). For scale, the widest signal these shots carry — the light baseline againstthe dark one — is 5.6% of the screen on Android and 6.3% on iOS, so 1% leaves room for
antialiasing while still catching a widget rendered in the wrong theme.
3. Each case slept through the load instead of waiting for it
render'stimeoutonly covers React commit plus a double rAF — it does not wait for theWebView to fetch anything, so the 10s there was decorative and the real wait was the blind
10s sleep. ~94% of each test's duration was that sleep.
Replaced with a signal-driven wait.
Hcaptcha.jsgains atestIDon the loading overlay,which is torn down when the page posts its first message, making its absence the one readiness
signal a UI-level test can observe. The wait also inspects
onMessageon every poll, so anSDK error fails the test with the SDK's own description rather than diffing a screenshot of a
spinner.
The readiness cap is 20s, deliberately above the SDK's own 15s
loadingTimeout: when thepage never comes up we want
loading timeoutreported throughonMessage, not the harnesskilling the test with an opaque message.
testTimeout: 60000stays as a backstop — it must live inrn-harness.config.mjs, which iswhere the harness actually reads it.
Stability
Five sequential CI rounds on
58135e3(Android ×5, iOS ×4 — 18 assertions): all green, noretries, no flakes.
Durations fell from ~10.7s to ~1.9–6.0s, and worst case is ~10% of the 60s backstop. The
spread is the intended consequence of waiting on a signal rather than a clock — runner noise
now shows through in duration, not in outcome.
Most importantly: these are the first runs in which the baselines were genuinely compared.
They held on both platforms.
Follow-up worth doing separately
screen.screenshot()accepts an element reference, and cropping to the widget raises thelight-vs-dark signal from ~5.6% to ~92.5%. Left out here because it changes the captured
dimensions and requires re-recording all four baselines on a real emulator and simulator.
🤖 Generated with Claude Code