Skip to content

Reduce listener delay for CI agents - #6782

Draft
evgenyrp wants to merge 8 commits into
mozilla:masterfrom
evgenyrp:worktree-reduce-listener-delay-6662
Draft

Reduce listener delay for CI agents#6782
evgenyrp wants to merge 8 commits into
mozilla:masterfrom
evgenyrp:worktree-reduce-listener-delay-6662

Conversation

@evgenyrp

@evgenyrp evgenyrp commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Another iteration of listener improvements with a focus on reducing the delay to trigger the agent. This will increase the number of runs when the agent analyzes an intermittent failure (capped at 100 per 24 hours).

Specifically:

  • do not perform lengthy extra checks if the test was never intermittent based on tests.firefox.dev data.
  • do not wait for the parent build to finish
  • some other fixes for the issues the agent found during the iterative process of dry runs and analysis of the logs

Stats from the last tested dry run:

Dry run (16.5 h, autoland)

Metric Value
Window Sep 2 17:07 → Sep 3 09:37 (16.5 h)
Distinct failing tasks seen 1315
Test-repair would-triggers 23 (~33 / day)
Build-repair would-triggers 4
Fast path fired 36
Fast-path fires that reached trigger 6
Job end → trigger, fast path 0.1 – 0.6 min
Job end → trigger, wait path 10.2 – 17.1 min
History check → trigger gap 1 – 5 s
Ancestor walk stepped over a pending parent 35
Ancestor walk fail-open after 600 s 3
Fast-path triggers the skipped wait would have dropped 0 (all classified 7–9 h later)
Log-parse wait exhausted (60 s) 2
Cross-push dedupe skips 11
Test triggers later classified intermittent 19 / 23 (fast path 5/6, wait path 14/17)
Budget hits (cap 100) 0
Errors / tracebacks 0

Redelpoyed live for testing

related to #6662

@marco-c

marco-c commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

do not perform lengthy extra checks if the test was never intermittent based on tests.firefox.dev data.

How many tests do we have that were never intermittent? Maybe we can have a configurable threshold instead of 0, say start with a very low number like 0.1%.

@evgenyrp

evgenyrp commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

do not perform lengthy extra checks if the test was never intermittent based on tests.firefox.dev data.

How many tests do we have that were never intermittent? Maybe we can have a configurable threshold instead of 0, say start with a very low number like 0.1%.

We do check the failure rate against the threshold; see https://github.com/mozilla/bugbug/pull/6782/changes#diff-c516aab8c954b9767286eefb7f8c2cb34ace3122326bbad4dd31760a5e28d413R250

Here's what we have in the logs now:

2026-09-03 14:10:21.024
2026-09-03 21:10:21 INFO    app.flakiness: Test devtools/client/inspector/rules/test/browser_rules_inherited-element-backed-pseudo-elements.js failed 5432 of its last 11690 runs (46.467%); it may be an intermittent
2026-09-03 14:04:08.844
2026-09-03 21:04:08 INFO    app.flakiness: Test toolkit/content/tests/chrome/test_findbar.xhtml failed 1166 of its last 13025 runs (8.952%); it may be an intermittent
2026-09-03 14:03:23.632
2026-09-03 21:03:23 INFO    app.flakiness: Test gfx/layers/apz/test/mochitest/test_group_touchevents-6.html failed 148 of its last 17102 runs (0.865%); it may be an intermittent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants