Repository navigation
fix: stop a <testsuites> wrapper lending a nested suite's run markers - #10
Open
JingHao-Leon wants to merge 2 commits into
Open
JingHao-Leon wants to merge 2 commits into
JingHao-Leon wants to merge 2 commits into
Conversation
A <testsuites> wrapper holding two suites, one with <property> run markers and one without, shows the unmarked suite borrowing the marked sibling's commit. The unmarked suite should fall back to the documented defaults and warn instead. Co-Authored-By: JingHao-Leon <102573344+JingHao-Leon@users.noreply.github.com>
_read_run_metadata looked up commit/attempt attributes on the element itself, but looked up <property> markers with ElementTree.iter, which descends into nested elements. A <testsuites> wrapper therefore reported the first nested suite's markers as its own, and parse_file's root level fallback handed them to any suite that carried no markers of its own. The result was silently wrong: a suite with no run identity was attributed to a sibling's commit, the missing metadata warning was suppressed, and a test that never flaked could be reported as a flake and charged for it. Restrict the root level fallback to the markers the root carries itself, so a suite without markers falls back to the documented defaults and warns, as documented. Co-Authored-By: JingHao-Leon <102573344+JingHao-Leon@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
A
<testsuites>wrapper no longer lends thecommitandattemptmarkers of a suite nested inside it to a suite that carries no markers of its own. Such a suite now falls back to the documented defaults and records the missing metadata warning, which is what the README already describes.Why
src/flakeledger/junit.py:74—_read_run_metadatareadscommitandattemptwithelem.get(...), which looks at the element itself, but reads the<property>markers withelem.iter("property"), which descends into nested elements. The two lookups had different scopes: the attribute lookup stopped at the element, the property lookup did not.src/flakeledger/junit.py:150— when a suite has no markers of its own,parse_filefell back to_read_run_metadata(root). The root is the<testsuites>wrapper, so the recursive property lookup found the<property>elements of the suites nested inside it and reported them as the root's own markers. The unmarked suite was given that commit and attempt, and because the fallback reported no missing metadata, the warning block atsrc/flakeledger/junit.py:155-160was skipped.The result was silently wrong: a suite with no run identity was attributed to a sibling suite's commit, the missing metadata warning was suppressed, and a test that never flaked was reported as a flake and charged for a flaky event that never happened.
The attribute marker style (
<testsuite commit="..." attempt="...">) was unaffected, becauseelem.getdoes not descend. Only the<property>style, which the README documents as the second accepted style, hit this.Fixes #9
Before
Fixture, the
betasuite carries no run markers:No warning is printed, even though the
betasuite has neither marker.After
The
betasuite falls back tounknown-commit, its failing attempt is the only observation for that commit, so it isundeterminedand not charged.Verification
PYTHONPATH=src python -m pytest -q, with the source at257573eand the new tests applied:With the fix:
PYTHONPATH=src python -m unittest discover -s testsreportsFAILED (failures=1)before the fix andRan 24 tests/OKafter it.python -m compileall -q srcpasses andPYTHONPATH=src python -m flakeledger --helpruns.The output over the bundled samples is unchanged:
ingeststill reports 25 cases over 4 commits,classifystill reports 3 flakes, 2 genuine failures and 9 stable, andcoststill reports a total flaky cost of 67.5008 USD.Checklist
python -m compileall -q srcpassespython -m pytest -qpassesCHANGELOG.mdgains a### Fixedentry under[Unreleased]. The README needed no change: it already documents the fallback tocommit=unknown-commit,attempt=1with a warning, and this makes the code do that.CI status
The
CIworkflow for this head has not run. GitHub recorded the run withconclusion: action_required— the run is blocked awaiting approval (forworkflows triggered from a fork, GitHub requires a maintainer to approve them
before they execute) — and
gh pr checksreports no checks for this branch.So there is no upstream CI result to report here. The test counts above are
from a local run against the patched tree.
This change was prepared with the assistance of an AI coding agent.