Repository navigation
ci: Test Python 3.12 coverage step times out about a third of the time #1381
Description
Test Python 3.12 is the only interpreter in the test matrix that runs the
suite under coverage, and its step budget now sits one minute above the slowest
run that fits inside it. The result is a job that fails on a timeout rather than
on anything in the change under test.
The asymmetry
.github/workflows/ci.yml:
- name: Run tests # 3.10, 3.11, 3.13
if: matrix.python-version != '3.12'
timeout-minutes: 45
run: python scripts/reporting_test_harness.py pytest tests/ -v -ra
- name: Run tests with coverage # 3.12 only
if: matrix.python-version == '3.12'
timeout-minutes: 55
run: python scripts/reporting_test_harness.py pytest tests/ -v -ra --cov=src/adcp --cov-report=term-missing3.12 carries --cov on top of the same suite and gets 10 extra minutes for it.
Measured step durations
Duration of the test step itself (not the job) across the seven most recent
non-cancelled ci.yml runs at the time of writing:
| step | cap | durations | margin |
|---|---|---|---|
Run tests 3.13 |
45 | 22, 22, 23, 26, 27, 29, 30 | 15 min |
Run tests 3.11 |
45 | 22, 28, 29, 33, 33, 34 | 11 min |
Run tests 3.10 |
45 | 40, 40, 40, 40, 41, 41 | 4 min |
Run tests with coverage 3.12 |
55 | 46, 50, 54, 54, 55✗, 55✗, 55✗ | 1 min |
Coverage costs about 13–14 minutes over the same suite on 3.10 (40–41 → 54), and
the extra budget granted for it is 10. Three of the seven runs hit the cap. All
three end on the same line:
##[error]The action 'Run tests with coverage' has timed out after 55 minutes.
In each case the log holds no FAILED line and no pytest summary — the suite was
still reporting PASSED when the step was killed, one of them at 86% of
collection. A branch's own content is not what decides which side of the cap it
lands on.
Two further notes:
- Pushes to
maindo not surface this, because they are cancelled by the
concurrency group before finishing, somainnever publishes a duration for
this job. It is only ever observable on a pull request. Run testson 3.10 has 4 minutes of margin on a 45-minute cap, so the same
failure mode reaches the non-coverage lane next.
Two remedies, either of which closes it
- Raise
timeout-minuteson the coverage step to clear the observed maximum
with real headroom — 70 would give it the same proportional margin that 3.11
and 3.13 have today, and 3.10 could use the same treatment. - Move coverage off the critical path: drop
--covfrom the matrix and collect
it in a separate job. That also removes the reason 3.12 differs from the other
three interpreters at all, and leaves four symmetric lanes.
Happy to send a PR for whichever you prefer, or to leave it with you if the
suite's runtime is the thing worth attacking instead.
Activity
KonstantinMirin commentedon Oct 3, 2026
Two more observations since filing, both the same step and the same line, which
tightens the sample to five timeouts across five unrelated branches.
conductor/sdk-types-freezeconductor/sdk-operational-routesconductor/sdk-mcp-lifespanfix/hierarchy-ratchetfix/scalar-str-codegenThe detail worth adding is where they stop. Four of the five print the same last
line before the step is killed:
The fifth reaches 88%. So these are not five branches each breaking something
different — they are five runs of the same suite reaching the same point and
running out of budget there, with the variation between them smaller than the
margin the cap allows. The diffs under test range from 3 files to 152, which is
further evidence the cap rather than the change decides the outcome.
For completeness on the earlier table: those durations are the step, taken
from the job API's per-step timestamps, not the job. The job adds roughly three
minutes of checkout, dependency install, lint and type-check on top.
bokelley commentedon Oct 3, 2026
Triage
Classification: Bug — CI configuration causes unreliable test failures unrelated to the change under test.
Bucket(s): (no "ci" label exists in this repo; bucket noted here as a gap)
Status: ready-to-implement
What the experts said:
continue-on-error: truesince coverage is informational, not a merge gate.My take: Both experts agree on the root cause and that the change is non-breaking (workflow config only, no SDK surface). Remedy 2 is the better fix: it removes the hidden coupling and makes each CI row's name match what it actually does. Remedy 1 is acceptable as a lower-effort interim fix provided step and job ceilings are bumped together.
Implementation scope (Remedy 2 — preferred):
.github/workflows/ci.yml: remove--covflags from theRun tests with coveragestep (lines 122–125) and make all four matrix lanes symmetric withtimeout-minutes: 50.coveragejob targeting Python 3.12 only, runningpytest tests/ -v -ra --cov=src/adcp --cov-report=term-missing,timeout-minutes: 80,continue-on-error: true.timeout-minutes: 60for the test matrix.pg-conformance-required-gateneeds:list (line 229) if the coverage job should also be a prerequisite — check whether coverage is currently a required status check.Implementation scope (Remedy 1 — interim, lower effort):
Triaged by Claude Code. Session: https://claude.ai/code/session_01LeGGBMmCwj6wwXR1WPsU1T
Generated by Claude Code
fix(ci): allow full coverage validation to finish before publication