ci: upload UT diagnostics on failure - #449
Merged
Merged
Conversation
XuPeng-SH
added a commit
to matrixorigin/matrixone
that referenced
this pull request
Sep 11, 2026
## Problem The single-runner race UT job can hit its outer timeout while the log only shows the last foreground line. A partial run needs to tell us which stage was active, which cases completed slowly, which cases were still active, and where the raw evidence is stored. ## Change - Keep the bounded single-runner UT schedule: HNSW completes first, light runs in a private report helper, and the issues package may overlap only through the explicit opt-in path (`UT_OVERLAP_LIGHT=0` remains the default). - Emit an append-only checkpoint for stage, package, process, admission, cancellation, and report ownership events. - Emit a bounded heartbeat every 60 seconds with current stage/label, active cases discovered from foreground and helper/shard reports, process count, and cgroup memory current/peak. - On timeout, cancellation, or test failure, preserve the partial Go JSON stream, checkpoint, stderr, and a top-20 summary of completed slow tests/packages. Active/incomplete cases are reported separately; truncated UTF-8/JSON lines do not discard the preceding events. - Stop child process groups and the heartbeat with bounded waits before collecting diagnostics; preserve helper reports until their single-owner merge succeeds. - Add scheduler, cancellation, parser, and heartbeat tests. A companion CI workflow change uploads `ut-report` and the raw `scratch/*-UT-Report.out` files on failed/cancelled jobs for download (matrixorigin/CI#449). ## Validation - `bash -n optools/run_ut.sh` - `python3 -m py_compile optools/summarize_ut_slow_cases.py` - `go test ./optools -count=1` - `go test -race ./optools -count=1` - `git diff --check` The scheduler change is opt-in until a same-runner A/B proves wall-time and resource benefit. The diagnostic path is active for all UT runs and does not change test coverage or the authoritative test result. --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a MatrixOne UT job times out or is cancelled, the runner preserves partial evidence under
ut-reportand helper streams underscratch. Upload those files as a short-lived, shard-specific artifact so a timeout can be downloaded and analyzed after the log window is gone.The failure/cancellation artifact includes:
ut-reportThe runner normally snapshots the main JSON into
ut-reportusing a same-filesystem hard link where possible, so the raw scratch copy is not uploaded twice. A best-effort pre-upload fallback recovers the nested scratch main report when an outer kill reaches cleanup before that snapshot exists.The upload copy is bounded to 250 MiB total, including a 64 KiB manifest reserve. Files beyond the budget retain a marked head/tail fragment, and
manifest.txtrecords original/stored sizes and omitted entries. The source reports remain untouched. Recovery is capped at 2 minutes and upload at 5 minutes. Diagnostic artifacts are retained for 1 day.Successful UT jobs upload only the small
top.txtand checkpoint baseline for 1 day. This provides historical stage/slow-case comparison without copying the raw JSON on every successful run.The recovery, staging, upload, and baseline steps are best effort and do not change the authoritative UT result. Missing diagnostics are warned and ignored.
Validation:
git diff --checkpasses