Skip to content

ci: cut motd and push-worker further and lint both groups in parallel - #62

Merged
LucaLin233 merged 6 commits into
mainfrom
luno/ci-shard-3
Oct 24, 2026
Merged

LucaLin233 merged 6 commits into
mainfrom
luno/ci-shard-3

Conversation

@LucaLin233

@LucaLin233 LucaLin233 commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

结果

main 的 Shell tests 墙钟 107 s → 84 s(run 40125741952,19 个前置作业,无排队)。各必需检查全绿。

作业(Ubuntu / Debian) 耗时
ShellCheck 75 s(单作业,生产组与测试组并行 + 组内 -P4)
motd-a / motd-b / motd-c 55 / 54 / 32;75 / 55 / 42
cloudflare-a / cloudflare-b 53 / 50;61 / 67
push-worker-a / push-worker-b 54 / 50;58 / 57
push 69;66
other 43;—

PASS 与基线一致:motd 648(Ubuntu)/ 653(Debian)、push-worker 303、cloudflare 312。

改动

  • motd 8→11 段 / 3 作业(4/4/3):旧段 4 在 building_phase 前切、旧段 6 在 illegal_legacy_name 前切、旧段 7 按 tool/module 拆。
  • cloudflare 8→12 段 / 2 作业(5/7,按实测成本均衡):旧段 3 按信号拆、旧段 6 分 2、旧段 7 分 3。
  • push-worker 4→7 段 / 2 作业(4/3):failed-reap 8+12、retry 10+10、publication 12+12。
  • ShellCheck 并行:生产组与测试组同时启动、组内 -n1 -P4,分别 wait,任一失败归一 exit 1;保留 Run ShellCheck 步骤名。
  • runner 契约:TEST_SEGMENTS 上限 12、token 有界词法校验;test-shards.sh 严格 token 校验、非法 token 负控使用真实段数。
  • README 拓扑说明同步更新。

验证

  • 双 OS CI:所有分段作业实际执行,PASS 多重集与基线一致;两个必需检查通过。
  • 本机(Debian 13):motd 串行 655 PASS = 候选串行 = 11 段并集;worker 303 PASS 同;cloudflare 312 PASS 同;串行顺序一致。
  • test-shards.sh 与分片负控、test-lint-coverage.sh 与 lint 负控通过。
  • 前导零覆盖变量(08/09/010)按十进制解释,LENS 定向复审通过。

未达成与后续

  • 目标 ~75 s 未完全达到(84 s)。剩余瓶颈:ShellCheck 75 s、Debian motd-a 75 s(段 2/3 未切)。
  • 下一步若要压到 ~70–75:把 ShellCheck 拆成生产/测试两个作业(+1 作业,仍为 20 并发),并切 motd 段 2/3(13 段);两者都要再动矩阵与段契约,未纳入本 PR。
  • cloudflare 曾试过 3 作业(12 段 4/4/4),但 21 个前置作业触发并发排队,墙钟反升到 109 s,故收回 2 作业。
  • 分片夹具是文本提取器,不解析任意 YAML 形态;矩阵结构改写会 fail-closed。
  • 性能随 runner 负载波动,84 s 为单次实测;建议合并后再观察 main 的 2–3 次运行。

FORGE and others added 6 commits October 25, 2026 01:25
The eight motd segments left two jobs at 98 s and 77 s wall clock and a
63 s slowest segment.  The three heaviest segments are split along their
case-lifecycle boundaries instead of by line count, giving eleven
segments spread over jobs of four, four and three:

* segment 4 keeps the target, pointer and pending hook cases, segment 5
  takes the six building phases with their three signals;
* segment 7 keeps the dependency, noop and final-group cases, segment 8
  takes the illegal legacy names, legacy backups and legal imports;
* segments 9 and 10 run one implementation of the rollback-damage cases
  each and keep the normal rollback cases with segment 10; those cases
  move verbatim into run_rollback_damage_cases, so both halves share one
  body instead of running it twice.

Every segment still builds its own fixtures, and with TEST_SEGMENT unset
the guards run in file order, which is the original order.

tests/run.sh keeps the plan precheck, the duplicate detection and the
child environment cleanup; the segment count is bounded lexically to
1..12 and every TEST_SEGMENT_SET entry to 0..11 before the arithmetic
range comparison.

tests/test-shards.sh follows the eleven-segment expectations, runs
segment 10 as executable evidence for two-digit indices, and validates
every segment_set token strictly, so "8 9 10x" is refused instead of
being filtered out by the coverage arithmetic.
The eight-segment split left the second job at 83 s because it carried the
three heaviest segments (30/61/61/66 s) while the first finished in 53 s.  Cut
the heavy tail instead of piling it up:

- the uninstall signal matrix (39 independent cases) splits by signal into
  segments 3 (HUP, INT) and 4 (TERM), which keeps the serial order HUP, INT,
  TERM;
- the metadata capture matrix becomes segment 7 and the anchor plus
  creation/publication matrix becomes segment 8;
- the last segment (bookkeeping, capture boundaries, restore publishing,
  disable and uninstall barriers) becomes segments 9, 10 and 11.

Every segment still builds its own fixture, serial mode keeps the original
order, and the PASS tally is unchanged at 312.  Both OS matrices now run four,
four and four segments in three jobs whose sets mix early and late segments so
the 58 s straggler never shares a job with another heavy half.
The twelve-segment split spread cloudflare over three jobs per OS, which put
twenty-one jobs in the plan (nine per OS plus lint).  GitHub Free runs twenty
at a time, so cloudflare-a waited 44 s for a runner and the wall clock rose to
109 s.

Cut the suite into two jobs instead, balancing the measured segment cost:

- cloudflare-a: 2 3 5 6 11 -> 1+42+25+58+23 = 149 s
- cloudflare-b: 0 1 4 7 8 9 10 -> 11+12+21+19+39+21+19 = 142 s

Every segment still runs exactly once per OS, the heavy tail (42 s and 58 s)
stays apart from the 39 s segment, and the plan drops to nineteen jobs, so no
job queues behind another.  tests/test-shards.sh expects the new pair; the
segment count stays twelve, so the token and count rejection cases are
unchanged.
@LucaLin233
LucaLin233 marked this pull request as ready for review October 24, 2026 19:37
@LucaLin233
LucaLin233 merged commit 5fc7806 into main Oct 24, 2026
21 checks passed
@LucaLin233
LucaLin233 deleted the luno/ci-shard-3 branch October 24, 2026 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant