Repository navigation
Conversation
hsliuustc0106
left a comment
There was a problem hiding this comment.
DRAFT review — PR #213 "test(vela): require attributed member replays and qualify 9B on L20"
Verdict: COMMENTED. The PR is in draft state (draft=true, author psv666, first-time author). Per policy, an explicitly draft PR receives a COMMENTED review regardless of findings; this review does not approve, request changes, or alter the draft state, and does not fill the contributor's self-review checkboxes.
Template and draft state: All required sections present (Summary, Required human contributor answers, Evidence, Purpose, Test Plan, Test Result, Demo/evidence, Merge Danger, Self-review). All four self-review checkboxes are unchecked, consistent with draft. Note: checkbox texts for items 2 and 4 are abbreviated paraphrases of the template's exact wording ("I have checked architecture and scope." / "I have checked that claims match implementation and evidence.") — same substance, but the canonical template sentences are not used verbatim (P3 note below).
Pinned metadata (observed via GitHub API and local git)
- Base:
codex/vela195-9b-acceptance-base@77b31fc2863c— verified to be the head of draft PR #210 ("fix(vela): preserve row ownership in decoder executor replay"). Stacked directly on top of the approved-family work: ancestry of the base verified for #195's heada8ecddd(the approved 4B/9B PR), #177's accepted headdf02f95, #205's head5679f595, #208's head1d460564, and mainf1e674f2— exactly as the body claims. Mergeable=MERGEABLE / CLEAN into the recorded base. - Head:
codex/vela195-9b-acceptance-20261120@422e74f8b303; head repo InferMatrixLab (PRs live in the InferMatrixLab parent). Two commits, both authoredpsv666 <2693925048@qq.com>; underlying author branches unchanged (bases verified). - Files: 3, +117/−4 —
tests/vela/native_executor.rs(+49),recipe/vela/native.md(+67/−3),src/models/vela/README.md(+1/−1). No production source, CUDA/ABI, or manifest changes — verified by diffstat.
Body gate (post-#138 required answers)
All four mandatory answers present and specific; "Is a PR needed? Yes." present. Gate status: present and complete (checkbox wording deviation noted below).
What "attributed member replays" enforces
A fail-closed call-attribution guard in the GPU replay test, inserted before any row readout (tests/vela/native_executor.rs:119):
- New helper
validate_call_rows(native_executor.rs:232–243), called asvalidate_call_rows(&calls, planned.rows.len(), m != "Vela-2.0-0.8B"). For any member other than the legacy 0.8B, every recorded executor call must carry an integer"row"with0 ≤ row < planned_row_count; missing, mixed (some rows present, some absent), null, negative, fractional, string, and out-of-range values all fail the assertion before any row is read out or matched. - The legacy 0.8B fallback is deliberately preserved: if no call carries any
rowkey, the historical kind-order matching still passes; but a mixed legacy recording (some attributed, some not) is rejected — theany(...is_some)trigger makes the all-calls rule apply. The rationale is documented in code comment and inrecipe/vela/native.md:108–112("Only the original 0.8B recordings may omit all row metadata…; this compatibility does not apply to a fresh 4B/9B capture"). - A dedicated weight-free unit test
fresh_member_replays_require_every_call_to_identify_a_valid_row(native_executor.rs:345–377) exercises the helper directly across 4B/9B members with each invalid shape, valid-ownership pass, and the legacy all-unattributed pass. This closes the disclosed red state ("the weight-free regression accepted a fresh member recording with every row omitted under the inherited legacy fallback") — that before/after demonstration is author-reported (exit-101 log retained off-repo invela195-9b-evidence.tar.gz, SHA2560d6f006a…).
This strengthens the replay gate; existing row/kind/head uniqueness, finite/shape/count assertions, and all numerical ceilings are untouched (verified in diff).
The 9B-on-L20 qualification: test gate or recorded run?
A recorded run, not a test gate. The only code change is the attribution guard above. The 9B qualification lives in documentation: a new ### Independent L20 9B library profile section in recipe/vela/native.md:210+ recording: the pinned 9B package vllm-sr/Vela-2.0-9B@068425bb…; FP32-resident parameters (31.8 GB) with BF16-autocast backbone and FP32 heads; nine reference-vs-worker exact cases; five head captures hash-frozen before native comparison (373-span positive window); FP32-head maxima 3.624e-5 / 1.311e-6 against the original 1.6e-4 / 1.2e-2 ceilings; response finishing 7.708e-7 at 1.5e-2; the CPU hidden-input probe 1.252e-6; and a one-run four-case GPU logit replay (worst 0.2521) against the original 0.35 gates, no retries or overrides. All GPU numbers are author-reported and disclosed: one L20 sm_89, driver 580.178.04, private library SHA256 63f3f519… with process-map/GPU-UUID binding claims, evidence archive SHA256 + byte size + member count, and an explicit unverified list (historical exact replay unrun because original paired tensors were unrecoverable; full-length inputs, native positive-window GPU, complete native GPU typed answers, HTTP lifecycle, other hardware). Memory figures are properly caveated ("exclude some driver/library allocations and are not a general capacity guarantee"). Acceptable as disclosed for a draft; none of it is independently verifiable by this review (no GPU), and no CI validates the numbers.
The README row for 9B (src/models/vela/README.md:37) is corrected from an understated "~18 GB + FP32 heads" to "~31.8 GB FP32 parameters plus caches/workspace; measured L20 peak reserved 45,026 MiB" with a working relative link to the recipe anchor — a conservative correction that aligns docs with the recorded measurement.
Gate preservation vs approved #195 (extends, does not weaken)
Verified ancestry: #195's approved head is in this PR's base, and the diff only makes the fresh-member replay stricter (attribution required) while preserving legacy behavior explicitly and documented. No tolerance loosened, no gate constant changed, no skipped or ignored tests added (the --ignored GPU opt-ins are pre-existing convention). The strict-4B/9B evidence wording replaces an over-broad "remains unverified until rerun" sentence with an accurate split (historical unverified; current-paired 4B in #210, 9B below) — clarification, not weakening. Extends the approved surface.
Findings
- P2 — no CI run for the head in either repository; Rust-side effect unverified remotely and not runnable in this review. Trigger: Actions API
total_count=0for422e74f8b303in both ThinkFlowLab and InferMatrixLab;gh pr checks 213reports no checks. Consequence: compile and behavior of the new guard + unit test at this head rest on author-reported runs only (body: ordinary 77/5, workspace 504/126, clippy/fmt/release all pass; the "168 Vela Python passed" claim was independently reproduced here). The body discloses that main-only workflow filters may skip this stacked base. Required before ready: one CI run (or maintainer-run cargo suite) at the eventual head. - P3 — self-review checkboxes 2 and 4 deviate from the canonical template sentences. Trigger: body uses abbreviated paraphrases. Consequence: template-checking tooling and the #138 gate expect the canonical four questions' checklist verbatim; restore the template text before marking ready.
- P3 — stacked on a draft (#210). Trigger: base
77b31fc2863cis #210's head, and #210 isdraft=true(verified). Consequence: this PR cannot land until #210 and the rest of the stack land; a retarget and revalidation of source/profile will be required, as the body itself states. Not a defect of this diff — a sequencing constraint reviewers should not lose. - Notes (informational):
recipe/vela/native.md:27cites draft PR #210 as the current-paired 4B evidence anchor; draft-PR evidence can change or vanish — fine for now, re-anchor at merge time.- The unit test's
std::panic::catch_unwindstyle is acceptable; assertion-based helpers would read cleaner. No action required. - The "Before" red demonstration (all-unattributed fresh-member capture accepted) is a valuable negative control; the retained log is off-repo, so its content is trusted from the body.
Provenance (separated)
- Source inspected: PR body/metadata via API; full diff
77b31fc2863c..422e74f8b303; native_executor.rs at base and head; native.md and README.md at base and head; #210/#195/#177/#205/#208 states and heads; #138 issue. - Commands run (this review, macOS, no cargo/GPU; Python 3.12.8 / pytest 8.4.1 / mkdocs 1.6.1): worktree at
422e74f8b303(removed after):PYTHONPATH=src python -m pytest tests/vela -q→ 168 passed (independently reproduces the body's "Vela Python 168 passed");PYTHONPATH=. python -m pytest tests/benchmarks -q→ 133 passed;mkdocs build --strict→ exit 0, 0 warnings. - Observed CI: none exists for this head (0 runs in both repos, verified per-SHA via Actions API on both ThinkFlowLab and InferMatrixLab).
- Author-reported (not verified here): all cargo suite counts and clippy/fmt/build results; every 9B/0.8B L20 CPU and GPU replay number and gate comparison; the red/green attribution demonstration; memory and allocator figures; all artifact/library/bundle hashes.
Bottom line
A genuinely strengthening supplement: fail-closed row attribution for fresh 4B/9B replays with the legacy 0.8B fallback explicitly scoped, plus a disclosed (but unverifiable here) recorded 9B L20 profile with unchanged original gates and a corrected memory table. COMMENTED. Before ready: obtain CI or maintainer-run cargo evidence at the head (P2), restore canonical checklist wording (P3), and re-stack after #210 and the underlying work land (P3).
Summary
Require valid row attribution in the 4B/9B GPU replay before any row readout, and document the independently measured 9B reference memory and current-paired library profile.
This is a focused stacked supplement to #210, based on its delivered head
77b31fc2863c41713002c083140f15629a930395. It retains MrDongsls's #195a8ecddd, accepted #177df02f95, #208 / #205 and mainf1e674f2. Original author branches are unchanged. The 4B captures in #210 already carry valid rows; this guard preserves their qualification. Retarget after the underlying stack lands and revalidate the resulting source/profile.Required human contributor answers
The contributor explicitly confirmed this model-acceptance scope, necessary draft supplements and the fresh-member attribution guard; the detailed confirmation is retained in the contributor's stage record.
Is a PR needed? Yes. Necessary focused draft supplements preserving other authors' commits were explicitly authorized.
Evidence
1.6e-4/ answer1.2e-2gates, including exact 373 positive-window spans and Unicode offsets; maxima3.624e-5/1.311e-6(stdout).1.5e-2, max7.708e-7(stdout). A separate probe runs native FP32 candidate and span heads from frozen reference hidden inputs, then native finishing: four complete typed JSON envelopes pass, max1.2516975402832031e-6, with usage independently summed from validated plans (1041/133/597/228).Purpose
A new member capture that loses every row must fail closed rather than inherit the historical 0.8B call-order fallback. The memory documentation now distinguishes FP32 parameter residency and autocast caches/workspace from checkpoint or uniform-BF16 export size. No production forward/scoring, CUDA/ABI or root manifest change is introduced by this supplement.
Test Plan
Head:
422e74f8b3030d5cf766cf3ce4ee281d44691d58, tree957b4bb97a52c42423fe2829891d4fbf12edd09d. Numerical Rust code is the validated3a8e054089196a16f264cf29d31aac878a45e98f; subsequent source changes are documentation only.From repository root (use the archived current-paired replay view for 9B opt-ins):
Test Result
Personally executed on L20: ordinary Vela 77 passed / 5 ignored; workspace 504 passed / 126 ignored; Vela Python 168 passed; all Rust CI commands and strict docs pass. Original 9 row-cap regressions pass in debug and release. Real 9B processing 2/2 covers all five plans plus structured-state tokens; the recorder independently verifies fresh plans equal those expectations. The actual tokenizer admits final 16383/16384-token rows and rejects the next row with whole-request 413/too_long. No workspace-wide ignored run or unrelated model GPU campaign.
Demo / evidence
Package
vllm-sr/Vela-2.0-9B@068425bb145c05884a98b870c219047da22d9aea. All 46 official-revision files, Git blobs and 15 LFS pointers/content hashes verified; all 426 faithful/rounded-BF16 backbone tensors, 37 FP32 heads, tokenizer/calibration/config and exact bundle file set checked. Manifest anchor2b57dd1bd9ee2b10d60fa8ff0567d13a45cb411fd67e739c5589c21cd05688c2.Reference: Python 3.12.13, torch 2.11.0+cu128, transformers 5.19.0, tokenizers 0.23.2, numpy 2.5.3, safetensors 0.8.0. All FP32 parameters, BF16-autocast backbone and FP32 heads; default batch 32 / token budget 24576, highest float32 matmul precision / CUDA matmul TF32 off, Transformers torch GDN fallback. Nine real reference TestClient versus production Python-worker cases match status/body/key order; all-kind warmup observes candidate/router/broad heads. Each of the five captures traces exactly one
_predict, including batches and same-run typed oracles. JSON+tensors frozen before any native numerical comparison; frozen manifest SHA256e1bb8eb7e8a8e97e14d837e084f5a1efd09fbacc38d0dfdb83d4b28fa86a0f40remains unchanged afterwards.Reference PyTorch peak allocated 44305.62 MiB / peak reserved 45026 MiB; CUDA free after the long-window capture 79.81 MiB. This fixed run completes with very little headroom; allocator figures exclude some driver/library allocations and do not establish general capacity or full 16K inference support.
Native: one L20 sm89, driver 580.178.04, Rust 1.100.0 / CUDA 12.8.61, uniform BF16 backbone / FP32 heads / individual fixed-M64 readouts, Graph/opt-ins unset. Library
63f3f5199420a22dec2976899939e6a749ef998202bad26fcf8928545b95235breused read-only after actual C/CUDA/loader source, symbols, ABI 9, sm89 and default-profile match. Actual process maps, /proc/exe hashes and GPU UUID samples bind execution to this private library/GPU1.Raw commands/exits, original failures, frozen pairs, historical deltas, typed responses, source/binary/package/library/env hashes and reproduction helpers are retained off-repository in vela195-9b-evidence.tar.gz, SHA256 0d6f006a8aab9d8ae150f0417416f1141ab4e584044dec33923e6780fefcf6d8 (67,364,164 bytes; 410 members). The contributor retains the artifact. No generated run dump or new binary fixture is added to product Git; all original #195 fixtures remain unchanged.
Historical exact replay remains unrun because original paired 9B tensors/hashes were unavailable. Fresh and historical long-window captures both have 373 spans but some positions differ; the archived delta is a reference-run difference, not a native failure. Current native comparisons use the frozen fresh pairs throughout.
CPU typed responses use reference-backbone hidden inputs. Complete native GPU typed answers, native HTTP/readiness/frontend/lifecycle, native positive-window GPU, full-length reference inputs and other hardware remain unrun. No performance claim; test wall time includes repeated full-bundle verification. Video/performance A/B: N/A for this harness/docs supplement.
Merge Danger
Door: two-way.
Blast Radius: validation.
Damaged member metadata is rejected earlier; valid finite-input production arithmetic is unchanged. This narrow qualification does not establish family-wide readiness.
Self-review
The focused complete diff, callers, ownership, original gate preservation, observed red/green failure and artifact hygiene were checked. No remaining blocking finding in this supplement. The inherited main-to-stack diff still exceeds 3,000 authored-code lines and requires full contributor/maintainer review. Keep draft; this is personally executed evidence, not maintainer approval or candidate CI (main-only workflow filters may skip this stacked base).