Repository navigation
Conversation
hsliuustc0106
left a comment
There was a problem hiding this comment.
Draft review — PR #210 "fix(vela): preserve row ownership in decoder executor replay"
Verdict: COMMENTED (first-pass review of a DRAFT; draft state must not change and no approval, merge or completed validation is implied). Nothing was posted; this is a local draft body.
Review state and provenance
- Draft state: OPEN,
isDraft: true(verified twice viagh pr view, start and end of review). Head unchanged at77b31fc2863c41713002c083140f15629a930395. Head repoInferMatrixLab/system1-omni, branchcodex/vela195-4b-acceptance-20261120; basecodex/vela195-4b-acceptance-base@4792f8c7dfe53be00cffd7a6beea67dd09e903af(verified in git: merge ofa8ecddd7= #195 head +df02f95c= accepted 0.8B snapshot, matching the body). Authorpsv666(first-time contributor on my record). - Template: body contains all four post-#138 answers, the necessity decision, bug-fix failure evidence (GPU test exit 101 on
example-request: span word count, native 23 / reference 27; base loader checks 2 passed/1 failed on accepted NaN),## Summary,## Required human contributor answers,## Evidence,## Test Planwith head/tree/base identifiers,## Demo / evidence(pinned 4B package, bundle anchor, library SHA-256, hardware/toolchain profile, off-repo archive SHA-256),## Merge Danger,## Self-reviewchecklist present and unchecked (left untouched). - Gate status: intent answers attributed to contributor confirmations 2026-11-19/20 retained in an off-repo task record — not independently verifiable from the repository; recorded as author-reported. Draft → COMMENTED regardless.
- Skill load record: local unpublished skill
/Users/hsliu/thinkflowlab/system1-omni/.agents/skills/system1-omni-review/SKILL.mdSHA-25658be3bc60806c4c5af7038a0c6a80fadb94f3360d714556a3e965c8903dde108;references/repository-contracts.mdSHA-256d6ae9650e4e18e8986926a790339412f52750fd1eeb19d358e5d91cd547f1202(byte-identical at this head);docs/architecture.md(head SHA-256e2635d6c...) andCONTRIBUTING.mdread at the head.
Base/head and merge-base
- Merge-base with mirror
origin/main(5fa5179d5):5fa5179d5(mirror head). Against canonical main (614b0abc, fork): merge-base isf1e674f2— canonical main is 1 version-bump commit ahead. - Ancestry verified: #208's head
1d460564is an ancestor of this PR's base4792f8c7; #195's heada8ecdddis also an ancestor. So this supplement sits on top of #208 and #195 in one integrated stack. - #195 is OPEN/unmerged per
gh pr view 195(state: OPEN,mergedAt: null) — noted because the review brief described #195 as merged; observed API state differs. Canonical main contains nosrc/models/vela/nativecrate, consistent with #195/#176/#205/#175/#152 all being OPEN.
Diff (focused: 4792f8c7..77b31fc28, 4 commits, all authored psv666)
5 files, +147/−30 (= 177 changed text lines; --numstat: recipe/vela/native.md +9/−0; src/models/vela/README.md +11/−7; src/models/vela/native/src/heads.rs +14/−7; tests/vela/native_executor.rs +108/−12; tests/vela/native_head_tensors.rs +5/−4) — matches PR metadata. The body's "150 changed Rust lines / 140 nonblank, noncomment code lines" is consistent. No production forward/scoring arithmetic, ABI, or tolerance changes in the diff — verified by reading every hunk. Well below large-change thresholds; the inherited stack's >3,000-line review obligation is acknowledged in the body and remains.
- Commit
19e76020(heads.rs + test formatting): loader fix, see below. - Commit
f7ed41f6: executor replay ownership fix + 3 weight-free regression tests. - Commits
83655f2a,77b31fc2: README/recipe docs corrections.
Review questions
- The ownership bug:
tests/vela/native_executor.rs::the_gpu_backbone_replays_the_reference_readoutsconsumed recorded reference head calls per kind in global recorded order (oldnext_calltook(calls, index, kind)and advanced a per-kind counter), while the recorder writes calls in actual execution order. When the executor's execution order differs from planned row order, replay compares a row's native forward output against another row's recorded call — on the real 4B example, a 23-word native row was compared with another row's 27-word call, trippingassert_eq!(got.len(), want.len(), "{case}: span word count")(native_executor.rs:200 at head) with exit 101 before the 0.35 logit ceiling. So rows were attributed to the wrong request/call: a false failure here (and a latent false-pass risk whenever shapes happen to match). - Fix mechanics: new
next_call(calls, consumed, row, kind, head)matches each planned row's call by unique, unconsumedrow+kind+headwhen fixtures carryrowattribution: exactly one unconsumed match is required (0 → panic "expected exactly one", >1 → panic duplicate); if any call carriesrow, all must carry a valid one (mixed fixtures fail loudly); after the loop,assert!(consumed.iter().all(|used| *used))rejects unused calls (native_executor.rs:219-222 at head). Legacy 0.8B fixtures withoutrowmetadata keep the original kind-order path (explicitly commented fallback). The per-kind counters are retained for the final count assertions (lines 210/215).heads.rs(19e76020) also restores finite FP32 tensor validation dropped by the member parameterization and rejects truncated F32 byte lengths:tensor()now errors on a non-emptyas_chunks::<4>()remainder ("has an invalid F32 byte length") and on any non-finite value ("contains non-finite values") instead of silently ignoring the remainder and accepting NaN/Inf (heads.rs:241-255 at head). - Regression test pinning the exact failure: yes, three weight-free tests:
attributed_calls_match_rows_when_execution_is_reordered— two equal-shape span calls recorded in reversed row order (row 1 first); the assertion requires row 0 to receive row 0's call's logits, which the old nth-by-kind selection would have failed (it would hand row 0 the row-1 call).legacy_unattributed_calls_keep_the_original_kind_orderpins the fallback.attributed_calls_reject_missing_duplicate_and_wrong_headspins the rejection cases viacatch_unwind. Plus the new consumed-all assertion in the GPU replay. Caveat (author-disclosed implicitly, stated here explicitly): the new tests use the newnext_callsignature, so they cannot literally compile/run against the old code — the "fails against the old selection" red state is author-reported reconstruction, not an observed run of this exact test file at the base.
Findings
No actionable findings (P0–P3) in the focused diff.
Informational notes:
- N1: The attributed matcher assumes the recorder's
rowvalue equals the index intoplanned.rowsused by the replay loop. The weight-free tests validate the matcher logic only with synthetic calls; real index alignment is pinned solely by the GPU replay itself (author-reported 4/4 cases passing). A recorder change that renumbers rows would surface as loud panics, not silent drift — acceptable. - N2: The new F32 byte-length-remainder rejection in
heads.rshas no dedicated regression test (the non-finite path does:tensor_rejects_non_finite_values, pre-existing at base and failing there because NaN was accepted — consistent with the body's "2 passed / 1 failed"). Minor coverage gap. - N3: Template shape: results are presented in a table under
## Evidenceand there is no literal## Test Resultheading (contents are present and more detailed than the template minimum). Template-polish only. - N4: The base-integration note (15 legacy unlabeled 0.8B replay inputs from #177 coexisting with #195's labeled inputs; dedup review deferred to stack integration) is correctly flagged by the body as future work, not hidden.
Evidence separation
- Source inspected: all four commits' hunks in full;
heads.rsat head (tensor loader, cfg-test registration ofnative_head_tensors.rs);tests/vela/native_executor.rsat head (replay loop, new matcher, count assertions,span word countassert);tests/vela/native_head_tensors.rsat head and base (non-finite test present and failing at base per body); README/recipe at head. - Commands personally run: ref/SHA/ancestry checks; worktree
/tmp/pr-review-pass194/wt210@77b31fc2;PYTHONPATH=src /Users/hsliu/miniconda3/bin/python -m pytest tests/vela -q→ 168 passed (9.3 s);PYTHONPATH=. ... pytest tests/benchmarks -q→ 133 passed (28.3 s);mkdocs build --strict(mkdocs 1.6.1) → exit 0. Nocargo/rustcand no GPU on this host — Rust tests and the GPU campaign were not runnable here. - Observed CI: none.
gh pr checks 210→ "no checks reported on the branch". Actions APIhead_sha=77b31fc28...→total_count: 0in bothThinkFlowLab/system1-omniandInferMatrixLab/system1-omni. The body itself anticipates this ("Main-only CI filters may not run against this stacked base"). - Author-reported (not verified here): the GPU replay campaign (4/4 cases at the unchanged 0.35 ceiling, drifts 0.1602/0.0211/0.0738/0.0254), the real 4B failure before / pass after, loader red state, all cargo fmt/clippy/workspace/release results (503 passed, 126 ignored), token plans, real-bundle processing, frozen reference pairs, HTTP client/worker parity (9/9), CPU finishing/long-window results, and the off-repo evidence archive.
Material unverified checks
- All Rust behavior (weight-free regressions included) and all GPU/native numerical claims — no Rust toolchain or GPU on the review host; the 0.35-ceiling GPU qualification is the core acceptance evidence and rests on the author's archived runs.
- Current-vs-historical long-window reference difference (82 vs 76 spans) is disclosed as separately reported; historical exact replay unrun — remains open per the body.
- Complete GPU typed answers, native HTTP/lifecycle, 9B, and other hardware profiles remain unrun (body-disclosed; unchanged by this supplement).
Cross-PR and integration
- Stacked on #208 (verified ancestor) and #195 (verified ancestor of the base). Docs-only overlap with #208 (
recipe/vela/native.md,src/models/vela/README.md; both normalize the same trailing "Checks:" line — consistent in stack order). No Rust file overlap with #208; relative to #195 this PR is the intended focused supplement onheads.rs/executor tests/docs. Since #195, #176, #205, #175, #152 are all OPEN, none of this stack can land until the parents do; retarget onto current main and revalidate per hop, as the body states.
Summary
The executor replay in #195 consumes reference calls in global execution order, while its single-predict recorder records actual row ownership. On the real 4B example, it compares a 23-word native row with another row's 27-word call and fails before its numerical gate. Match each call by unique, unconsumed
row + kind + head, retain the legacy 0.8B path, and reject missing, duplicate or unused calls.Also restore finite FP32 tensor validation dropped by the member parameterization and correct the README's stale packed-forward/test-count claims. The recipe explains how to preserve complete reference JSON/tensor pairs.
This stacked supplement preserves the author's #195 head
a8ecdddand accepted 0.8B snapshotdf02f95(accepted #176/#208/#205 plus current mainf1e674f2). Its contributor-owned base is4792f8c; the focused diff is 5 files, +147/-30, 177 total text lines, with 150 changed Rust lines / 140 nonblank, noncomment code lines. No production forward/scoring arithmetic, shared CUDA/ABI, numerical tolerance or committed expectation changes.Required human contributor answers
The contributor explicitly confirmed the common intent and necessary-stage-draft decision on 2026-11-19, and the independent 4B stage on 2026-11-20; the traceable source remains in the contributor's task record.
Is a PR needed? Yes. Necessary draft supplements preserving original authors' history are authorized. The observed loader and replay-ownership gaps are separate from the delivered 0.8B cap and #205 recipe/test-placement work.
Evidence
example-request: span word count, native 23 / reference 27. A weight-free regression with reordered, equal-shape calls also fails against the old selection. The inherited loader checks at base4792f8creport 2 passed / 1 failed because NaN is accepted.Fresh 4B reference JSON and hidden tensors were frozen together before native comparison, from one real
_predictper case, including inputs/logits/calibrated answers/execution batches. Same-run SystemOne finishing consumes the captured triple without another forward. Original product fixtures stay unchanged; an isolated replay view holds the current pairs. Current reference long-window output has 82 spans versus the historical committed 76. That reference-run difference is reported separately; historical exact replay is unrun because its original paired tensors/hash assets were unavailable.too_longTest Plan
Head:
77b31fc2863c41713002c083140f15629a930395, tree4929dc8e84c787ad2295e206ffa7305a373a76eb. Base4792f8c7dfe53be00cffd7a6beea67dd09e903af. The final recipe-only descendant has the same Rust source as testedf7ed41fand was rechecked with strict MkDocs.From repository root:
Demo / evidence
Pinned 4B package:
d692394127f6f87fc581d8634c9558fe4776efce; actual extended bundle anchor:948160dd331442f1cb9739793e2d78406feab20de4831f2d96fc091f87f8bef9. Present assets match fresh pinned HF Git/LFS metadata; all 426 faithful/rounded-BF16 backbone tensors, 37 FP32 heads, tokenizer/calibration/config were independently checked. Reference: Python 3.12.13, torch 2.11.0+cu128, transformers 5.19.0, tokenizers 0.23.2, numpy 2.5.3, safetensors 0.8.0; BF16-autocast backbone / FP32 heads.Native profile: one NVIDIA L20, sm_89, driver 580.178.04, Rust 1.100.0 / CUDA 12.8.61; uniform BF16 backbone, FP32 heads, individual fixed-M64 readouts, no Graph or other opt-ins. Library SHA256
63f3f5199420a22dec2976899939e6a749ef998202bad26fcf8928545b95235bwas reused only after actual source/symbol/ABI/target/profile match to accepted #177. Process maps, running-binary hashes and GPU UUID sampling bind the executed profile.Raw commands/exits, original failures, frozen pairs, old-vs-new deltas, probes and source/binary/package/library hashes are retained off-repository in vela195-4b-evidence.tar.gz, SHA256 ee5602ddf1aeb39864720c81740620554ebe5ac7bc357e8fb8a8a557ad7dd4d7. The archive contains 256 files; the contributor retains the artifact. No generated run dump or new binary fixture is added by this focused supplement. The integration base retains 15 legacy unlabeled 0.8B replay inputs from accepted #177 alongside #195's labeled inputs; all 51 original #195 fixture blobs are unchanged. Current member tests use labeled paths. Final stack integration should review legacy deduplication while preserving #177 reproducibility.
These GPU results qualify small-case logits. CPU native responses use frozen GPU-reference hidden inputs, not native GPU-backbone outputs. Complete GPU typed answers, native HTTP/readiness/frontend/lifecycle, native positive long-window GPU backbone, 9B and other hardware remain unrun. Video/performance A/B: N/A for this loader/test/docs supplement; no latency or throughput claim.
Merge Danger
Door: two-way.
Blast Radius: validation.
Valid finite-input arithmetic is unchanged. Non-finite head tensors now fail at load, and executor parity checks compare the correct recorded owners. Retarget/revalidate after the underlying stack lands; this narrow profile does not establish family-wide readiness.
Self-review
The focused diff, architecture boundaries, failure/pass evidence and artifact hygiene were checked. No remaining blocking finding in this supplement. The inherited branch-to-main stack still exceeds 3,000 authored-code lines and requires full contributor/maintainer review. Keep draft; no reviewer request or merge is implied. Main-only CI filters may not run against this stacked base; the results above are personally executed checks.