Skip to content

modernbert: reject zero attention heads before deriving geometry - #207

Draft
psv666 wants to merge 1 commit into
codex/vela152-main-sync-20261120from
codex/vela152-cpu-sync-20261120
Draft

psv666 wants to merge 1 commit into
codex/vela152-main-sync-20261120from
codex/vela152-cpu-sync-20261120

Conversation

@psv666

@psv666 psv666 commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Required human contributor answers

The contributor confirmed this stage on 2026-11-20: synchronize the existing Vela ModernBERT PR with main and validate its CPU structure/loader contracts, preserving the original author's work. The shared contributor intent was confirmed on 2026-11-19 and remains applicable; detailed confirmation is retained in the contributor's task record. Public parent context: #137 and #152.

  1. Outcome: Bring assigned model support to reproducible, reviewable acceptance so supported models work reliably.
  2. Highest-leverage variable: Reliable model coverage and numerical/contract consistency.
  3. How this moves it: Reuse the existing implementation, close contract and evidence gaps, and add only necessary focused corrections before performance work.
  4. User and need: Model, application and agent developers who need stable interfaces, reliable outputs and reproducible deployment.

Is a PR needed? Yes. The contributor authorized necessary small supplements and stage draft PRs while preserving others' commits. A real CPU loader failure was reproduced here.

Bug-fix evidence: On base 90a1beff, the new integration test fails with adapter.rs:90:13: attempt to divide by zero. The same command passes after this guard. Reproduction and verbatim excerpts are below.

Purpose

Reject num_attention_heads: 0 as a field-specific configuration error before deriving hidden_size / num_attention_heads. Without an explicit head_dim, the adapter currently panics before structure validation can reject the input.

This is a 2-file, +21-line stacked supplement, based on codex/vela152-main-sync-20261120 at 90a1beff35e4a8228593ae166073d8c7af9ce46d. That base preserves MrDongsls's #152 head 022c62af and main f1e674f2 as its two parents. Its sole manual merge resolution unions the workspace members in root Cargo.toml; Cargo.lock is auto-merged. Qwen ABI stays 9.

The base carries the original 74-file implementation so this draft's review diff stays limited to the guard and regression. Integration into main requires #152 to land, followed by retargeting and fresh checks. No changes were pushed to the author's branch.

Summary

 derive_head_dim(hidden, heads, config)
+  reject heads == 0 with num_attention_heads error
   derive or parse head_dim
   validate the complete structure

The root integration test covers the 1024-wide family fixture and Vela 0.3B fixture, each with derived and explicit head_dim. Existing golden fixtures, geometry, checkpoint census, precision classes and numerical tolerances are unchanged.

Evidence

Run from the repository root on Linux, with Rust 1.100.0 and no weights or GPU:

cargo test --locked -p omni-modernbert --test structure \
  zero_attention_heads_is_rejected_without_panicking -- --exact --nocapture

Before, base plus only the regression test:

thread 'zero_attention_heads_is_rejected_without_panicking' panicked at
src/models/modernbert/src/adapter.rs:90:13:
attempt to divide by zero
test result: FAILED. 0 passed; 1 failed

After, head a11335aeed99d30ebdef123d673b569eb8a1c299:

test zero_attention_heads_is_rejected_without_panicking ... ok
test result: ok. 1 passed; 0 failed

Test Plan

CPU-only on the L20 host, Rust 1.100.0, private cargo-home/target/Python environments; CUDA_VISIBLE_DEVICES empty. Dependencies came from verified local caches; Cargo checks used --locked --offline. No checkpoint download, forward pass or GPU execution.

  • cargo fmt --all --check
  • cargo clippy --workspace --locked --all-targets --features omni-laya/serve -- -D warnings
  • cargo test --workspace --locked --features omni-laya/serve
  • cargo build --workspace --release --locked --features omni-laya/serve
  • Component test discovery/tests for ModernBERT, Vela, Laya and shared Qwen; Vela cuda-feature compilation/CPU checks.
  • CUDA text contract and root Python tests; Vela Python/comparison-tool checks; benchmark tests; strict MkDocs.

Test Result

Check Personally executed result
Focused components 132 passed, 0 failed, 74 ignored
Full workspace, serialized follow-up 472 passed, 0 failed, 121 ignored
fmt, strict workspace Clippy, release build PASS
Vela Python / comparison tool 157 passed / PASS
Benchmark Python 133 passed; 185 subtests passed
CUDA text contract / root Python checks 0 errors, existing ABI-define warning / 27 passed
Strict MkDocs PASS using a separately copied and hash-verified docs environment

The first full workspace run failed the unchanged JEV-VL test stats_and_prefix_publication_finish_concurrently at its 10-second timeout. It subsequently passed in isolation on exact main (0.69 s) and this candidate (0.62 s); one serialized full-workspace retry passed. The original failure is retained; its cause remains unclassified and the timeout was not changed.

The initial docs attempt used a Python environment without MkDocs; that failure is retained. The final strict build used the existing independent docs environment, without installing or modifying shared packages.

Demo / evidence

The before/after CPU regression above is the evidence for this patch. Original Laya golden plans and the 20-configuration mock launch-trace comparison pass; these checks do not establish device numerical parity. The pinned Vela census (134 encoder + 18 head tensors) and YaRN assertions pass without fixture regeneration.

The Vela cuda-feature command reports 13 ok, but its one device test explicitly early-returns without VELA_NATIVE_CHECKPOINT/VELA_NATIVE_LIBRARY: 12 CPU checks, device execution unrun. L20 is sm_89 while the inherited bundle requires sm_90a; no L20 native Vela result is claimed. Historical H100 evidence belongs to its recorded source.

Raw logs, failures, source bundles, hashes and the component review map remain in the contributor-local/L20 evidence archive; there is no public raw-archive URL. The repository test and excerpts above are independently reproducible. Performance A/B and video are N/A for this invalid-config guard. The complete original #152 implementation still requires its own large-change and hardware review.

Merge Danger

Door: two-way; ordinary revert, no stored-data migration.

Blast Radius: loader.

Only malformed zero-head ModernBERT configurations change from panic to error. Valid structures retain their existing parsing, geometry and execution plans.

Self-review

Agent precheck reviewed the complete 21-line supplement, the caller through Vela read_structure, root test discovery, artifact placement and the unchanged positive-profile gates. No blocking finding was identified for this small guard. The full parent PR, future decoder path/crate integration and target-hardware qualification remain separate. Keep draft for contributor/maintainer review.

  • I have reviewed the full diff and addressed the issues I found.
  • I have checked that the change follows the project's architecture and stays focused on the stated purpose.
  • I have run the checks appropriate to this change and reported commands, results, and anything I could not verify above.
  • I have checked that the PR description, documentation, and any accuracy or performance claims match the implementation and available evidence.

A CPU loader regression test on the synced #152 tree reproduced a division-by-zero panic for num_attention_heads=0 without head_dim. Return a field-specific error before division; cover the 1024 family and Vela 0.3B HF layouts with derived and explicit head dimensions. Preserve all positive-profile geometry and numerical assertions.

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Draft review — PR #207 "modernbert: reject zero attention heads before deriving geometry"

Verdict: COMMENTED (first-pass review of a DRAFT; draft state must not change and no approval, merge or completed validation is implied). Nothing was posted; this is a local draft body.

Review state and provenance

  • Draft state: OPEN, isDraft: true (verified twice via gh pr view, start and end of review). Head unchanged at a11335aeed99d30ebdef123d673b569eb8a1c299. Head repo InferMatrixLab/system1-omni, branch codex/vela152-cpu-sync-20261120; base codex/vela152-main-sync-20261120 @ 90a1beff35e4a8228593ae166073d8c7af9ce46d. Author psv666 (first-time contributor on my record).
  • Template: canonical .github/PULL_REQUEST_TEMPLATE.md (fork/main 614b0abc). Body contains all four post-#138 answers (outcome / highest-leverage variable / how it moves the variable / user and need), the necessity decision ("Is a PR needed? Yes — ..."), bug-fix failure evidence (divide-by-zero panic excerpt at adapter.rs:90 on base, passing at head), ## Purpose, ## Test Plan, ## Test Result (table), ## Demo / evidence, ## Merge Danger, ## Self-review with the 4-item checklist present and unchecked (left untouched).
  • Gate status: human contributor-intent answers present, attributed to contributor confirmations of 2026-11-19/20 with source "retained in the contributor's task record" — that record is off-repo and not independently verifiable from the repository; recorded as author-reported. Draft → COMMENTED regardless.
  • Skill load record: /Users/hsliu/thinkflowlab/system1-omni/.agents/skills/system1-omni-review/SKILL.md, SHA-256 58be3bc60806c4c5af7038a0c6a80fadb94f3360d714556a3e965c8903dde108 (local working-tree copy of the unpublished skill, working tree at ca5f69b5); references/repository-contracts.md SHA-256 d6ae9650e4e18e8986926a790339412f52750fd1eeb19d358e5d91cd547f1202 (byte-identical at the PR head). docs/architecture.md and CONTRIBUTING.md read at the PR head.

Base/head and merge-base

  • Merge-base with mirror origin/main (5fa5179d5): 5fa5179d5 — the mirror head itself, i.e. not stale relative to the mirror.
  • Mirror lag check: canonical main fetched from git@github.com:InferMatrixLab/system1-omni.git is 614b0abc (#214, 2026-11-20). Merge-base of origin/pr/207 with canonical main is f1e674f2 — 1 commit behind canonical main (version bump only). Trivial drift; retarget needed after #152 lands.
  • Base parentage verified in git: 90a1beff = merge of 022c62af (#152 head) + f1e674f2 (main) — matches the body's claim. gh pr view 152 → OPEN, unmerged.

Diff (focused: 90a1beff..a11335ae)

2 files, +21/−0 — matches PR metadata. One commit, authored psv666 <2693925048@qq.com>. Authored-code count = total count = 21 added lines (far below any large-change threshold; the inherited #152 stack is explicitly excluded from this diff and remains separately reviewable).

  • src/models/modernbert/src/adapter.rs (+4): ensure!(heads > 0, "num_attention_heads must be positive, got {heads}") as the first statement of derive_head_dim (adapter.rs:81-83 at head).
  • tests/modernbert/structure.rs (+17): regression zero_attention_heads_is_rejected_without_panicking.

Review questions

  • Division-by-zero guard: yes. The only integer division by num_attention_heads in the file is hidden / heads inside derive_head_dim (adapter.rs:95 region); heads is read from config at adapter.rs:244 via uint(config, "num_attention_heads")?, so a missing field errors earlier and only an explicit 0 reaches the guard.
  • Where the rejection happens: at config parse, inside derive_head_dim (called from parse_hf_config, adapter.rs:245), before structure validation — exactly where the panic occurred. The guard is placed before the head_dim match, so it also rejects heads == 0 when an explicit head_dim is present. That is stricter than strictly necessary to prevent the divide-by-zero, but is a reasonable field-level validity rule and the test pins it deliberately.
  • Test pinned: yes. zero_attention_heads_is_rejected_without_panicking covers family_config() (structure.rs:13) and vela_encoder_config() (structure.rs:28) × {derived, explicit head_dim} (4 combinations), asserting the error contains num_attention_heads. ModernBertConfigAdapter::from_value is exercised end-to-end.

Findings

No actionable findings (P0–P3) in the focused diff.

Informational notes:

  • N1: The body's Test Plan omits the literal template field "System1-Omni Version / Commit:"; the tested base/head commits are identified in Purpose/Evidence instead. Template-polish only.
  • N2: Guard placement rejects heads == 0 even with explicit head_dim — intentional per the test; noted so future readers do not "relax" it accidentally.
  • N3: The body discloses one unclassified full-workspace JEV-VL timeout that passed on isolated retry, and one docs-environment misstep, with original failures retained. Appropriate disclosure for a +21-line guard.

Evidence separation

  • Source inspected: focused diff (both hunks, full text); adapter.rs at head (guard, call sites, sole division, require_modernbert, provenance flow); tests/modernbert/structure.rs at head (helpers at lines 13/28); PR body; PR template at canonical main.
  • Commands personally run: ref/SHA verification; merge-bases (mirror and canonical); git diff/git show above; worktree /tmp/pr-review-pass194/wt207 @ a11335ae; PYTHONPATH=src /Users/hsliu/miniconda3/bin/python -m pytest tests/vela -q → 157 passed (9.86 s); PYTHONPATH=. ... pytest tests/benchmarks -q → 133 passed (28.4 s). These suites are untouched by this diff; runs recorded as head-state sanity only. mkdocs build --strict not run — no docs touched (correct per instructions). No cargo/rustc on this host, so no Rust check was runnable here.
  • Observed CI: none. gh pr checks 207 --repo ThinkFlowLab/system1-omni → "no checks reported on the branch". Actions API head_sha=a11335aee... → total_count: 0 in both ThinkFlowLab/system1-omni and InferMatrixLab/system1-omni. There is no CI signal at this head.
  • Author-reported (not verified here): all cargo fmt/clippy/test/build results (Rust 1.100.0, --locked --offline, L20 host), before/after regression outputs, focused-component counts (132/472 passed), Vela Python/comparison-tool, CUDA text-contract and strict MkDocs results.

Material unverified checks

  • The pinned Rust regression and workspace checks at the head (no Rust toolchain on the review host).
  • Everything inherited from unmerged #152 (74-file ModernBERT implementation): explicitly outside this supplement's diff, still requires its own large-change and hardware review after it lands.
  • No GPU/forward-pass claims are made by this PR; none were needed.

Cross-PR and integration

  • No file overlap with #208/#210 (different model crate). gh pr view reports MERGEABLE into its base, but the base itself is an unmerged stack (#152 OPEN) — this PR cannot integrate into main until #152 lands, then must be retargeted and rechecked against current main.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants