Skip to content

omnijev: add opt-in BF16 reference vision arithmetic - #209

Draft
psv666 wants to merge 1 commit into
InferMatrixLab:mainfrom
psv666:codex/omnijev114-vision-20261120
Draft

psv666 wants to merge 1 commit into
InferMatrixLab:mainfrom
psv666:codex/omnijev114-vision-20261120

Conversation

@psv666

@psv666 psv666 commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Purpose

OmniJev's legacy BF16 vision arithmetic drifts from the pinned official tower through normalization, attention and GEMM reduction boundaries. Add an explicit OMNIJEV_VISION_NUMERICS=reference path that matches all 40 observed visual boundaries on the original protected PNG case, while retaining legacy as the default. This is the bounded vision stage of #114; the original full 280-answer precision gate still fails, so this PR stays draft.

Required human contributor answers

The contributor confirmed these answers and the draft-PR decision during preparation; the public scope is RFC #114. Private preparation records are retained with the contributor.

  1. Outcome: Advance reliable native model support to a reproducible state with real-weight validation and complete numerical evidence.
  2. Highest-leverage variable: Trustworthy support coverage and numerical consistency, before performance optimization.
  3. How this moves it: Reuse existing implementations, resolve reference/contract/numerical gaps, and make the smallest evidence-backed change rather than add another worker or unsupported model claims.
  4. User and actual need: Model/application/agent developers using System1-Omni need stable interfaces, reliable output and reproducible deployment on the target hardware.

Is a PR needed? Yes: the contributor authorized necessary implementation, L20 validation, self-review and a reviewable draft PR for this stage.

Observed failure: Original video.png/noul_region, unchanged question/region: official unmerged BF16 0.5443, legacy native 0.4997, drift 0.0446 and a protected true→false decision. The strict gate actually exits 1. Raw before/after results and failing intermediate stage regression are linked below.

Summary

OmniJev Executor (legacy default)
  reference opt-in -> shared VisionModel, unadapted 4B only
    cuDNN BF16 patch -> Welford norm -> framework GEMMs (1 MiB)
    FP32 rotary angles -> head-64 reference attention -> BF16 merger
  existing merged language/readout -> existing heads/calibration

Reuse the existing reference kernels/GEMM handle; the 1 MiB heuristic budget matches the actual Torch addmm boundary. A 32 MiB budget selects a different split-K in the merger: the intermediate exact-stage regression fails there, and the single budget control makes it exact. No change to gemm.cu, language/LoRA execution, weights, calibration, corpus or gates.

Add separately resolved optional ABI-9 symbols, retaining old-library loading and JEMM's 1152/head-72 guards. The original patch constructor still fixes 1152. Reject adapted/incorrect layout, bounded-cache accounting and late profile selection. Keep the existing trace events; add opt-in selected-block operator tracing and root tests/oracle generator. This builds on the native worker and reuse implementation from merged #129/#134, preserving their author attribution.

Test Plan

Base / head: f1e674f2d324f5bb4e1ba2162551d0b633f20be0 / 6410e7634fd5e077000810ae07dc2f251b4f044d. 17 files; 730 changed authored-code lines, 859 total. No generated run dumps or matrices committed.

Pinned artifacts: official code 14dbec4f71e194852c8d7b88ab36ef639493f400; checkpoint ffe5f436eaf22e20e2f041f8e74e121fd057a6cb; Qwen3.5-4B 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a; export manifest SHA256 2bab1a7d6ac96859fa07af932f1bafb448acd4abe21a36d9b204e5e2a56c6dfe. Original BF16 unmerged reference, 248079 vocabulary, 602112 pixels, calibration and all original 28 requests/280 answers remain fixed.

Hardware/runtime: one L20 GPU0 (sm_89), driver 580.178.04; Rust 1.100, nvcc 12.8, native cuBLASLt 120803 and cuDNN 92400. Actual native maps also load cuDNN's bundled cuBLASLt13. Official Python 3.10.12, Torch 2.14.0+cu130, torchvision 0.29.0+cu130, Transformers 5.17.0, PEFT 0.19.1, Pillow 12.3.0, no FLA. Full lock/library hashes are included. These stacks are not claimed library-identical. Cooperative GPU lock and occupancy checks were used; no performance campaign.

Test Result

  • Exact-head format, workspace Clippy (--locked --all-targets --features omni-laya/serve -- -D warnings), ordinary workspace tests, workspace release build, test discovery and strict MkDocs pass. Ordinary tests: 427 passed, 122 ignored; skipped device tests are not counted as executed passes. CUDA backend/checker tests: 27 + 68 pass; the existing Laya ABI-declaration warning remains.
  • Named GPU checks all pass: five Torch/operator/legacy compatibility tests, five existing head-72 vision tests, fixed-GEMM row invariance, exact real-weight stage/Graph regression, and two OmniJev worker order/reuse tests. The stage test includes actual capture/replay, changed pixels at the same grid, resident readback and error recovery. Existing 4B/JEMM operator bits match the pre-change library. The legacy minimum preserves every observed value/visual byte.
  • Six first-ready/direct/frontend parity and malformed-input recovery checks pass. The protected minimum now returns 0.5525 against official 0.5443 (drift 0.0082); all 40 visual boundaries, including all 24 blocks and merger, are bitwise exact on this case.
  • Original full corpus is freshly replayed with final native eager/reuse and FP32 head capture. All 28 eager/reuse responses agree apart from timing; image hashes, questions, fields and input usage match. Official full captures are reused read-only from the completed documented-stack prerequisite, not claimed freshly re-executed here.
Population Answers Within 0.01 Max returned drift Protected flips
demo PNG 84 68 0.0355 0
gate PNG 56 52 0.0151 0
demo JPEG 84 29 0.1758 1
gate JPEG 56 28 0.1456 2
Total 280 177 0.1758 3

Unrounded: 177/280 within 0.01, max 0.17586451768875122, same three protected failures; all raw results round back to production. The original strict full gate exits 1. The previous documented baseline (113/280 within 0.01, max 0.3066, four protected flips) remains separately retained; the JPEG failure set changes and does not uniformly improve.

Remaining protected cases: demo-jpg/board.jpg/q8 margin 0.0397 (check first→safe); gate-jpg/arcade.jpg/long_start margin 0.0935 (third→second); gate-jpg/web.jpg/one_token margin 0.0746 (click OK→click Close). All 140 PNG decisions agree, but 20 PNG answers still exceed 0.01. In the original 12-question demo-png/video.png/q2 Score request, confidence remains 0.5613460540771484 versus 0.5786595344543457 (raw drift 0.017313480377197266) with the same harmless decision. Its full context/raw/control pointers are preserved for the next stage. Remaining errors are not attributed wholesale to JPEG or LoRA; JPEG input factors and language/selected-readout/merged-vs-unmerged residuals need separate controls.

Demo / evidence

Numerical evidence archive, 485264 bytes, 197 inventoried files. SHA256 85e39a9d95762dec3cc34447ffb40d2e12562a0ae8d416810138ecc24fa786da. Contains the protocol, all original gate rows/raw numeric results, failures, source helpers, dependency lock, actual runtime/library hashes and reproduction commands. Screenshot bytes, embedded image requests, weights, credentials and private preparation discussions are omitted; complete replay inputs/BF16 tensors remain in the contributor's private archive. Maintained test commands and profile scope are documented in the recipe and root tests README.

Video: N/A, numerical backend change. Performance/startup/memory improvements: unmeasured. Other GPUs/Metal, full 27B/Cua checkpoints and whole unmerged-language parity: unverified. No ready/merge request or issue-closing claim.

Merge Danger

Door: two-way. Revert the change or select legacy; exports and request/response schema are unchanged.

Blast Radius: Vision. New arithmetic is selected explicitly for unadapted OmniJev4B; shared optional bindings and the patch-constructor wrapper have device compatibility coverage. This does not qualify real 27B models or enable the new profile by default. Whole #114 acceptance remains blocked by the disclosed original-gate failures.

Self-review

Agent self-review/precheck completed against the full pinned diff; contributor/maintainer review remains separate. Kept draft. Human checklist is intentionally left for the contributor:

  • I have reviewed the full diff and addressed the issues I found.
  • I have checked architecture and scope.
  • I have checked the reported validation and remaining gaps.
  • I have checked the description, documentation and numerical claims.

Align patch, LayerNorm, rotary, head-64 attention and vision GEMM boundaries with the pinned BF16 reference. Keep legacy/JEMM behavior and optional ABI compatibility. The original 280-answer whole-model precision gate still fails; this is the bounded vision stage for InferMatrixLab#114.

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review draft — #209 "omnijev: add opt-in BF16 reference vision arithmetic"

Verdict: COMMENTED. All four PRs in this series are OPEN and explicitly DRAFT; this is a first review of a draft, not an approval and not a completed validation. The human self-review checklist in the body is intentionally left unchecked by the contributor and must stay untouched by reviewers.

Provenance and pins

  • Repository: ThinkFlowLab/system1-omni resolves via rename redirect to InferMatrixLab/system1-omni (same repo, API id 1386454853). CI run URLs therefore appear under the InferMatrixLab name; both API hostnames return identical runs.
  • Base (GitHub baseRefOid and body pin, verified identical): f1e674f2d324f5bb4e1ba2162551d0b633f20be0 ("Add a Valen-Decision-1.5 reference worker (#204)").
  • Head: 6410e7634fd5e077000810ae07dc2f251b4f044d (matches origin/pr/209 and GitHub headRefOid). Author psv666 (new author in this series), head repo psv666/system1-omni, branch codex/omnijev114-vision-20261120.
  • Diff: 17 files, +841/−18 (GitHub and local git diff --stat agree). Body's "730 changed authored-code lines, 859 total" reproduces exactly (code-file numstat +712/−18 = 730; total 841+18 = 859).
  • Local checkout used for skill/contract loading: branch skill-add-new-model-evidence-pointer @ ca5f69b5c104ff2ed0c64ed2f195b49bec92e32c (note: this is ahead of the PR base; see contract-provenance note below).

Position in the series (cross-reference)

Verified by git merge-base --is-ancestor: 209 ⊂ 211 ⊂ 212 is a clean linear stack (one commit each, first-parent chains verified). #215 is NOT a superset: git merge-base origin/pr/212 origin/pr/215 is the shared base f1e674f2d, and 209/211/212 are not ancestors of 215. The series is actually two families: the OmniJev stack #209→#211→#212, and a separate Vela carrier PR #215 (own base branch codex/vela152-coexist-base, see the #215 review). Review order applied: #209 → #211 → #212 → #215.

Template gates

  • Four required contributor answers: present and explicit (#209 carries them; #211/#212/#215 restate and link back). Necessity answer ("Is a PR needed?"): present, "Yes", with scope of authorization. Draft decision: explicit ("this PR stays draft").
  • Self-review checklist: present, all four boxes unchecked — correct for a draft; not filled by this review.
  • Evidence pins: base/head SHAs correct (verified against GitHub metadata and local git). Pinned artifact hashes (official code 14dbec4f7…, checkpoint ffe5f436e…, base 851bf6e8…, export manifest SHA256) are author-reported; the manifest/vocabulary/geometry pins match the constants in code (src/models/omnijev/native/src/export.rs: CHECKPOINT_REVISION, BASE_REVISION).

Findings

No P0/P1/P2 findings. The contract questions resolve cleanly:

  • Opt-in mechanism: environment variable OMNIJEV_VISION_NUMERICS parsed in Executor::load (src/models/omnijev/native/src/executor.rs:120-131 at head); NotPresent → legacy; unknown values fail startup (tested in tests/omnijev/window_graph.rs::vision_profile_is_explicit_and_rejects_unknown_settings, including "Reference", "reference " variants). Library callers use Executor::load_with_vision_numerics. Default is unchanged legacy behavior; opt-out default preserved.
  • Opt-in is device-guarded: VisionModel::enable_omnijev_numerics (src/models/qwen3_5/native/src/vision/model.rs:231-263) requires unadapted 4B (hidden 1024, no LoRA), selection before first forward, rejects bounded-cache accounting, requires the cuDNN-9 capability probe, and is idempotent. All four guards are exercised in tests/qwen3_5/vision_omnijev.rs (real-weight, #[ignore]-gated).
  • BF16 rounding locations: BF16 outputs at every operator boundary retained; FP32 rotary angles (cs1_omnijev_vision_rope, unused si documented in src/backends/cuda/qwen3_5/ops.h:272); Welford LayerNorm kernel; framework-style GEMMs via a dedicated cuBLASLt handle with Torch's 1 MiB heuristic budget (the body documents that 32 MiB changes merger split-K and fails the exact-stage gate — a measured, not assumed, rationale); head-64 FlashAttention with FP32 accumulation → BF16 output (src/backends/cuda/qwen3_5/reference_vision_attention.cu:198-214, scale 0.125f, rows ≤ 4608, heads == 16, dh == 64 guards).
  • Parity evidence is measured-then-declared: kernel-level tests assert bitwise torch.equal against pinned-Torch SDPA/layer_norm (tests/qwen3_5/test_vision_omnijev.py, pinned Torch 2.14.0 enforced); stage-level test compares BF16 bytes against a frozen oracle (tests/qwen3_5/vision_omnijev.rs); legacy-operator byte preservation is tested against an immutable pre-change library. The declared corpus result (177/280 within 0.01, max drift 0.1758, 3 protected flips, strict gate exits 1) is repeated consistently in body, recipe/omnijev/native.md and src/models/omnijev/README.md, and the PR honestly keeps the 280-answer gate failed and stays draft.
  • ABI: purely additive at ABI 9 — five new optional symbols (cs1_reference_patch_create_v2, cs1_omnijev_vision_norm/rope/attention_workspace_floats/attention, src/backends/cuda/qwen3_5/ops.h:267-276). The original 27B constructor is preserved as a wrapper calling v2 with hidden 1152 (reference_patch.cu:63-66); old libraries remain loadable (optional resolve, explicit rebuild-instruction error via Api::omnijev_vision()). The V-pointer offset qkv.at(h*2*2) is correct because DeviceBuffer::at is byte-offsetting (src/models/qwen3_5/native/src/cuda.rs:767-770), matching the kernel's compact-Q/K / V-in-QKV layout (also verified bitwise from Python at rows 4..4608).
  • Informational (no action required): cudaFuncSetAttribute's return in cs1_omnijev_vision_attention is unchecked (reference_vision_attention.cu:203-204); a failure still surfaces through the subsequent cudaGetLastError on launch, so the error path is covered. The 1 MiB budget and parity claims are explicitly scoped to the pinned stack (Torch 2.14.0/cuDNN 9.24/sm_89) — the docs state this.

Evidence provenance

  • Source inspected: full 17-file diff at f1e674f2d..6410e7634; ops.h/cuda.rs/executor.rs/vision model.rs/reference*.cu in depth; ops.h cross-checked against the Rust extern signatures.
  • Commands run (personally): ref/ancestry verification; git diff --stat/--numstat per PR; tree/SHA pin checks; numstat authored-line reproduction; binary/non-source file scan (none committed — no run dumps, no fixtures, matching the body claim).
  • Observed CI (Actions API, both repo hostnames, exact head_sha 6410e7634…): CI (rust, benchmarks, kev-export, cuda-contract, cuda-compile jobs per .github/workflows/ci.yml) and Docs — both success, run 44674667802 / 44674667780. Fork repo (psv666/system1-omni) has 0 runs for this SHA. These are CPU-level checks; no GPU model test ran.
  • Author-reported (not independently verifiable here): all L20 GPU measurements (40-boundary bitwise match, 280-answer table, protected-case margins), the evidence archive (release asset omnijev114-vision-public-evidence.tgz, SHA256 85e39a9d…, 197 files — not downloaded/verified), CUDA library/operator hash pins.

Unverified checks / evidence gaps

  • No local GPU and no cargo in this environment: the weight-bearing Rust suites (vision_omnijev.rs, vision_omnijev_reference.py) and the 280-answer replay are author-reported only. Missing hardware is an evidence gap, not a PASS.
  • The declared corpus gate fails (177/280) by design and the PR says so; acceptance of #114 remains blocked. No merging decision is requested or implied here.

Skill load record

system1-omni-review SKILL.md loaded from /Users/hsliu/thinkflowlab/system1-omni/.agents/skills/system1-omni-review/SKILL.md, SHA-256 58be3bc60806c4c5af7038a0c6a80fadb94f3360d714556a3e965c8903dde108, last touched by commit ef927963, working tree matches HEAD. Also read: references/repository-contracts.md (SHA-256 d6ae9650…), self-review/SKILL.md (SHA-256 a795b008…), docs/architecture.md and CONTRIBUTING.md sections at the refs named above. Provenance note: the "Large code changes" CONTRIBUTING section exists at the local checkout (ca5f69b5) but not in fork/main:CONTRIBUTING.md; the 3,000-authored-line requirements live in .agents/skills/self-review/SKILL.md at canonical main. Not load-bearing for #209 (859 total lines).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants