Skip to content

test(vela): clarify #169 span coordinates and contract acceptance - #206

Draft
psv666 wants to merge 2 commits into
mainfrom
codex/l20-b-vela-contract
Draft

psv666 wants to merge 2 commits into
mainfrom
codex/l20-b-vela-contract

Conversation

@psv666

@psv666 psv666 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Required human contributor answers

The contributor confirmed this single stage on 2026-11-19: Vela #169 contract audit and acceptance wording. #137 and the native PRs provide background/evidence; implementation of the native stack and other models is outside this draft's scope. Detailed human confirmation is retained in the contributor's task record.

  1. Outcome: A consistent Vela contract and reproducible, reviewable acceptance record.
  2. Highest-leverage variable: Reliable model behavior and numerical/contract consistency.
  3. How this moves it: Reuse the existing reference, resolve its wording/evidence gaps, and add focused documentation and acceptance cases.
  4. User and need: Model, application and agent developers who need stable interfaces, reliable outputs and reproducible deployment.

Is a PR needed? Yes. The contributor authorized necessary contract documentation/tests and a stage draft PR.

Observed gaps: #169 records the conflict between #137's old span/set rejection/no-truncation proposal and the shipped reference. The recipe suggested token usage as a tail-coverage check, which the actual assembly/window policy does not guarantee. Source inspection and a preparation-only repro also show distinct span coordinate origins and an unresolved cross-part-list failure. This PR changes no inference or transport behavior.

Summary

Document Vela's existing five-kind policy, clarify span coordinate origins and pending decisions, and bind a real non-BMP CPU oracle to maintained acceptance tests.

over: "request"    -> field-local code-point offsets
over: ["request"]  -> rendered-part offsets when fields share a part
over: "user"       -> rendered-part offsets
cross-part list   -> unresolved upstream preparation failure

Preserve span/set extensions, upstream state truncation, model-defined confidence and actual rendered-row token accounting. The shared RFC #61 remains open. Related: #169, #137, merged #151/#174. This does not mark native/family acceptance complete.

Evidence

  • The real five-kind CJK/non-BMP oracle retains EMAIL_ADDRESS [10,30) after an emoji; slicing the named request field gives zhang.wei@example.cn. Independent set probabilities sum to more than1.3. The actual partial PERSON prediction is preserved without a task-accuracy claim.
  • Opt-in tests exercise the hash-pinned upstream code: validation/precedence, field versus part coordinate origins, actual8192-token assembly/truncation and schema-only overflow. Controlled outputs/token IDs are explicitly distinguished from model-forward evidence.
  • Cross-part reproduction: state has request and source; a span uses over: ["request","source"]. Mapping resolves target_part=["user","context"]; upstream _to_generic -> _prep raises TypeError: unhashable type: 'list' before a model forward. This is recorded as unresolved scope/error behavior, not blessed as a client-error contract.

Personally executed on L20 Linux, baseline f1e674f2d324f5bb4e1ba2162551d0b633f20be0, final candidate bfc766c103ccf529b3f277cba38550c42f04ae2e:

Check Result
Original main ordinary Vela suite 135 passed
Final candidate with pinned-package opt-in 161 passed,0 failed
Hash-pinned upstream contract cases 14 cases included in the161
Original reference-vs-worker comparison All9 original cases exact in status/body/key order
Extra Unicode reference comparison HTTP200 exact;6 ordered answer entries; usage155/0
Actual worker/current-main Rust frontend First Unicode200, invalid span422, invalid set422, state-error precedence422, schema-only overflow413:5/5 byte-equal status/content-type/body
Strict MkDocs, Python compileall, whitespace and model-README links PASS

The real-model/HTTP runs preceded the final documentation/controlled-coordinate follow-up; their inference/transport source is unchanged. With no package env, the14 upstream checks skip explicitly; ordinary tests need no weights/torch/GPU and perform no download.

case: example-request-unicode.json
both paths: HTTP 200
answers: 6 key(s) in pinned order
input_tokens: 155 == reference
exact match (status, body, key order): PASS

first-unicode PASS 200
invalid-span PASS 422
invalid-set PASS 422
error-precedence PASS 422
schema-only-overflow PASS 413

161 passed in 10.35s

Purpose

Provide #169's Vela-specific current-state/acceptance record, correct the tail-coverage advice and relative recipe links, and name the remaining decisions without changing external behavior.

Pending decisions

  • Formal [RFC]: Write down which parts of the /v1/systemone contract all workers share #61 adoption of model-defined extension fields, confidence/usage/truncation and reconciliation of [New Model]: Support the Vela 2.0 series #137's old wording. Discussion favors following the model reference, but is not a completed shared-contract decision.
  • Whether the accepted span contract permits part/list coordinates, requires a single named field, or needs explicit origin metadata. The current response correlates target through the request; no new response field is introduced.
  • Supported behavior and client-error status for cross-part span lists; the current upstream preparation failure remains unfixed.
  • Shared error-envelope policy versus Vela's reference detail list and documented transport differences.
  • Native member/oracle/precision, router/broad execution and structural parity remain independently assigned acceptance work.

Test Plan

  • Pinned reference/runtime/tokenizer/calibration: vllm-sr/Vela-2.0-0.3B@d6f03aa9baca4f9017fa8f8e9c2764d3bed3f409; repository package gate plus full model LFS SHA256 bd37dd0db177d766bc554eded78ae8493bc248172d0480f6a543b30932a0cc8e.
  • CPU oracle: Python3.12.13, official torch2.11.0+cpu, transformers5.19.0, tokenizers0.23.2, numpy2.5.3, safetensors0.8.0; OMP/MKL8 threads, no FLA. No GPU execution is needed for this contract stage.
  • Actual HTTP: one warmed Python CPU worker and current-main Rust frontend, serial loopback requests; Rust1.100.0 binary SHA256 a56fc7bebdd3bfbb4edbbb80083c6a0421a7d51ab232533e869a14348f02bd0f.
  • Recipe documents all reproduction commands. Original four oracle files remain unchanged. Only the small consumed Unicode request/oracle is added.
  • Original raw archive vela03-evidence.tar.gz, SHA256 bcdaf65b82cfc10e9fd0ca8e91cae4e03b9ff077c960e8b5c1914e97ad12b1a2, is retained on contributor-local and L20 storage, with final coordinate repro/test/docs output retained separately. Raw machine logs remain outside product Git.

Test Result

Scoped CPU/reference/HTTP/documentation checks pass. The earlier Windows ordinary suite had128 pass/7 failures from Linux-extension assumptions, symlink privileges and default CJK decoding; the original suite passed on Linux. New fixture consumers use explicit UTF8; unrelated platform tests are unchanged.

No external behavior, calibration or numerical gate is changed. Formal contract adoption, issue closure and native acceptance are not claimed.

Demo / evidence

The committed oracle is an actual CPU prediction captured by the documented exporter. Real direct/frontend comparisons preserve complete response bodies with no timing-field exemption. Video and performance A/B are inapplicable to this docs/contract-test change.

Merge Danger

Door: two-way.

Blast Radius: contract.

230 added Python test lines; total+616/-10, including a215-line consumed oracle. Tests import only a package accepted by the existing hash gate. The docs describe current behavior and pending policy; they do not standardize other workers.

Self-review

Agent precheck reviewed the six-file diff, owning upstream functions, production response gate, fixture provenance/consumers, ordinary-test independence and docs links. All test bodies/fixtures are under root tests. The contributor checklist remains for human review; keep draft.

  • I have reviewed the full diff and addressed the issues I found.
  • I have checked that the change follows the project's architecture and stays focused on the stated purpose.
  • I have run the checks appropriate to this change and reported commands, results, and anything that I could not verify above.
  • I have checked that the PR description, documentation, and any accuracy or performance claims match the implementation and available evidence.

@psv666 psv666 changed the title test(vela): record span/set policy and pin Unicode contract acceptance test(vela): record #169 contract decisions and Unicode acceptance Nov 19, 2026
@psv666 psv666 changed the title test(vela): record #169 contract decisions and Unicode acceptance test(vela): clarify #169 span coordinates and contract acceptance Nov 19, 2026

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DRAFT review — PR #206 "test(vela): clarify #169 span coordinates and contract acceptance"

Verdict: COMMENTED. The PR is in draft state (draft=true, author psv666, first-time author). Per policy, an explicitly draft PR receives a COMMENTED review regardless of findings; this review does not approve, request changes, or alter the draft state, and does not fill the contributor's self-review checkboxes.

Template and draft state: All template sections present (Purpose / Test Plan / Test Result / Demo-evidence / Merge Danger / Self-review). All four self-review checkboxes are unchecked, consistent with draft ("keep draft" stated in the body).

Pinned metadata (observed via GitHub API and local git)

  • Base: main @ f1e674f2d324 ("Add a Valen-Decision-1.5 reference worker (#204)") — verified an ancestor of current main (614b0abc, #214). Main has advanced since creation, but the PR reports mergeable=MERGEABLE / CLEAN.
  • Head: codex/l20-b-vela-contract @ bfc766c103cc; head repo InferMatrixLab (PRs live in the InferMatrixLab parent). Two commits, both authored psv666 <2693925048@qq.com>.
  • Files: 6, +616/−10 — recipe/vela/example-request-unicode.json (new, +46), tests/vela/fixtures/oracle-example-request-unicode.json (new, +215), tests/vela/test_oracle_fixtures.py (new, +76), tests/vela/test_runtime_contract.py (new, +154), recipe/vela/reference.md (+110/−3), src/models/vela/README.md (+15/−7).

Body gate (post-#138 required answers)

All four mandatory answers present and specific, "Is a PR needed? Yes." present, plus an "Observed gaps" paragraph in the bug-fix-evidence slot (records the #169 conflict and the incorrect tail-coverage recipe advice; no runtime defect claimed). Gate status: present and complete.

Diff review — new tests vs docs, and what they pin about span coordinates

Both. +616 lines split between new tests (230 Python test lines) and documentation, with one consumed 215-line frozen oracle fixture.

New tests (all weight-free; verified to run locally):

  • tests/vela/test_oracle_fixtures.py (ordinary, run everywhere): binds the new Unicode oracle to the maintained recipe request (key order preserved) and to the production envelope gate (finish_response from src/models/vela/postprocess.py — verified present at main). What it pins about coordinates, at test_oracle_fixtures.py:45–63: EMAIL_ADDRESS span [10, 30) with text zhang.wei@example.cn positioned after a non-BMP emoji, and for every span that slicing the named field state["request"][start:end] == span["text"], with explicit negative checks that offsets are neither UTF-8 bytes (len(field[:start].encode("utf-8")) != start) nor UTF-16 code units (len(field[:start].encode("utf-16-le")) // 2 != start) — exactly the discriminator #169's acceptance-fixture checklist demands ("Include CJK and non-BMP characters so byte/UTF-16 offsets cannot pass"). It also pins set independence (sum(probabilities) > 1.3, impossible after cross-label softmax), preserved per-label noul views, threshold echo, and single-sequence usage 155/0 — matching #169's items on set semantics, Noul views, and single-rendered-sequence token counting.
  • tests/vela/test_runtime_contract.py (opt-in; skips explicitly without VELA_CONTRACT_PACKAGE, test_runtime_contract.py:25 — observed 8+3+1+1+1 = 14 skips locally): 8 parametrized upstream validation/error-precedence cases with loc path assertions; the coordinate-origin test (test_runtime_contract.py:80) pinning the claimed triad — over: "request" → field-local code-point offsets, over: ["request"] and over: "user" → rendered-part offsets including field headers, with a controlled predict stub asserting the upstream span_limits computation; and the actual 8,192-token assembly boundary, unprotected-before-protected cut accounting, and schema-only overflow as a distinct error. These execute the hash-pinned upstream runtime (no model forward), with controlled outputs explicitly distinguished from model-forward evidence.

Docs:

  • recipe/vela/reference.md:283+ new "Contract decisions and acceptance" section: a Vela-policy table (five-kind forwarding incl. span/set; span identity/offsets; independent set selection; length limits; confidence; rendered-row usage; validation), four separate acceptance gates, the opt-in command, fixture provenance and regeneration command, and a "Pending contract decisions" list (reference.md:365+). The −3 lines replace incorrect advice: the old "check usage.input_tokens against the expected budget" tail-coverage suggestion becomes "usage.input_tokens is work accounting and cannot prove that the complete state was examined" (reference.md:401–402) — a correction, not a gate change; the assembly policy indeed cannot guarantee tail coverage.
  • src/models/vela/README.md:108–113 mirrors the coordinate-origin clarification and the unresolved cross-part case; also fixes genuinely broken relative links (../../ → ../../../ from src/models/vela/ — the old targets do not exist) and adds the recipe anchor link (README.md:211).

Accuracy of #169 references against actual merged code at main

  • #169 is an OPEN issue ("[Contract]: Reconcile Vela span/set and truncation semantics before native parity"), not a merged PR. Its acceptance-fixture checklist (CJK+non-BMP span offsets, set independence, Noul views, invalid span/set + precedence + schema-vs-state + truncation boundary, pinned 0.3B CPU FP32 oracle) is delivered item-by-item by the new tests/fixture; the PR correctly does not claim #61 adoption or #169 resolution and keeps cross-part behavior recorded as unresolved rather than blessed as a client-error contract (reference.md:365+, body).
  • The merged carriers it cites are verified merged: #151 (0.3B reference worker, merged 2026-10-25) and #174 (0.8B hybrid, merged 2026-10-27); #137 is an open issue; #61 open with the linked comment. Accurate.
  • The field-local claim is consistent with the pre-existing README wording at main ("Unicode code-point offsets into the targeted field's text (clipped and re-trimmed to the field when it shares a part)"). The part-target nuance (["request"] ≠ "request" when fields share a part) is new documentation; the rendering/span-limit logic lives in the hash-pinned upstream package (vela2_inference.py pinned in src/models/vela/engine.py:46, sha256 d3e140c7…), not in-repo, so it is pinned by the opt-in tests only (see P3 below). The repo-side plumbing the tests rely on (verify_package, verify_no_foreign_runtime, put_first_on_sys_path in engine.py:200/218/233) is verified present at main. The cross-part _to_generic → _prep TypeError is body-reported repro only, with no committed test — deliberately, since the PR refuses to codify it as a contract (disclosed).

Gate preservation (extends, does not weaken)

Additive only: no existing test or fixture modified (the original four oracle files untouched per diffstat), no tolerance or gate touched, and the one semantic doc edit removes a false assurance rather than loosening a requirement. The 14 new skips are explicit, package-gated opt-ins matching the repository's existing opt-in conventions (same pattern as --ignored GPU tests), and are disclosed in the body with the exact count. Extends the approved-family surface (#195's native acceptance is explicitly left separate).

Findings

  • P3 — the span-coordinate origin triad is pinned only by opt-in tests that ordinary CI never executes. Trigger: test_runtime_contract.py requires VELA_CONTRACT_PACKAGE; locally and in CI without that env the 14 checks skip (observed). Consequence: the reference.md/README statements about ["request"] vs "request" offsets rest on author-reported opt-in runs (body: 14 cases included in 161 passed on L20), not on any gate that runs unattended. Disclosed, but before this becomes the documented contract, at least one recorded run at the eventual merge head should be linked, or the package env wired into an opt-in CI lane.
  • P3 — cross-part span-list failure has no committed regression. Trigger: over: ["request","source"] upstream TypeError is described in the body and reference.md:365+ but has no test. Consequence: the eventual contract decision (error status? rejection?) will lack a pinned repro; acceptable for now because the PR deliberately does not bless it, but track it under the pending-decision list.
  • Notes (informational):
    • Main advanced from f1e674f2 to 614b0abc (workspace version bumps); no conflict observed for these files, but a rebase/retarget is normal hygiene before ready.
    • "span_heads" not in response (test_oracle_fixtures.py:59) covers the 0.3B's single-span-head case; hybrid router/broad identity remains with the pre-existing hybrid fixtures — coverage note, not a gap introduced here.
    • The committed oracle retains a partial PERSON prediction and the body explicitly disclaims task-accuracy meaning — correct framing.

Provenance (separated)

  • Source inspected: PR body/metadata via API; full diff f1e674f2d324..bfc766c103cc; #169 issue text; #151/#174/#137/#61 states; src/models/vela/{engine,postprocess,preprocess}.py at main; README/reference.md at main and head; CONTRIBUTING.md.
  • Commands run (this review, macOS, no GPU; Python 3.12.8 / pytest 8.4.1 / mkdocs 1.6.1): worktree at bfc766c103cc (removed after): PYTHONPATH=src python -m pytest tests/vela -q → 147 passed, 14 skipped (all 14 verified to be the disclosed VELA_CONTRACT_PACKAGE skips; consistent with the body's 135 original + 12 new ordinary = 147, and 161 = 147+14 with package); PYTHONPATH=. python -m pytest tests/benchmarks -q → 133 passed; mkdocs build --strict → exit 0, 0 warnings.
  • Observed CI (head bfc766c103cc, verified per-SHA in BOTH repos): ThinkFlowLab Actions: CI (ci.yml) success + Docs (docs.yml) success, event=pull_request, 2 runs; InferMatrixLab: same 2 runs success; gh pr checks: rust, benchmarks, build, cuda-compile, cuda-contract, kev-export pass, deploy skipping. This is the only PR of the three with current-head CI evidence.
  • Author-reported (not verified here): the L20 opt-in package run (161 passed), the reference-vs-worker HTTP comparisons, the byte-equality frontend loop, the cross-part TypeError repro, fixture capture provenance.

Bottom line

A well-scoped contract-documentation and fixture PR that accurately records #169's current state, adds 12 ordinary + 14 opt-in tests without touching existing gates, and corrects incorrect tail-coverage advice. COMMENTED. Before ready: link a recorded opt-in contract run at the merge head (P3), and keep the cross-part decision tracked (P3).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants