From 75c2eab5685a2f6d40f80f1cfa08b792aeaa06fd Mon Sep 17 00:00:00 2001 From: zuiho-kai <31877877+zuiho-kai@users.noreply.github.com> Date: Sat, 21 Nov 2026 14:01:31 +0900 Subject: [PATCH 01/11] docs: inherit scheduling RSI lessons (Ascend queue order, agent KV eviction) Co-Authored-By: Claude Opus 5.5 --- docs/scheduling-rsi-lessons.md | 85 ++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 docs/scheduling-rsi-lessons.md diff --git a/docs/scheduling-rsi-lessons.md b/docs/scheduling-rsi-lessons.md new file mode 100644 index 00000000..567dd9a6 --- /dev/null +++ b/docs/scheduling-rsi-lessons.md @@ -0,0 +1,85 @@ +# Scheduling RSI lessons: Ascend queue ordering and agent KV eviction + +**An LLM-proposed waiting-queue order (prefix-cache hit share of the prompt, +highest first) raised simulated capacity at a 10 s mean TTFT bound from about +3.3 to 15.3 req/s (4.61×, repeats 4.2–4.8×) over the Ascend shortest-job-first +baseline.** This is a virtual-clock simulation result from the maintainers' +campaign ledger. It has no acceptance issue, was not run on NPU hardware and is +not an InferRSI PASS. + +This page records two earlier campaigns of the same RSI line (model proposes a +policy → a frozen scorer measures it → keep the better, drop the worse) so that +later campaigns inherit the measurement rules and the failure modes, not only +the headline. Apple-silicon lessons from later work are in the +[Apple silicon playbook](apple-silicon-playbook.md) and +[Inkling 2 Mini on Apple silicon](inkling2-mini-apple-silicon.md). + +## Sources and labels + +| Label | Meaning | +| --- | --- | +| **public** | In a public repository or dataset at the link given in the row. | +| **ledger** | From the maintainers' unpublished campaign report or run log; not reproducible from this page. | +| **inferred** | Analysis made during the work and not measured. | + +## Campaign 1: vLLM-Ascend waiting-queue order (simulation) + +Setup (ledger): real vLLM engine with only the device operators replaced by +stand-ins; a virtual clock calibrated from another team's operator timing +table, step time ≈ 1.313 ms + 0.00468 ms × tokens in the step, error ≤ 3%; +393 multi-turn sessions in which 98.7% of requests contain the whole previous +request; baseline prefix reuse about 14%. Eight candidates ran in parallel; +more saturated the CPU. Wall time of the simulator was 5–45× faster than the +device and was never used as the metric. + +| Observation | Cause | Action | Source | +| --- | --- | --- | --- | +| Ordering by *absolute* tokens saved, fewest first: 1.75×; most first: 2.47×; by *share* of the prompt already cached, highest first: 4.61×. | Share favours requests whose remaining prefill is small relative to what is already resident, so cache pages are reused before eviction. | Try the normalised form of a key as well as the absolute one. | ledger | +| 4.61× is the load at which the *mean* TTFT equals 10 s. Requiring every run to stay under the bound gives about 3.9×. | Mean-at-bound and every-run-under-bound are different operating points. | Publish the conservative point next to the headline and name which one is quoted. | ledger | +| At the same 10 s mean, median TTFT moved 1.5 → 5.7 s and the shortest quarter of prompts 1.8 → 9.5 s; the longest-to-mean ratio fell 1.80 → 1.28. | The policy equalises latency: long prompts gain, short cold prompts wait. | Report latency by prompt-length band, not only capacity. State the precondition: prefill-dominated load and no hard sub-second SLO for short requests. | ledger | +| Run-to-run noise was about 4%. Four successive "guaranteed floor" values were each inside the noise and were all withdrawn. | Single-seed differences below the noise are not results. | Below 5% difference: no conclusion. A "better" verdict also needs a mechanism signal (hit rate up, recompute down, queue shorter). | ledger | +| Split validation: sessions halved into an evolution half and an exam half gave 2.06× and 2.01× at the baseline's bound load. | Checks the policy is not fitted to the sessions it was found on. | Always hold out sessions. Note that this split used a larger cache pool than the 4.61× figure; the two numbers must not be combined. | ledger | +| Parameter-level search: −11.2% mean TTFT in 4 generations, then 30 generations without change; the last 80-generation round produced 1 new candidate and 79 duplicates. | The parameter plane was exhausted; the proposer kept regenerating known candidates. | De-duplicate before scoring. When a plane stops moving, switch planes rather than spend generations. | ledger | +| Structure-level search, first round of 5 candidates: only gate tuning helped (10.5%). Later: place by free slots 33.2%, add a starvation quota (1 slot in 8 for the longest waiter) 38.1%, reserve one quarter of slots 28.2%, order by waiting time only 23.6%. | Structural changes reach gains the parameter plane cannot. | Expect most first-round structural ideas to fail; keep the failures in the trial index. | ledger | +| The champion came from a human-written requirement route, not from the automatic mutation chain. | — | Attribute candidates to their actual origin; write "measured on the same scorer", not "found by the model", unless the chain produced it. | ledger | +| Most human effort went into making candidates run on both the NPU path and the simulator (interfaces, parameter adaptation, debugging). | Two execution targets. | Budget the adaptation work explicitly; it is the cost driver, not proposal generation. | ledger | +| At the highest load, recompute was only ten-odd percentage points from perfect reuse with ordering alone, and cold short requests remained slower. | The ordering plane is close to its ceiling. | Next plane: cache residency and admission. | ledger, inferred | + +Not established: NPU hardware confirmation of the simulated ranking; paired +per-session statistics instead of the 5% rule; a separate held-out audit split +for the champion. + +## Campaign 2: agent-aware KV eviction (CPU replay) + +Setup: CPU replay of public agent traces, policy proposed by a model, scored by +hit rate. Code was contributed to +[JiusiServe/vllm-evolve#1](https://github.com/JiusiServe/vllm-evolve/pull/1) +(public). Primary trace: Weka cc-traces with sub-agents +([HF dataset](https://huggingface.co/datasets/semianalysisai/cc-traces-weka-with-subagents-051926), public); +219 traces, about 78,000 requests. + +| Observation | Cause | Action | Source | +| --- | --- | --- | --- | +| Trace chains are idle 91% of the time; root agents idle p50 2.0 s / p90 35 s, sub-agents p50 0.54 s. | Tool calls and model turns alternate. | Expected return time is a usable eviction feature; sub-agent KV should stay resident before root-agent KV. | ledger, inferred | +| With 32 sessions and device memory at 10% of the active working set, LRU hit rate 51% against an upper bound of 83% (Belady 78%, prefetch-only 53%); at 25%: 88% against 97%. | The headroom comes from eviction order by expected return, not from prefetch. | Agent-aware eviction is worth searching only when memory is tight; measure the gap to Belady before searching. | ledger | +| After pinning the blocks of running requests, LRU and Belady differed by only 0.7 points (10%) and 3.3 points (25%); without pinning, by 26 points. | The unpinned replay credited evictions a real engine cannot make. | Model engine constraints (pinning, admission queueing) in the scorer before quoting headroom. | ledger | +| A 30-generation exploratory search cost 5.15 USD and moved mean hit rate 60.93% → 61.13%. | Small remaining headroom after pinning; the gain is inside noise. | Compute the reachable headroom first; do not spend search budget on a plane with < 1 point to gain. | ledger | +| On a real vLLM 0.20 engine with Qwen2.5-7B, the evolved eviction function loaded through `--scheduler-cls`, evicted by score, and produced 192/192 identical outputs under batch-invariant mode. | — | The hardware test proves the plug-in runs and keeps outputs; it does not score the policy. Gains come from the RSI scorer only. | ledger | +| Locally collected agent sessions were rejected as a dataset. | They have no external standing. | Use public traces already used by papers or benchmarks (Weka cc-traces, FreeInference, TraceLab). | ledger | + +## Campaign gates inherited from both + +These are the pre-run checks the maintainers now apply before any search +(ledger, from a closed research campaign that spent most of its time on its +harness): + +1. **Mechanism first.** Write the hypothesis as "variant X beats Y because part + P makes them differ", and name P in code. No P, no run. +2. **Power before running.** Baseline rate, samples per arm and the smallest + difference that would be visible against the measured run-to-run noise. +3. **Harness time box.** Harness and framework acceptance together at most one + working day or one quarter of the budget. +4. **Smallest discriminating run first.** Two arms, one seed. A tie sends the + work back to step 1, not to more seeds. +5. **Daily split.** Record how much of the day went to the hypothesis and how + much to the harness. From 4bfec49f43347ed382aa4b5e6641c64d64de5a8d Mon Sep 17 00:00:00 2001 From: zuiho-kai <31877877+zuiho-kai@users.noreply.github.com> Date: Sat, 21 Nov 2026 14:03:10 +0900 Subject: [PATCH 02/11] docs: add kernel-vs-e2e and frozen-rival gates Co-Authored-By: Claude Opus 5.5 --- docs/scheduling-rsi-lessons.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/scheduling-rsi-lessons.md b/docs/scheduling-rsi-lessons.md index 567dd9a6..d7bed180 100644 --- a/docs/scheduling-rsi-lessons.md +++ b/docs/scheduling-rsi-lessons.md @@ -83,3 +83,9 @@ harness): work back to step 1, not to more seeds. 5. **Daily split.** Record how much of the day went to the hypothesis and how much to the harness. +6. **Kernel gain is not end-to-end gain.** In a sibling InferMatrixLab engine + a kernel that was 13% faster in isolation moved end-to-end time by 0.4% + (public, `InferMatrixLab/system1-omni`). Score candidates on the full + generation path only. +7. **Freeze the rival.** Thresholds written before the run, rival + configuration frozen, inputs byte-identical, failed candidates kept. From 1cc0c08ed413faef6841a121b09c84cfca900e98 Mon Sep 17 00:00:00 2001 From: zuiho-kai <31877877+zuiho-kai@users.noreply.github.com> Date: Sat, 21 Nov 2026 14:33:50 +0900 Subject: [PATCH 03/11] docs: Inkling 2 Mini Apple silicon page skeleton (WIP) Co-Authored-By: Claude Opus 5.5 --- docs/inkling2-mini-apple-silicon.md | 48 +++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 docs/inkling2-mini-apple-silicon.md diff --git a/docs/inkling2-mini-apple-silicon.md b/docs/inkling2-mini-apple-silicon.md new file mode 100644 index 00000000..4a64419c --- /dev/null +++ b/docs/inkling2-mini-apple-silicon.md @@ -0,0 +1,48 @@ +# Inkling 2 Mini on Apple silicon: local adaptation and one RSI round + +**RESULTS_HEADLINE** + +[Inkling 2 Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) is a +26B-total, MoE, looped transformer from Thinking Machines Lab (24 layers; the +first and last 6 run once, the middle 12 run three times; about 3.4B active +parameters, 5.4B effective with the loop). The release checkpoint is ModelOpt +NVFP4 experts with MXFP8 tail layers (about 20 GB) and does not load in Apple +runtimes directly. This page records how it was brought up on one Mac with +community conversions, which runtime was faster, and one InferRSI-style search +round on that runtime. Measurement rules follow the +[scheduling RSI lessons](scheduling-rsi-lessons.md) and the +[Apple silicon playbook](apple-silicon-playbook.md). Nothing here is an +InferRSI acceptance result. + +## Machine and sources + +- Apple M5 Max, 128 GB unified memory, macOS 27.0 (26A425). +- Labels: **measured** = run on this machine, raw logs in the maintainers' + run directory (not published); **public** = model card or repository at the + link given. + +| Route | Weights | Runtime | Source | +| --- | --- | --- | --- | +| MLX | [mlx-community/Inkling-2-Mini-4bit](https://huggingface.co/mlx-community/Inkling-2-Mini-4bit) (17.3 GB) | mlx-lm from [ml-explore/mlx-lm#2086](https://github.com/ml-explore/mlx-lm/pull/2086) (`0c4c991`), mlx 0.32.4 | public | +| llama.cpp | [lmstudio-community/Inkling-2-Mini-GGUF](https://huggingface.co/lmstudio-community/Inkling-2-Mini-GGUF) Q4_K_M (17.5 GB) | llama.cpp main `418fe97`, Metal | public | + +Not run: Ollama (not installed on the machine), vLLM/SGLang (no Apple +backend for this model), the variable-loop research model +`TML-d24-varloop-research` (released only in the ModelOpt format, no MLX or +GGUF conversion at the time). + +## Bring-up + +BRINGUP_ROWS + +## Route comparison + +ROUTE_TABLE + +## RSI round + +RSI_SECTION + +## What we did not establish + +LIMITS From 7c1f7d793bab42d86a87e92fae2122406ec01212 Mon Sep 17 00:00:00 2001 From: zuiho-kai <31877877+zuiho-kai@users.noreply.github.com> Date: Sat, 21 Nov 2026 15:30:43 +0900 Subject: [PATCH 04/11] docs(inkling2): bring-up and route comparison (measured) Co-Authored-By: Claude Opus 5.5 --- docs/inkling2-mini-apple-silicon.md | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/docs/inkling2-mini-apple-silicon.md b/docs/inkling2-mini-apple-silicon.md index 4a64419c..9272307a 100644 --- a/docs/inkling2-mini-apple-silicon.md +++ b/docs/inkling2-mini-apple-silicon.md @@ -33,11 +33,32 @@ GGUF conversion at the time). ## Bring-up -BRINGUP_ROWS +| Observation | Cause | Action | Source | +| --- | --- | --- | --- | +| The release checkpoint does not load in MLX or llama.cpp. | Experts are ModelOpt NVFP4 and the last 4 layers MXFP8; Apple runtimes need their own formats. | Use a community conversion and pin it by repository and file hash. | public | +| Released mlx-lm 0.32.2 has no `inkling2_loop` model. | Support is in an open pull request. | Install mlx-lm from [#2086](https://github.com/ml-explore/mlx-lm/pull/2086) (`0c4c991`); llama.cpp main already has `inkling2_loop`. | measured | +| Two shards arrived larger than their published size (5,140,845,290 bytes against 5,031,793,948; two GGUF segments over-long). | The segmented downloader appended curl's own `--retry` restarts, which resend the range from its start. | Verify every shard against the Hub `lfs.oid` SHA-256 before the first load; re-fetch only segments whose size is wrong. | measured | +| With `reasoning_effort="none"` (template value 0) the model still thinks first: a 512-token cap ended inside the thinking block on both runtimes. | The chat template only adds a system line; the model is not forced to skip thinking. | Give code-generation tests a cap that covers thinking plus answer (4,096 here); count thinking and answer tokens separately. | measured | +| A complete code-generation run on MLX produced 2,244 tokens (thinking, then a 78-line Python file with 3 pytest cases); the 3 tests pass. | — | Keep one end-to-end generation with a checked answer next to any speed number. | measured | +| llama-server's streamed answer appears in `reasoning_content`, not `content`, until thinking ends; repeated requests reused the cached prompt (4 of 61 prompt tokens processed). | Server-side reasoning parsing and prompt cache. | Read both fields for time to first token and send `cache_prompt: false` when timing. | measured | ## Route comparison -ROUTE_TABLE +Same machine, one runtime at a time (a lock file kept them apart), greedy +decoding, fresh process per route. Idle ComfyUI and a small mlx-lm server were +resident but not working during the runs. + +| Metric | MLX 4-bit | llama.cpp Q4_K_M | Command | +| --- | ---: | ---: | --- | +| Prefill, 512-token prompt (tok/s, mean of 3) | 1,677 | 998 | `mlx_lm.benchmark -p 512 -g 128 -n 3`; `llama-bench -p 512 -n 128 -r 3` | +| Decode, 128 tokens (tok/s, mean of 3) | 47.0 | 41.7 | same | +| Decode in a 512-token chat answer (tok/s, 3 runs) | 45.3–47.6 | 41.5–43.4 | `stream_generate`; llama-server `/v1/chat/completions` | +| Time to first token, 61-token chat prompt, warm (s) | 0.24 | 0.29–0.31 | same | +| Peak memory | 17.6 GB (`mx.get_peak_memory`), 19.0 GB in the benchmark | 17.6 GB resident, of which weights are file-mapped | | + +MLX was faster on both prefill and decode, so the RSI round below uses MLX. +The first MLX request of a process took 2.17 s to first token because Metal +kernels compile on first use; warm numbers exclude it. ## RSI round From 7aa11120db6e9bec7ae4ab3464b6fbddf03a4d8a Mon Sep 17 00:00:00 2001 From: zuiho-kai <31877877+zuiho-kai@users.noreply.github.com> Date: Sat, 21 Nov 2026 15:43:58 +0900 Subject: [PATCH 05/11] docs(inkling2): RSI round on MLX, playbook rows, scripts and records Co-Authored-By: Claude Opus 5.5 --- README.md | 1 + docs/apple-silicon-playbook.md | 12 ++ docs/inkling2-mini-apple-silicon.md | 73 +++++++++++- examples/inkling2_mini_mlx/README.md | 26 +++++ examples/inkling2_mini_mlx/bench_llama.sh | 8 ++ examples/inkling2_mini_mlx/bench_mlx.py | 45 ++++++++ examples/inkling2_mini_mlx/chat_bench.py | 19 ++++ examples/inkling2_mini_mlx/mechanism.py | 21 ++++ .../rounds/r1/cands/_patch.py | 14 +++ .../rounds/r1/cands/c1_compile_gates.py | 5 + .../rounds/r1/cands/c2_compile_moe.py | 5 + .../rounds/r1/cands/c3_compile_gates_moe.py | 5 + .../rounds/r1/cands/c4_compile_residual.py | 5 + .../rounds/r1/log-evolve.jsonl | 5 + .../rounds/r1/score-evolve.json | 85 ++++++++++++++ .../rounds/r2/cands/_patch.py | 14 +++ .../rounds/r2/cands/r2a_gates_moe.py | 5 + .../rounds/r2/cands/r2b_gates_moe_residual.py | 7 ++ .../rounds/r2/cands/r2c_fused_mlp_half.py | 56 ++++++++++ .../rounds/r2/cands/r2d_fused_half_gates.py | 6 + .../rounds/r2/log-evolve.jsonl | 5 + .../rounds/r2/log-holdout.jsonl | 5 + .../inkling2_mini_mlx/rounds/r2/mechanism.txt | 4 + .../rounds/r2/score-evolve.json | 105 ++++++++++++++++++ .../rounds/r2/score-holdout.json | 105 ++++++++++++++++++ examples/inkling2_mini_mlx/rsi_mlx.py | 101 +++++++++++++++++ 26 files changed, 739 insertions(+), 3 deletions(-) create mode 100644 examples/inkling2_mini_mlx/README.md create mode 100755 examples/inkling2_mini_mlx/bench_llama.sh create mode 100644 examples/inkling2_mini_mlx/bench_mlx.py create mode 100644 examples/inkling2_mini_mlx/chat_bench.py create mode 100644 examples/inkling2_mini_mlx/mechanism.py create mode 100644 examples/inkling2_mini_mlx/rounds/r1/cands/_patch.py create mode 100644 examples/inkling2_mini_mlx/rounds/r1/cands/c1_compile_gates.py create mode 100644 examples/inkling2_mini_mlx/rounds/r1/cands/c2_compile_moe.py create mode 100644 examples/inkling2_mini_mlx/rounds/r1/cands/c3_compile_gates_moe.py create mode 100644 examples/inkling2_mini_mlx/rounds/r1/cands/c4_compile_residual.py create mode 100644 examples/inkling2_mini_mlx/rounds/r1/log-evolve.jsonl create mode 100644 examples/inkling2_mini_mlx/rounds/r1/score-evolve.json create mode 100644 examples/inkling2_mini_mlx/rounds/r2/cands/_patch.py create mode 100644 examples/inkling2_mini_mlx/rounds/r2/cands/r2a_gates_moe.py create mode 100644 examples/inkling2_mini_mlx/rounds/r2/cands/r2b_gates_moe_residual.py create mode 100644 examples/inkling2_mini_mlx/rounds/r2/cands/r2c_fused_mlp_half.py create mode 100644 examples/inkling2_mini_mlx/rounds/r2/cands/r2d_fused_half_gates.py create mode 100644 examples/inkling2_mini_mlx/rounds/r2/log-evolve.jsonl create mode 100644 examples/inkling2_mini_mlx/rounds/r2/log-holdout.jsonl create mode 100644 examples/inkling2_mini_mlx/rounds/r2/mechanism.txt create mode 100644 examples/inkling2_mini_mlx/rounds/r2/score-evolve.json create mode 100644 examples/inkling2_mini_mlx/rounds/r2/score-holdout.json create mode 100644 examples/inkling2_mini_mlx/rsi_mlx.py diff --git a/README.md b/README.md index 877c255c..6f38656d 100644 --- a/README.md +++ b/README.md @@ -162,6 +162,7 @@ request per engine and measure audio frames per second and median time to first - **▶️ Run and review.** [Operations](docs/operations.md) · [Packaged optimization benchmarks](docs/optimization-benchmarks.md) · [Result delivery](docs/result-delivery.md) - **🧠 Use knowledge.** [Skill library](src/inferrsi/knowledge/data/skill_library/README.md) · [CANNbot setup and revocation](docs/external-channels.md#setting-one-up) · [MindIE setup and revocation](docs/codex-plugin.md#enable-and-revoke) +- **📓 Field lessons.** [Apple silicon playbook](docs/apple-silicon-playbook.md) · [Inkling 2 Mini on Apple silicon](docs/inkling2-mini-apple-silicon.md) · [Scheduling RSI lessons](docs/scheduling-rsi-lessons.md) · [h3-metal lessons](docs/h3-metal-apple-silicon-lessons.md) - **🧭 Understand and extend.** [Module layout and import migration](docs/module-layout.md) · [Architecture and directory map](docs/architecture.md) · [Standalone engine template](docs/template/README.md) - **📌 Follow changes.** [Releases](https://github.com/InferMatrixLab/InferRSI/releases) · [Changelog](CHANGELOG.md) diff --git a/docs/apple-silicon-playbook.md b/docs/apple-silicon-playbook.md index b2699496..0d6a1b97 100644 --- a/docs/apple-silicon-playbook.md +++ b/docs/apple-silicon-playbook.md @@ -58,6 +58,18 @@ acceptance result. | A candidate kernel used 42,240 bytes of threadgroup memory and failed only at run time. | The limit is 32 KB. | Build every Metal pipeline before timing so the failure is a build failure. | ledger | | PyTorch/diffusers could not run the model on MPS. | `aten::_int_mm` is not implemented for MPS. | Check the int8 operator set before choosing PyTorch MPS as a baseline. | ledger (2026-10-24) | +## Bringing up a new model + +From [Inkling 2 Mini on Apple silicon](inkling2-mini-apple-silicon.md) (measured on the same M5 Max). + +| Observation | Cause | Action | Source | +| --- | --- | --- | --- | +| The release checkpoint (ModelOpt NVFP4/MXFP8) loads in neither MLX nor llama.cpp. | Apple runtimes need their own quantized formats. | Start from a community conversion; pin repository, revision and file SHA-256. | measured | +| Shards arrived larger than published after a segmented download. | curl `--retry` inside a ranged, appending segment resends from the range start. | Verify each file against the Hub `lfs.oid` before the first load. | measured | +| A 512-token cap ended inside the thinking block although reasoning effort was set to `none`. | The template value only adds a system line. | Size generation caps for thinking plus answer; count both. | measured | +| A looped MoE model decoded at about 48 tok/s on MLX while weight traffic allowed far more. | 48 layer visits per token, each with many small ops: launch- and graph-building-bound. | Search compilation of pure sub-blocks first; per-module compiles gave 1–3%, a fused per-layer half plus gates 6.4% on held-out prompts. | measured | +| During a search round the default path drifted from 47.4 to 48.9 tok/s. | Warm-up/thermal drift over minutes. | Pair each candidate run with the adjacent default run; an all-runs median inflated one candidate from 1.048 to 1.053. | measured | + ## Benchmarking a shared Mac | Observation | Cause | Action | Source | diff --git a/docs/inkling2-mini-apple-silicon.md b/docs/inkling2-mini-apple-silicon.md index 9272307a..48cfe412 100644 --- a/docs/inkling2-mini-apple-silicon.md +++ b/docs/inkling2-mini-apple-silicon.md @@ -1,6 +1,11 @@ # Inkling 2 Mini on Apple silicon: local adaptation and one RSI round -**RESULTS_HEADLINE** +**On one M5 Max, MLX ran Inkling 2 Mini at 47.0 tok/s decode and 1,677 tok/s +prefill (512-token prompt), ahead of llama.cpp at 41.7 and 998. Two search +rounds then found a compile patch that raised MLX decode by 6.4% (paired +median, held-out prompts, run-to-run noise 3.9%) with token-identical greedy +output.** These are maintainer measurements without an acceptance issue; they +are not an InferRSI PASS. [Inkling 2 Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) is a 26B-total, MoE, looped transformer from Thinking Machines Lab (24 layers; the @@ -62,8 +67,70 @@ kernels compile on first use; warm numbers exclude it. ## RSI round -RSI_SECTION +Scripts, candidates and per-round records: +[examples/inkling2_mini_mlx](../examples/inkling2_mini_mlx/README.md). + +**Plane and why.** Decode on this model is launch-bound rather than +bandwidth-bound: a token visits 48 layers (6 + 12 × 3 + 6), each with a MoE +block, two depth-attention residuals and, in loop passes 2 and 3, four +message gates, so many small kernels and Python graph-building per step. +The searched plane is therefore *model-side graph compilation* (`mx.compile` +around pure sub-blocks), not scheduling: one user, one stream, no queue to +order. Loop count was not searched because the variable-loop model has no +Apple-format weights and changing `num_passes` on the released model changes +its output. + +**Gates frozen before the run** (inherited from the +[scheduling RSI lessons](scheduling-rsi-lessons.md)): + +- greedy decoding; a candidate whose token ids differ from the default path on + any prompt is rejected; +- each candidate run is paired with the default run immediately before it, 3 + pairs per candidate, score = median paired ratio of decode tok/s; +- below 5% difference: no conclusion; +- candidates are selected on two "evolve" prompts and reported on two + different "holdout" prompts (TypeScript and Rust code generation); +- a "better" verdict needs a mechanism signal. + +**Round 1** (evolve, 256 generated tokens, noise 2.9%). Paired medians: +gates 1.031, MoE 1.015, depth residual 1.012, gates + MoE 1.048. All +token-identical, all below 5%: no conclusion. The first version of the scorer +compared with the median of all default runs; the default path drifted from +47.4 to 48.9 tok/s during the round, which inflated gates + MoE to 1.053. The +scorer was changed to adjacent pairing before round 2. + +**Round 2** (proposed from round 1: combine the compile points; fuse the whole +MLP half of each layer). + +| Candidate | Evolve (noise 2.3%) | Holdout (noise 3.9%) | Output | +| --- | ---: | ---: | --- | +| gates + MoE (round-1 best, re-run) | 1.043 | 1.054 | identical | +| gates + MoE + depth residual | 1.057 | 1.056 | identical | +| fused MLP half per layer | 1.044 | 1.039 | identical | +| **fused MLP half + q/k/v gates** | **1.068** | **1.064** (1.068, 1.064, 1.060) | identical | + +Mechanism (one decode step, synchronous, median of 50 steps, two alternating +processes each): 22.07 → 20.94 ms per step; Python graph building +3.89 → 3.34 ms. About half of the saving is CPU graph construction and the +rest is GPU time from fewer, fused kernels. The winning patch is +`rounds/r2/cands/r2d_fused_half_gates.py`; it changes no weights and no +numerics outside the compiled regions. + +**How candidates were produced.** The candidates were written by the +assistant running the campaign (Claude Code, `claude-opus-5-5`) from reading +the model source; no automatic mutation chain was used. Round 2 was proposed +after reading round 1's scores. ## What we did not establish -LIMITS +- One machine, one macOS build, 128 GB; nothing on 32–64 GB Macs, which the + model card lists as supported. +- Single stream only; no batching, concurrency or long-context runs (the + advertised 1M context was not exercised). Prompts are short (61 tokens); + prefill-heavy work was measured only by the 512-token benchmark. +- Quality was checked only as greedy-token identity on four prompts and one + passing generated test file; no benchmark accuracy. +- The compile patch was not measured inside `mlx_lm.server` and was not + proposed upstream. +- llama.cpp was run with default settings; its flags were not searched, so + the route comparison is default against default. diff --git a/examples/inkling2_mini_mlx/README.md b/examples/inkling2_mini_mlx/README.md new file mode 100644 index 00000000..55296e06 --- /dev/null +++ b/examples/inkling2_mini_mlx/README.md @@ -0,0 +1,26 @@ +# Inkling 2 Mini on MLX: benchmark and RSI round scripts + +Scripts and records behind [Inkling 2 Mini on Apple silicon](../../docs/inkling2-mini-apple-silicon.md). +They are maintainer tools, not part of the `inferrsi` package, and need an +Apple-silicon Mac with mlx-lm from ml-explore/mlx-lm#2086 (`0c4c991`). + +| File | Purpose | +| --- | --- | +| `bench_mlx.py` | One code-generation request on MLX: TTFT, prefill/decode tok/s, peak memory, full output | +| `chat_bench.py` | The same request through llama-server's streaming OpenAI API | +| `bench_llama.sh` | `llama-bench` 512/128 × 3 with `/usr/bin/time -l` | +| `rsi_mlx.py` | One search round: load once, alternate each candidate with the default path, paired scoring, token-identity gate, evolve/holdout split | +| `mechanism.py` | Per-step graph-building and step time, default vs one candidate | +| `rounds/r1`, `rounds/r2` | Candidates (`cands/*.py`) and the JSONL/JSON records each round wrote | + +```bash +python rsi_mlx.py rounds/r2 --split evolve --max-tokens 256 --reps 3 +python rsi_mlx.py rounds/r2 --split holdout --max-tokens 256 --reps 3 +python mechanism.py rounds/r2/cands/r2d_fused_half_gates.py +``` + +Round 1 records predate adjacent pairing; its paired ratios in the doc were +recomputed from the logged run order (default run `1 + 3k + i` precedes +repetition `i` of candidate `k`). Workload prompts are kept verbatim, +including the Chinese ones. Run one runtime at a time; another GPU job on the +machine invalidates the round. diff --git a/examples/inkling2_mini_mlx/bench_llama.sh b/examples/inkling2_mini_mlx/bench_llama.sh new file mode 100755 index 00000000..26d9099a --- /dev/null +++ b/examples/inkling2_mini_mlx/bench_llama.sh @@ -0,0 +1,8 @@ +#!/bin/bash +# llama.cpp route: llama-bench prefill/decode with the peak memory reported by /usr/bin/time -l +# Usage: bench_llama.sh (LLAMA_BIN = llama.cpp build/bin) +set -e; G=$1; O=$2; mkdir -p $O; B=${LLAMA_BIN:?set LLAMA_BIN to llama.cpp build/bin} +P='用 Python 写一个线程安全的 LRU 缓存类,支持 get/put/容量上限和过期时间(TTL),附 3 个 pytest 用例。只输出代码。' +/usr/bin/time -l $B/llama-bench -m $G -p 512 -n 128 -r 3 -o json > $O/bench.json 2> $O/bench.err # 512/128, 3 repetitions: llama-bench defaults +cat $O/bench.json | python3 -c "import sys,json;[print(x['n_prompt'],x['n_gen'],round(x['avg_ts'],2),'tok/s') for x in json.load(sys.stdin)]" +grep -E "peak memory footprint" $O/bench.err diff --git a/examples/inkling2_mini_mlx/bench_mlx.py b/examples/inkling2_mini_mlx/bench_mlx.py new file mode 100644 index 00000000..36dc5af5 --- /dev/null +++ b/examples/inkling2_mini_mlx/bench_mlx.py @@ -0,0 +1,45 @@ +# Usage: python bench_mlx.py --max-tokens N [--effort none] [--out result.json] [stream_generate key=value ...] +# Measures TTFT, prefill tok/s, decode tok/s and peak memory; the full output goes to .txt +import argparse, json, time +import mlx.core as mx +from mlx_lm import load, stream_generate +from mlx_lm.sample_utils import make_sampler + +PROMPT = ("用 Python 写一个线程安全的 LRU 缓存类,支持 get/put/容量上限和过期时间(TTL)," + "附 3 个 pytest 用例。只输出代码。") + +ap = argparse.ArgumentParser() +ap.add_argument("model") +ap.add_argument("--max-tokens", type=int, required=True) # always given on the command line +ap.add_argument("--effort", default="none") # a key of effort_map in the model's chat template +ap.add_argument("--out", default=None) +ap.add_argument("--runs", type=int, default=1) # single run unless asked +ap.add_argument("--prompt", default=PROMPT) +ap.add_argument("kw", nargs="*") +a = ap.parse_args() +kw = {} +for s in a.kw: + k, v = s.split("=", 1) + kw[k] = int(v) if v.lstrip("-").isdigit() else v + +t0 = time.time(); model, tok = load(a.model); load_s = time.time() - t0 +msgs = [{"role": "user", "content": a.prompt}] +prompt = tok.apply_chat_template(msgs, add_generation_prompt=True, reasoning_effort=a.effort) +sampler = make_sampler(temp=0.0) # greedy decoding: reproducible output, so changed results are detectable +res = [] +for r in range(a.runs): + mx.reset_peak_memory() + t0 = time.time(); ttft = None; text = ""; last = None + for resp in stream_generate(model, tok, prompt, max_tokens=a.max_tokens, sampler=sampler, **kw): + if ttft is None: ttft = time.time() - t0 + text += resp.text; last = resp + wall = time.time() - t0 + row = dict(run=r, load_s=load_s, ttft_s=ttft, prompt_tokens=last.prompt_tokens, + prefill_tps=last.prompt_tps, gen_tokens=last.generation_tokens, + decode_tps=last.generation_tps, peak_gb=mx.get_peak_memory() / 1e9, + wall_s=wall, kw=kw) + print(json.dumps(row, ensure_ascii=False), flush=True); res.append(row) +if a.out: + json.dump(res, open(a.out, "w"), ensure_ascii=False, indent=1) + open(a.out.rsplit(".", 1)[0] + ".txt", "w").write(text) +print("----- output -----\n" + text) diff --git a/examples/inkling2_mini_mlx/chat_bench.py b/examples/inkling2_mini_mlx/chat_bench.py new file mode 100644 index 00000000..c3230b68 --- /dev/null +++ b/examples/inkling2_mini_mlx/chat_bench.py @@ -0,0 +1,19 @@ +# One code-generation request through llama-server's streaming OpenAI API; records time to first token and server timings. +import sys, json, time, urllib.request +url, maxtok = sys.argv[1], int(sys.argv[2]) +P = "用 Python 写一个线程安全的 LRU 缓存类,支持 get/put/容量上限和过期时间(TTL),附 3 个 pytest 用例。只输出代码。" +body = dict(messages=[{"role": "user", "content": P}], max_tokens=maxtok, temperature=0, stream=True, + chat_template_kwargs={"reasoning_effort": "none"}, timings_per_token=False, cache_prompt=False) # no prompt-cache reuse, so time to first token is comparable across runs +req = urllib.request.Request(url + "/v1/chat/completions", json.dumps(body).encode(), {"Content-Type": "application/json"}) +t0 = time.time(); ttft = None; text = ""; think = ""; timings = None +for line in urllib.request.urlopen(req): + line = line.decode().strip() + if not line.startswith("data: ") or line == "data: [DONE]": continue + d = json.loads(line[6:]) + if d.get("timings"): timings = d["timings"] + for c in d.get("choices", []): + dl = c.get("delta") or {} + t = dl.get("content") or ""; r = dl.get("reasoning_content") or "" + if (t or r) and ttft is None: ttft = time.time() - t0 + text += t; think += r +print(json.dumps(dict(ttft_s=ttft, wall_s=time.time() - t0, timings=timings, text=text, reasoning=think), ensure_ascii=False)) diff --git a/examples/inkling2_mini_mlx/mechanism.py b/examples/inkling2_mini_mlx/mechanism.py new file mode 100644 index 00000000..b2066170 --- /dev/null +++ b/examples/inkling2_mini_mlx/mechanism.py @@ -0,0 +1,21 @@ +# Mechanism check: per decode step, time spent building the lazy graph in Python vs the whole step. +# A compile candidate should cut graph-building time (fewer small ops) and GPU time (fused kernels). +# Usage: python mechanism.py [candidate.py] +import sys, time, importlib.util, statistics +import mlx.core as mx +from mlx_lm import load +from mlx_lm.models.cache import make_prompt_cache + +model, tok = load(sys.argv[1]); cand = sys.argv[2] if len(sys.argv) > 2 else None +if cand: + spec = importlib.util.spec_from_file_location("c", cand); m = importlib.util.module_from_spec(spec) + spec.loader.exec_module(m); m.apply(model) +ids = mx.array(tok.encode("def lru_cache(capacity):"))[None] +cache = make_prompt_cache(model); out = model(ids, cache=cache); mx.eval(out) +y = mx.argmax(out[:, -1:], -1) +build, total = [], [] +for i in range(60): # 60 steps; the first 10 are warm-up (the first includes compilation) + t0 = time.perf_counter(); out = model(y, cache=cache); t1 = time.perf_counter() + y = mx.argmax(out[:, -1:], -1); mx.eval(y); t2 = time.perf_counter() + if i >= 10: build.append(t1 - t0); total.append(t2 - t0) +print(f"{'cand' if cand else 'base'} build_ms={statistics.median(build)*1e3:.2f} step_ms={statistics.median(total)*1e3:.2f}") diff --git a/examples/inkling2_mini_mlx/rounds/r1/cands/_patch.py b/examples/inkling2_mini_mlx/rounds/r1/cands/_patch.py new file mode 100644 index 00000000..150ea04e --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r1/cands/_patch.py @@ -0,0 +1,14 @@ +# Shared helper: replace a module class __call__ with one mx.compile per instance; revert_all restores it. +import mlx.core as mx +_saved = {} +def compile_class(cls): + orig = cls.__call__; _saved[cls] = orig; cache = {} + def call(self, *a, **k): + f = cache.get(id(self)) + if f is None: + f = cache[id(self)] = mx.compile(lambda *aa: orig(self, *aa)) + return f(*a) if not k else orig(self, *a, **k) + cls.__call__ = call +def revert_all(): + for cls, orig in _saved.items(): cls.__call__ = orig + _saved.clear() diff --git a/examples/inkling2_mini_mlx/rounds/r1/cands/c1_compile_gates.py b/examples/inkling2_mini_mlx/rounds/r1/cands/c1_compile_gates.py new file mode 100644 index 00000000..5652dd52 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r1/cands/c1_compile_gates.py @@ -0,0 +1,5 @@ +"""Compile each loop message gate (MessageGate: two rms_norms, concat, three projections, two tanh) into one graph. Passes 2 and 3 of the loop call 4 gates per layer: about 96 calls per token.""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _patch +from mlx_lm.models.inkling2 import layers as L +def apply(model): _patch.compile_class(L.MessageGate); return {} +def revert(model): _patch.revert_all() diff --git a/examples/inkling2_mini_mlx/rounds/r1/cands/c2_compile_moe.py b/examples/inkling2_mini_mlx/rounds/r1/cands/c2_compile_moe.py new file mode 100644 index 00000000..01bac2b2 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r1/cands/c2_compile_moe.py @@ -0,0 +1,5 @@ +"""Compile the whole MoE block (router, top-k, two expert matmuls, soft-capped SwiGLU, weighted sum). 48 layer visits per token, one MoE block each.""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _patch +from mlx_lm.models.inkling2 import layers as L +def apply(model): _patch.compile_class(L.SparseMoeBlock); return {} +def revert(model): _patch.revert_all() diff --git a/examples/inkling2_mini_mlx/rounds/r1/cands/c3_compile_gates_moe.py b/examples/inkling2_mini_mlx/rounds/r1/cands/c3_compile_gates_moe.py new file mode 100644 index 00000000..5863770e --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r1/cands/c3_compile_gates_moe.py @@ -0,0 +1,5 @@ +"""c1 + c2 together.""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _patch +from mlx_lm.models.inkling2 import layers as L +def apply(model): _patch.compile_class(L.MessageGate); _patch.compile_class(L.SparseMoeBlock); return {} +def revert(model): _patch.revert_all() diff --git a/examples/inkling2_mini_mlx/rounds/r1/cands/c4_compile_residual.py b/examples/inkling2_mini_mlx/rounds/r1/cands/c4_compile_residual.py new file mode 100644 index 00000000..d510f1db --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r1/cands/c4_compile_residual.py @@ -0,0 +1,5 @@ +"""Compile the depth-attention residual (BlockAttentionResidual, twice per layer).""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _patch +from mlx_lm.models.inkling2 import layers as L +def apply(model): _patch.compile_class(L.BlockAttentionResidual); return {} +def revert(model): _patch.revert_all() diff --git a/examples/inkling2_mini_mlx/rounds/r1/log-evolve.jsonl b/examples/inkling2_mini_mlx/rounds/r1/log-evolve.jsonl new file mode 100644 index 00000000..5ea4b25f --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r1/log-evolve.jsonl @@ -0,0 +1,5 @@ +{"cand": "c1_compile_gates", "dec_med": 49.40115411275421, "dec_all": [49.40115411275421, 49.3791565718586, 49.51521835056296], "ttft_med": 0.21953141689300537, "peak": 17.569098932, "same_output": true, "err": null, "doc": "把循环层的消息门(MessageGate:两次 rms_norm、拼接、三次投影、两次 tanh)整段编译成一张图,减少解码时每步的小算子发射。循环段后两遍每层 4 个门,每 token 约 96 次调用。"} +{"cand": "c2_compile_moe", "dec_med": 49.02118691839245, "dec_all": [48.85107143881726, 49.09644688548954, 49.02118691839245], "ttft_med": 0.22214102745056152, "peak": 17.560419744, "same_output": true, "err": null, "doc": "把 MoE 块(路由打分、top-k、两次专家矩阵乘、软截断 SwiGLU、加权求和)整段编译。每 token 走 48 次层访问,每次一个 MoE 块。"} +{"cand": "c3_compile_gates_moe", "dec_med": 51.02677239497174, "dec_all": [51.108004003752164, 50.98362410796302, 51.02677239497174], "ttft_med": 0.22019445896148682, "peak": 17.567906632, "same_output": true, "err": null, "doc": "c1 + c2 同时上。"} +{"cand": "c4_compile_residual", "dec_med": 49.07735030987658, "dec_all": [49.112221671001905, 49.07735030987658, 49.05263728135563], "ttft_med": 0.2178339958190918, "peak": 17.559941956, "same_output": true, "err": null, "doc": "把深度注意力残差(BlockAttentionResidual,每层两次)编译。"} +{"cand": "baseline", "dec_med": 48.44602831890364, "dec_all": [47.446792327900155, 47.84514640626699, 48.09577184676405, 48.003700957422566, 48.013395006973006, 48.38342647762863, 48.468703240708905, 48.76026873686101, 48.64142996610397, 48.743378020755145, 48.44602831890364, 48.871949743256266, 48.47134355045931], "ttft_med": 0.21846795082092285, "peak": 17.5599129, "noise_range": 0.02941742522162574} diff --git a/examples/inkling2_mini_mlx/rounds/r1/score-evolve.json b/examples/inkling2_mini_mlx/rounds/r1/score-evolve.json new file mode 100644 index 00000000..46958f0b --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r1/score-evolve.json @@ -0,0 +1,85 @@ +{ + "baseline": { + "dec_med": 48.44602831890364, + "dec_all": [ + 47.446792327900155, + 47.84514640626699, + 48.09577184676405, + 48.003700957422566, + 48.013395006973006, + 48.38342647762863, + 48.468703240708905, + 48.76026873686101, + 48.64142996610397, + 48.743378020755145, + 48.44602831890364, + 48.871949743256266, + 48.47134355045931 + ], + "ttft_med": 0.21846795082092285, + "peak": 17.5599129 + }, + "noise_range": 0.02941742522162574, + "cands": { + "c1_compile_gates": { + "dec_med": 49.40115411275421, + "dec_all": [ + 49.40115411275421, + 49.3791565718586, + 49.51521835056296 + ], + "ttft_med": 0.21953141689300537, + "peak": 17.569098932, + "same_output": true, + "err": null, + "doc": "把循环层的消息门(MessageGate:两次 rms_norm、拼接、三次投影、两次 tanh)整段编译成一张图,减少解码时每步的小算子发射。循环段后两遍每层 4 个门,每 token 约 96 次调用。", + "speedup": 1.0197152548308668, + "verdict": "差距<5%,不下结论" + }, + "c2_compile_moe": { + "dec_med": 49.02118691839245, + "dec_all": [ + 48.85107143881726, + 49.09644688548954, + 49.02118691839245 + ], + "ttft_med": 0.22214102745056152, + "peak": 17.560419744, + "same_output": true, + "err": null, + "doc": "把 MoE 块(路由打分、top-k、两次专家矩阵乘、软截断 SwiGLU、加权求和)整段编译。每 token 走 48 次层访问,每次一个 MoE 块。", + "speedup": 1.0118721517417844, + "verdict": "差距<5%,不下结论" + }, + "c3_compile_gates_moe": { + "dec_med": 51.02677239497174, + "dec_all": [ + 51.108004003752164, + 50.98362410796302, + 51.02677239497174 + ], + "ttft_med": 0.22019445896148682, + "peak": 17.567906632, + "same_output": true, + "err": null, + "doc": "c1 + c2 同时上。", + "speedup": 1.0532704984416874, + "verdict": "留" + }, + "c4_compile_residual": { + "dec_med": 49.07735030987658, + "dec_all": [ + 49.112221671001905, + 49.07735030987658, + 49.05263728135563 + ], + "ttft_med": 0.2178339958190918, + "peak": 17.559941956, + "same_output": true, + "err": null, + "doc": "把深度注意力残差(BlockAttentionResidual,每层两次)编译。", + "speedup": 1.0130314499016753, + "verdict": "差距<5%,不下结论" + } + } +} \ No newline at end of file diff --git a/examples/inkling2_mini_mlx/rounds/r2/cands/_patch.py b/examples/inkling2_mini_mlx/rounds/r2/cands/_patch.py new file mode 100644 index 00000000..150ea04e --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/cands/_patch.py @@ -0,0 +1,14 @@ +# Shared helper: replace a module class __call__ with one mx.compile per instance; revert_all restores it. +import mlx.core as mx +_saved = {} +def compile_class(cls): + orig = cls.__call__; _saved[cls] = orig; cache = {} + def call(self, *a, **k): + f = cache.get(id(self)) + if f is None: + f = cache[id(self)] = mx.compile(lambda *aa: orig(self, *aa)) + return f(*a) if not k else orig(self, *a, **k) + cls.__call__ = call +def revert_all(): + for cls, orig in _saved.items(): cls.__call__ = orig + _saved.clear() diff --git a/examples/inkling2_mini_mlx/rounds/r2/cands/r2a_gates_moe.py b/examples/inkling2_mini_mlx/rounds/r2/cands/r2a_gates_moe.py new file mode 100644 index 00000000..5863770e --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/cands/r2a_gates_moe.py @@ -0,0 +1,5 @@ +"""c1 + c2 together.""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _patch +from mlx_lm.models.inkling2 import layers as L +def apply(model): _patch.compile_class(L.MessageGate); _patch.compile_class(L.SparseMoeBlock); return {} +def revert(model): _patch.revert_all() diff --git a/examples/inkling2_mini_mlx/rounds/r2/cands/r2b_gates_moe_residual.py b/examples/inkling2_mini_mlx/rounds/r2/cands/r2b_gates_moe_residual.py new file mode 100644 index 00000000..551b5632 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/cands/r2b_gates_moe_residual.py @@ -0,0 +1,7 @@ +"""All three round-1 compile points (message gate, MoE, depth residual).""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _patch +from mlx_lm.models.inkling2 import layers as L +def apply(model): + for c in (L.MessageGate, L.SparseMoeBlock, L.BlockAttentionResidual): _patch.compile_class(c) + return {} +def revert(model): _patch.revert_all() diff --git a/examples/inkling2_mini_mlx/rounds/r2/cands/r2c_fused_mlp_half.py b/examples/inkling2_mini_mlx/rounds/r2/cands/r2c_fused_mlp_half.py new file mode 100644 index 00000000..ac1c2a43 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/cands/r2c_fused_mlp_half.py @@ -0,0 +1,56 @@ +"""Fuse the second half of every layer (depth residual for the MLP input -> message gate -> RMSNorm -> MoE) +into one compiled graph per layer instance. The attention half and the convolution state (cache writes) are untouched.""" +import mlx.core as mx +from mlx_lm.models.inkling2 import layers as L +_orig = L.DecoderLayer.__call__ +_cache = {} + +def _half(self, has_msg): + key = (id(self), has_msg) + f = _cache.get(key) + if f is None: + def body(hidden_states, block_states, *msg): + dtype = self._dtype + normed = L.unweighted_rms_norm(hidden_states) + h = self.mlp_input(block_states, normed)[..., 0, :] + if msg: + h = self.mlp_message_gate(h, msg[0][..., 3, :]) + return self.mlp(self.post_attention_layernorm(h).astype(dtype)) + f = _cache[key] = mx.compile(body) + return f + +def call(self, hidden_states, block_states, *, cache=None, conv_mask=None, messages=None): + dtype = hidden_states.dtype + self._dtype = dtype + conv = cache[1] if cache is not None else None + conv_states = list(conv.cache) if conv is not None else [None] * 4 + block_residual = hidden_states + normed_residual = L.unweighted_rms_norm(block_residual.astype(mx.float32)) + attention_inputs = None + if self.attention_input is not None: + attention_inputs = self.attention_input(block_states, normed_residual) + hidden_states = self._self_attention(hidden_states, attention_inputs, cache, conv_states, conv_mask, messages) + hidden_states, conv_states[2] = self.post_attention_convolution(hidden_states, conv_states[2], conv_mask) + residual = hidden_states = hidden_states.astype(mx.float32) + if self.is_block_boundary: + block = normed_residual[None] + block_states = block if block_states is None else mx.concatenate([block_states, block]) + else: + residual = hidden_states = block_residual.astype(mx.float32) + hidden_states + if messages is not None: + hidden_states = _half(self, True)(hidden_states, block_states, messages) + else: + hidden_states = _half(self, False)(hidden_states, block_states) + hidden_states, conv_states[3] = self.post_mlp_convolution(hidden_states, conv_states[3], conv_mask) + if conv is not None: + n = self.post_mlp_convolution.kernel_size - 1 + for i, window in enumerate(conv_states): + conv[i] = L._conv_state(window, conv.lengths, n) + conv.advance(block_residual.shape[1]) + hidden_states = residual + hidden_states.astype(mx.float32) + return hidden_states.astype(dtype), block_states + +def apply(model): + L.DecoderLayer.__call__ = call; return {} +def revert(model): + L.DecoderLayer.__call__ = _orig diff --git a/examples/inkling2_mini_mlx/rounds/r2/cands/r2d_fused_half_gates.py b/examples/inkling2_mini_mlx/rounds/r2/cands/r2d_fused_half_gates.py new file mode 100644 index 00000000..a398af4a --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/cands/r2d_fused_half_gates.py @@ -0,0 +1,6 @@ +"""r2c plus compiled q/k/v message gates in the attention half.""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _patch +import r2c_fused_mlp_half as H +from mlx_lm.models.inkling2 import layers as L +def apply(model): H.apply(model); _patch.compile_class(L.MessageGate); return {} +def revert(model): H.revert(model); _patch.revert_all() diff --git a/examples/inkling2_mini_mlx/rounds/r2/log-evolve.jsonl b/examples/inkling2_mini_mlx/rounds/r2/log-evolve.jsonl new file mode 100644 index 00000000..8bcf51a1 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/log-evolve.jsonl @@ -0,0 +1,5 @@ +{"cand": "r2a_gates_moe", "dec_med": 50.424029905177704, "dec_all": [50.424029905177704, 50.04613092519887, 50.62708168991173], "ttft_med": 0.220139741897583, "peak": 17.567414856, "paired": [1.0428356184966039, 1.0320574751381653, 1.0474635424160863], "same_output": true, "err": null, "doc": "c1 + c2 同时上。"} +{"cand": "r2b_gates_moe_residual", "dec_med": 51.21983893528104, "dec_all": [51.43588232496421, 51.21983893528104, 51.17689779327035], "ttft_med": 0.22014844417572021, "peak": 17.566595896, "paired": [1.0569113877913525, 1.0565390347402557, 1.055181811728994], "same_output": true, "err": null, "doc": "第 1 轮三个编译点(消息门、MoE、深度残差)全上。"} +{"cand": "r2c_fused_mlp_half", "dec_med": 49.797729329488135, "dec_all": [49.797729329488135, 49.55705031401985, 50.197880166144515], "ttft_med": 0.21745789051055908, "peak": 17.56299792, "paired": [1.0473919742965152, 1.035651301526283, 1.0442818871213315], "same_output": true, "err": null, "doc": "把每层后半段(深度残差取 MLP 输入 → 消息门 → RMSNorm → MoE)合成一张编译图,每层实例一份;\n前半段注意力和卷积状态(带缓存写入)不动。比单独编译各模块多合掉模块之间的接缝。"} +{"cand": "r2d_fused_half_gates", "dec_med": 51.4059422007786, "dec_all": [51.54030730625611, 51.4059422007786, 51.042283946977946], "ttft_med": 0.21700644493103027, "peak": 17.569096812, "paired": [1.0677181572871433, 1.072562396095457, 1.0630550931504787], "same_output": true, "err": null, "doc": "r2c 再加上前半段的消息门编译(q/k/v 三个门)。"} +{"cand": "baseline", "dec_med": 48.27145343037871, "dec_all": [48.17797630886772, 48.35280749028416, 48.491612270430004, 48.333025102845085, 48.666220195101445, 48.47888932742855, 48.500549596673956, 47.5445015348098, 47.8510964462513, 48.06928166160197, 48.27145343037871, 47.9281600659469, 48.014711820540384], "ttft_med": 0.21836340427398682, "peak": 17.5599953, "noise_range": 0.023237722930996654} diff --git a/examples/inkling2_mini_mlx/rounds/r2/log-holdout.jsonl b/examples/inkling2_mini_mlx/rounds/r2/log-holdout.jsonl new file mode 100644 index 00000000..a7073f9d --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/log-holdout.jsonl @@ -0,0 +1,5 @@ +{"cand": "r2a_gates_moe", "dec_med": 51.04474074076323, "dec_all": [50.85660088211513, 51.04474074076323, 51.08489211463147], "ttft_med": 0.19790959358215332, "peak": 17.521604464, "paired": [1.054492129303139, 1.0870758620919294, 1.048187714673263], "same_output": true, "err": null, "doc": "c1 + c2 同时上。"} +{"cand": "r2b_gates_moe_residual", "dec_med": 51.562731237754896, "dec_all": [51.562731237754896, 51.5694744945997, 51.031208850454746], "ttft_med": 0.198167085647583, "peak": 17.52170266, "paired": [1.0562378572332667, 1.0557891451819792, 1.0479394689598494], "same_output": true, "err": null, "doc": "第 1 轮三个编译点(消息门、MoE、深度残差)全上。"} +{"cand": "r2c_fused_mlp_half", "dec_med": 50.094175012824834, "dec_all": [50.13723459977536, 50.094175012824834, 49.96048055644977], "ttft_med": 0.19697952270507812, "peak": 17.516487316, "paired": [1.0280730946681598, 1.043505733952328, 1.0386180183034315], "same_output": true, "err": null, "doc": "把每层后半段(深度残差取 MLP 输入 → 消息门 → RMSNorm → MoE)合成一张编译图,每层实例一份;\n前半段注意力和卷积状态(带缓存写入)不动。比单独编译各模块多合掉模块之间的接缝。"} +{"cand": "r2d_fused_half_gates", "dec_med": 51.2338418260509, "dec_all": [51.49597712697293, 51.2338418260509, 50.81507864179749], "ttft_med": 0.19498789310455322, "peak": 17.521638616, "paired": [1.0679889472833788, 1.0639873985016102, 1.059518895826631], "same_output": true, "err": null, "doc": "r2c 再加上前半段的消息门编译(q/k/v 三个门)。"} +{"cand": "baseline", "dec_med": 48.228525817185286, "dec_all": [48.47356339461423, 48.228525817185286, 46.95600603488204, 48.73639654377695, 48.817348180285336, 48.8444825654189, 48.69671423017082, 48.768161388328714, 48.00565380996111, 48.102844044685014, 48.21770605207308, 48.1526772762557, 47.96052136677736], "ttft_med": 0.19630658626556396, "peak": 17.514715684, "noise_range": 0.039156837131935344} diff --git a/examples/inkling2_mini_mlx/rounds/r2/mechanism.txt b/examples/inkling2_mini_mlx/rounds/r2/mechanism.txt new file mode 100644 index 00000000..03ea6e71 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/mechanism.txt @@ -0,0 +1,4 @@ +base build_ms=3.92 step_ms=22.09 +cand build_ms=3.31 step_ms=20.75 +base build_ms=3.86 step_ms=22.05 +cand build_ms=3.37 step_ms=21.13 diff --git a/examples/inkling2_mini_mlx/rounds/r2/score-evolve.json b/examples/inkling2_mini_mlx/rounds/r2/score-evolve.json new file mode 100644 index 00000000..6b08a1bd --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/score-evolve.json @@ -0,0 +1,105 @@ +{ + "baseline": { + "dec_med": 48.27145343037871, + "dec_all": [ + 48.17797630886772, + 48.35280749028416, + 48.491612270430004, + 48.333025102845085, + 48.666220195101445, + 48.47888932742855, + 48.500549596673956, + 47.5445015348098, + 47.8510964462513, + 48.06928166160197, + 48.27145343037871, + 47.9281600659469, + 48.014711820540384 + ], + "ttft_med": 0.21836340427398682, + "peak": 17.5599953 + }, + "noise_range": 0.023237722930996654, + "cands": { + "r2a_gates_moe": { + "dec_med": 50.424029905177704, + "dec_all": [ + 50.424029905177704, + 50.04613092519887, + 50.62708168991173 + ], + "ttft_med": 0.220139741897583, + "peak": 17.567414856, + "paired": [ + 1.0428356184966039, + 1.0320574751381653, + 1.0474635424160863 + ], + "same_output": true, + "err": null, + "doc": "c1 + c2 同时上。", + "speedup": 1.0428356184966039, + "verdict": "差距<5%,不下结论" + }, + "r2b_gates_moe_residual": { + "dec_med": 51.21983893528104, + "dec_all": [ + 51.43588232496421, + 51.21983893528104, + 51.17689779327035 + ], + "ttft_med": 0.22014844417572021, + "peak": 17.566595896, + "paired": [ + 1.0569113877913525, + 1.0565390347402557, + 1.055181811728994 + ], + "same_output": true, + "err": null, + "doc": "第 1 轮三个编译点(消息门、MoE、深度残差)全上。", + "speedup": 1.0565390347402557, + "verdict": "留" + }, + "r2c_fused_mlp_half": { + "dec_med": 49.797729329488135, + "dec_all": [ + 49.797729329488135, + 49.55705031401985, + 50.197880166144515 + ], + "ttft_med": 0.21745789051055908, + "peak": 17.56299792, + "paired": [ + 1.0473919742965152, + 1.035651301526283, + 1.0442818871213315 + ], + "same_output": true, + "err": null, + "doc": "把每层后半段(深度残差取 MLP 输入 → 消息门 → RMSNorm → MoE)合成一张编译图,每层实例一份;\n前半段注意力和卷积状态(带缓存写入)不动。比单独编译各模块多合掉模块之间的接缝。", + "speedup": 1.0442818871213315, + "verdict": "差距<5%,不下结论" + }, + "r2d_fused_half_gates": { + "dec_med": 51.4059422007786, + "dec_all": [ + 51.54030730625611, + 51.4059422007786, + 51.042283946977946 + ], + "ttft_med": 0.21700644493103027, + "peak": 17.569096812, + "paired": [ + 1.0677181572871433, + 1.072562396095457, + 1.0630550931504787 + ], + "same_output": true, + "err": null, + "doc": "r2c 再加上前半段的消息门编译(q/k/v 三个门)。", + "speedup": 1.0677181572871433, + "verdict": "留" + } + } +} \ No newline at end of file diff --git a/examples/inkling2_mini_mlx/rounds/r2/score-holdout.json b/examples/inkling2_mini_mlx/rounds/r2/score-holdout.json new file mode 100644 index 00000000..201a0807 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/score-holdout.json @@ -0,0 +1,105 @@ +{ + "baseline": { + "dec_med": 48.228525817185286, + "dec_all": [ + 48.47356339461423, + 48.228525817185286, + 46.95600603488204, + 48.73639654377695, + 48.817348180285336, + 48.8444825654189, + 48.69671423017082, + 48.768161388328714, + 48.00565380996111, + 48.102844044685014, + 48.21770605207308, + 48.1526772762557, + 47.96052136677736 + ], + "ttft_med": 0.19630658626556396, + "peak": 17.514715684 + }, + "noise_range": 0.039156837131935344, + "cands": { + "r2a_gates_moe": { + "dec_med": 51.04474074076323, + "dec_all": [ + 50.85660088211513, + 51.04474074076323, + 51.08489211463147 + ], + "ttft_med": 0.19790959358215332, + "peak": 17.521604464, + "paired": [ + 1.054492129303139, + 1.0870758620919294, + 1.048187714673263 + ], + "same_output": true, + "err": null, + "doc": "c1 + c2 同时上。", + "speedup": 1.054492129303139, + "verdict": "留" + }, + "r2b_gates_moe_residual": { + "dec_med": 51.562731237754896, + "dec_all": [ + 51.562731237754896, + 51.5694744945997, + 51.031208850454746 + ], + "ttft_med": 0.198167085647583, + "peak": 17.52170266, + "paired": [ + 1.0562378572332667, + 1.0557891451819792, + 1.0479394689598494 + ], + "same_output": true, + "err": null, + "doc": "第 1 轮三个编译点(消息门、MoE、深度残差)全上。", + "speedup": 1.0557891451819792, + "verdict": "留" + }, + "r2c_fused_mlp_half": { + "dec_med": 50.094175012824834, + "dec_all": [ + 50.13723459977536, + 50.094175012824834, + 49.96048055644977 + ], + "ttft_med": 0.19697952270507812, + "peak": 17.516487316, + "paired": [ + 1.0280730946681598, + 1.043505733952328, + 1.0386180183034315 + ], + "same_output": true, + "err": null, + "doc": "把每层后半段(深度残差取 MLP 输入 → 消息门 → RMSNorm → MoE)合成一张编译图,每层实例一份;\n前半段注意力和卷积状态(带缓存写入)不动。比单独编译各模块多合掉模块之间的接缝。", + "speedup": 1.0386180183034315, + "verdict": "差距<5%,不下结论" + }, + "r2d_fused_half_gates": { + "dec_med": 51.2338418260509, + "dec_all": [ + 51.49597712697293, + 51.2338418260509, + 50.81507864179749 + ], + "ttft_med": 0.19498789310455322, + "peak": 17.521638616, + "paired": [ + 1.0679889472833788, + 1.0639873985016102, + 1.059518895826631 + ], + "same_output": true, + "err": null, + "doc": "r2c 再加上前半段的消息门编译(q/k/v 三个门)。", + "speedup": 1.0639873985016102, + "verdict": "留" + } + } +} \ No newline at end of file diff --git a/examples/inkling2_mini_mlx/rsi_mlx.py b/examples/inkling2_mini_mlx/rsi_mlx.py new file mode 100644 index 00000000..64654df0 --- /dev/null +++ b/examples/inkling2_mini_mlx/rsi_mlx.py @@ -0,0 +1,101 @@ +# One InferRSI-style round on a Mac: load the model once, alternate each candidate with the default path, score by measured decode speed. +# Usage: python rsi_mlx.py --split evolve|holdout --max-tokens N --reps R +# Each /cands/*.py (not starting with "_") is a candidate defining apply(model) -> dict of stream_generate kwargs; +# it may patch the model in place (swap ops, wrap mx.compile) and defines revert(model). +# Score: median paired ratio candidate/adjacent-default decode tok/s. Gate: greedy token ids identical to the default path. +# Noise: range of the default path's own runs. Below 5% difference: no conclusion. +import argparse, glob, importlib.util, json, os, statistics, time +import mlx.core as mx +from mlx_lm import load, stream_generate +from mlx_lm.sample_utils import make_sampler + +# Code-generation workload. Split validation, as in earlier InferRSI campaigns: +# candidates are selected on "evolve"; the reported number is the "holdout" split. +SPLITS = { + "evolve": [ + "用 Python 写一个线程安全的 LRU 缓存类,支持 get/put/容量上限和过期时间(TTL),附 3 个 pytest 用例。只输出代码。", + "Write a Go HTTP server with a /healthz endpoint and a rate-limited /api/echo endpoint using a token bucket. Code only.", + ], + "holdout": [ + "用 TypeScript 写一个带重试和指数退避的 fetch 封装,支持超时取消,附用法示例。只输出代码。", + "Write a Rust function that parses a CSV line with quoted fields and escaped quotes, with unit tests. Code only.", + ], +} + +ap = argparse.ArgumentParser() +ap.add_argument("model"); ap.add_argument("round_dir") +ap.add_argument("--max-tokens", type=int, required=True) +ap.add_argument("--reps", type=int, required=True) +ap.add_argument("--only", default=None) +ap.add_argument("--split", choices=list(SPLITS), required=True) +a = ap.parse_args() + +model, tok = load(a.model) +sampler = make_sampler(temp=0.0) +prompts = [tok.apply_chat_template([{"role": "user", "content": p}], add_generation_prompt=True, + reasoning_effort="none") for p in SPLITS[a.split]] + + +def run_once(kw): + out = [] + for p in prompts: + mx.reset_peak_memory(); t0 = time.time(); ttft = None; ids = []; last = None + for r in stream_generate(model, tok, p, max_tokens=a.max_tokens, sampler=sampler, **kw): + if ttft is None: ttft = time.time() - t0 + ids.append(r.token); last = r + out.append(dict(ttft=ttft, dec=last.generation_tps, pre=last.prompt_tps, n=last.generation_tokens, + peak=mx.get_peak_memory() / 1e9, ids=ids)) + return out + + +def summary(runs): + dec = [statistics.mean(x["dec"] for x in r) for r in runs] + ttft = [statistics.mean(x["ttft"] for x in r) for r in runs] + return dict(dec_med=statistics.median(dec), dec_all=dec, ttft_med=statistics.median(ttft), + peak=max(x["peak"] for r in runs for x in r)) + + +log = open(os.path.join(a.round_dir, f"log-{a.split}.jsonl"), "a") +def emit(d): + print(json.dumps(d, ensure_ascii=False), flush=True); log.write(json.dumps(d, ensure_ascii=False) + "\n"); log.flush() + +run_once({}) # warm-up: the first pass compiles Metal kernels; not scored +base_runs = [run_once({})] +ref_ids = [x["ids"] for x in base_runs[0]] +results = {} +cands = sorted(p for p in glob.glob(os.path.join(a.round_dir, "cands", "*.py")) if not os.path.basename(p).startswith("_")) +if a.only: cands = [c for c in cands if a.only in c] +for path in cands: + name = os.path.basename(path)[:-3] + spec = importlib.util.spec_from_file_location(name, path); mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + runs = []; pairs = []; same = True; err = None + try: + for i in range(a.reps): + base_runs.append(run_once({})) # default path, alternated with the candidate + kw = mod.apply(model) + r = run_once(kw); runs.append(r) + pairs.append(summary([r])["dec_med"] / summary([base_runs[-1]])["dec_med"]) # paired with the adjacent default run, cancelling thermal drift + if hasattr(mod, "revert"): mod.revert(model) + same = same and all(x["ids"] == ref for x, ref in zip(r, ref_ids)) + except Exception as e: + err = repr(e) + if hasattr(mod, "revert"): mod.revert(model) + s = summary(runs) if runs else {} + results[name] = dict(s, paired=pairs, same_output=same, err=err, doc=(mod.__doc__ or "").strip()) + emit(dict(cand=name, **{k: v for k, v in results[name].items()})) + +b = summary(base_runs) +noise = (max(b["dec_all"]) - min(b["dec_all"])) / b["dec_med"] +emit(dict(cand="baseline", **b, noise_range=noise)) +for n, r in results.items(): + if r.get("dec_med"): + r["speedup"] = statistics.median(r["paired"]) + r["verdict"] = ("output changed: rejected" if not r["same_output"] else + "<5%: no conclusion" if abs(r["speedup"] - 1) < 0.05 else + "keep" if r["speedup"] > 1 else "reject") +json.dump(dict(baseline=b, noise_range=noise, cands=results), open(os.path.join(a.round_dir, f"score-{a.split}.json"), "w"), + ensure_ascii=False, indent=1) +for n, r in sorted(results.items(), key=lambda kv: -kv[1].get("dec_med", 0)): + print(f"{n:24s} {r.get('dec_med', 0):7.2f} tok/s x{r.get('speedup', 0):.3f} same_output={r['same_output']} {r.get('verdict')} {r['err'] or ''}") +print(f"baseline {b['dec_med']:.2f} tok/s noise range {noise:.1%}") From d2515d61405283080d4190059701569e9bd1307d Mon Sep 17 00:00:00 2001 From: zuiho-kai <31877877+zuiho-kai@users.noreply.github.com> Date: Sat, 21 Nov 2026 16:18:53 +0900 Subject: [PATCH 06/11] docs(inkling2): RSI rounds 3-4 on MLX: 1.491x decode on holdout, token-identical Co-Authored-By: Claude Opus 5.5 --- docs/apple-silicon-playbook.md | 2 +- docs/inkling2-mini-apple-silicon.md | 65 +++- examples/inkling2_mini_mlx/README.md | 9 +- examples/inkling2_mini_mlx/count_ops2.py | 19 ++ .../rounds/r3/cands/_fast.py | 256 ++++++++++++++ .../rounds/r3/cands/r3a_split.py | 4 + .../rounds/r3/cands/r3b_mask.py | 4 + .../rounds/r3/cands/r3c_mask_fuse.py | 4 + .../rounds/r3/cands/r3d_all6.py | 4 + .../rounds/r3/cands/r3e_msg.py | 4 + .../rounds/r3/log-evolve.jsonl | 6 + .../rounds/r3/log-holdout.jsonl | 6 + .../inkling2_mini_mlx/rounds/r3/mechanism.txt | 6 + .../rounds/r3/score-evolve.json | 128 +++++++ .../rounds/r3/score-holdout.json | 128 +++++++ .../rounds/r4/cands/_fast.py | 315 ++++++++++++++++++ .../rounds/r4/cands/r4a_conv_kernel.py | 4 + .../rounds/r4/cands/r4b_ref_r3e.py | 4 + .../rounds/r4/cands/r4c_conv_batchmsg.py | 4 + .../rounds/r4/log-evolve.jsonl | 4 + .../rounds/r4/log-holdout.jsonl | 4 + .../inkling2_mini_mlx/rounds/r4/mechanism.txt | 6 + .../rounds/r4/score-evolve.json | 82 +++++ .../rounds/r4/score-holdout.json | 82 +++++ 24 files changed, 1137 insertions(+), 13 deletions(-) create mode 100644 examples/inkling2_mini_mlx/count_ops2.py create mode 100644 examples/inkling2_mini_mlx/rounds/r3/cands/_fast.py create mode 100644 examples/inkling2_mini_mlx/rounds/r3/cands/r3a_split.py create mode 100644 examples/inkling2_mini_mlx/rounds/r3/cands/r3b_mask.py create mode 100644 examples/inkling2_mini_mlx/rounds/r3/cands/r3c_mask_fuse.py create mode 100644 examples/inkling2_mini_mlx/rounds/r3/cands/r3d_all6.py create mode 100644 examples/inkling2_mini_mlx/rounds/r3/cands/r3e_msg.py create mode 100644 examples/inkling2_mini_mlx/rounds/r3/log-evolve.jsonl create mode 100644 examples/inkling2_mini_mlx/rounds/r3/log-holdout.jsonl create mode 100644 examples/inkling2_mini_mlx/rounds/r3/mechanism.txt create mode 100644 examples/inkling2_mini_mlx/rounds/r3/score-evolve.json create mode 100644 examples/inkling2_mini_mlx/rounds/r3/score-holdout.json create mode 100644 examples/inkling2_mini_mlx/rounds/r4/cands/_fast.py create mode 100644 examples/inkling2_mini_mlx/rounds/r4/cands/r4a_conv_kernel.py create mode 100644 examples/inkling2_mini_mlx/rounds/r4/cands/r4b_ref_r3e.py create mode 100644 examples/inkling2_mini_mlx/rounds/r4/cands/r4c_conv_batchmsg.py create mode 100644 examples/inkling2_mini_mlx/rounds/r4/log-evolve.jsonl create mode 100644 examples/inkling2_mini_mlx/rounds/r4/log-holdout.jsonl create mode 100644 examples/inkling2_mini_mlx/rounds/r4/mechanism.txt create mode 100644 examples/inkling2_mini_mlx/rounds/r4/score-evolve.json create mode 100644 examples/inkling2_mini_mlx/rounds/r4/score-holdout.json diff --git a/docs/apple-silicon-playbook.md b/docs/apple-silicon-playbook.md index 0d6a1b97..cddd32f2 100644 --- a/docs/apple-silicon-playbook.md +++ b/docs/apple-silicon-playbook.md @@ -67,7 +67,7 @@ From [Inkling 2 Mini on Apple silicon](inkling2-mini-apple-silicon.md) (measured | The release checkpoint (ModelOpt NVFP4/MXFP8) loads in neither MLX nor llama.cpp. | Apple runtimes need their own quantized formats. | Start from a community conversion; pin repository, revision and file SHA-256. | measured | | Shards arrived larger than published after a segmented download. | curl `--retry` inside a ranged, appending segment resends from the range start. | Verify each file against the Hub `lfs.oid` before the first load. | measured | | A 512-token cap ended inside the thinking block although reasoning effort was set to `none`. | The template value only adds a system line. | Size generation caps for thinking plus answer; count both. | measured | -| A looped MoE model decoded at about 48 tok/s on MLX while weight traffic allowed far more. | 48 layer visits per token, each with many small ops: launch- and graph-building-bound. | Search compilation of pure sub-blocks first; per-module compiles gave 1–3%, a fused per-layer half plus gates 6.4% on held-out prompts. | measured | +| A looped MoE model decoded at about 48 tok/s on MLX while weight traffic allowed far more. | 48 layer visits per token, each with many small ops: launch- and graph-building-bound. | Profile one layer visit at decode shape before compiling. Per-module compiles gave 1–3% and a fused layer half 6.4%; two compiled graphs per layer visit around the cache write, a one-kernel mask, fused projections and one-kernel short convs gave 1.49× on held-out prompts (ops per step 7,624 → 3,664). | measured | | During a search round the default path drifted from 47.4 to 48.9 tok/s. | Warm-up/thermal drift over minutes. | Pair each candidate run with the adjacent default run; an all-runs median inflated one candidate from 1.048 to 1.053. | measured | ## Benchmarking a shared Mac diff --git a/docs/inkling2-mini-apple-silicon.md b/docs/inkling2-mini-apple-silicon.md index 48cfe412..40adff2c 100644 --- a/docs/inkling2-mini-apple-silicon.md +++ b/docs/inkling2-mini-apple-silicon.md @@ -1,10 +1,14 @@ -# Inkling 2 Mini on Apple silicon: local adaptation and one RSI round +# Inkling 2 Mini on Apple silicon: local adaptation and four RSI rounds **On one M5 Max, MLX ran Inkling 2 Mini at 47.0 tok/s decode and 1,677 tok/s -prefill (512-token prompt), ahead of llama.cpp at 41.7 and 998. Two search -rounds then found a compile patch that raised MLX decode by 6.4% (paired -median, held-out prompts, run-to-run noise 3.9%) with token-identical greedy -output.** These are maintainer measurements without an acceptance issue; they +prefill (512-token prompt), ahead of llama.cpp at 41.7 and 998. Four search +rounds then raised MLX decode 1.491× (paired median, held-out prompts, +run-to-run noise 9.2%) with token-identical greedy output: about 47 → 70 +tok/s. Rounds 1–2 compiled sub-blocks (1.064×); rounds 3–4 rebuilt the +single-token layer path (two compiled graphs per layer visit around the cache +write, a one-kernel attention mask, fused projections, one-kernel short +convolutions) and halved the ops per step.** The patch is published as +[zuiho-kai/inkling2-mlx-fastpath](https://github.com/zuiho-kai/inkling2-mlx-fastpath). These are maintainer measurements without an acceptance issue; they are not an InferRSI PASS. [Inkling 2 Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) is a @@ -116,10 +120,52 @@ rest is GPU time from fewer, fused kernels. The winning patch is `rounds/r2/cands/r2d_fused_half_gates.py`; it changes no weights and no numerics outside the compiled regions. +**Round 3** (proposed from a per-module profile: at decode shape the +attention half of a layer visit cost ~240 µs against ~90 µs for the MoE, +nearly all in small ops). Each layer visit for B = 1, L = 1 is split into +three parts: everything before the KV-cache write as one compiled graph, the +cache write and attention eager (their length changes every step), everything +after as a second compiled graph; conv states become explicit inputs and +outputs. Options on top: the attention mask/bias (arange, compare, gather, +where, cast) as one Metal kernel; q/r/gate/sink projections concatenated into +one quantized matmul (all six in sliding layers without loop messages); +message-gate forget/message projections as one matmul. + +| Candidate | Evolve (noise 9.1%) | Holdout (noise 6.8%) | Output | +| --- | ---: | ---: | --- | +| three-part layer path | 1.290 | 1.290 | identical | +| + mask kernel | 1.354 | 1.353 | identical | +| + fused q/r/gate/sink | 1.430 | **1.419** | identical | +| + all six projections in one | 1.424 | 1.396 | identical | +| + fused message-gate projections | 1.375 | 1.392 | identical | + +The last two are within 5% of the third: no conclusion. + +**Round 4** (proposed from the op count after round 3: the 4-tap short +convolutions, four per layer visit, were each a concatenate, two casts, a +generic convolution, an add/divide and a state slice). + +| Candidate | Evolve (noise 12.2%) | Holdout (noise 9.2%) | Output | +| --- | ---: | ---: | --- | +| **round-3 options + one-kernel short convs** | **1.486** | **1.491** (1.462, 1.491, 1.515) | identical | +| round-3 options, re-run as reference | 1.327 | 1.409 | identical | +| + q/k/v message gates as one batched matmul | 1.516 | 1.501 | identical | + +Batched gates are within 5% of the shipped candidate: no conclusion, not +shipped. Default-path tok/s varied 43–48 across rounds (thermal); only the +adjacent-paired ratios are compared. + +Mechanism, round 4 candidate against default (one decode step, median of 50 +steps, two alternating processes each; non-view ops counted from the step +graph): step 22.1 → 15.3 ms, Python graph building 3.9 → 1.8 ms, non-view +ops per step 7,624 → 3,664. The weights read per token (~2.5 GB) would take +about 5 ms at the machine's bandwidth, so the step remains launch-bound. + **How candidates were produced.** The candidates were written by the assistant running the campaign (Claude Code, `claude-opus-5-5`) from reading -the model source; no automatic mutation chain was used. Round 2 was proposed -after reading round 1's scores. +the model source; no automatic mutation chain was used. Each round was +proposed after reading the previous round's scores and, for rounds 3–4, a +per-module profile and an op count of one decode step. ## What we did not establish @@ -130,7 +176,8 @@ after reading round 1's scores. prefill-heavy work was measured only by the 512-token benchmark. - Quality was checked only as greedy-token identity on four prompts and one passing generated test file; no benchmark accuracy. -- The compile patch was not measured inside `mlx_lm.server` and was not - proposed upstream. +- The patch was not measured inside `mlx_lm.server` and was not proposed to + mlx-lm. "Identical" means identical greedy tokens on the prompts tested; the + compiled graphs are not proven bit-exact for every input. - llama.cpp was run with default settings; its flags were not searched, so the route comparison is default against default. diff --git a/examples/inkling2_mini_mlx/README.md b/examples/inkling2_mini_mlx/README.md index 55296e06..dcd44079 100644 --- a/examples/inkling2_mini_mlx/README.md +++ b/examples/inkling2_mini_mlx/README.md @@ -11,16 +11,19 @@ Apple-silicon Mac with mlx-lm from ml-explore/mlx-lm#2086 (`0c4c991`). | `bench_llama.sh` | `llama-bench` 512/128 × 3 with `/usr/bin/time -l` | | `rsi_mlx.py` | One search round: load once, alternate each candidate with the default path, paired scoring, token-identity gate, evolve/holdout split | | `mechanism.py` | Per-step graph-building and step time, default vs one candidate | -| `rounds/r1`, `rounds/r2` | Candidates (`cands/*.py`) and the JSONL/JSON records each round wrote | +| `count_ops2.py` | Ops in one decode step's graph (total and non-view), default or one candidate | +| `rounds/r1` … `rounds/r4` | Candidates (`cands/*.py`) and the JSONL/JSON records each round wrote | ```bash python rsi_mlx.py rounds/r2 --split evolve --max-tokens 256 --reps 3 python rsi_mlx.py rounds/r2 --split holdout --max-tokens 256 --reps 3 -python mechanism.py rounds/r2/cands/r2d_fused_half_gates.py +python mechanism.py rounds/r4/cands/r4a_conv_kernel.py +python count_ops2.py rounds/r4/cands/r4a_conv_kernel.py ``` Round 1 records predate adjacent pairing; its paired ratios in the doc were recomputed from the logged run order (default run `1 + 3k + i` precedes repetition `i` of candidate `k`). Workload prompts are kept verbatim, including the Chinese ones. Run one runtime at a time; another GPU job on the -machine invalidates the round. +machine invalidates the round. The round-4 winner is packaged as +[zuiho-kai/inkling2-mlx-fastpath](https://github.com/zuiho-kai/inkling2-mlx-fastpath). diff --git a/examples/inkling2_mini_mlx/count_ops2.py b/examples/inkling2_mini_mlx/count_ops2.py new file mode 100644 index 00000000..6954cc5a --- /dev/null +++ b/examples/inkling2_mini_mlx/count_ops2.py @@ -0,0 +1,19 @@ +# Count op nodes in one decode-step graph (total and non-view, the latter ~ GPU kernels); an optional candidate file is applied first. +import sys, collections, re, importlib.util +import mlx.core as mx +from mlx_lm import load +from mlx_lm.models.cache import make_prompt_cache +model, tok = load(sys.argv[1]) +if len(sys.argv) > 2: + spec = importlib.util.spec_from_file_location("c", sys.argv[2]); m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m); m.apply(model) +ids = mx.array(tok.encode("def lru_cache(capacity):"))[None] +cache = make_prompt_cache(model); out = model(ids, cache=cache); mx.eval(out) +for _ in range(3): + y = mx.argmax(out[:, -1:], -1); out = model(y, cache=cache); mx.eval(out) +out = model(y, cache=cache) +mx.export_to_dot("step.dot", out) +txt = open("step.dot").read() +ops = re.findall(r'label ="?([A-Za-z_]+)', txt) +VIEW = {"Broadcast","Transpose","Reshape","Squeeze","Flatten","Unflatten","ExpandDims","StopGradient","Slice","Split"} +c = collections.Counter(o if not o.startswith("Compiled") else "Compiled*" for o in ops) +print("total", sum(c.values()), "non-view", sum(v for k, v in c.items() if k not in VIEW)); print(c.most_common(30)) diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/_fast.py b/examples/inkling2_mini_mlx/rounds/r3/cands/_fast.py new file mode 100644 index 00000000..c3c1f3ee --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/_fast.py @@ -0,0 +1,256 @@ +# 第 3 轮共用:解码一步(B=1, L=1)的每层走「前段合图 → 写缓存+注意力 → 后段合图」三段。 +# 前段:残差取输入、消息门、归一化、q/k/v/r/门/汇点投影、短卷积、裁剪、相对位置偏置,一张编译图; +# 中段:KV 缓存写入和注意力(长度每步变,不编译); +# 后段:门控、输出投影、卷积、残差、MoE、卷积,一张编译图。卷积状态当作显式输入输出。 +# 选项:mask_kernel 把注意力遮罩的十几个小算子换成一个 Metal 内核;fuse_proj 把共享输入的投影拼成一次量化矩阵乘。 +import mlx.core as mx +from mlx import nn +from mlx_lm.models.inkling2 import layers as L +from mlx_lm.models.cache import RotatingKVCache + +OPTS = dict(mask_kernel=False, fuse_proj=False, fuse_all6=False, fuse_msg=False, conv_ew=False) +_DEF = dict(OPTS) +_orig = L.DecoderLayer.__call__ +_cache = {} +_fused = {} + + +def _sliding(self): + return isinstance(self, L.SlidingDecoderLayer) + + +def _proj(self, attn, qi, ki, vi, same_kv): + if OPTS["fuse_proj"]: + f = _fused.get(id(attn)) + if f is not None: + outs = f["qrgs"](qi) + qp, rp, gate, sinks = mx.split(outs, f["qrgs_split"], axis=-1) + if same_kv and OPTS["fuse_all6"]: # 滑窗层无消息:q/k/v 同一输入,六个投影一次 + qp, kp, vp, rp, gate, sinks = mx.split(f["all6"](qi), f["all6_split"], axis=-1) + return qp, kp, vp, rp, gate, sinks + kp = attn.k_proj(ki); vp = attn.v_proj(vi) + return qp, kp, vp, rp, gate, sinks + return attn.q_proj(qi), attn.k_proj(ki), attn.v_proj(vi), attn.r_proj(qi), attn.gate_proj(qi), attn.sink_proj(qi) + + +def _pre(self, has_bs, has_msg): + sliding = _sliding(self); attn = self.self_attn + + def body(*args): + it = iter(args) + h = next(it); bs = next(it) if has_bs else None + cs0 = next(it); cs1 = next(it) + msg = next(it) if has_msg else None + pos = None if sliding else next(it) + dtype = h.dtype + nr = L.unweighted_rms_norm(h.astype(mx.float32)) + ai = self.attention_input(bs, nr) if self.attention_input is not None else None + if sliding: + hs = ai[..., 0, :] if ai is not None else h + if msg is None: + qi = ki = vi = self.input_layernorm(hs).astype(dtype) + else: + st = self._mix_messages((hs,) * 3, msg) + qi, ki, vi = [self.input_layernorm(s).astype(dtype) for s in st] + else: + q, k, v = (ai[..., i, :] for i in range(3)) + if msg is not None: + q, k, v = self._mix_messages((q, k, v), msg) + qi = self.q_input_layernorm(q).astype(dtype) + ki = self.k_input_layernorm(k).astype(dtype) + vi = self.v_input_layernorm(v).astype(dtype) + qp, kp, vp, rp, gate, sinks = _proj(self, attn, qi, ki, vi, sliding and msg is None) + if sliding: + ks, w0 = attn.k_sconv(kp, cs0, None); vs, w1 = attn.v_sconv(vp, cs1, None) + else: + ks, w0 = attn.k_sconv(kp, ki, cs0, None); vs, w1 = attn.v_sconv(vp, vi, cs1, None) + B, Ln = 1, 1 + qs = qp.reshape(B, Ln, attn.num_heads, -1) + ks = ks.reshape(B, Ln, attn.num_key_value_heads, -1) + vs = vs.reshape(B, Ln, attn.num_key_value_heads, -1) + rs = rp.reshape(B, Ln, attn.num_heads, -1) + qs = attn.q_l2_clip(qs); ks = attn.k_l2_clip(ks) + vs = L.unweighted_rms_norm(vs.astype(mx.float32), attn.eps).astype(vs.dtype) + rs = attn.rel_l2_clip(rs) + if not sliding: + tau = 1.0 + attn.log_scaling_alpha * mx.log(mx.maximum((pos + 1) / attn.log_scaling_n_floor, 1.0)) + tau = tau[..., None, None] + qs = (qs.astype(mx.float32) * tau).astype(qs.dtype) + rs = (rs.astype(mx.float32) * tau).astype(rs.dtype) + pb = attn.rel_logits_proj(rs) + return (qs.transpose(0, 2, 1, 3), ks.transpose(0, 2, 1, 3), vs.transpose(0, 2, 1, 3), + pb, sinks, gate, nr, w0[:, 1:], w1[:, 1:]) + return mx.compile(body) + + +def _post(self, has_bs, has_msg): + attn = self.self_attn + + def body(*args): + it = iter(args) + out = next(it); gate = next(it); h = next(it); nr = next(it) + bs = next(it) if has_bs else None + cs2 = next(it); cs3 = next(it) + msg = next(it) if has_msg else None + dtype = h.dtype + ao = out.reshape(1, 1, -1).astype(mx.float32) * mx.sigmoid(gate.astype(mx.float32)) + hs = attn.o_proj(ao.astype(gate.dtype)) + hs, w2 = self.post_attention_convolution(hs, cs2, None) + residual = hs = hs.astype(mx.float32) + if self.is_block_boundary: + block = nr[None] + bs = block if bs is None else mx.concatenate([bs, block]) + else: + residual = hs = h.astype(mx.float32) + hs + ns = L.unweighted_rms_norm(hs) + hs = self.mlp_input(bs, ns)[..., 0, :] + if msg is not None: + hs = self.mlp_message_gate(hs, msg[..., 3, :]) + hs = self.mlp(self.post_attention_layernorm(hs).astype(dtype)) + hs, w3 = self.post_mlp_convolution(hs, cs3, None) + hs = residual + hs.astype(mx.float32) + return hs.astype(dtype), bs, w2[:, 1:], w3[:, 1:] + return mx.compile(body) + + +def _get(kind, self, has_bs, has_msg): + key = (kind, id(self), has_bs, has_msg) + f = _cache.get(key) + if f is None: + f = _cache[key] = (_pre if kind == "pre" else _post)(self, has_bs, has_msg) + return f + + +# ---- 遮罩内核:mask[h, s] = 可见 ? (距离= S) return; + int k = (int)s; + if (rot) { k = ((k - idx) % S + S) % S; } + int kpos = k + (end - S); + int dist = (end - 1) - kpos; + bool vis = (dist >= 0) && (kpos >= 0); + if (window > 0) vis = vis && (dist < window); + T v; + if (!vis) v = T(-INFINITY); + else if (dist < R) v = pb[h * R + dist]; + else v = T(0); + out[h * S + s] = v; +""" +_mask_k = None + + +def _attention_fast(attn, q, keys, values, pb, sinks, kvc): + global _mask_k + if _mask_k is None: + _mask_k = mx.fast.metal_kernel(name="ink_mask", input_names=["pb", "params"], output_names=["out"], source=_MASK_SRC) + S = keys.shape[2]; end = kvc.offset; H = pb.shape[2]; R = pb.shape[-1] + rot = isinstance(kvc, RotatingKVCache) + params = mx.array([S, end, kvc._idx if rot else 0, int(rot), attn.sliding_window, R], mx.int32) + mask = _mask_k(inputs=[pb[0, 0], params], template=[("T", q.dtype)], grid=(S, H, 1), threadgroup=(min(S, 256), 1, 1), + output_shapes=[(H, S)], output_dtypes=[q.dtype])[0] + o = mx.fast.scaled_dot_product_attention(q, keys, values, scale=attn.scaling, mask=mask[None, :, None, :], + sinks=sinks[0, 0].astype(q.dtype)) + return o.transpose(0, 2, 1, 3) + + +def call(self, hidden_states, block_states, *, cache=None, conv_mask=None, messages=None): + conv = cache[1] if cache is not None else None + if conv is None or hidden_states.shape[:2] != (1, 1) or conv.cache[0] is None or conv_mask is not None: + return _orig(self, hidden_states, block_states, cache=cache, conv_mask=conv_mask, messages=messages) + kvc = cache[0]; attn = self.self_attn + has_bs = block_states is not None; has_msg = messages is not None + args = [hidden_states] + ([block_states] if has_bs else []) + [conv[0], conv[1]] + ([messages] if has_msg else []) + if not _sliding(self): + args.append(mx.array([kvc.offset], mx.float32)) + q, k, v, pb, sinks, gate, nr, c0, c1 = _get("pre", self, has_bs, has_msg)(*args) + keys, values = kvc.update_and_fetch(k, v) + if OPTS["mask_kernel"]: + out = _attention_fast(attn, q, keys, values, pb, sinks, kvc) + else: + out = attn._attention(q, keys, values, pb, sinks, kvc) + args = [out, gate, hidden_states, nr] + ([block_states] if has_bs else []) + [conv[2], conv[3]] + ([messages] if has_msg else []) + hs, bs, c2, c3 = _get("post", self, has_bs, has_msg)(*args) + conv[0], conv[1], conv[2], conv[3] = c0, c1, c2, c3 + return hs, bs + + +def _cat_q(ps): + f = nn.QuantizedLinear.__new__(nn.QuantizedLinear); nn.Module.__init__(f) + f.group_size, f.bits, f.mode = ps[0].group_size, ps[0].bits, getattr(ps[0], "mode", "affine") + f.weight = mx.concatenate([p.weight for p in ps]); f.scales = mx.concatenate([p.scales for p in ps]) + f.biases = mx.concatenate([p.biases for p in ps]) + f.freeze(); mx.eval(f.parameters()) + sizes = [p.weight.shape[0] for p in ps] + return f, [sum(sizes[:i + 1]) for i in range(len(ps) - 1)] + + +def _fuse(model): + for layer in model.model.language_model.layers: + a = layer.self_attn + if id(a) in _fused: continue + ps = [a.q_proj, a.r_proj, a.gate_proj, a.sink_proj] + if not all(isinstance(p, nn.QuantizedLinear) for p in ps): continue + d = _fused[id(a)] = {} + d["qrgs"], d["qrgs_split"] = _cat_q(ps) + d["all6"], d["all6_split"] = _cat_q([a.q_proj, a.k_proj, a.v_proj, a.r_proj, a.gate_proj, a.sink_proj]) + + +# ---- 消息门:遗忘门和消息门两个投影共享输入,拼成一次矩阵乘 ---- +_orig_msg = L.MessageGate.__call__ +_msgw = {} + + +def _fuse_msg(model): + for layer in model.model.language_model.layers: + for n in ("q_message_gate", "k_message_gate", "v_message_gate", "mlp_message_gate"): + g = getattr(layer, n, None) + if g is None or id(g) in _msgw: continue + assert not isinstance(g.forget_gate_proj, nn.QuantizedLinear) + w = mx.concatenate([g.forget_gate_proj.weight, g.message_gate_proj.weight]); mx.eval(w) + _msgw[id(g)] = w + + +def _msg_call(self, hidden_states, message): + states = mx.concatenate([L.unweighted_rms_norm(hidden_states), L.unweighted_rms_norm(message)], axis=-1) + states = self.down_proj(states.astype(self.down_proj.weight.dtype)).astype(mx.float32) + w = _msgw.get(id(self)) + if w is None: + return _orig_msg(self, hidden_states, message) + fm = mx.tanh(states @ w.T) + forget_gate, message_gate = mx.split(fm, 2, axis=-1) + return (1 - forget_gate) * hidden_states + message_gate * message + + +# ---- 短卷积:深度卷积核 4 抽头,L=1 时写成逐元素乘加,免掉通用卷积内核 ---- +_orig_conv = L.ShortConvolution.__call__ + + +def _conv_call(self, hidden_states, conv_state, conv_mask): + if hidden_states.shape[1] != 1 or conv_state is None or conv_mask is not None: + return _orig_conv(self, hidden_states, conv_state, conv_mask) + K = self.kernel_size + residual = hidden_states.astype(mx.float32) + window = L._conv_input(hidden_states, conv_state, conv_mask, K) + w = self.conv1d.weight # (C, K, 1) + x = window.astype(w.dtype) + out = (x.astype(mx.float32) * w[:, :, 0].T[None].astype(mx.float32)).sum(axis=1, keepdims=True).astype(w.dtype) + out = (out.astype(mx.float32) + residual) / (K + 1) ** 0.5 + return out.astype(hidden_states.dtype), window + + +def apply(model, **opts): + OPTS.update(_DEF); OPTS.update(opts); _cache.clear() + if OPTS["fuse_proj"]: _fuse(model) + L.DecoderLayer.__call__ = call + if OPTS["fuse_msg"]: _fuse_msg(model); L.MessageGate.__call__ = _msg_call + if OPTS["conv_ew"]: L.ShortConvolution.__call__ = _conv_call + return {} + + +def revert(model): + L.DecoderLayer.__call__ = _orig; _cache.clear() + L.MessageGate.__call__ = _orig_msg; L.ShortConvolution.__call__ = _orig_conv diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3a_split.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3a_split.py new file mode 100644 index 00000000..975fd5af --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3a_split.py @@ -0,0 +1,4 @@ +"""第 3 轮 a_split:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 []。""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, ) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3b_mask.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3b_mask.py new file mode 100644 index 00000000..4b20a7db --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3b_mask.py @@ -0,0 +1,4 @@ +"""第 3 轮 b_mask:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True]。""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, mask_kernel=True) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3c_mask_fuse.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3c_mask_fuse.py new file mode 100644 index 00000000..6825bbf3 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3c_mask_fuse.py @@ -0,0 +1,4 @@ +"""第 3 轮 c_mask_fuse:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True]。""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3d_all6.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3d_all6.py new file mode 100644 index 00000000..e0f4b62a --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3d_all6.py @@ -0,0 +1,4 @@ +"""第 3 轮 d_all6:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True]。""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3e_msg.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3e_msg.py new file mode 100644 index 00000000..c2d359a7 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3e_msg.py @@ -0,0 +1,4 @@ +"""第 3 轮 e_msg:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r3/log-evolve.jsonl b/examples/inkling2_mini_mlx/rounds/r3/log-evolve.jsonl new file mode 100644 index 00000000..fd610258 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/log-evolve.jsonl @@ -0,0 +1,6 @@ +{"cand": "r3a_split", "dec_med": 58.597822881124515, "dec_all": [58.597822881124515, 59.36159852090833, 57.87520297279241], "ttft_med": 0.22028803825378418, "peak": 17.559962404, "paired": [1.2901202935255751, 1.328994706191247, 1.2751990680281606], "same_output": true, "err": null, "doc": "第 3 轮 a_split:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 []。"} +{"cand": "r3b_mask", "dec_med": 64.39439576221922, "dec_all": [64.39439576221922, 64.50176422427727, 62.196659874623435], "ttft_med": 0.21064341068267822, "peak": 17.55992794, "paired": [1.3974349092167846, 1.3538424803419962, 1.337859365648511], "same_output": true, "err": null, "doc": "第 3 轮 b_mask:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True]。"} +{"cand": "r3c_mask_fuse", "dec_med": 65.87582522354171, "dec_all": [66.85689733335586, 60.07978068716797, 65.87582522354171], "ttft_med": 0.21292757987976074, "peak": 18.031822244, "paired": [1.4295186687269676, 1.3107599987142853, 1.475380798263453], "same_output": true, "err": null, "doc": "第 3 轮 c_mask_fuse:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True]。"} +{"cand": "r3d_all6", "dec_med": 62.6047822368255, "dec_all": [62.6047822368255, 59.526029134274545, 65.52385727844654], "ttft_med": 0.21091711521148682, "peak": 18.031755524, "paired": [1.4405800103914217, 1.2814983090199807, 1.4239628466388194], "same_output": true, "err": null, "doc": "第 3 轮 d_all6:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True]。"} +{"cand": "r3e_msg", "dec_med": 63.991903394997216, "dec_all": [63.92982974662121, 65.6092751424128, 63.991903394997216], "ttft_med": 0.21320867538452148, "peak": 18.097880132, "paired": [1.3747761961567637, 1.4176718066732137, 1.37302375595508], "same_output": true, "err": null, "doc": "第 3 轮 e_msg:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。"} +{"cand": "baseline", "dec_med": 46.047785588104894, "dec_all": [44.99154466912073, 45.42043340857104, 44.666542495893125, 45.38522998004131, 46.08042588424395, 47.64347784978898, 46.48968454503763, 46.76881722215917, 45.83583626758505, 44.65004919481033, 43.458038974048435, 46.450337636260144, 46.01514529196584, 46.50199059697087, 46.27959365036335, 46.60655223003351], "ttft_med": 0.2206212282180786, "peak": 18.0883913, "noise_range": 0.09089338004609131} diff --git a/examples/inkling2_mini_mlx/rounds/r3/log-holdout.jsonl b/examples/inkling2_mini_mlx/rounds/r3/log-holdout.jsonl new file mode 100644 index 00000000..84524c79 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/log-holdout.jsonl @@ -0,0 +1,6 @@ +{"cand": "r3a_split", "dec_med": 59.815185922194104, "dec_all": [59.815185922194104, 58.116759316609006, 60.00709217739403], "ttft_med": 0.1961524486541748, "peak": 17.51477316, "paired": [1.2901830930466067, 1.2212808959664423, 1.2992045891748238], "same_output": true, "err": null, "doc": "第 3 轮 a_split:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 []。"} +{"cand": "r3b_mask", "dec_med": 64.60535174358287, "dec_all": [64.47242975670096, 64.88611319582448, 64.60535174358287], "ttft_med": 0.18941152095794678, "peak": 17.514705052, "paired": [1.3532531059417017, 1.355208418973997, 1.349479906786514], "same_output": true, "err": null, "doc": "第 3 轮 b_mask:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True]。"} +{"cand": "r3c_mask_fuse", "dec_med": 66.82440670116867, "dec_all": [66.61798282077788, 66.86022128070815, 66.82440670116867], "ttft_med": 0.18910002708435059, "peak": 17.98661582, "paired": [1.4237253603067872, 1.4040673680084226, 1.419207536882739], "same_output": true, "err": null, "doc": "第 3 轮 c_mask_fuse:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True]。"} +{"cand": "r3d_all6", "dec_med": 66.66493160403824, "dec_all": [66.66493160403824, 66.80098053708868, 64.81861677574682], "ttft_med": 0.18864834308624268, "peak": 17.98495168, "paired": [1.4005889493981019, 1.3960647918871552, 1.3468629042926332], "same_output": true, "err": null, "doc": "第 3 轮 d_all6:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True]。"} +{"cand": "r3e_msg", "dec_med": 66.20176969252506, "dec_all": [66.20176969252506, 66.30925957405773, 64.2510970652737], "ttft_med": 0.19022655487060547, "peak": 18.048848016, "paired": [1.4179043613368552, 1.3922370512388482, 1.3556241642347557], "same_output": true, "err": null, "doc": "第 3 轮 e_msg:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。"} +{"cand": "baseline", "dec_med": 47.59225545047404, "dec_all": [44.875001491678844, 46.36178093215281, 47.58672596005785, 46.1875617415321, 47.642550734687504, 47.87906589670432, 47.87425986758568, 46.7913157116362, 47.618955332282226, 47.085718589084685, 47.597784940890236, 47.84948444032406, 48.12562330520884, 46.68986956927579, 47.627851532218635, 47.395951444656625], "ttft_med": 0.19784337282180786, "peak": 18.043229216, "noise_range": 0.0683014869281136} diff --git a/examples/inkling2_mini_mlx/rounds/r3/mechanism.txt b/examples/inkling2_mini_mlx/rounds/r3/mechanism.txt new file mode 100644 index 00000000..2d32a483 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/mechanism.txt @@ -0,0 +1,6 @@ +base build_ms=3.95 step_ms=22.01 +cand build_ms=2.13 step_ms=16.43 +base build_ms=3.99 step_ms=22.55 +cand build_ms=1.87 step_ms=15.95 +total 12184 non-view 7624 +total 7176 non-view 4224 diff --git a/examples/inkling2_mini_mlx/rounds/r3/score-evolve.json b/examples/inkling2_mini_mlx/rounds/r3/score-evolve.json new file mode 100644 index 00000000..9de69c82 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/score-evolve.json @@ -0,0 +1,128 @@ +{ + "baseline": { + "dec_med": 46.047785588104894, + "dec_all": [ + 44.99154466912073, + 45.42043340857104, + 44.666542495893125, + 45.38522998004131, + 46.08042588424395, + 47.64347784978898, + 46.48968454503763, + 46.76881722215917, + 45.83583626758505, + 44.65004919481033, + 43.458038974048435, + 46.450337636260144, + 46.01514529196584, + 46.50199059697087, + 46.27959365036335, + 46.60655223003351 + ], + "ttft_med": 0.2206212282180786, + "peak": 18.0883913 + }, + "noise_range": 0.09089338004609131, + "cands": { + "r3a_split": { + "dec_med": 58.597822881124515, + "dec_all": [ + 58.597822881124515, + 59.36159852090833, + 57.87520297279241 + ], + "ttft_med": 0.22028803825378418, + "peak": 17.559962404, + "paired": [ + 1.2901202935255751, + 1.328994706191247, + 1.2751990680281606 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 a_split:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 []。", + "speedup": 1.2901202935255751, + "verdict": "留" + }, + "r3b_mask": { + "dec_med": 64.39439576221922, + "dec_all": [ + 64.39439576221922, + 64.50176422427727, + 62.196659874623435 + ], + "ttft_med": 0.21064341068267822, + "peak": 17.55992794, + "paired": [ + 1.3974349092167846, + 1.3538424803419962, + 1.337859365648511 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 b_mask:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True]。", + "speedup": 1.3538424803419962, + "verdict": "留" + }, + "r3c_mask_fuse": { + "dec_med": 65.87582522354171, + "dec_all": [ + 66.85689733335586, + 60.07978068716797, + 65.87582522354171 + ], + "ttft_med": 0.21292757987976074, + "peak": 18.031822244, + "paired": [ + 1.4295186687269676, + 1.3107599987142853, + 1.475380798263453 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 c_mask_fuse:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True]。", + "speedup": 1.4295186687269676, + "verdict": "留" + }, + "r3d_all6": { + "dec_med": 62.6047822368255, + "dec_all": [ + 62.6047822368255, + 59.526029134274545, + 65.52385727844654 + ], + "ttft_med": 0.21091711521148682, + "peak": 18.031755524, + "paired": [ + 1.4405800103914217, + 1.2814983090199807, + 1.4239628466388194 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 d_all6:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True]。", + "speedup": 1.4239628466388194, + "verdict": "留" + }, + "r3e_msg": { + "dec_med": 63.991903394997216, + "dec_all": [ + 63.92982974662121, + 65.6092751424128, + 63.991903394997216 + ], + "ttft_med": 0.21320867538452148, + "peak": 18.097880132, + "paired": [ + 1.3747761961567637, + 1.4176718066732137, + 1.37302375595508 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 e_msg:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。", + "speedup": 1.3747761961567637, + "verdict": "留" + } + } +} \ No newline at end of file diff --git a/examples/inkling2_mini_mlx/rounds/r3/score-holdout.json b/examples/inkling2_mini_mlx/rounds/r3/score-holdout.json new file mode 100644 index 00000000..3ff64c97 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/score-holdout.json @@ -0,0 +1,128 @@ +{ + "baseline": { + "dec_med": 47.59225545047404, + "dec_all": [ + 44.875001491678844, + 46.36178093215281, + 47.58672596005785, + 46.1875617415321, + 47.642550734687504, + 47.87906589670432, + 47.87425986758568, + 46.7913157116362, + 47.618955332282226, + 47.085718589084685, + 47.597784940890236, + 47.84948444032406, + 48.12562330520884, + 46.68986956927579, + 47.627851532218635, + 47.395951444656625 + ], + "ttft_med": 0.19784337282180786, + "peak": 18.043229216 + }, + "noise_range": 0.0683014869281136, + "cands": { + "r3a_split": { + "dec_med": 59.815185922194104, + "dec_all": [ + 59.815185922194104, + 58.116759316609006, + 60.00709217739403 + ], + "ttft_med": 0.1961524486541748, + "peak": 17.51477316, + "paired": [ + 1.2901830930466067, + 1.2212808959664423, + 1.2992045891748238 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 a_split:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 []。", + "speedup": 1.2901830930466067, + "verdict": "留" + }, + "r3b_mask": { + "dec_med": 64.60535174358287, + "dec_all": [ + 64.47242975670096, + 64.88611319582448, + 64.60535174358287 + ], + "ttft_med": 0.18941152095794678, + "peak": 17.514705052, + "paired": [ + 1.3532531059417017, + 1.355208418973997, + 1.349479906786514 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 b_mask:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True]。", + "speedup": 1.3532531059417017, + "verdict": "留" + }, + "r3c_mask_fuse": { + "dec_med": 66.82440670116867, + "dec_all": [ + 66.61798282077788, + 66.86022128070815, + 66.82440670116867 + ], + "ttft_med": 0.18910002708435059, + "peak": 17.98661582, + "paired": [ + 1.4237253603067872, + 1.4040673680084226, + 1.419207536882739 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 c_mask_fuse:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True]。", + "speedup": 1.419207536882739, + "verdict": "留" + }, + "r3d_all6": { + "dec_med": 66.66493160403824, + "dec_all": [ + 66.66493160403824, + 66.80098053708868, + 64.81861677574682 + ], + "ttft_med": 0.18864834308624268, + "peak": 17.98495168, + "paired": [ + 1.4005889493981019, + 1.3960647918871552, + 1.3468629042926332 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 d_all6:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True]。", + "speedup": 1.3960647918871552, + "verdict": "留" + }, + "r3e_msg": { + "dec_med": 66.20176969252506, + "dec_all": [ + 66.20176969252506, + 66.30925957405773, + 64.2510970652737 + ], + "ttft_med": 0.19022655487060547, + "peak": 18.048848016, + "paired": [ + 1.4179043613368552, + 1.3922370512388482, + 1.3556241642347557 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 e_msg:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。", + "speedup": 1.3922370512388482, + "verdict": "留" + } + } +} \ No newline at end of file diff --git a/examples/inkling2_mini_mlx/rounds/r4/cands/_fast.py b/examples/inkling2_mini_mlx/rounds/r4/cands/_fast.py new file mode 100644 index 00000000..9b952730 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/cands/_fast.py @@ -0,0 +1,315 @@ +# 第 3 轮共用:解码一步(B=1, L=1)的每层走「前段合图 → 写缓存+注意力 → 后段合图」三段。 +# 前段:残差取输入、消息门、归一化、q/k/v/r/门/汇点投影、短卷积、裁剪、相对位置偏置,一张编译图; +# 中段:KV 缓存写入和注意力(长度每步变,不编译); +# 后段:门控、输出投影、卷积、残差、MoE、卷积,一张编译图。卷积状态当作显式输入输出。 +# 选项:mask_kernel 把注意力遮罩的十几个小算子换成一个 Metal 内核;fuse_proj 把共享输入的投影拼成一次量化矩阵乘。 +import mlx.core as mx +from mlx import nn +from mlx_lm.models.inkling2 import layers as L +from mlx_lm.models.cache import RotatingKVCache + +OPTS = dict(mask_kernel=False, fuse_proj=False, fuse_all6=False, fuse_msg=False, conv_ew=False, conv_kernel=False, batch_msg=False) +_DEF = dict(OPTS) +_orig = L.DecoderLayer.__call__ +_cache = {} +_fused = {} + + +def _sliding(self): + return isinstance(self, L.SlidingDecoderLayer) + + +def _proj(self, attn, qi, ki, vi, same_kv): + if OPTS["fuse_proj"]: + f = _fused.get(id(attn)) + if f is not None: + outs = f["qrgs"](qi) + qp, rp, gate, sinks = mx.split(outs, f["qrgs_split"], axis=-1) + if same_kv and OPTS["fuse_all6"]: # 滑窗层无消息:q/k/v 同一输入,六个投影一次 + qp, kp, vp, rp, gate, sinks = mx.split(f["all6"](qi), f["all6_split"], axis=-1) + return qp, kp, vp, rp, gate, sinks + kp = attn.k_proj(ki); vp = attn.v_proj(vi) + return qp, kp, vp, rp, gate, sinks + return attn.q_proj(qi), attn.k_proj(ki), attn.v_proj(vi), attn.r_proj(qi), attn.gate_proj(qi), attn.sink_proj(qi) + + +# ---- 短卷积内核(第 4 轮):拼窗口、转权重精度、4 抽头深度卷积、加残差、除 sqrt(K+1)、出新状态,一个内核 ---- +_CONV_SRC = """ + uint c = thread_position_in_grid.x; + if (c >= (uint)C) return; + float xs[4]; + for (int k = 0; k < 3; ++k) xs[k] = state[k * C + c]; + float xr = static_cast(x[c]); + xs[3] = xr; + float acc = 0.0f; + for (int k = 0; k < 4; ++k) acc += static_cast(static_cast(xs[k])) * static_cast(w[c * 4 + k]); + WT o = static_cast(acc); + float r = (static_cast(o) + xr) / 2.2360679774997896f; + out[c] = static_cast(r); + for (int k = 0; k < 3; ++k) new_state[k * C + c] = xs[k + 1]; +""" +_conv_k = None + + +def _sconv(conv, x, st): + """返回 (输出, 新卷积状态)。""" + global _conv_k + if OPTS["conv_kernel"] and conv.kernel_size == 4: + if _conv_k is None: + _conv_k = mx.fast.metal_kernel(name="ink_sconv", input_names=["x", "state", "w"], + output_names=["out", "new_state"], source=_CONV_SRC) + C = x.shape[-1]; w = conv.conv1d.weight + o, ns = _conv_k(inputs=[x, st, w], template=[("C", C), ("WT", w.dtype), ("OT", x.dtype)], grid=(C, 1, 1), + threadgroup=(256, 1, 1), output_shapes=[x.shape, st.shape], output_dtypes=[x.dtype, mx.float32]) + return o, ns + o, win = conv(x, st, None) + return o, win[:, 1:] + + +# ---- q/k/v 三个消息门批成一次批量矩阵乘(第 4 轮)---- +_bw = {} + + +def _mix(self, states, msg): + if not OPTS["batch_msg"]: + return self._mix_messages(states, msg) + gs = (self.q_message_gate, self.k_message_gate, self.v_message_gate) + w = _bw.get(id(self)) + if w is None: + wd = mx.stack([g.down_proj.weight for g in gs]) # (3, r, 2D) + wf = mx.stack([mx.concatenate([g.forget_gate_proj.weight, g.message_gate_proj.weight]) for g in gs]) # (3, 2D, r) + mx.eval(wd, wf); w = _bw[id(self)] = (wd, wf) + wd, wf = w + h = mx.stack(list(states)) # (3, B, L, D) + m = mx.stack([msg[..., i, :] for i in range(3)]) + x = mx.concatenate([L.unweighted_rms_norm(h), L.unweighted_rms_norm(m)], axis=-1) + x = x.astype(wd.dtype).reshape(3, -1, x.shape[-1]) + st = (x @ wd.swapaxes(-1, -2)).astype(mx.float32) + fm = mx.tanh(st @ wf.swapaxes(-1, -2)).reshape(*h.shape[:-1], -1) + fg, mg = mx.split(fm, 2, axis=-1) + out = (1 - fg) * h + mg * m + return out[0], out[1], out[2] + + +def _pre(self, has_bs, has_msg): + sliding = _sliding(self); attn = self.self_attn + + def body(*args): + it = iter(args) + h = next(it); bs = next(it) if has_bs else None + cs0 = next(it); cs1 = next(it) + msg = next(it) if has_msg else None + pos = None if sliding else next(it) + dtype = h.dtype + nr = L.unweighted_rms_norm(h.astype(mx.float32)) + ai = self.attention_input(bs, nr) if self.attention_input is not None else None + if sliding: + hs = ai[..., 0, :] if ai is not None else h + if msg is None: + qi = ki = vi = self.input_layernorm(hs).astype(dtype) + else: + st = _mix(self, (hs,) * 3, msg) + qi, ki, vi = [self.input_layernorm(s).astype(dtype) for s in st] + else: + q, k, v = (ai[..., i, :] for i in range(3)) + if msg is not None: + q, k, v = _mix(self, (q, k, v), msg) + qi = self.q_input_layernorm(q).astype(dtype) + ki = self.k_input_layernorm(k).astype(dtype) + vi = self.v_input_layernorm(v).astype(dtype) + qp, kp, vp, rp, gate, sinks = _proj(self, attn, qi, ki, vi, sliding and msg is None) + if sliding: + ks, n0 = _sconv(attn.k_sconv, kp, cs0); vs, n1 = _sconv(attn.v_sconv, vp, cs1) + else: + ks, w0 = attn.k_sconv(kp, ki, cs0, None); vs, w1 = attn.v_sconv(vp, vi, cs1, None) + n0, n1 = w0[:, 1:], w1[:, 1:] + B, Ln = 1, 1 + qs = qp.reshape(B, Ln, attn.num_heads, -1) + ks = ks.reshape(B, Ln, attn.num_key_value_heads, -1) + vs = vs.reshape(B, Ln, attn.num_key_value_heads, -1) + rs = rp.reshape(B, Ln, attn.num_heads, -1) + qs = attn.q_l2_clip(qs); ks = attn.k_l2_clip(ks) + vs = L.unweighted_rms_norm(vs.astype(mx.float32), attn.eps).astype(vs.dtype) + rs = attn.rel_l2_clip(rs) + if not sliding: + tau = 1.0 + attn.log_scaling_alpha * mx.log(mx.maximum((pos + 1) / attn.log_scaling_n_floor, 1.0)) + tau = tau[..., None, None] + qs = (qs.astype(mx.float32) * tau).astype(qs.dtype) + rs = (rs.astype(mx.float32) * tau).astype(rs.dtype) + pb = attn.rel_logits_proj(rs) + return (qs.transpose(0, 2, 1, 3), ks.transpose(0, 2, 1, 3), vs.transpose(0, 2, 1, 3), + pb, sinks, gate, nr, n0, n1) + return mx.compile(body) + + +def _post(self, has_bs, has_msg): + attn = self.self_attn + + def body(*args): + it = iter(args) + out = next(it); gate = next(it); h = next(it); nr = next(it) + bs = next(it) if has_bs else None + cs2 = next(it); cs3 = next(it) + msg = next(it) if has_msg else None + dtype = h.dtype + ao = out.reshape(1, 1, -1).astype(mx.float32) * mx.sigmoid(gate.astype(mx.float32)) + hs = attn.o_proj(ao.astype(gate.dtype)) + hs, n2 = _sconv(self.post_attention_convolution, hs, cs2) + residual = hs = hs.astype(mx.float32) + if self.is_block_boundary: + block = nr[None] + bs = block if bs is None else mx.concatenate([bs, block]) + else: + residual = hs = h.astype(mx.float32) + hs + ns = L.unweighted_rms_norm(hs) + hs = self.mlp_input(bs, ns)[..., 0, :] + if msg is not None: + hs = self.mlp_message_gate(hs, msg[..., 3, :]) + hs = self.mlp(self.post_attention_layernorm(hs).astype(dtype)) + hs, n3 = _sconv(self.post_mlp_convolution, hs, cs3) + hs = residual + hs.astype(mx.float32) + return hs.astype(dtype), bs, n2, n3 + return mx.compile(body) + + +def _get(kind, self, has_bs, has_msg): + key = (kind, id(self), has_bs, has_msg) + f = _cache.get(key) + if f is None: + f = _cache[key] = (_pre if kind == "pre" else _post)(self, has_bs, has_msg) + return f + + +# ---- 遮罩内核:mask[h, s] = 可见 ? (距离= S) return; + int k = (int)s; + if (rot) { k = ((k - idx) % S + S) % S; } + int kpos = k + (end - S); + int dist = (end - 1) - kpos; + bool vis = (dist >= 0) && (kpos >= 0); + if (window > 0) vis = vis && (dist < window); + T v; + if (!vis) v = T(-INFINITY); + else if (dist < R) v = pb[h * R + dist]; + else v = T(0); + out[h * S + s] = v; +""" +_mask_k = None + + +def _attention_fast(attn, q, keys, values, pb, sinks, kvc): + global _mask_k + if _mask_k is None: + _mask_k = mx.fast.metal_kernel(name="ink_mask", input_names=["pb", "params"], output_names=["out"], source=_MASK_SRC) + S = keys.shape[2]; end = kvc.offset; H = pb.shape[2]; R = pb.shape[-1] + rot = isinstance(kvc, RotatingKVCache) + params = mx.array([S, end, kvc._idx if rot else 0, int(rot), attn.sliding_window, R], mx.int32) + mask = _mask_k(inputs=[pb[0, 0], params], template=[("T", q.dtype)], grid=(S, H, 1), threadgroup=(min(S, 256), 1, 1), + output_shapes=[(H, S)], output_dtypes=[q.dtype])[0] + o = mx.fast.scaled_dot_product_attention(q, keys, values, scale=attn.scaling, mask=mask[None, :, None, :], + sinks=sinks[0, 0].astype(q.dtype)) + return o.transpose(0, 2, 1, 3) + + +def call(self, hidden_states, block_states, *, cache=None, conv_mask=None, messages=None): + conv = cache[1] if cache is not None else None + if conv is None or hidden_states.shape[:2] != (1, 1) or conv.cache[0] is None or conv_mask is not None: + return _orig(self, hidden_states, block_states, cache=cache, conv_mask=conv_mask, messages=messages) + kvc = cache[0]; attn = self.self_attn + has_bs = block_states is not None; has_msg = messages is not None + args = [hidden_states] + ([block_states] if has_bs else []) + [conv[0], conv[1]] + ([messages] if has_msg else []) + if not _sliding(self): + args.append(mx.array([kvc.offset], mx.float32)) + q, k, v, pb, sinks, gate, nr, c0, c1 = _get("pre", self, has_bs, has_msg)(*args) + keys, values = kvc.update_and_fetch(k, v) + if OPTS["mask_kernel"]: + out = _attention_fast(attn, q, keys, values, pb, sinks, kvc) + else: + out = attn._attention(q, keys, values, pb, sinks, kvc) + args = [out, gate, hidden_states, nr] + ([block_states] if has_bs else []) + [conv[2], conv[3]] + ([messages] if has_msg else []) + hs, bs, c2, c3 = _get("post", self, has_bs, has_msg)(*args) + conv[0], conv[1], conv[2], conv[3] = c0, c1, c2, c3 + return hs, bs + + +def _cat_q(ps): + f = nn.QuantizedLinear.__new__(nn.QuantizedLinear); nn.Module.__init__(f) + f.group_size, f.bits, f.mode = ps[0].group_size, ps[0].bits, getattr(ps[0], "mode", "affine") + f.weight = mx.concatenate([p.weight for p in ps]); f.scales = mx.concatenate([p.scales for p in ps]) + f.biases = mx.concatenate([p.biases for p in ps]) + f.freeze(); mx.eval(f.parameters()) + sizes = [p.weight.shape[0] for p in ps] + return f, [sum(sizes[:i + 1]) for i in range(len(ps) - 1)] + + +def _fuse(model): + for layer in model.model.language_model.layers: + a = layer.self_attn + if id(a) in _fused: continue + ps = [a.q_proj, a.r_proj, a.gate_proj, a.sink_proj] + if not all(isinstance(p, nn.QuantizedLinear) for p in ps): continue + d = _fused[id(a)] = {} + d["qrgs"], d["qrgs_split"] = _cat_q(ps) + d["all6"], d["all6_split"] = _cat_q([a.q_proj, a.k_proj, a.v_proj, a.r_proj, a.gate_proj, a.sink_proj]) + + +# ---- 消息门:遗忘门和消息门两个投影共享输入,拼成一次矩阵乘 ---- +_orig_msg = L.MessageGate.__call__ +_msgw = {} + + +def _fuse_msg(model): + for layer in model.model.language_model.layers: + for n in ("q_message_gate", "k_message_gate", "v_message_gate", "mlp_message_gate"): + g = getattr(layer, n, None) + if g is None or id(g) in _msgw: continue + assert not isinstance(g.forget_gate_proj, nn.QuantizedLinear) + w = mx.concatenate([g.forget_gate_proj.weight, g.message_gate_proj.weight]); mx.eval(w) + _msgw[id(g)] = w + + +def _msg_call(self, hidden_states, message): + states = mx.concatenate([L.unweighted_rms_norm(hidden_states), L.unweighted_rms_norm(message)], axis=-1) + states = self.down_proj(states.astype(self.down_proj.weight.dtype)).astype(mx.float32) + w = _msgw.get(id(self)) + if w is None: + return _orig_msg(self, hidden_states, message) + fm = mx.tanh(states @ w.T) + forget_gate, message_gate = mx.split(fm, 2, axis=-1) + return (1 - forget_gate) * hidden_states + message_gate * message + + +# ---- 短卷积:深度卷积核 4 抽头,L=1 时写成逐元素乘加,免掉通用卷积内核 ---- +_orig_conv = L.ShortConvolution.__call__ + + +def _conv_call(self, hidden_states, conv_state, conv_mask): + if hidden_states.shape[1] != 1 or conv_state is None or conv_mask is not None: + return _orig_conv(self, hidden_states, conv_state, conv_mask) + K = self.kernel_size + residual = hidden_states.astype(mx.float32) + window = L._conv_input(hidden_states, conv_state, conv_mask, K) + w = self.conv1d.weight # (C, K, 1) + x = window.astype(w.dtype) + out = (x.astype(mx.float32) * w[:, :, 0].T[None].astype(mx.float32)).sum(axis=1, keepdims=True).astype(w.dtype) + out = (out.astype(mx.float32) + residual) / (K + 1) ** 0.5 + return out.astype(hidden_states.dtype), window + + +def apply(model, **opts): + OPTS.update(_DEF); OPTS.update(opts); _cache.clear() + if OPTS["fuse_proj"]: _fuse(model) + L.DecoderLayer.__call__ = call + if OPTS["fuse_msg"]: _fuse_msg(model); L.MessageGate.__call__ = _msg_call + if OPTS["conv_ew"]: L.ShortConvolution.__call__ = _conv_call + return {} + + +def revert(model): + L.DecoderLayer.__call__ = _orig; _cache.clear() + L.MessageGate.__call__ = _orig_msg; L.ShortConvolution.__call__ = _orig_conv diff --git a/examples/inkling2_mini_mlx/rounds/r4/cands/r4a_conv_kernel.py b/examples/inkling2_mini_mlx/rounds/r4/cands/r4a_conv_kernel.py new file mode 100644 index 00000000..4f51c685 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/cands/r4a_conv_kernel.py @@ -0,0 +1,4 @@ +"""第 4 轮 a_conv_kernel:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True]。""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r4/cands/r4b_ref_r3e.py b/examples/inkling2_mini_mlx/rounds/r4/cands/r4b_ref_r3e.py new file mode 100644 index 00000000..6cee6e70 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/cands/r4b_ref_r3e.py @@ -0,0 +1,4 @@ +"""第 4 轮 b_ref_r3e:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r4/cands/r4c_conv_batchmsg.py b/examples/inkling2_mini_mlx/rounds/r4/cands/r4c_conv_batchmsg.py new file mode 100644 index 00000000..378cdb5a --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/cands/r4c_conv_batchmsg.py @@ -0,0 +1,4 @@ +"""第 4 轮 c_conv_batchmsg:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True]。""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r4/log-evolve.jsonl b/examples/inkling2_mini_mlx/rounds/r4/log-evolve.jsonl new file mode 100644 index 00000000..bd643dbd --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/log-evolve.jsonl @@ -0,0 +1,4 @@ +{"cand": "r4a_conv_kernel", "dec_med": 70.37613505324236, "dec_all": [70.37613505324236, 63.407270571352086, 70.53740472243632], "ttft_med": 0.21026098728179932, "peak": 18.097880836, "paired": [1.4781394375224184, 1.4864271727446174, 1.5782824063347054], "same_output": true, "err": null, "doc": "第 4 轮 a_conv_kernel:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True]。"} +{"cand": "r4b_ref_r3e", "dec_med": 64.05252272468896, "dec_all": [64.05252272468896, 66.85197717882195, 62.98588519987612], "ttft_med": 0.2118755578994751, "peak": 18.097897604, "paired": [1.3271082612873688, 1.388945637421096, 1.3089572241154328], "same_output": true, "err": null, "doc": "第 4 轮 b_ref_r3e:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。"} +{"cand": "r4c_conv_batchmsg", "dec_med": 67.27773522180502, "dec_all": [67.27773522180502, 64.80468996476651, 67.576769573957], "ttft_med": 0.21914458274841309, "peak": 18.161532996, "paired": [1.516376817631209, 1.5187029693126763, 1.4963789746016443], "same_output": true, "err": null, "doc": "第 4 轮 c_conv_batchmsg:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True]。"} +{"cand": "baseline", "dec_med": 45.971434473146694, "dec_all": [46.78267174962613, 47.611296516926195, 42.65750232100074, 44.69251158051463, 48.26473061252324, 48.13145696828593, 48.11913180924665, 44.36742532565367, 42.67107609205199, 45.160197196667255], "ttft_med": 0.2195042371749878, "peak": 18.152008372, "noise_range": 0.12197201057099595} diff --git a/examples/inkling2_mini_mlx/rounds/r4/log-holdout.jsonl b/examples/inkling2_mini_mlx/rounds/r4/log-holdout.jsonl new file mode 100644 index 00000000..bf284570 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/log-holdout.jsonl @@ -0,0 +1,4 @@ +{"cand": "r4a_conv_kernel", "dec_med": 66.5639386058893, "dec_all": [66.5639386058893, 64.83010099014217, 66.91319649713606], "ttft_med": 0.192732572555542, "peak": 18.048849136, "paired": [1.4621318225938569, 1.4912991034038232, 1.5152283342410546], "same_output": true, "err": null, "doc": "第 4 轮 a_conv_kernel:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True]。"} +{"cand": "r4b_ref_r3e", "dec_med": 63.21071599250109, "dec_all": [63.48963303775635, 60.665830819532324, 63.21071599250109], "ttft_med": 0.19359838962554932, "peak": 18.049061552, "paired": [1.4089705684626987, 1.3472671620694185, 1.4668349117184647], "same_output": true, "err": null, "doc": "第 4 轮 b_ref_r3e:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。"} +{"cand": "r4c_conv_batchmsg", "dec_med": 70.28057320251098, "dec_all": [70.28057320251098, 69.37463871972089, 70.67514817849448], "ttft_med": 0.18816149234771729, "peak": 18.11250072, "paired": [1.507655637687247, 1.5007724118719972, 1.4964355952280801], "same_output": true, "err": null, "doc": "第 4 轮 c_conv_batchmsg:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True]。"} +{"cand": "baseline", "dec_med": 45.04490632221295, "dec_all": [43.06871846686166, 45.5252649434873, 43.472232258552545, 44.16047072578763, 45.06100727641791, 45.02880536800799, 43.09327211093362, 46.615799686407044, 46.225955495267954, 47.22899428740365], "ttft_med": 0.20153599977493286, "peak": 18.105733216, "noise_range": 0.09235840764731344} diff --git a/examples/inkling2_mini_mlx/rounds/r4/mechanism.txt b/examples/inkling2_mini_mlx/rounds/r4/mechanism.txt new file mode 100644 index 00000000..f92e9883 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/mechanism.txt @@ -0,0 +1,6 @@ +base build_ms=3.94 step_ms=22.17 +cand build_ms=1.75 step_ms=15.28 +base build_ms=3.91 step_ms=22.05 +cand build_ms=1.79 step_ms=15.28 +total 12184 non-view 7624 +total 6440 non-view 3664 diff --git a/examples/inkling2_mini_mlx/rounds/r4/score-evolve.json b/examples/inkling2_mini_mlx/rounds/r4/score-evolve.json new file mode 100644 index 00000000..677bb8e8 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/score-evolve.json @@ -0,0 +1,82 @@ +{ + "baseline": { + "dec_med": 45.971434473146694, + "dec_all": [ + 46.78267174962613, + 47.611296516926195, + 42.65750232100074, + 44.69251158051463, + 48.26473061252324, + 48.13145696828593, + 48.11913180924665, + 44.36742532565367, + 42.67107609205199, + 45.160197196667255 + ], + "ttft_med": 0.2195042371749878, + "peak": 18.152008372 + }, + "noise_range": 0.12197201057099595, + "cands": { + "r4a_conv_kernel": { + "dec_med": 70.37613505324236, + "dec_all": [ + 70.37613505324236, + 63.407270571352086, + 70.53740472243632 + ], + "ttft_med": 0.21026098728179932, + "peak": 18.097880836, + "paired": [ + 1.4781394375224184, + 1.4864271727446174, + 1.5782824063347054 + ], + "same_output": true, + "err": null, + "doc": "第 4 轮 a_conv_kernel:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True]。", + "speedup": 1.4864271727446174, + "verdict": "留" + }, + "r4b_ref_r3e": { + "dec_med": 64.05252272468896, + "dec_all": [ + 64.05252272468896, + 66.85197717882195, + 62.98588519987612 + ], + "ttft_med": 0.2118755578994751, + "peak": 18.097897604, + "paired": [ + 1.3271082612873688, + 1.388945637421096, + 1.3089572241154328 + ], + "same_output": true, + "err": null, + "doc": "第 4 轮 b_ref_r3e:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。", + "speedup": 1.3271082612873688, + "verdict": "留" + }, + "r4c_conv_batchmsg": { + "dec_med": 67.27773522180502, + "dec_all": [ + 67.27773522180502, + 64.80468996476651, + 67.576769573957 + ], + "ttft_med": 0.21914458274841309, + "peak": 18.161532996, + "paired": [ + 1.516376817631209, + 1.5187029693126763, + 1.4963789746016443 + ], + "same_output": true, + "err": null, + "doc": "第 4 轮 c_conv_batchmsg:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True]。", + "speedup": 1.516376817631209, + "verdict": "留" + } + } +} \ No newline at end of file diff --git a/examples/inkling2_mini_mlx/rounds/r4/score-holdout.json b/examples/inkling2_mini_mlx/rounds/r4/score-holdout.json new file mode 100644 index 00000000..6df13d2a --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/score-holdout.json @@ -0,0 +1,82 @@ +{ + "baseline": { + "dec_med": 45.04490632221295, + "dec_all": [ + 43.06871846686166, + 45.5252649434873, + 43.472232258552545, + 44.16047072578763, + 45.06100727641791, + 45.02880536800799, + 43.09327211093362, + 46.615799686407044, + 46.225955495267954, + 47.22899428740365 + ], + "ttft_med": 0.20153599977493286, + "peak": 18.105733216 + }, + "noise_range": 0.09235840764731344, + "cands": { + "r4a_conv_kernel": { + "dec_med": 66.5639386058893, + "dec_all": [ + 66.5639386058893, + 64.83010099014217, + 66.91319649713606 + ], + "ttft_med": 0.192732572555542, + "peak": 18.048849136, + "paired": [ + 1.4621318225938569, + 1.4912991034038232, + 1.5152283342410546 + ], + "same_output": true, + "err": null, + "doc": "第 4 轮 a_conv_kernel:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True]。", + "speedup": 1.4912991034038232, + "verdict": "留" + }, + "r4b_ref_r3e": { + "dec_med": 63.21071599250109, + "dec_all": [ + 63.48963303775635, + 60.665830819532324, + 63.21071599250109 + ], + "ttft_med": 0.19359838962554932, + "peak": 18.049061552, + "paired": [ + 1.4089705684626987, + 1.3472671620694185, + 1.4668349117184647 + ], + "same_output": true, + "err": null, + "doc": "第 4 轮 b_ref_r3e:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。", + "speedup": 1.4089705684626987, + "verdict": "留" + }, + "r4c_conv_batchmsg": { + "dec_med": 70.28057320251098, + "dec_all": [ + 70.28057320251098, + 69.37463871972089, + 70.67514817849448 + ], + "ttft_med": 0.18816149234771729, + "peak": 18.11250072, + "paired": [ + 1.507655637687247, + 1.5007724118719972, + 1.4964355952280801 + ], + "same_output": true, + "err": null, + "doc": "第 4 轮 c_conv_batchmsg:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True]。", + "speedup": 1.5007724118719972, + "verdict": "留" + } + } +} \ No newline at end of file From fc17889e1adf1f065813b9f77bca6786ecf801ec Mon Sep 17 00:00:00 2001 From: zuiho-kai <31877877+zuiho-kai@users.noreply.github.com> Date: Sat, 21 Nov 2026 16:19:17 +0900 Subject: [PATCH 07/11] examples(inkling2): English comments in round 3-4 candidates Co-Authored-By: Claude Opus 5.5 --- .../rounds/r3/cands/_fast.py | 18 +++++++------- .../rounds/r3/cands/r3a_split.py | 2 +- .../rounds/r3/cands/r3b_mask.py | 2 +- .../rounds/r3/cands/r3c_mask_fuse.py | 2 +- .../rounds/r3/cands/r3d_all6.py | 2 +- .../rounds/r3/cands/r3e_msg.py | 2 +- .../rounds/r4/cands/_fast.py | 24 +++++++++---------- .../rounds/r4/cands/r4a_conv_kernel.py | 2 +- .../rounds/r4/cands/r4b_ref_r3e.py | 2 +- .../rounds/r4/cands/r4c_conv_batchmsg.py | 2 +- 10 files changed, 29 insertions(+), 29 deletions(-) diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/_fast.py b/examples/inkling2_mini_mlx/rounds/r3/cands/_fast.py index c3c1f3ee..8fbad890 100644 --- a/examples/inkling2_mini_mlx/rounds/r3/cands/_fast.py +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/_fast.py @@ -1,8 +1,8 @@ -# 第 3 轮共用:解码一步(B=1, L=1)的每层走「前段合图 → 写缓存+注意力 → 后段合图」三段。 -# 前段:残差取输入、消息门、归一化、q/k/v/r/门/汇点投影、短卷积、裁剪、相对位置偏置,一张编译图; -# 中段:KV 缓存写入和注意力(长度每步变,不编译); -# 后段:门控、输出投影、卷积、残差、MoE、卷积,一张编译图。卷积状态当作显式输入输出。 -# 选项:mask_kernel 把注意力遮罩的十几个小算子换成一个 Metal 内核;fuse_proj 把共享输入的投影拼成一次量化矩阵乘。 +# Shared by rounds 3-4: each layer visit of a decode step (B=1, L=1) runs as pre graph -> cache write + attention -> post graph. +# pre : residual read-in, message gates, norms, q/k/v/r/gate/sink projections, short convs, clips, relative bias; one compiled graph; +# mid : KV-cache write and attention (length changes every step, not compiled); +# post: output gate, o_proj, convs, residual, MoE, convs; one compiled graph. Conv states are explicit inputs/outputs. +# Options: mask_kernel = attention mask in one Metal kernel; fuse_proj = projections sharing an input in one quantized matmul. import mlx.core as mx from mlx import nn from mlx_lm.models.inkling2 import layers as L @@ -25,7 +25,7 @@ def _proj(self, attn, qi, ki, vi, same_kv): if f is not None: outs = f["qrgs"](qi) qp, rp, gate, sinks = mx.split(outs, f["qrgs_split"], axis=-1) - if same_kv and OPTS["fuse_all6"]: # 滑窗层无消息:q/k/v 同一输入,六个投影一次 + if same_kv and OPTS["fuse_all6"]: # sliding layer, no messages: q/k/v share the input, six projections at once qp, kp, vp, rp, gate, sinks = mx.split(f["all6"](qi), f["all6_split"], axis=-1) return qp, kp, vp, rp, gate, sinks kp = attn.k_proj(ki); vp = attn.v_proj(vi) @@ -121,7 +121,7 @@ def _get(kind, self, has_bs, has_msg): return f -# ---- 遮罩内核:mask[h, s] = 可见 ? (距离 one matmul ---- _orig_msg = L.MessageGate.__call__ _msgw = {} @@ -225,7 +225,7 @@ def _msg_call(self, hidden_states, message): return (1 - forget_gate) * hidden_states + message_gate * message -# ---- 短卷积:深度卷积核 4 抽头,L=1 时写成逐元素乘加,免掉通用卷积内核 ---- +# ---- short conv as elementwise multiply-add for L=1 ---- _orig_conv = L.ShortConvolution.__call__ diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3a_split.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3a_split.py index 975fd5af..fc833844 100644 --- a/examples/inkling2_mini_mlx/rounds/r3/cands/r3a_split.py +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3a_split.py @@ -1,4 +1,4 @@ -"""第 3 轮 a_split:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 []。""" +"""Round 3 a_split: three-part layer decode path (pre graph / cache + attention / post graph), options [].""" import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast def apply(model): return _fast.apply(model, ) def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3b_mask.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3b_mask.py index 4b20a7db..d0ead909 100644 --- a/examples/inkling2_mini_mlx/rounds/r3/cands/r3b_mask.py +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3b_mask.py @@ -1,4 +1,4 @@ -"""第 3 轮 b_mask:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True]。""" +"""Round 3 b_mask: three-part layer decode path (pre graph / cache + attention / post graph), options [mask_kernel=True].""" import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast def apply(model): return _fast.apply(model, mask_kernel=True) def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3c_mask_fuse.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3c_mask_fuse.py index 6825bbf3..e2132854 100644 --- a/examples/inkling2_mini_mlx/rounds/r3/cands/r3c_mask_fuse.py +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3c_mask_fuse.py @@ -1,4 +1,4 @@ -"""第 3 轮 c_mask_fuse:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True]。""" +"""Round 3 c_mask_fuse: three-part layer decode path (pre graph / cache + attention / post graph), options [mask_kernel=True, fuse_proj=True].""" import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True) def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3d_all6.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3d_all6.py index e0f4b62a..67212375 100644 --- a/examples/inkling2_mini_mlx/rounds/r3/cands/r3d_all6.py +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3d_all6.py @@ -1,4 +1,4 @@ -"""第 3 轮 d_all6:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True]。""" +"""Round 3 d_all6: three-part layer decode path (pre graph / cache + attention / post graph), options [mask_kernel=True, fuse_proj=True, fuse_all6=True].""" import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True) def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3e_msg.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3e_msg.py index c2d359a7..a9232e4f 100644 --- a/examples/inkling2_mini_mlx/rounds/r3/cands/r3e_msg.py +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3e_msg.py @@ -1,4 +1,4 @@ -"""第 3 轮 e_msg:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。""" +"""Round 3 e_msg: three-part layer decode path (pre graph / cache + attention / post graph), options [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True].""" import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True) def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r4/cands/_fast.py b/examples/inkling2_mini_mlx/rounds/r4/cands/_fast.py index 9b952730..dd64db47 100644 --- a/examples/inkling2_mini_mlx/rounds/r4/cands/_fast.py +++ b/examples/inkling2_mini_mlx/rounds/r4/cands/_fast.py @@ -1,8 +1,8 @@ -# 第 3 轮共用:解码一步(B=1, L=1)的每层走「前段合图 → 写缓存+注意力 → 后段合图」三段。 -# 前段:残差取输入、消息门、归一化、q/k/v/r/门/汇点投影、短卷积、裁剪、相对位置偏置,一张编译图; -# 中段:KV 缓存写入和注意力(长度每步变,不编译); -# 后段:门控、输出投影、卷积、残差、MoE、卷积,一张编译图。卷积状态当作显式输入输出。 -# 选项:mask_kernel 把注意力遮罩的十几个小算子换成一个 Metal 内核;fuse_proj 把共享输入的投影拼成一次量化矩阵乘。 +# Shared by rounds 3-4: each layer visit of a decode step (B=1, L=1) runs as pre graph -> cache write + attention -> post graph. +# pre : residual read-in, message gates, norms, q/k/v/r/gate/sink projections, short convs, clips, relative bias; one compiled graph; +# mid : KV-cache write and attention (length changes every step, not compiled); +# post: output gate, o_proj, convs, residual, MoE, convs; one compiled graph. Conv states are explicit inputs/outputs. +# Options: mask_kernel = attention mask in one Metal kernel; fuse_proj = projections sharing an input in one quantized matmul. import mlx.core as mx from mlx import nn from mlx_lm.models.inkling2 import layers as L @@ -25,7 +25,7 @@ def _proj(self, attn, qi, ki, vi, same_kv): if f is not None: outs = f["qrgs"](qi) qp, rp, gate, sinks = mx.split(outs, f["qrgs_split"], axis=-1) - if same_kv and OPTS["fuse_all6"]: # 滑窗层无消息:q/k/v 同一输入,六个投影一次 + if same_kv and OPTS["fuse_all6"]: # sliding layer, no messages: q/k/v share the input, six projections at once qp, kp, vp, rp, gate, sinks = mx.split(f["all6"](qi), f["all6_split"], axis=-1) return qp, kp, vp, rp, gate, sinks kp = attn.k_proj(ki); vp = attn.v_proj(vi) @@ -33,7 +33,7 @@ def _proj(self, attn, qi, ki, vi, same_kv): return attn.q_proj(qi), attn.k_proj(ki), attn.v_proj(vi), attn.r_proj(qi), attn.gate_proj(qi), attn.sink_proj(qi) -# ---- 短卷积内核(第 4 轮):拼窗口、转权重精度、4 抽头深度卷积、加残差、除 sqrt(K+1)、出新状态,一个内核 ---- +# ---- short conv kernel (round 4): window, cast to weight dtype, 4-tap depthwise conv, residual, / sqrt(K+1), new state: one kernel ---- _CONV_SRC = """ uint c = thread_position_in_grid.x; if (c >= (uint)C) return; @@ -52,7 +52,7 @@ def _proj(self, attn, qi, ki, vi, same_kv): def _sconv(conv, x, st): - """返回 (输出, 新卷积状态)。""" + """Return (output, new conv state).""" global _conv_k if OPTS["conv_kernel"] and conv.kernel_size == 4: if _conv_k is None: @@ -66,7 +66,7 @@ def _sconv(conv, x, st): return o, win[:, 1:] -# ---- q/k/v 三个消息门批成一次批量矩阵乘(第 4 轮)---- +# ---- q/k/v message gates as one batched matmul (round 4) ---- _bw = {} @@ -180,7 +180,7 @@ def _get(kind, self, has_bs, has_msg): return f -# ---- 遮罩内核:mask[h, s] = 可见 ? (距离 one matmul ---- _orig_msg = L.MessageGate.__call__ _msgw = {} @@ -284,7 +284,7 @@ def _msg_call(self, hidden_states, message): return (1 - forget_gate) * hidden_states + message_gate * message -# ---- 短卷积:深度卷积核 4 抽头,L=1 时写成逐元素乘加,免掉通用卷积内核 ---- +# ---- short conv as elementwise multiply-add for L=1 ---- _orig_conv = L.ShortConvolution.__call__ diff --git a/examples/inkling2_mini_mlx/rounds/r4/cands/r4a_conv_kernel.py b/examples/inkling2_mini_mlx/rounds/r4/cands/r4a_conv_kernel.py index 4f51c685..1c4f1f00 100644 --- a/examples/inkling2_mini_mlx/rounds/r4/cands/r4a_conv_kernel.py +++ b/examples/inkling2_mini_mlx/rounds/r4/cands/r4a_conv_kernel.py @@ -1,4 +1,4 @@ -"""第 4 轮 a_conv_kernel:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True]。""" +"""Round 4 a_conv_kernel: round-3 three-part decode path, options [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True].""" import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True) def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r4/cands/r4b_ref_r3e.py b/examples/inkling2_mini_mlx/rounds/r4/cands/r4b_ref_r3e.py index 6cee6e70..c14bb7ba 100644 --- a/examples/inkling2_mini_mlx/rounds/r4/cands/r4b_ref_r3e.py +++ b/examples/inkling2_mini_mlx/rounds/r4/cands/r4b_ref_r3e.py @@ -1,4 +1,4 @@ -"""第 4 轮 b_ref_r3e:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。""" +"""Round 4 b_ref_r3e: round-3 three-part decode path, options [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True].""" import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True) def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r4/cands/r4c_conv_batchmsg.py b/examples/inkling2_mini_mlx/rounds/r4/cands/r4c_conv_batchmsg.py index 378cdb5a..709f4cfb 100644 --- a/examples/inkling2_mini_mlx/rounds/r4/cands/r4c_conv_batchmsg.py +++ b/examples/inkling2_mini_mlx/rounds/r4/cands/r4c_conv_batchmsg.py @@ -1,4 +1,4 @@ -"""第 4 轮 c_conv_batchmsg:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True]。""" +"""Round 4 c_conv_batchmsg: round-3 three-part decode path, options [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True].""" import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True) def revert(model): _fast.revert(model) From e29c2cfdc15b23fb290fd760f79109449272e628 Mon Sep 17 00:00:00 2001 From: zuiho-kai <31877877+zuiho-kai@users.noreply.github.com> Date: Sat, 21 Nov 2026 17:29:30 +0900 Subject: [PATCH 08/11] examples(inkling2): incomplete pairs are INCONCLUSIVE; speed win is only a screen - A candidate with an error, fewer than --reps pairs or a failed revert gets verdict INCONCLUSIVE; its error and samples stay in the record. - Token gate compares length as well as ids (zip compared a prefix only). - A complete, token-identical win >=5% now prints "speed screen passed, pending mechanism check" instead of "keep"; the mechanism gate (mechanism.py, count_ops2.py) is separate. - Scoring moved into functions with MLX imported inside main(), so it is testable on CPU; tests/test_inkling2_mlx_round_scoring.py covers first pair ok / second pair raises, a complete win, and a shorter output with an equal prefix. --- examples/inkling2_mini_mlx/rsi_mlx.py | 162 ++++++++++++++--------- tests/test_inkling2_mlx_round_scoring.py | 65 +++++++++ 2 files changed, 161 insertions(+), 66 deletions(-) create mode 100644 tests/test_inkling2_mlx_round_scoring.py diff --git a/examples/inkling2_mini_mlx/rsi_mlx.py b/examples/inkling2_mini_mlx/rsi_mlx.py index 64654df0..3b464ee3 100644 --- a/examples/inkling2_mini_mlx/rsi_mlx.py +++ b/examples/inkling2_mini_mlx/rsi_mlx.py @@ -1,13 +1,14 @@ # One InferRSI-style round on a Mac: load the model once, alternate each candidate with the default path, score by measured decode speed. # Usage: python rsi_mlx.py --split evolve|holdout --max-tokens N --reps R # Each /cands/*.py (not starting with "_") is a candidate defining apply(model) -> dict of stream_generate kwargs; -# it may patch the model in place (swap ops, wrap mx.compile) and defines revert(model). -# Score: median paired ratio candidate/adjacent-default decode tok/s. Gate: greedy token ids identical to the default path. +# it may patch the model in place (swap ops, wrap mx.compile) and must define revert(model). +# Score: median paired ratio candidate/adjacent-default decode tok/s. Gate: greedy token ids identical to the default path +# (same length and same ids on every prompt). # Noise: range of the default path's own runs. Below 5% difference: no conclusion. +# A candidate with an error, fewer than --reps pairs, or a failed revert is INCONCLUSIVE; its error and samples are kept. +# A speed win is only a screen: "speed screen passed, pending mechanism check". The mechanism gate +# (mechanism.py, count_ops2.py) is checked separately and is not read by this script. import argparse, glob, importlib.util, json, os, statistics, time -import mlx.core as mx -from mlx_lm import load, stream_generate -from mlx_lm.sample_utils import make_sampler # Code-generation workload. Split validation, as in earlier InferRSI campaigns: # candidates are selected on "evolve"; the reported number is the "holdout" split. @@ -22,30 +23,9 @@ ], } -ap = argparse.ArgumentParser() -ap.add_argument("model"); ap.add_argument("round_dir") -ap.add_argument("--max-tokens", type=int, required=True) -ap.add_argument("--reps", type=int, required=True) -ap.add_argument("--only", default=None) -ap.add_argument("--split", choices=list(SPLITS), required=True) -a = ap.parse_args() - -model, tok = load(a.model) -sampler = make_sampler(temp=0.0) -prompts = [tok.apply_chat_template([{"role": "user", "content": p}], add_generation_prompt=True, - reasoning_effort="none") for p in SPLITS[a.split]] - - -def run_once(kw): - out = [] - for p in prompts: - mx.reset_peak_memory(); t0 = time.time(); ttft = None; ids = []; last = None - for r in stream_generate(model, tok, p, max_tokens=a.max_tokens, sampler=sampler, **kw): - if ttft is None: ttft = time.time() - t0 - ids.append(r.token); last = r - out.append(dict(ttft=ttft, dec=last.generation_tps, pre=last.prompt_tps, n=last.generation_tokens, - peak=mx.get_peak_memory() / 1e9, ids=ids)) - return out +def same_ids(run, ref_ids): + """Quality gate: every prompt's token ids equal the default path's, including length.""" + return len(run) == len(ref_ids) and all(x["ids"] == ref for x, ref in zip(run, ref_ids)) def summary(runs): @@ -55,47 +35,97 @@ def summary(runs): peak=max(x["peak"] for r in runs for x in r)) -log = open(os.path.join(a.round_dir, f"log-{a.split}.jsonl"), "a") -def emit(d): - print(json.dumps(d, ensure_ascii=False), flush=True); log.write(json.dumps(d, ensure_ascii=False) + "\n"); log.flush() - -run_once({}) # warm-up: the first pass compiles Metal kernels; not scored -base_runs = [run_once({})] -ref_ids = [x["ids"] for x in base_runs[0]] -results = {} -cands = sorted(p for p in glob.glob(os.path.join(a.round_dir, "cands", "*.py")) if not os.path.basename(p).startswith("_")) -if a.only: cands = [c for c in cands if a.only in c] -for path in cands: - name = os.path.basename(path)[:-3] - spec = importlib.util.spec_from_file_location(name, path); mod = importlib.util.module_from_spec(spec) - spec.loader.exec_module(mod) - runs = []; pairs = []; same = True; err = None +def pair_candidate(mod, model, run_once, base_runs, ref_ids, reps): + """Alternate the default path with the candidate `reps` times. Keeps every sample and the error, if any.""" + runs = []; pairs = []; same = True; err = None; reverted = True try: - for i in range(a.reps): + for i in range(reps): base_runs.append(run_once({})) # default path, alternated with the candidate - kw = mod.apply(model) + kw = mod.apply(model); reverted = False r = run_once(kw); runs.append(r) pairs.append(summary([r])["dec_med"] / summary([base_runs[-1]])["dec_med"]) # paired with the adjacent default run, cancelling thermal drift - if hasattr(mod, "revert"): mod.revert(model) - same = same and all(x["ids"] == ref for x, ref in zip(r, ref_ids)) + mod.revert(model); reverted = True + same = same and same_ids(r, ref_ids) except Exception as e: err = repr(e) - if hasattr(mod, "revert"): mod.revert(model) + if not reverted: + try: + mod.revert(model); reverted = True + except Exception as e2: + err += f"; revert failed: {e2!r}" s = summary(runs) if runs else {} - results[name] = dict(s, paired=pairs, same_output=same, err=err, doc=(mod.__doc__ or "").strip()) - emit(dict(cand=name, **{k: v for k, v in results[name].items()})) - -b = summary(base_runs) -noise = (max(b["dec_all"]) - min(b["dec_all"])) / b["dec_med"] -emit(dict(cand="baseline", **b, noise_range=noise)) -for n, r in results.items(): - if r.get("dec_med"): - r["speedup"] = statistics.median(r["paired"]) - r["verdict"] = ("output changed: rejected" if not r["same_output"] else - "<5%: no conclusion" if abs(r["speedup"] - 1) < 0.05 else - "keep" if r["speedup"] > 1 else "reject") -json.dump(dict(baseline=b, noise_range=noise, cands=results), open(os.path.join(a.round_dir, f"score-{a.split}.json"), "w"), - ensure_ascii=False, indent=1) -for n, r in sorted(results.items(), key=lambda kv: -kv[1].get("dec_med", 0)): - print(f"{n:24s} {r.get('dec_med', 0):7.2f} tok/s x{r.get('speedup', 0):.3f} same_output={r['same_output']} {r.get('verdict')} {r['err'] or ''}") -print(f"baseline {b['dec_med']:.2f} tok/s noise range {noise:.1%}") + return dict(s, paired=pairs, same_output=same, err=err, reverted=reverted, doc=(mod.__doc__ or "").strip()) + + +def score(r, reps): + """Verdict for one candidate. Only complete, error-free, reverted runs that pass the token gate are scored.""" + if r["err"] or not r["reverted"] or len(r["paired"]) != reps: + return dict(verdict="INCONCLUSIVE") + speedup = statistics.median(r["paired"]) + if not r["same_output"]: + return dict(speedup=speedup, verdict="output changed: rejected") + if abs(speedup - 1) < 0.05: + return dict(speedup=speedup, verdict="<5%: no conclusion") + return dict(speedup=speedup, verdict="speed screen passed, pending mechanism check" if speedup > 1 else "reject") + + +def main(): + import mlx.core as mx + from mlx_lm import load, stream_generate + from mlx_lm.sample_utils import make_sampler + + ap = argparse.ArgumentParser() + ap.add_argument("model"); ap.add_argument("round_dir") + ap.add_argument("--max-tokens", type=int, required=True) + ap.add_argument("--reps", type=int, required=True) + ap.add_argument("--only", default=None) + ap.add_argument("--split", choices=list(SPLITS), required=True) + a = ap.parse_args() + + model, tok = load(a.model) + sampler = make_sampler(temp=0.0) + prompts = [tok.apply_chat_template([{"role": "user", "content": p}], add_generation_prompt=True, + reasoning_effort="none") for p in SPLITS[a.split]] + + def run_once(kw): + out = [] + for p in prompts: + mx.reset_peak_memory(); t0 = time.time(); ttft = None; ids = []; last = None + for r in stream_generate(model, tok, p, max_tokens=a.max_tokens, sampler=sampler, **kw): + if ttft is None: ttft = time.time() - t0 + ids.append(r.token); last = r + out.append(dict(ttft=ttft, dec=last.generation_tps, pre=last.prompt_tps, n=last.generation_tokens, + peak=mx.get_peak_memory() / 1e9, ids=ids)) + return out + + log = open(os.path.join(a.round_dir, f"log-{a.split}.jsonl"), "a") + def emit(d): + print(json.dumps(d, ensure_ascii=False), flush=True); log.write(json.dumps(d, ensure_ascii=False) + "\n"); log.flush() + + run_once({}) # warm-up: the first pass compiles Metal kernels; not scored + base_runs = [run_once({})] + ref_ids = [x["ids"] for x in base_runs[0]] + results = {} + cands = sorted(p for p in glob.glob(os.path.join(a.round_dir, "cands", "*.py")) if not os.path.basename(p).startswith("_")) + if a.only: cands = [c for c in cands if a.only in c] + for path in cands: + name = os.path.basename(path)[:-3] + spec = importlib.util.spec_from_file_location(name, path); mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + results[name] = pair_candidate(mod, model, run_once, base_runs, ref_ids, a.reps) + emit(dict(cand=name, **results[name])) + + b = summary(base_runs) + noise = (max(b["dec_all"]) - min(b["dec_all"])) / b["dec_med"] + emit(dict(cand="baseline", **b, noise_range=noise)) + for n, r in results.items(): + r.update(score(r, a.reps)) + json.dump(dict(baseline=b, noise_range=noise, cands=results), open(os.path.join(a.round_dir, f"score-{a.split}.json"), "w"), + ensure_ascii=False, indent=1) + for n, r in sorted(results.items(), key=lambda kv: -kv[1].get("dec_med", 0)): + print(f"{n:24s} {r.get('dec_med', 0):7.2f} tok/s x{r.get('speedup', 0):.3f} same_output={r['same_output']} {r.get('verdict')} {r['err'] or ''}") + print(f"baseline {b['dec_med']:.2f} tok/s noise range {noise:.1%}") + + +if __name__ == "__main__": + main() diff --git a/tests/test_inkling2_mlx_round_scoring.py b/tests/test_inkling2_mlx_round_scoring.py new file mode 100644 index 00000000..64ce989e --- /dev/null +++ b/tests/test_inkling2_mlx_round_scoring.py @@ -0,0 +1,65 @@ +"""CPU checks for the Inkling 2 Mini MLX round scorer; no MLX import needed.""" + +import importlib.util +from pathlib import Path + +path = Path(__file__).resolve().parents[1] / "examples/inkling2_mini_mlx/rsi_mlx.py" +spec = importlib.util.spec_from_file_location("inkling2_rsi_mlx", path) +rsi = importlib.util.module_from_spec(spec) +spec.loader.exec_module(rsi) + +REF = [[1, 2, 3], [4, 5]] + + +def sample(dec, ids=REF): + return [dict(ttft=0.1, dec=dec, pre=100.0, n=len(x), peak=1.0, ids=list(x)) for x in ids] + + +class Cand: + """Candidate whose second pass raises; records apply/revert calls.""" + + def __init__(self, fail_on=None, cand_ids=REF): + self.fail_on, self.cand_ids, self.applied, self.reverted = fail_on, cand_ids, 0, 0 + + def apply(self, model): + self.applied += 1 + return {"cand": True} + + def revert(self, model): + self.reverted += 1 + + +def runner(cand): + def run_once(kw): + if not kw: + return sample(40.0) + if cand.applied == cand.fail_on: + raise RuntimeError("Metal command buffer aborted") + return sample(44.0, cand.cand_ids) # 10% faster than the default path + + return run_once + + +def test_first_pair_ok_second_pair_raises_is_inconclusive_and_keeps_samples(): + cand = Cand(fail_on=2) + r = rsi.pair_candidate(cand, None, runner(cand), [sample(40.0)], REF, reps=3) + assert r["err"] and "Metal command buffer aborted" in r["err"] + assert r["paired"] == [1.1] and r["dec_all"] == [44.0] + assert r["reverted"] and cand.reverted == cand.applied == 2 + assert rsi.score(r, reps=3) == {"verdict": "INCONCLUSIVE"} + + +def test_complete_faster_identical_run_is_only_a_speed_screen(): + cand = Cand() + r = rsi.pair_candidate(cand, None, runner(cand), [sample(40.0)], REF, reps=3) + assert r["err"] is None and len(r["paired"]) == 3 and r["same_output"] + s = rsi.score(r, reps=3) + assert s["verdict"] == "speed screen passed, pending mechanism check" + assert "keep" not in s["verdict"] + + +def test_shorter_output_with_equal_prefix_fails_token_gate(): + cand = Cand(cand_ids=[[1, 2], [4, 5]]) + r = rsi.pair_candidate(cand, None, runner(cand), [sample(40.0)], REF, reps=3) + assert not r["same_output"] + assert rsi.score(r, reps=3)["verdict"] == "output changed: rejected" From f229ed596617402f2c151b74fcc3f9e9625fe27b Mon Sep 17 00:00:00 2001 From: zuiho-kai <31877877+zuiho-kai@users.noreply.github.com> Date: Sat, 21 Nov 2026 17:31:36 +0900 Subject: [PATCH 09/11] docs(inkling2): mark speed figures as unverified exploration records MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per the public performance claim rules in docs/contributing.md: - Inkling page: drop the quantitative claims from title and lead; state that it is an exploratory record without an acceptance issue; link raw route comparison outputs (examples/inkling2_mini_mlx/route/); describe the scorer's speed screen vs the separate mechanism gate; note the committed verdict strings (keep / 留, round-1 1.053) are from superseded scorers; report evolve and holdout as exploratory data (no record that the candidate was fixed before holdout was read). - Scheduling lessons: remove the 4.61x headline; ledger figures unverified. - Playbook and README rows labelled exploratory / unverified. --- README.md | 2 +- docs/apple-silicon-playbook.md | 4 +- docs/inkling2-mini-apple-silicon.md | 61 ++++++++---- docs/scheduling-rsi-lessons.md | 17 ++-- examples/inkling2_mini_mlx/README.md | 10 +- .../route/llama-bench-summary.txt | 3 + .../inkling2_mini_mlx/route/llama-bench.json | 94 +++++++++++++++++++ .../inkling2_mini_mlx/route/llama-chat.jsonl | 3 + .../inkling2_mini_mlx/route/mlx-benchmark.log | 7 ++ .../inkling2_mini_mlx/route/mlx-chat.jsonl | 15 +++ 10 files changed, 185 insertions(+), 31 deletions(-) create mode 100644 examples/inkling2_mini_mlx/route/llama-bench-summary.txt create mode 100644 examples/inkling2_mini_mlx/route/llama-bench.json create mode 100644 examples/inkling2_mini_mlx/route/llama-chat.jsonl create mode 100644 examples/inkling2_mini_mlx/route/mlx-benchmark.log create mode 100644 examples/inkling2_mini_mlx/route/mlx-chat.jsonl diff --git a/README.md b/README.md index 6f38656d..d3f4ab40 100644 --- a/README.md +++ b/README.md @@ -162,7 +162,7 @@ request per engine and measure audio frames per second and median time to first - **▶️ Run and review.** [Operations](docs/operations.md) · [Packaged optimization benchmarks](docs/optimization-benchmarks.md) · [Result delivery](docs/result-delivery.md) - **🧠 Use knowledge.** [Skill library](src/inferrsi/knowledge/data/skill_library/README.md) · [CANNbot setup and revocation](docs/external-channels.md#setting-one-up) · [MindIE setup and revocation](docs/codex-plugin.md#enable-and-revoke) -- **📓 Field lessons.** [Apple silicon playbook](docs/apple-silicon-playbook.md) · [Inkling 2 Mini on Apple silicon](docs/inkling2-mini-apple-silicon.md) · [Scheduling RSI lessons](docs/scheduling-rsi-lessons.md) · [h3-metal lessons](docs/h3-metal-apple-silicon-lessons.md) +- **📓 Field lessons (exploratory, unverified).** [Apple silicon playbook](docs/apple-silicon-playbook.md) · [Inkling 2 Mini on Apple silicon](docs/inkling2-mini-apple-silicon.md) · [Scheduling RSI lessons](docs/scheduling-rsi-lessons.md) · [h3-metal lessons](docs/h3-metal-apple-silicon-lessons.md) - **🧭 Understand and extend.** [Module layout and import migration](docs/module-layout.md) · [Architecture and directory map](docs/architecture.md) · [Standalone engine template](docs/template/README.md) - **📌 Follow changes.** [Releases](https://github.com/InferMatrixLab/InferRSI/releases) · [Changelog](CHANGELOG.md) diff --git a/docs/apple-silicon-playbook.md b/docs/apple-silicon-playbook.md index cddd32f2..a8a9d0f6 100644 --- a/docs/apple-silicon-playbook.md +++ b/docs/apple-silicon-playbook.md @@ -60,14 +60,14 @@ acceptance result. ## Bringing up a new model -From [Inkling 2 Mini on Apple silicon](inkling2-mini-apple-silicon.md) (measured on the same M5 Max). +From [Inkling 2 Mini on Apple silicon](inkling2-mini-apple-silicon.md) (measured on the same M5 Max; exploratory record without an acceptance issue, unverified). | Observation | Cause | Action | Source | | --- | --- | --- | --- | | The release checkpoint (ModelOpt NVFP4/MXFP8) loads in neither MLX nor llama.cpp. | Apple runtimes need their own quantized formats. | Start from a community conversion; pin repository, revision and file SHA-256. | measured | | Shards arrived larger than published after a segmented download. | curl `--retry` inside a ranged, appending segment resends from the range start. | Verify each file against the Hub `lfs.oid` before the first load. | measured | | A 512-token cap ended inside the thinking block although reasoning effort was set to `none`. | The template value only adds a system line. | Size generation caps for thinking plus answer; count both. | measured | -| A looped MoE model decoded at about 48 tok/s on MLX while weight traffic allowed far more. | 48 layer visits per token, each with many small ops: launch- and graph-building-bound. | Profile one layer visit at decode shape before compiling. Per-module compiles gave 1–3% and a fused layer half 6.4%; two compiled graphs per layer visit around the cache write, a one-kernel mask, fused projections and one-kernel short convs gave 1.49× on held-out prompts (ops per step 7,624 → 3,664). | measured | +| A looped MoE model decoded at about 48 tok/s on MLX while weight traffic allowed far more. | 48 layer visits per token, each with many small ops: launch- and graph-building-bound. | Profile one layer visit at decode shape before compiling. Per-module compiles gave 1–3% and a fused layer half 6.4%; two compiled graphs per layer visit around the cache write, a one-kernel mask, fused projections and one-kernel short convs measured 1.49× (paired median, two prompts, exploratory) with ops per step 7,624 → 3,664. | measured, unverified | | During a search round the default path drifted from 47.4 to 48.9 tok/s. | Warm-up/thermal drift over minutes. | Pair each candidate run with the adjacent default run; an all-runs median inflated one candidate from 1.048 to 1.053. | measured | ## Benchmarking a shared Mac diff --git a/docs/inkling2-mini-apple-silicon.md b/docs/inkling2-mini-apple-silicon.md index 40adff2c..38c83f2a 100644 --- a/docs/inkling2-mini-apple-silicon.md +++ b/docs/inkling2-mini-apple-silicon.md @@ -1,15 +1,14 @@ -# Inkling 2 Mini on Apple silicon: local adaptation and four RSI rounds - -**On one M5 Max, MLX ran Inkling 2 Mini at 47.0 tok/s decode and 1,677 tok/s -prefill (512-token prompt), ahead of llama.cpp at 41.7 and 998. Four search -rounds then raised MLX decode 1.491× (paired median, held-out prompts, -run-to-run noise 9.2%) with token-identical greedy output: about 47 → 70 -tok/s. Rounds 1–2 compiled sub-blocks (1.064×); rounds 3–4 rebuilt the -single-token layer path (two compiled graphs per layer visit around the cache -write, a one-kernel attention mask, fused projections, one-kernel short -convolutions) and halved the ops per step.** The patch is published as -[zuiho-kai/inkling2-mlx-fastpath](https://github.com/zuiho-kai/inkling2-mlx-fastpath). These are maintainer measurements without an acceptance issue; they -are not an InferRSI PASS. +# Inkling 2 Mini on Apple silicon: local adaptation and four exploratory RSI rounds + +This is an unverified exploration record, not a performance claim. It has no +acceptance issue and is not an InferRSI PASS. It describes how Inkling 2 Mini +was brought up on one M5 Max with MLX and llama.cpp, and four InferRSI-style +search rounds on MLX. Every number below holds only for the stated machine, +versions, settings and prompts. The round numbers can be recomputed from the +committed records in +[examples/inkling2_mini_mlx](../examples/inkling2_mini_mlx/README.md); the +route comparison links its raw outputs; other measured rows rest on +unpublished maintainer logs. [Inkling 2 Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) is a 26B-total, MoE, looped transformer from Thinking Machines Lab (24 layers; the @@ -26,9 +25,9 @@ InferRSI acceptance result. ## Machine and sources - Apple M5 Max, 128 GB unified memory, macOS 27.0 (26A425). -- Labels: **measured** = run on this machine, raw logs in the maintainers' - run directory (not published); **public** = model card or repository at the - link given. +- Labels: **measured** = run on this machine; unless a row links raw outputs, + the logs are in the maintainers' run directory (not published), so treat it + as unverified; **public** = model card or repository at the link given. | Route | Weights | Runtime | Source | | --- | --- | --- | --- | @@ -55,7 +54,11 @@ GGUF conversion at the time). Same machine, one runtime at a time (a lock file kept them apart), greedy decoding, fresh process per route. Idle ComfyUI and a small mlx-lm server were -resident but not working during the runs. +resident but not working during the runs. Raw outputs (benchmark logs, +llama-bench JSON, per-run chat timings) are in +[examples/inkling2_mini_mlx/route](../examples/inkling2_mini_mlx/route/). +Both runtimes ran with default settings; this is one exploratory +default-against-default comparison, not a runtime ranking. | Metric | MLX 4-bit | llama.cpp Q4_K_M | Command | | --- | ---: | ---: | --- | @@ -65,7 +68,8 @@ resident but not working during the runs. | Time to first token, 61-token chat prompt, warm (s) | 0.24 | 0.29–0.31 | same | | Peak memory | 17.6 GB (`mx.get_peak_memory`), 19.0 GB in the benchmark | 17.6 GB resident, of which weights are file-mapped | | -MLX was faster on both prefill and decode, so the RSI round below uses MLX. +In this comparison MLX measured faster on both prefill and decode, so the +search rounds below used MLX. The first MLX request of a process took 2.17 s to first token because Metal kernels compile on first use; warm numbers exclude it. @@ -94,14 +98,31 @@ its output. - below 5% difference: no conclusion; - candidates are selected on two "evolve" prompts and reported on two different "holdout" prompts (TypeScript and Rust code generation); -- a "better" verdict needs a mechanism signal. +- a "better" verdict needs a mechanism signal. `rsi_mlx.py` performs only the + speed screen and prints "speed screen passed, pending mechanism check"; + the mechanism is measured separately with `mechanism.py` and `count_ops2.py` + (reported per round below). A candidate with an error, fewer pairs than + `--reps` or a failed revert is INCONCLUSIVE. + +The verdict strings stored in the committed `score-*.json` records ("keep", +and "留" in round 1) were written by earlier versions of the scorer and are +superseded by the rules above; the paired ratios in the records are what this +page uses. + +**Selection and holdout.** Within each round, every candidate was run on the +evolve and the holdout prompts in the same session. There is no record that +the shipped candidate was fixed from the evolve scores and the 5% rule before +the holdout results were read, so both splits are reported as exploratory +data, not as a held-out confirmation. **Round 1** (evolve, 256 generated tokens, noise 2.9%). Paired medians: gates 1.031, MoE 1.015, depth residual 1.012, gates + MoE 1.048. All token-identical, all below 5%: no conclusion. The first version of the scorer compared with the median of all default runs; the default path drifted from 47.4 to 48.9 tok/s during the round, which inflated gates + MoE to 1.053. The -scorer was changed to adjacent pairing before round 2. +scorer was changed to adjacent pairing before round 2. The "留" verdict and +1.053 still stored in `rounds/r1/score-evolve.json` come from that superseded +scorer; the paired figures above replace them. **Round 2** (proposed from round 1: combine the compile points; fuse the whole MLP half of each layer). @@ -169,6 +190,8 @@ per-module profile and an op count of one decode step. ## What we did not establish +- No acceptance issue, frozen campaign evidence bundle or independent rerun; + all speed figures are exploratory observations under the setup above. - One machine, one macOS build, 128 GB; nothing on 32–64 GB Macs, which the model card lists as supported. - Single stream only; no batching, concurrency or long-context runs (the diff --git a/docs/scheduling-rsi-lessons.md b/docs/scheduling-rsi-lessons.md index d7bed180..6ed95377 100644 --- a/docs/scheduling-rsi-lessons.md +++ b/docs/scheduling-rsi-lessons.md @@ -1,11 +1,14 @@ # Scheduling RSI lessons: Ascend queue ordering and agent KV eviction -**An LLM-proposed waiting-queue order (prefix-cache hit share of the prompt, -highest first) raised simulated capacity at a 10 s mean TTFT bound from about -3.3 to 15.3 req/s (4.61×, repeats 4.2–4.8×) over the Ascend shortest-job-first -baseline.** This is a virtual-clock simulation result from the maintainers' -campaign ledger. It has no acceptance issue, was not run on NPU hardware and is -not an InferRSI PASS. +This is an unverified exploration record. The quantitative findings below come +from the maintainers' unpublished campaign ledger or from replays whose +outputs are not published here. They have no acceptance issue, are not +reproducible from this page and are not InferRSI PASS results; Campaign 1 is a +simulation and was not run on NPU hardware. They are kept as lessons about method, not +as established performance gains. For example, the best waiting-queue order +in Campaign 1 (prefix-cache hit share of the prompt, highest first) was +recorded in the ledger at 4.61× simulated capacity over the Ascend +shortest-job-first baseline; that figure is unverified. This page records two earlier campaigns of the same RSI line (model proposes a policy → a frozen scorer measures it → keep the better, drop the worse) so that @@ -19,7 +22,7 @@ the headline. Apple-silicon lessons from later work are in the | Label | Meaning | | --- | --- | | **public** | In a public repository or dataset at the link given in the row. | -| **ledger** | From the maintainers' unpublished campaign report or run log; not reproducible from this page. | +| **ledger** | From the maintainers' unpublished campaign report or run log; not reproducible from this page; unverified. | | **inferred** | Analysis made during the work and not measured. | ## Campaign 1: vLLM-Ascend waiting-queue order (simulation) diff --git a/examples/inkling2_mini_mlx/README.md b/examples/inkling2_mini_mlx/README.md index dcd44079..20ba79fd 100644 --- a/examples/inkling2_mini_mlx/README.md +++ b/examples/inkling2_mini_mlx/README.md @@ -12,6 +12,7 @@ Apple-silicon Mac with mlx-lm from ml-explore/mlx-lm#2086 (`0c4c991`). | `rsi_mlx.py` | One search round: load once, alternate each candidate with the default path, paired scoring, token-identity gate, evolve/holdout split | | `mechanism.py` | Per-step graph-building and step time, default vs one candidate | | `count_ops2.py` | Ops in one decode step's graph (total and non-view), default or one candidate | +| `route/` | Raw outputs of the MLX / llama.cpp route comparison (`mlx_lm.benchmark` log, `llama-bench` JSON, per-run chat timings) | | `rounds/r1` … `rounds/r4` | Candidates (`cands/*.py`) and the JSONL/JSON records each round wrote | ```bash @@ -21,9 +22,14 @@ python mechanism.py rounds/r4/cands/r4a_conv_kernel.py python count_ops2.py rounds/r4/cands/r4a_conv_kernel.py ``` -Round 1 records predate adjacent pairing; its paired ratios in the doc were +`rsi_mlx.py` prints "speed screen passed, pending mechanism check" for a +complete, token-identical candidate at least 5% faster, and INCONCLUSIVE for +one with an error, fewer pairs than `--reps` or a failed revert. The verdict +strings in the committed `score-*.json` files ("keep", "留") come from earlier +scorer versions and are superseded. Round 1 records predate adjacent pairing; its paired ratios in the doc were recomputed from the logged run order (default run `1 + 3k + i` precedes repetition `i` of candidate `k`). Workload prompts are kept verbatim, including the Chinese ones. Run one runtime at a time; another GPU job on the -machine invalidates the round. The round-4 winner is packaged as +machine invalidates the round. These records are exploratory and +unverified (no acceptance issue). The round-4 candidate is packaged as [zuiho-kai/inkling2-mlx-fastpath](https://github.com/zuiho-kai/inkling2-mlx-fastpath). diff --git a/examples/inkling2_mini_mlx/route/llama-bench-summary.txt b/examples/inkling2_mini_mlx/route/llama-bench-summary.txt new file mode 100644 index 00000000..60cdbc15 --- /dev/null +++ b/examples/inkling2_mini_mlx/route/llama-bench-summary.txt @@ -0,0 +1,3 @@ +512 0 997.58 tok/s +0 128 41.65 tok/s + 459666632 peak memory footprint diff --git a/examples/inkling2_mini_mlx/route/llama-bench.json b/examples/inkling2_mini_mlx/route/llama-bench.json new file mode 100644 index 00000000..777551ef --- /dev/null +++ b/examples/inkling2_mini_mlx/route/llama-bench.json @@ -0,0 +1,94 @@ +[ + { + "build_commit": "418fe97", + "build_number": 1, + "cpu_info": "Accelerate, Apple M5 Max", + "gpu_info": "Apple M5 Max", + "backends": "MTL,BLAS", + "model_filename": "/lmstudio-community/Inkling-2-Mini-GGUF/Inkling-2-Mini-Q4_K_M.gguf", + "model_type": "inkling2_loop ?B Q4_K - Medium", + "model_size": 17466116736, + "model_n_params": 26070385152, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 6, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "repack": true, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-11-21T06:20:11Z", + "avg_ns": 513241972, + "stddev_ns": 426397, + "avg_ts": 997.580602, + "stddev_ts": 0.828454, + "samples_ns": [ 513724666, 513084708, 512916542 ], + "samples_ts": [ 996.643, 997.886, 998.213 ] + }, + { + "build_commit": "418fe97", + "build_number": 1, + "cpu_info": "Accelerate, Apple M5 Max", + "gpu_info": "Apple M5 Max", + "backends": "MTL,BLAS", + "model_filename": "/lmstudio-community/Inkling-2-Mini-GGUF/Inkling-2-Mini-Q4_K_M.gguf", + "model_type": "inkling2_loop ?B Q4_K - Medium", + "model_size": 17466116736, + "model_n_params": 26070385152, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 6, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "repack": true, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-11-21T06:20:14Z", + "avg_ns": 3073455430, + "stddev_ns": 6380650, + "avg_ts": 41.647055, + "stddev_ts": 0.086542, + "samples_ns": [ 3066224375, 3075850125, 3078291792 ], + "samples_ts": [ 41.7452, 41.6145, 41.5815 ] + } +] diff --git a/examples/inkling2_mini_mlx/route/llama-chat.jsonl b/examples/inkling2_mini_mlx/route/llama-chat.jsonl new file mode 100644 index 00000000..a8e5420c --- /dev/null +++ b/examples/inkling2_mini_mlx/route/llama-chat.jsonl @@ -0,0 +1,3 @@ +{"ttft_s": 0.3104572296142578, "wall_s": 12.131134986877441, "timings": {"cache_n": 0, "prompt_n": 61, "prompt_ms": 253.471, "prompt_per_token_ms": 4.155262295081967, "prompt_per_second": 240.65869468302094, "predicted_n": 512, "predicted_ms": 11870.061, "predicted_per_token_ms": 23.229082191780822, "predicted_per_second": 43.04948390745423}, "text": "", "reasoning": "The user asks in Chinese: write a thread-safe LRU cache class in Python supporting get/put/capacity limit and TTL, with 3 pytest cases. Only output code. Need provide code only, no explanation. Need ensure correctness.\n\nDesign: use OrderedDict for LRU, Lock (RLock), time.time. put(key,value,ttl=None): if ttl is None use default_ttl? Class init capacity, default_ttl maybe. get(key): check expiry, move to end, return value or None. put: if key exists delete then insert end; if len>capacity popitem(last=False). TTL: store (value, expire_at). get if expire_at <= now: delete and return None. Also maybe put with ttl=0? expire immediately. Need thread-safe.\n\nTests: 3 pytest cases: capacity eviction LRU order; TTL expiry; thread safety maybe concurrent puts/gets no exceptions and size <= capacity. Need only code. Include imports. Use pytest.\n\nNeed be careful: only output code, no markdown? User says 只输出代码. So final should be code only, no prose. Could include code block? \"只输出代码\" likely no explanation; code block is okay but maybe raw code safer. Provide raw Python code.\n\nImplementation details:\nfrom collections import OrderedDict\nimport threading, time\nimport pytest\n\nclass LRUCache:\n def __init__(self, capacity, default_ttl=None):\n if capacity <=0: raise ValueError\n self.capacity=capacity\n self.default_ttl=default_ttl\n self._data=OrderedDict()\n self._lock=threading.RLock()\n def get(self,key,default=None):\n with self._lock:\n item=self._data.get(key)\n if item is None: return default\n value, expires_at = item\n if expires_at is not None and expires_at <= time.monotonic():\n del self._data[key]\n return default\n self._data.move_to_end(key)\n return value\n"} +{"ttft_s": 0.29821109771728516, "wall_s": 12.032907009124756, "timings": {"cache_n": 0, "prompt_n": 61, "prompt_ms": 237.324, "prompt_per_token_ms": 3.8905573770491806, "prompt_per_second": 257.0325799329187, "predicted_n": 512, "predicted_ms": 11784.265, "predicted_per_token_ms": 23.061183953033268, "predicted_per_second": 43.36290808124224}, "text": "", "reasoning": "The user asks in Chinese: write a thread-safe LRU cache class in Python supporting get/put/capacity limit and TTL, with 3 pytest cases. Only output code. Need provide code only, no explanation. Need ensure correctness.\n\nDesign: use OrderedDict for LRU, Lock (RLock), time.time. put(key,value,ttl=None): if ttl is None use default_ttl? Class init capacity, default_ttl maybe. get(key): check expiry, move to end, return value or None. put: if key exists delete then insert end; if len>capacity popitem(last=False). TTL: store (value, expire_at). get if expire_at <= now: delete and return None. Also maybe put with ttl=0? expire immediately. Need thread-safe.\n\nTests: 3 pytest cases: capacity eviction LRU order; TTL expiry; thread safety maybe concurrent puts/gets no exceptions and size <= capacity. Need only code. Include imports. Use pytest.\n\nNeed be careful: only output code, no markdown? User says 只输出代码. So final should be code only, no prose. Could include code block? \"只输出代码\" likely no explanation; code block is okay but maybe raw code safer. Provide raw Python code.\n\nImplementation details:\nfrom collections import OrderedDict\nimport threading, time\nimport pytest\n\nclass LRUCache:\n def __init__(self, capacity, default_ttl=None):\n if capacity <=0: raise ValueError\n self.capacity=capacity\n self.default_ttl=default_ttl\n self._data=OrderedDict()\n self._lock=threading.RLock()\n def get(self,key,default=None):\n with self._lock:\n item=self._data.get(key)\n if item is None: return default\n value, expires_at = item\n if expires_at is not None and expires_at <= time.monotonic():\n del self._data[key]\n return default\n self._data.move_to_end(key)\n return value\n"} +{"ttft_s": 0.29129791259765625, "wall_s": 12.558490991592407, "timings": {"cache_n": 0, "prompt_n": 61, "prompt_ms": 236.752, "prompt_per_token_ms": 3.8811803278688526, "prompt_per_second": 257.65357842805975, "predicted_n": 512, "predicted_ms": 12316.508, "predicted_per_token_ms": 24.102755381604695, "predicted_per_second": 41.48903244328668}, "text": "", "reasoning": "The user asks in Chinese: write a thread-safe LRU cache class in Python supporting get/put/capacity limit and TTL, with 3 pytest cases. Only output code. Need provide code only, no explanation. Need ensure correctness.\n\nDesign: use OrderedDict for LRU, Lock (RLock), time.time. put(key,value,ttl=None): if ttl is None use default_ttl? Class init capacity, default_ttl maybe. get(key): check expiry, move to end, return value or None. put: if key exists delete then insert end; if len>capacity popitem(last=False). TTL: store (value, expire_at). get if expire_at <= now: delete and return None. Also maybe put with ttl=0? expire immediately. Need thread-safe.\n\nTests: 3 pytest cases: capacity eviction LRU order; TTL expiry; thread safety maybe concurrent puts/gets no exceptions and size <= capacity. Need only code. Include imports. Use pytest.\n\nNeed be careful: only output code, no markdown? User says 只输出代码. So final should be code only, no prose. Could include code block? \"只输出代码\" likely no explanation; code block is okay but maybe raw code safer. Provide raw Python code.\n\nImplementation details:\nfrom collections import OrderedDict\nimport threading, time\nimport pytest\n\nclass LRUCache:\n def __init__(self, capacity, default_ttl=None):\n if capacity <=0: raise ValueError\n self.capacity=capacity\n self.default_ttl=default_ttl\n self._data=OrderedDict()\n self._lock=threading.RLock()\n def get(self,key,default=None):\n with self._lock:\n item=self._data.get(key)\n if item is None: return default\n value, expires_at = item\n if expires_at is not None and expires_at <= time.monotonic():\n del self._data[key]\n return default\n self._data.move_to_end(key)\n return value\n"} diff --git a/examples/inkling2_mini_mlx/route/mlx-benchmark.log b/examples/inkling2_mini_mlx/route/mlx-benchmark.log new file mode 100644 index 00000000..94169643 --- /dev/null +++ b/examples/inkling2_mini_mlx/route/mlx-benchmark.log @@ -0,0 +1,7 @@ +[transformers] You are using a model of type `inkling2_loop` to instantiate a model of type ``. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. +Running warmup.. +Timing with prompt_tokens=512, generation_tokens=128, batch_size=1. +Trial 1: prompt_tps=1701.590, generation_tps=47.270, peak_memory=19.044, total_time=3.011 +Trial 2: prompt_tps=1704.370, generation_tps=46.170, peak_memory=19.044, total_time=3.075 +Trial 3: prompt_tps=1625.348, generation_tps=47.467, peak_memory=19.045, total_time=3.014 +Averages: prompt_tps=1677.103, generation_tps=46.969, peak_memory=19.044 diff --git a/examples/inkling2_mini_mlx/route/mlx-chat.jsonl b/examples/inkling2_mini_mlx/route/mlx-chat.jsonl new file mode 100644 index 00000000..f3f8ac92 --- /dev/null +++ b/examples/inkling2_mini_mlx/route/mlx-chat.jsonl @@ -0,0 +1,15 @@ +{"run": 0, "load_s": 2.161633014678955, "ttft_s": 2.1704137325286865, "prompt_tokens": 61, "prefill_tps": 29.1318650445503, "gen_tokens": 512, "decode_tps": 47.640906850511996, "peak_gb": 17.559900508, "wall_s": 12.918859958648682, "kw": {}} +{"run": 1, "load_s": 2.161633014678955, "ttft_s": 0.245164155960083, "prompt_tokens": 61, "prefill_tps": 249.52940491322087, "gen_tokens": 512, "decode_tps": 45.252409289809215, "peak_gb": 17.56069738, "wall_s": 11.560818910598755, "kw": {}} +{"run": 2, "load_s": 2.161633014678955, "ttft_s": 0.24292707443237305, "prompt_tokens": 61, "prefill_tps": 251.78264390018035, "gen_tokens": 512, "decode_tps": 47.34382293487728, "peak_gb": 17.560628916, "wall_s": 11.058743953704834, "kw": {}} +----- 输出 ----- +<|message_model|><|content_thinking|>The user asks in Chinese: write a thread-safe LRU cache class in Python supporting get/put/capacity limit and TTL expiration, with 3 pytest cases. Only output code. Need provide code only. Need ensure correctness. + +Design: class LRUCache with OrderedDict, Lock. get(key): with lock, if key in cache: move_to_end, check expiry: if expired delete and return None (or default). else return value. put(key,value,ttl=None): with lock, if key exists delete; set expiry = time.time()+ttl if ttl else None; cache[key]=(value,expiry); move_to_end; while len>capacity: popitem(last=False). Need capacity maybe maxsize. TTL default from constructor? Support per-put ttl and default ttl. get returns default maybe None. Thread-safe. + +Tests: 3 pytest cases: capacity eviction LRU order; TTL expiration (use small ttl and monkeypatch time or sleep? Better avoid sleep by injecting clock? Could use time.sleep(0.05) with ttl=0.01; acceptable but flaky. Better design class accepts time_func for tests? But user asked class; can include optional clock param. Tests can use fake clock. Simpler: class uses time.time; tests use ttl=0.05 and time.sleep(0.06). 3 cases: 1) LRU eviction order; 2) TTL expiry; 3) thread safety? Need 3 pytest cases. Could do: get/put basic, capacity eviction, TTL expiry. Thread safety test maybe with threads incrementing? But only 3 cases. Include thread-safety in one? Need 3 pytest用例. We can write 3 tests: test_lru_eviction, test_ttl_expiry, test_thread_safety. Thread safety test: create cache capacity large, spawn threads putting/getting, assert no exception and final size <= capacity. Deterministic enough. + +Need only output code, no explanation. Ensure code block only? User says 只输出代码. So final should be code only, likely a single code block. No prose. + +Need be careful with OrderedDict move_to_end and expiry. Implementation: + + From e6a6c03eba9b3838e3a2e7223cdac68ba6e4d543 Mon Sep 17 00:00:00 2001 From: zuiho-kai <31877877+zuiho-kai@users.noreply.github.com> Date: Sat, 21 Nov 2026 17:51:36 +0900 Subject: [PATCH 10/11] examples(inkling2): mark the model as possibly changed before apply(); stop the round on an unconfirmed revert - pair_candidate() sets reverted=False before calling apply(), so an apply() that changes the model and then raises is still reverted. - run_round() stops the round when a revert is not confirmed, so later default runs are not taken from a possibly changed model. - Two CPU tests for these paths. - route/mlx-chat.jsonl keeps only the timing records; the model output moves to route/mlx-chat-output.txt. Docs updated. - README: the exploratory label is on the two new pages only. --- README.md | 2 +- docs/inkling2-mini-apple-silicon.md | 2 +- examples/inkling2_mini_mlx/README.md | 2 +- .../route/mlx-chat-output.txt | 12 ++++++ .../inkling2_mini_mlx/route/mlx-chat.jsonl | 12 ------ examples/inkling2_mini_mlx/rsi_mlx.py | 23 ++++++++--- tests/test_inkling2_mlx_round_scoring.py | 39 +++++++++++++++++++ 7 files changed, 72 insertions(+), 20 deletions(-) create mode 100644 examples/inkling2_mini_mlx/route/mlx-chat-output.txt diff --git a/README.md b/README.md index d3f4ab40..a22200ac 100644 --- a/README.md +++ b/README.md @@ -162,7 +162,7 @@ request per engine and measure audio frames per second and median time to first - **▶️ Run and review.** [Operations](docs/operations.md) · [Packaged optimization benchmarks](docs/optimization-benchmarks.md) · [Result delivery](docs/result-delivery.md) - **🧠 Use knowledge.** [Skill library](src/inferrsi/knowledge/data/skill_library/README.md) · [CANNbot setup and revocation](docs/external-channels.md#setting-one-up) · [MindIE setup and revocation](docs/codex-plugin.md#enable-and-revoke) -- **📓 Field lessons (exploratory, unverified).** [Apple silicon playbook](docs/apple-silicon-playbook.md) · [Inkling 2 Mini on Apple silicon](docs/inkling2-mini-apple-silicon.md) · [Scheduling RSI lessons](docs/scheduling-rsi-lessons.md) · [h3-metal lessons](docs/h3-metal-apple-silicon-lessons.md) +- **📓 Field lessons.** [Apple silicon playbook](docs/apple-silicon-playbook.md) · [Inkling 2 Mini on Apple silicon](docs/inkling2-mini-apple-silicon.md) (exploratory, unverified) · [Scheduling RSI lessons](docs/scheduling-rsi-lessons.md) (exploratory, unverified) · [h3-metal lessons](docs/h3-metal-apple-silicon-lessons.md) - **🧭 Understand and extend.** [Module layout and import migration](docs/module-layout.md) · [Architecture and directory map](docs/architecture.md) · [Standalone engine template](docs/template/README.md) - **📌 Follow changes.** [Releases](https://github.com/InferMatrixLab/InferRSI/releases) · [Changelog](CHANGELOG.md) diff --git a/docs/inkling2-mini-apple-silicon.md b/docs/inkling2-mini-apple-silicon.md index 38c83f2a..c0a2bcfe 100644 --- a/docs/inkling2-mini-apple-silicon.md +++ b/docs/inkling2-mini-apple-silicon.md @@ -55,7 +55,7 @@ GGUF conversion at the time). Same machine, one runtime at a time (a lock file kept them apart), greedy decoding, fresh process per route. Idle ComfyUI and a small mlx-lm server were resident but not working during the runs. Raw outputs (benchmark logs, -llama-bench JSON, per-run chat timings) are in +llama-bench JSON, per-run chat timings as JSONL, the model output as a separate text file) are in [examples/inkling2_mini_mlx/route](../examples/inkling2_mini_mlx/route/). Both runtimes ran with default settings; this is one exploratory default-against-default comparison, not a runtime ranking. diff --git a/examples/inkling2_mini_mlx/README.md b/examples/inkling2_mini_mlx/README.md index 20ba79fd..a1890431 100644 --- a/examples/inkling2_mini_mlx/README.md +++ b/examples/inkling2_mini_mlx/README.md @@ -12,7 +12,7 @@ Apple-silicon Mac with mlx-lm from ml-explore/mlx-lm#2086 (`0c4c991`). | `rsi_mlx.py` | One search round: load once, alternate each candidate with the default path, paired scoring, token-identity gate, evolve/holdout split | | `mechanism.py` | Per-step graph-building and step time, default vs one candidate | | `count_ops2.py` | Ops in one decode step's graph (total and non-view), default or one candidate | -| `route/` | Raw outputs of the MLX / llama.cpp route comparison (`mlx_lm.benchmark` log, `llama-bench` JSON, per-run chat timings) | +| `route/` | Raw outputs of the MLX / llama.cpp route comparison (`mlx_lm.benchmark` log, `llama-bench` JSON, per-run chat timings in `mlx-chat.jsonl` / `llama-chat.jsonl`, the MLX model output in `mlx-chat-output.txt`) | | `rounds/r1` … `rounds/r4` | Candidates (`cands/*.py`) and the JSONL/JSON records each round wrote | ```bash diff --git a/examples/inkling2_mini_mlx/route/mlx-chat-output.txt b/examples/inkling2_mini_mlx/route/mlx-chat-output.txt new file mode 100644 index 00000000..7a58887c --- /dev/null +++ b/examples/inkling2_mini_mlx/route/mlx-chat-output.txt @@ -0,0 +1,12 @@ +----- 输出 ----- +<|message_model|><|content_thinking|>The user asks in Chinese: write a thread-safe LRU cache class in Python supporting get/put/capacity limit and TTL expiration, with 3 pytest cases. Only output code. Need provide code only. Need ensure correctness. + +Design: class LRUCache with OrderedDict, Lock. get(key): with lock, if key in cache: move_to_end, check expiry: if expired delete and return None (or default). else return value. put(key,value,ttl=None): with lock, if key exists delete; set expiry = time.time()+ttl if ttl else None; cache[key]=(value,expiry); move_to_end; while len>capacity: popitem(last=False). Need capacity maybe maxsize. TTL default from constructor? Support per-put ttl and default ttl. get returns default maybe None. Thread-safe. + +Tests: 3 pytest cases: capacity eviction LRU order; TTL expiration (use small ttl and monkeypatch time or sleep? Better avoid sleep by injecting clock? Could use time.sleep(0.05) with ttl=0.01; acceptable but flaky. Better design class accepts time_func for tests? But user asked class; can include optional clock param. Tests can use fake clock. Simpler: class uses time.time; tests use ttl=0.05 and time.sleep(0.06). 3 cases: 1) LRU eviction order; 2) TTL expiry; 3) thread safety? Need 3 pytest cases. Could do: get/put basic, capacity eviction, TTL expiry. Thread safety test maybe with threads incrementing? But only 3 cases. Include thread-safety in one? Need 3 pytest用例. We can write 3 tests: test_lru_eviction, test_ttl_expiry, test_thread_safety. Thread safety test: create cache capacity large, spawn threads putting/getting, assert no exception and final size <= capacity. Deterministic enough. + +Need only output code, no explanation. Ensure code block only? User says 只输出代码. So final should be code only, likely a single code block. No prose. + +Need be careful with OrderedDict move_to_end and expiry. Implementation: + + diff --git a/examples/inkling2_mini_mlx/route/mlx-chat.jsonl b/examples/inkling2_mini_mlx/route/mlx-chat.jsonl index f3f8ac92..99ff0da6 100644 --- a/examples/inkling2_mini_mlx/route/mlx-chat.jsonl +++ b/examples/inkling2_mini_mlx/route/mlx-chat.jsonl @@ -1,15 +1,3 @@ {"run": 0, "load_s": 2.161633014678955, "ttft_s": 2.1704137325286865, "prompt_tokens": 61, "prefill_tps": 29.1318650445503, "gen_tokens": 512, "decode_tps": 47.640906850511996, "peak_gb": 17.559900508, "wall_s": 12.918859958648682, "kw": {}} {"run": 1, "load_s": 2.161633014678955, "ttft_s": 0.245164155960083, "prompt_tokens": 61, "prefill_tps": 249.52940491322087, "gen_tokens": 512, "decode_tps": 45.252409289809215, "peak_gb": 17.56069738, "wall_s": 11.560818910598755, "kw": {}} {"run": 2, "load_s": 2.161633014678955, "ttft_s": 0.24292707443237305, "prompt_tokens": 61, "prefill_tps": 251.78264390018035, "gen_tokens": 512, "decode_tps": 47.34382293487728, "peak_gb": 17.560628916, "wall_s": 11.058743953704834, "kw": {}} ------ 输出 ----- -<|message_model|><|content_thinking|>The user asks in Chinese: write a thread-safe LRU cache class in Python supporting get/put/capacity limit and TTL expiration, with 3 pytest cases. Only output code. Need provide code only. Need ensure correctness. - -Design: class LRUCache with OrderedDict, Lock. get(key): with lock, if key in cache: move_to_end, check expiry: if expired delete and return None (or default). else return value. put(key,value,ttl=None): with lock, if key exists delete; set expiry = time.time()+ttl if ttl else None; cache[key]=(value,expiry); move_to_end; while len>capacity: popitem(last=False). Need capacity maybe maxsize. TTL default from constructor? Support per-put ttl and default ttl. get returns default maybe None. Thread-safe. - -Tests: 3 pytest cases: capacity eviction LRU order; TTL expiration (use small ttl and monkeypatch time or sleep? Better avoid sleep by injecting clock? Could use time.sleep(0.05) with ttl=0.01; acceptable but flaky. Better design class accepts time_func for tests? But user asked class; can include optional clock param. Tests can use fake clock. Simpler: class uses time.time; tests use ttl=0.05 and time.sleep(0.06). 3 cases: 1) LRU eviction order; 2) TTL expiry; 3) thread safety? Need 3 pytest cases. Could do: get/put basic, capacity eviction, TTL expiry. Thread safety test maybe with threads incrementing? But only 3 cases. Include thread-safety in one? Need 3 pytest用例. We can write 3 tests: test_lru_eviction, test_ttl_expiry, test_thread_safety. Thread safety test: create cache capacity large, spawn threads putting/getting, assert no exception and final size <= capacity. Deterministic enough. - -Need only output code, no explanation. Ensure code block only? User says 只输出代码. So final should be code only, likely a single code block. No prose. - -Need be careful with OrderedDict move_to_end and expiry. Implementation: - - diff --git a/examples/inkling2_mini_mlx/rsi_mlx.py b/examples/inkling2_mini_mlx/rsi_mlx.py index 3b464ee3..4dc951d6 100644 --- a/examples/inkling2_mini_mlx/rsi_mlx.py +++ b/examples/inkling2_mini_mlx/rsi_mlx.py @@ -41,7 +41,8 @@ def pair_candidate(mod, model, run_once, base_runs, ref_ids, reps): try: for i in range(reps): base_runs.append(run_once({})) # default path, alternated with the candidate - kw = mod.apply(model); reverted = False + reverted = False # apply() may change the model before it raises + kw = mod.apply(model) r = run_once(kw); runs.append(r) pairs.append(summary([r])["dec_med"] / summary([base_runs[-1]])["dec_med"]) # paired with the adjacent default run, cancelling thermal drift mod.revert(model); reverted = True @@ -57,6 +58,19 @@ def pair_candidate(mod, model, run_once, base_runs, ref_ids, reps): return dict(s, paired=pairs, same_output=same, err=err, reverted=reverted, doc=(mod.__doc__ or "").strip()) +def run_round(cands, model, run_once, base_runs, ref_ids, reps, emit): + """Pair each (name, module) in turn. If a candidate could not be reverted, the model may no longer be + the default path, so later default runs would be a polluted baseline: stop the round there.""" + results = {} + for name, mod in cands: + results[name] = pair_candidate(mod, model, run_once, base_runs, ref_ids, reps) + emit(dict(cand=name, **results[name])) + if not results[name]["reverted"]: + emit(dict(stopped=f"{name}: revert not confirmed; round stopped, reload the model before continuing")) + break + return results + + def score(r, reps): """Verdict for one candidate. Only complete, error-free, reverted runs that pass the token gate are scored.""" if r["err"] or not r["reverted"] or len(r["paired"]) != reps: @@ -105,15 +119,14 @@ def emit(d): run_once({}) # warm-up: the first pass compiles Metal kernels; not scored base_runs = [run_once({})] ref_ids = [x["ids"] for x in base_runs[0]] - results = {} cands = sorted(p for p in glob.glob(os.path.join(a.round_dir, "cands", "*.py")) if not os.path.basename(p).startswith("_")) if a.only: cands = [c for c in cands if a.only in c] - for path in cands: + def load_cand(path): name = os.path.basename(path)[:-3] spec = importlib.util.spec_from_file_location(name, path); mod = importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) - results[name] = pair_candidate(mod, model, run_once, base_runs, ref_ids, a.reps) - emit(dict(cand=name, **results[name])) + return name, mod + results = run_round((load_cand(p) for p in cands), model, run_once, base_runs, ref_ids, a.reps, emit) b = summary(base_runs) noise = (max(b["dec_all"]) - min(b["dec_all"])) / b["dec_med"] diff --git a/tests/test_inkling2_mlx_round_scoring.py b/tests/test_inkling2_mlx_round_scoring.py index 64ce989e..1a818117 100644 --- a/tests/test_inkling2_mlx_round_scoring.py +++ b/tests/test_inkling2_mlx_round_scoring.py @@ -63,3 +63,42 @@ def test_shorter_output_with_equal_prefix_fails_token_gate(): r = rsi.pair_candidate(cand, None, runner(cand), [sample(40.0)], REF, reps=3) assert not r["same_output"] assert rsi.score(r, reps=3)["verdict"] == "output changed: rejected" + + +class ApplyRaises(Cand): + """apply() changes the model, then raises.""" + + def apply(self, model): + self.applied += 1 + raise RuntimeError("apply half done") + + +class RevertFails(Cand): + def revert(self, model): + self.reverted += 1 + raise RuntimeError("revert failed") + + +def test_apply_that_raises_after_changing_the_model_is_reverted(): + cand = ApplyRaises() + r = rsi.pair_candidate(cand, None, runner(cand), [sample(40.0)], REF, reps=3) + assert "apply half done" in r["err"] + assert cand.reverted == 1 and r["reverted"] + assert rsi.score(r, reps=3) == {"verdict": "INCONCLUSIVE"} + + +def test_unconfirmed_revert_stops_the_round(): + bad, good = RevertFails(), Cand() + calls = [] + + def run_once(kw): + calls.append(kw) + return sample(44.0 if kw else 40.0) + + events = [] + results = rsi.run_round([("bad", bad), ("good", good)], None, run_once, [sample(40.0)], REF, 3, events.append) + assert list(results) == ["bad"] and not results["bad"]["reverted"] + assert "revert failed" in results["bad"]["err"] + assert good.applied == 0 and len(calls) == 2 # no default run after the failed revert + assert "round stopped" in events[-1]["stopped"] + assert rsi.score(results["bad"], reps=3) == {"verdict": "INCONCLUSIVE"} From 655c3dc8268278d115585494dc7e4279b50457c8 Mon Sep 17 00:00:00 2001 From: zuiho-kai <31877877+zuiho-kai@users.noreply.github.com> Date: Sat, 21 Nov 2026 17:59:39 +0900 Subject: [PATCH 11/11] docs(inkling2, scheduling): drop unaccepted performance figures; fix test line length AGENTS.md (Issues and claims, Research lock): no published numbers without an acceptance issue. Removed every speed, latency, memory, hit-rate, capacity and speedup figure from the Inkling page, the scheduling lessons and the playbook rows; kept method and failure lessons. Raw records under examples/inkling2_mini_mlx/rounds and route stay as unverified artifacts. Wrap a test line over the 100-column ruff limit. --- docs/apple-silicon-playbook.md | 4 +- docs/inkling2-mini-apple-silicon.md | 141 ++++++++--------------- docs/scheduling-rsi-lessons.md | 54 ++++----- tests/test_inkling2_mlx_round_scoring.py | 4 +- 4 files changed, 81 insertions(+), 122 deletions(-) diff --git a/docs/apple-silicon-playbook.md b/docs/apple-silicon-playbook.md index a8a9d0f6..84dd74d3 100644 --- a/docs/apple-silicon-playbook.md +++ b/docs/apple-silicon-playbook.md @@ -67,8 +67,8 @@ From [Inkling 2 Mini on Apple silicon](inkling2-mini-apple-silicon.md) (measured | The release checkpoint (ModelOpt NVFP4/MXFP8) loads in neither MLX nor llama.cpp. | Apple runtimes need their own quantized formats. | Start from a community conversion; pin repository, revision and file SHA-256. | measured | | Shards arrived larger than published after a segmented download. | curl `--retry` inside a ranged, appending segment resends from the range start. | Verify each file against the Hub `lfs.oid` before the first load. | measured | | A 512-token cap ended inside the thinking block although reasoning effort was set to `none`. | The template value only adds a system line. | Size generation caps for thinking plus answer; count both. | measured | -| A looped MoE model decoded at about 48 tok/s on MLX while weight traffic allowed far more. | 48 layer visits per token, each with many small ops: launch- and graph-building-bound. | Profile one layer visit at decode shape before compiling. Per-module compiles gave 1–3% and a fused layer half 6.4%; two compiled graphs per layer visit around the cache write, a one-kernel mask, fused projections and one-kernel short convs measured 1.49× (paired median, two prompts, exploratory) with ops per step 7,624 → 3,664. | measured, unverified | -| During a search round the default path drifted from 47.4 to 48.9 tok/s. | Warm-up/thermal drift over minutes. | Pair each candidate run with the adjacent default run; an all-runs median inflated one candidate from 1.048 to 1.053. | measured | +| A looped MoE model was far below what its weight traffic would allow at decode. | Many layer visits per token, each with many small ops: launch- and graph-building-bound. | Profile one layer visit at decode shape before compiling. Per-module compiles moved little; rebuilding the single-token layer path (compiled graphs around the cache write, a one-kernel mask, fused projections, one-kernel short convs) cut ops per step. No speedup is claimed. | measured, unverified | +| During a search round the default path drifted upward. | Warm-up/thermal drift over minutes. | Pair each candidate run with the adjacent default run; an all-runs median inflated one candidate. | measured, unverified | ## Benchmarking a shared Mac diff --git a/docs/inkling2-mini-apple-silicon.md b/docs/inkling2-mini-apple-silicon.md index c0a2bcfe..12474676 100644 --- a/docs/inkling2-mini-apple-silicon.md +++ b/docs/inkling2-mini-apple-silicon.md @@ -1,14 +1,13 @@ # Inkling 2 Mini on Apple silicon: local adaptation and four exploratory RSI rounds This is an unverified exploration record, not a performance claim. It has no -acceptance issue and is not an InferRSI PASS. It describes how Inkling 2 Mini -was brought up on one M5 Max with MLX and llama.cpp, and four InferRSI-style -search rounds on MLX. Every number below holds only for the stated machine, -versions, settings and prompts. The round numbers can be recomputed from the -committed records in -[examples/inkling2_mini_mlx](../examples/inkling2_mini_mlx/README.md); the -route comparison links its raw outputs; other measured rows rest on -unpublished maintainer logs. +acceptance issue and is not an InferRSI PASS, so it publishes no speed, +latency or memory figures and no speedup ratios. It records how Inkling 2 +Mini was brought up on one M5 Max with MLX and llama.cpp, how four +InferRSI-style search rounds on MLX were run, and the method and failure +lessons from them. The raw measurements stay as unverified records in +[examples/inkling2_mini_mlx](../examples/inkling2_mini_mlx/README.md) +(`route/` and `rounds/r*/`). [Inkling 2 Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) is a 26B-total, MoE, looped transformer from Thinking Machines Lab (24 layers; the @@ -16,8 +15,8 @@ first and last 6 run once, the middle 12 run three times; about 3.4B active parameters, 5.4B effective with the loop). The release checkpoint is ModelOpt NVFP4 experts with MXFP8 tail layers (about 20 GB) and does not load in Apple runtimes directly. This page records how it was brought up on one Mac with -community conversions, which runtime was faster, and one InferRSI-style search -round on that runtime. Measurement rules follow the +community conversions, how the two runtimes were compared, and how +InferRSI-style search rounds were run on MLX. Measurement rules follow the [scheduling RSI lessons](scheduling-rsi-lessons.md) and the [Apple silicon playbook](apple-silicon-playbook.md). Nothing here is an InferRSI acceptance result. @@ -60,18 +59,14 @@ llama-bench JSON, per-run chat timings as JSONL, the model output as a separate Both runtimes ran with default settings; this is one exploratory default-against-default comparison, not a runtime ranking. -| Metric | MLX 4-bit | llama.cpp Q4_K_M | Command | -| --- | ---: | ---: | --- | -| Prefill, 512-token prompt (tok/s, mean of 3) | 1,677 | 998 | `mlx_lm.benchmark -p 512 -g 128 -n 3`; `llama-bench -p 512 -n 128 -r 3` | -| Decode, 128 tokens (tok/s, mean of 3) | 47.0 | 41.7 | same | -| Decode in a 512-token chat answer (tok/s, 3 runs) | 45.3–47.6 | 41.5–43.4 | `stream_generate`; llama-server `/v1/chat/completions` | -| Time to first token, 61-token chat prompt, warm (s) | 0.24 | 0.29–0.31 | same | -| Peak memory | 17.6 GB (`mx.get_peak_memory`), 19.0 GB in the benchmark | 17.6 GB resident, of which weights are file-mapped | | +Commands: `mlx_lm.benchmark -p 512 -g 128 -n 3` and +`llama-bench -p 512 -n 128 -r 3`, then three 512-token chat answers through +`stream_generate` and llama-server `/v1/chat/completions`. -In this comparison MLX measured faster on both prefill and decode, so the -search rounds below used MLX. -The first MLX request of a process took 2.17 s to first token because Metal -kernels compile on first use; warm numbers exclude it. +The search rounds below used MLX, which was ahead of llama.cpp on both +prefill and decode in this one comparison. The first MLX request of a process +is slow to first token because Metal kernels compile on first use; time warm +requests separately from the first one. ## RSI round @@ -106,8 +101,8 @@ its output. The verdict strings stored in the committed `score-*.json` records ("keep", and "留" in round 1) were written by earlier versions of the scorer and are -superseded by the rules above; the paired ratios in the records are what this -page uses. +superseded by the rules above; the raw paired ratios in the records are kept +unchanged. **Selection and holdout.** Within each round, every candidate was run on the evolve and the holdout prompts in the same session. There is no record that @@ -115,72 +110,38 @@ the shipped candidate was fixed from the evolve scores and the 5% rule before the holdout results were read, so both splits are reported as exploratory data, not as a held-out confirmation. -**Round 1** (evolve, 256 generated tokens, noise 2.9%). Paired medians: -gates 1.031, MoE 1.015, depth residual 1.012, gates + MoE 1.048. All -token-identical, all below 5%: no conclusion. The first version of the scorer -compared with the median of all default runs; the default path drifted from -47.4 to 48.9 tok/s during the round, which inflated gates + MoE to 1.053. The -scorer was changed to adjacent pairing before round 2. The "留" verdict and -1.053 still stored in `rounds/r1/score-evolve.json` come from that superseded -scorer; the paired figures above replace them. - -**Round 2** (proposed from round 1: combine the compile points; fuse the whole -MLP half of each layer). - -| Candidate | Evolve (noise 2.3%) | Holdout (noise 3.9%) | Output | -| --- | ---: | ---: | --- | -| gates + MoE (round-1 best, re-run) | 1.043 | 1.054 | identical | -| gates + MoE + depth residual | 1.057 | 1.056 | identical | -| fused MLP half per layer | 1.044 | 1.039 | identical | -| **fused MLP half + q/k/v gates** | **1.068** | **1.064** (1.068, 1.064, 1.060) | identical | - -Mechanism (one decode step, synchronous, median of 50 steps, two alternating -processes each): 22.07 → 20.94 ms per step; Python graph building -3.89 → 3.34 ms. About half of the saving is CPU graph construction and the -rest is GPU time from fewer, fused kernels. The winning patch is -`rounds/r2/cands/r2d_fused_half_gates.py`; it changes no weights and no -numerics outside the compiled regions. - -**Round 3** (proposed from a per-module profile: at decode shape the -attention half of a layer visit cost ~240 µs against ~90 µs for the MoE, -nearly all in small ops). Each layer visit for B = 1, L = 1 is split into -three parts: everything before the KV-cache write as one compiled graph, the -cache write and attention eager (their length changes every step), everything -after as a second compiled graph; conv states become explicit inputs and -outputs. Options on top: the attention mask/bias (arange, compare, gather, -where, cast) as one Metal kernel; q/r/gate/sink projections concatenated into -one quantized matmul (all six in sliding layers without loop messages); -message-gate forget/message projections as one matmul. - -| Candidate | Evolve (noise 9.1%) | Holdout (noise 6.8%) | Output | -| --- | ---: | ---: | --- | -| three-part layer path | 1.290 | 1.290 | identical | -| + mask kernel | 1.354 | 1.353 | identical | -| + fused q/r/gate/sink | 1.430 | **1.419** | identical | -| + all six projections in one | 1.424 | 1.396 | identical | -| + fused message-gate projections | 1.375 | 1.392 | identical | - -The last two are within 5% of the third: no conclusion. - -**Round 4** (proposed from the op count after round 3: the 4-tap short -convolutions, four per layer visit, were each a concatenate, two casts, a -generic convolution, an add/divide and a state slice). - -| Candidate | Evolve (noise 12.2%) | Holdout (noise 9.2%) | Output | -| --- | ---: | ---: | --- | -| **round-3 options + one-kernel short convs** | **1.486** | **1.491** (1.462, 1.491, 1.515) | identical | -| round-3 options, re-run as reference | 1.327 | 1.409 | identical | -| + q/k/v message gates as one batched matmul | 1.516 | 1.501 | identical | - -Batched gates are within 5% of the shipped candidate: no conclusion, not -shipped. Default-path tok/s varied 43–48 across rounds (thermal); only the -adjacent-paired ratios are compared. - -Mechanism, round 4 candidate against default (one decode step, median of 50 -steps, two alternating processes each; non-view ops counted from the step -graph): step 22.1 → 15.3 ms, Python graph building 3.9 → 1.8 ms, non-view -ops per step 7,624 → 3,664. The weights read per token (~2.5 GB) would take -about 5 ms at the machine's bandwidth, so the step remains launch-bound. +**What each round tried and learned.** The per-candidate paired ratios, +noise ranges and mechanism timings are in the committed `rounds/r*/` +records; they are not summarised here. + +- Round 1: `mx.compile` around single sub-blocks (message gates, MoE, depth + residual, gates + MoE). Every candidate was token-identical and below the + 5% rule: no conclusion. The default path drifted upward during the round; + the first scorer compared each candidate with the median of all default + runs, which inflated one candidate, so the scorer was changed to adjacent + pairing before round 2. The "留" verdict and ratio stored in + `rounds/r1/score-evolve.json` come from that superseded scorer. +- Round 2 (proposed from round 1): combined compile points and a fused MLP + half per layer. Lesson: per-module compiles move little on a launch-bound + model; the mechanism probe attributed part of the saving to CPU graph + construction and part to fewer GPU kernels. +- Round 3 (proposed from a per-module profile at decode shape, where the + attention half of a layer visit was dominated by small ops): each layer + visit for B = 1, L = 1 split into a compiled graph before the KV-cache write, + the cache write and attention eager (their length changes every step), and a + compiled graph after; conv states become explicit inputs and outputs. + Options on top: the attention mask/bias as one Metal kernel; q/r/gate/sink + projections in one quantized matmul; message-gate projections in one + matmul. Several options were within 5% of each other: no conclusion between + them. +- Round 4 (proposed from the op count after round 3: each 4-tap short + convolution was a chain of small ops): one-kernel short convolutions on top + of the round-3 options. The mechanism probe showed fewer non-view ops and + less Python graph building per step, and the step stayed launch-bound + rather than bandwidth-bound. A batched message-gate variant was within 5% + of it: no conclusion, not shipped. +- Across rounds the default path's own speed moved with temperature, so only + adjacent-paired ratios are compared, never absolute speeds across rounds. **How candidates were produced.** The candidates were written by the assistant running the campaign (Claude Code, `claude-opus-5-5`) from reading @@ -191,7 +152,7 @@ per-module profile and an op count of one decode step. ## What we did not establish - No acceptance issue, frozen campaign evidence bundle or independent rerun; - all speed figures are exploratory observations under the setup above. + no performance figure is established, and none is published here. - One machine, one macOS build, 128 GB; nothing on 32–64 GB Macs, which the model card lists as supported. - Single stream only; no batching, concurrency or long-context runs (the diff --git a/docs/scheduling-rsi-lessons.md b/docs/scheduling-rsi-lessons.md index 6ed95377..d4ff1f23 100644 --- a/docs/scheduling-rsi-lessons.md +++ b/docs/scheduling-rsi-lessons.md @@ -1,14 +1,11 @@ # Scheduling RSI lessons: Ascend queue ordering and agent KV eviction -This is an unverified exploration record. The quantitative findings below come -from the maintainers' unpublished campaign ledger or from replays whose -outputs are not published here. They have no acceptance issue, are not -reproducible from this page and are not InferRSI PASS results; Campaign 1 is a -simulation and was not run on NPU hardware. They are kept as lessons about method, not -as established performance gains. For example, the best waiting-queue order -in Campaign 1 (prefix-cache hit share of the prompt, highest first) was -recorded in the ledger at 4.61× simulated capacity over the Ascend -shortest-job-first baseline; that figure is unverified. +This is an unverified exploration record. The two campaigns below have no +acceptance issue, their measurements are in the maintainers' unpublished +campaign ledger, Campaign 1 is a simulation that was not run on NPU hardware, +and nothing here is an InferRSI PASS. The page therefore publishes no +capacity, latency, hit-rate or speedup figures; it keeps only the method and +the failure modes. This page records two earlier campaigns of the same RSI line (model proposes a policy → a frozen scorer measures it → keep the better, drop the worse) so that @@ -29,24 +26,23 @@ the headline. Apple-silicon lessons from later work are in the Setup (ledger): real vLLM engine with only the device operators replaced by stand-ins; a virtual clock calibrated from another team's operator timing -table, step time ≈ 1.313 ms + 0.00468 ms × tokens in the step, error ≤ 3%; -393 multi-turn sessions in which 98.7% of requests contain the whole previous -request; baseline prefix reuse about 14%. Eight candidates ran in parallel; -more saturated the CPU. Wall time of the simulator was 5–45× faster than the -device and was never used as the metric. +table (step time as a linear function of tokens in the step); multi-turn +sessions in which almost every request contains the whole previous request. +Several candidates ran in parallel until the CPU saturated. Simulator wall time +was never used as the metric. | Observation | Cause | Action | Source | | --- | --- | --- | --- | -| Ordering by *absolute* tokens saved, fewest first: 1.75×; most first: 2.47×; by *share* of the prompt already cached, highest first: 4.61×. | Share favours requests whose remaining prefill is small relative to what is already resident, so cache pages are reused before eviction. | Try the normalised form of a key as well as the absolute one. | ledger | -| 4.61× is the load at which the *mean* TTFT equals 10 s. Requiring every run to stay under the bound gives about 3.9×. | Mean-at-bound and every-run-under-bound are different operating points. | Publish the conservative point next to the headline and name which one is quoted. | ledger | -| At the same 10 s mean, median TTFT moved 1.5 → 5.7 s and the shortest quarter of prompts 1.8 → 9.5 s; the longest-to-mean ratio fell 1.80 → 1.28. | The policy equalises latency: long prompts gain, short cold prompts wait. | Report latency by prompt-length band, not only capacity. State the precondition: prefill-dominated load and no hard sub-second SLO for short requests. | ledger | -| Run-to-run noise was about 4%. Four successive "guaranteed floor" values were each inside the noise and were all withdrawn. | Single-seed differences below the noise are not results. | Below 5% difference: no conclusion. A "better" verdict also needs a mechanism signal (hit rate up, recompute down, queue shorter). | ledger | -| Split validation: sessions halved into an evolution half and an exam half gave 2.06× and 2.01× at the baseline's bound load. | Checks the policy is not fitted to the sessions it was found on. | Always hold out sessions. Note that this split used a larger cache pool than the 4.61× figure; the two numbers must not be combined. | ledger | -| Parameter-level search: −11.2% mean TTFT in 4 generations, then 30 generations without change; the last 80-generation round produced 1 new candidate and 79 duplicates. | The parameter plane was exhausted; the proposer kept regenerating known candidates. | De-duplicate before scoring. When a plane stops moving, switch planes rather than spend generations. | ledger | -| Structure-level search, first round of 5 candidates: only gate tuning helped (10.5%). Later: place by free slots 33.2%, add a starvation quota (1 slot in 8 for the longest waiter) 38.1%, reserve one quarter of slots 28.2%, order by waiting time only 23.6%. | Structural changes reach gains the parameter plane cannot. | Expect most first-round structural ideas to fail; keep the failures in the trial index. | ledger | +| Ordering by the *share* of the prompt already cached, highest first, ranked above ordering by *absolute* tokens saved (either direction). | Share favours requests whose remaining prefill is small relative to what is already resident, so cache pages are reused before eviction. | Try the normalised form of a key as well as the absolute one. | ledger | +| The capacity at which the *mean* TTFT meets the bound was clearly higher than the capacity at which *every* run stays under it. | Mean-at-bound and every-run-under-bound are different operating points. | Publish the conservative point next to any headline and name which one is quoted. | ledger | +| At the same mean TTFT, median TTFT and the shortest prompts' TTFT got worse while the longest-to-mean ratio fell. | The policy equalises latency: long prompts gain, short cold prompts wait. | Report latency by prompt-length band, not only capacity. State the precondition: prefill-dominated load and no hard sub-second SLO for short requests. | ledger | +| Several successive "guaranteed floor" values were each inside the run-to-run noise and were all withdrawn. | Single-seed differences below the noise are not results. | Below 5% difference: no conclusion. A "better" verdict also needs a mechanism signal (hit rate up, recompute down, queue shorter). | ledger | +| Split validation (sessions halved into an evolution half and an exam half) gave similar results on both halves, but under a larger cache pool than the headline run. | Checks the policy is not fitted to the sessions it was found on. | Always hold out sessions, and never combine figures measured under different cache pools. | ledger | +| Parameter-level search improved for a few generations and then stopped moving; a late round produced almost only duplicates of known candidates. | The parameter plane was exhausted; the proposer kept regenerating known candidates. | De-duplicate before scoring. When a plane stops moving, switch planes rather than spend generations. | ledger | +| In the first structure-level round only gate tuning helped; later structural ideas (placing by free slots, a starvation quota for the longest waiter, reserving slots, ordering by waiting time) varied widely. | Structural changes reach gains the parameter plane cannot. | Expect most first-round structural ideas to fail; keep the failures in the trial index. | ledger | | The champion came from a human-written requirement route, not from the automatic mutation chain. | — | Attribute candidates to their actual origin; write "measured on the same scorer", not "found by the model", unless the chain produced it. | ledger | | Most human effort went into making candidates run on both the NPU path and the simulator (interfaces, parameter adaptation, debugging). | Two execution targets. | Budget the adaptation work explicitly; it is the cost driver, not proposal generation. | ledger | -| At the highest load, recompute was only ten-odd percentage points from perfect reuse with ordering alone, and cold short requests remained slower. | The ordering plane is close to its ceiling. | Next plane: cache residency and admission. | ledger, inferred | +| At the highest load, ordering alone came close to perfect reuse, and cold short requests remained slower. | The ordering plane is close to its ceiling. | Next plane: cache residency and admission. | ledger, inferred | Not established: NPU hardware confirmation of the simulated ranking; paired per-session statistics instead of the 5% rule; a separate held-out audit split @@ -59,15 +55,15 @@ hit rate. Code was contributed to [JiusiServe/vllm-evolve#1](https://github.com/JiusiServe/vllm-evolve/pull/1) (public). Primary trace: Weka cc-traces with sub-agents ([HF dataset](https://huggingface.co/datasets/semianalysisai/cc-traces-weka-with-subagents-051926), public); -219 traces, about 78,000 requests. +219 traces. | Observation | Cause | Action | Source | | --- | --- | --- | --- | -| Trace chains are idle 91% of the time; root agents idle p50 2.0 s / p90 35 s, sub-agents p50 0.54 s. | Tool calls and model turns alternate. | Expected return time is a usable eviction feature; sub-agent KV should stay resident before root-agent KV. | ledger, inferred | -| With 32 sessions and device memory at 10% of the active working set, LRU hit rate 51% against an upper bound of 83% (Belady 78%, prefetch-only 53%); at 25%: 88% against 97%. | The headroom comes from eviction order by expected return, not from prefetch. | Agent-aware eviction is worth searching only when memory is tight; measure the gap to Belady before searching. | ledger | -| After pinning the blocks of running requests, LRU and Belady differed by only 0.7 points (10%) and 3.3 points (25%); without pinning, by 26 points. | The unpinned replay credited evictions a real engine cannot make. | Model engine constraints (pinning, admission queueing) in the scorer before quoting headroom. | ledger | -| A 30-generation exploratory search cost 5.15 USD and moved mean hit rate 60.93% → 61.13%. | Small remaining headroom after pinning; the gain is inside noise. | Compute the reachable headroom first; do not spend search budget on a plane with < 1 point to gain. | ledger | -| On a real vLLM 0.20 engine with Qwen2.5-7B, the evolved eviction function loaded through `--scheduler-cls`, evicted by score, and produced 192/192 identical outputs under batch-invariant mode. | — | The hardware test proves the plug-in runs and keeps outputs; it does not score the policy. Gains come from the RSI scorer only. | ledger | +| Trace chains are idle most of the time; sub-agents return sooner than root agents. | Tool calls and model turns alternate. | Expected return time is a usable eviction feature; sub-agent KV should stay resident before root-agent KV. | ledger, inferred | +| With device memory small relative to the active working set, LRU left a large gap to the offline upper bound, and the gap came from eviction order, not prefetch; with more memory the gap shrank. | The headroom comes from eviction order by expected return, not from prefetch. | Agent-aware eviction is worth searching only when memory is tight; measure the gap to Belady before searching. | ledger | +| After pinning the blocks of running requests, the gap between LRU and Belady almost disappeared; without pinning it looked large. | The unpinned replay credited evictions a real engine cannot make. | Model engine constraints (pinning, admission queueing) in the scorer before quoting headroom. | ledger | +| An exploratory search after pinning moved mean hit rate only inside the noise. | Small remaining headroom after pinning. | Compute the reachable headroom first; do not spend search budget on a plane with almost nothing to gain. | ledger | +| On a real vLLM 0.20 engine with Qwen2.5-7B, the evolved eviction function loaded through `--scheduler-cls`, evicted by score, and kept outputs identical under batch-invariant mode. | — | The hardware test proves the plug-in runs and keeps outputs; it does not score the policy. Gains come from the RSI scorer only. | ledger | | Locally collected agent sessions were rejected as a dataset. | They have no external standing. | Use public traces already used by papers or benchmarks (Weka cc-traces, FreeInference, TraceLab). | ledger | ## Campaign gates inherited from both @@ -87,7 +83,7 @@ harness): 5. **Daily split.** Record how much of the day went to the hypothesis and how much to the harness. 6. **Kernel gain is not end-to-end gain.** In a sibling InferMatrixLab engine - a kernel that was 13% faster in isolation moved end-to-end time by 0.4% + a kernel that was clearly faster in isolation barely moved end-to-end time (public, `InferMatrixLab/system1-omni`). Score candidates on the full generation path only. 7. **Freeze the rival.** Thresholds written before the run, rival diff --git a/tests/test_inkling2_mlx_round_scoring.py b/tests/test_inkling2_mlx_round_scoring.py index 1a818117..59e0727c 100644 --- a/tests/test_inkling2_mlx_round_scoring.py +++ b/tests/test_inkling2_mlx_round_scoring.py @@ -96,7 +96,9 @@ def run_once(kw): return sample(44.0 if kw else 40.0) events = [] - results = rsi.run_round([("bad", bad), ("good", good)], None, run_once, [sample(40.0)], REF, 3, events.append) + results = rsi.run_round( + [("bad", bad), ("good", good)], None, run_once, [sample(40.0)], REF, 3, events.append + ) assert list(results) == ["bad"] and not results["bad"]["reverted"] assert "revert failed" in results["bad"]["err"] assert good.applied == 0 and len(calls) == 2 # no default run after the failed revert