diff --git a/README.md b/README.md index 877c255c..a22200ac 100644 --- a/README.md +++ b/README.md @@ -162,6 +162,7 @@ request per engine and measure audio frames per second and median time to first - **▶️ Run and review.** [Operations](docs/operations.md) · [Packaged optimization benchmarks](docs/optimization-benchmarks.md) · [Result delivery](docs/result-delivery.md) - **🧠 Use knowledge.** [Skill library](src/inferrsi/knowledge/data/skill_library/README.md) · [CANNbot setup and revocation](docs/external-channels.md#setting-one-up) · [MindIE setup and revocation](docs/codex-plugin.md#enable-and-revoke) +- **📓 Field lessons.** [Apple silicon playbook](docs/apple-silicon-playbook.md) · [Inkling 2 Mini on Apple silicon](docs/inkling2-mini-apple-silicon.md) (exploratory, unverified) · [Scheduling RSI lessons](docs/scheduling-rsi-lessons.md) (exploratory, unverified) · [h3-metal lessons](docs/h3-metal-apple-silicon-lessons.md) - **🧭 Understand and extend.** [Module layout and import migration](docs/module-layout.md) · [Architecture and directory map](docs/architecture.md) · [Standalone engine template](docs/template/README.md) - **📌 Follow changes.** [Releases](https://github.com/InferMatrixLab/InferRSI/releases) · [Changelog](CHANGELOG.md) diff --git a/docs/apple-silicon-playbook.md b/docs/apple-silicon-playbook.md index b2699496..84dd74d3 100644 --- a/docs/apple-silicon-playbook.md +++ b/docs/apple-silicon-playbook.md @@ -58,6 +58,18 @@ acceptance result. | A candidate kernel used 42,240 bytes of threadgroup memory and failed only at run time. | The limit is 32 KB. | Build every Metal pipeline before timing so the failure is a build failure. | ledger | | PyTorch/diffusers could not run the model on MPS. | `aten::_int_mm` is not implemented for MPS. | Check the int8 operator set before choosing PyTorch MPS as a baseline. | ledger (2026-10-24) | +## Bringing up a new model + +From [Inkling 2 Mini on Apple silicon](inkling2-mini-apple-silicon.md) (measured on the same M5 Max; exploratory record without an acceptance issue, unverified). + +| Observation | Cause | Action | Source | +| --- | --- | --- | --- | +| The release checkpoint (ModelOpt NVFP4/MXFP8) loads in neither MLX nor llama.cpp. | Apple runtimes need their own quantized formats. | Start from a community conversion; pin repository, revision and file SHA-256. | measured | +| Shards arrived larger than published after a segmented download. | curl `--retry` inside a ranged, appending segment resends from the range start. | Verify each file against the Hub `lfs.oid` before the first load. | measured | +| A 512-token cap ended inside the thinking block although reasoning effort was set to `none`. | The template value only adds a system line. | Size generation caps for thinking plus answer; count both. | measured | +| A looped MoE model was far below what its weight traffic would allow at decode. | Many layer visits per token, each with many small ops: launch- and graph-building-bound. | Profile one layer visit at decode shape before compiling. Per-module compiles moved little; rebuilding the single-token layer path (compiled graphs around the cache write, a one-kernel mask, fused projections, one-kernel short convs) cut ops per step. No speedup is claimed. | measured, unverified | +| During a search round the default path drifted upward. | Warm-up/thermal drift over minutes. | Pair each candidate run with the adjacent default run; an all-runs median inflated one candidate. | measured, unverified | + ## Benchmarking a shared Mac | Observation | Cause | Action | Source | diff --git a/docs/inkling2-mini-apple-silicon.md b/docs/inkling2-mini-apple-silicon.md new file mode 100644 index 00000000..12474676 --- /dev/null +++ b/docs/inkling2-mini-apple-silicon.md @@ -0,0 +1,167 @@ +# Inkling 2 Mini on Apple silicon: local adaptation and four exploratory RSI rounds + +This is an unverified exploration record, not a performance claim. It has no +acceptance issue and is not an InferRSI PASS, so it publishes no speed, +latency or memory figures and no speedup ratios. It records how Inkling 2 +Mini was brought up on one M5 Max with MLX and llama.cpp, how four +InferRSI-style search rounds on MLX were run, and the method and failure +lessons from them. The raw measurements stay as unverified records in +[examples/inkling2_mini_mlx](../examples/inkling2_mini_mlx/README.md) +(`route/` and `rounds/r*/`). + +[Inkling 2 Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) is a +26B-total, MoE, looped transformer from Thinking Machines Lab (24 layers; the +first and last 6 run once, the middle 12 run three times; about 3.4B active +parameters, 5.4B effective with the loop). The release checkpoint is ModelOpt +NVFP4 experts with MXFP8 tail layers (about 20 GB) and does not load in Apple +runtimes directly. This page records how it was brought up on one Mac with +community conversions, how the two runtimes were compared, and how +InferRSI-style search rounds were run on MLX. Measurement rules follow the +[scheduling RSI lessons](scheduling-rsi-lessons.md) and the +[Apple silicon playbook](apple-silicon-playbook.md). Nothing here is an +InferRSI acceptance result. + +## Machine and sources + +- Apple M5 Max, 128 GB unified memory, macOS 27.0 (26A425). +- Labels: **measured** = run on this machine; unless a row links raw outputs, + the logs are in the maintainers' run directory (not published), so treat it + as unverified; **public** = model card or repository at the link given. + +| Route | Weights | Runtime | Source | +| --- | --- | --- | --- | +| MLX | [mlx-community/Inkling-2-Mini-4bit](https://huggingface.co/mlx-community/Inkling-2-Mini-4bit) (17.3 GB) | mlx-lm from [ml-explore/mlx-lm#2086](https://github.com/ml-explore/mlx-lm/pull/2086) (`0c4c991`), mlx 0.32.4 | public | +| llama.cpp | [lmstudio-community/Inkling-2-Mini-GGUF](https://huggingface.co/lmstudio-community/Inkling-2-Mini-GGUF) Q4_K_M (17.5 GB) | llama.cpp main `418fe97`, Metal | public | + +Not run: Ollama (not installed on the machine), vLLM/SGLang (no Apple +backend for this model), the variable-loop research model +`TML-d24-varloop-research` (released only in the ModelOpt format, no MLX or +GGUF conversion at the time). + +## Bring-up + +| Observation | Cause | Action | Source | +| --- | --- | --- | --- | +| The release checkpoint does not load in MLX or llama.cpp. | Experts are ModelOpt NVFP4 and the last 4 layers MXFP8; Apple runtimes need their own formats. | Use a community conversion and pin it by repository and file hash. | public | +| Released mlx-lm 0.32.2 has no `inkling2_loop` model. | Support is in an open pull request. | Install mlx-lm from [#2086](https://github.com/ml-explore/mlx-lm/pull/2086) (`0c4c991`); llama.cpp main already has `inkling2_loop`. | measured | +| Two shards arrived larger than their published size (5,140,845,290 bytes against 5,031,793,948; two GGUF segments over-long). | The segmented downloader appended curl's own `--retry` restarts, which resend the range from its start. | Verify every shard against the Hub `lfs.oid` SHA-256 before the first load; re-fetch only segments whose size is wrong. | measured | +| With `reasoning_effort="none"` (template value 0) the model still thinks first: a 512-token cap ended inside the thinking block on both runtimes. | The chat template only adds a system line; the model is not forced to skip thinking. | Give code-generation tests a cap that covers thinking plus answer (4,096 here); count thinking and answer tokens separately. | measured | +| A complete code-generation run on MLX produced 2,244 tokens (thinking, then a 78-line Python file with 3 pytest cases); the 3 tests pass. | — | Keep one end-to-end generation with a checked answer next to any speed number. | measured | +| llama-server's streamed answer appears in `reasoning_content`, not `content`, until thinking ends; repeated requests reused the cached prompt (4 of 61 prompt tokens processed). | Server-side reasoning parsing and prompt cache. | Read both fields for time to first token and send `cache_prompt: false` when timing. | measured | + +## Route comparison + +Same machine, one runtime at a time (a lock file kept them apart), greedy +decoding, fresh process per route. Idle ComfyUI and a small mlx-lm server were +resident but not working during the runs. Raw outputs (benchmark logs, +llama-bench JSON, per-run chat timings as JSONL, the model output as a separate text file) are in +[examples/inkling2_mini_mlx/route](../examples/inkling2_mini_mlx/route/). +Both runtimes ran with default settings; this is one exploratory +default-against-default comparison, not a runtime ranking. + +Commands: `mlx_lm.benchmark -p 512 -g 128 -n 3` and +`llama-bench -p 512 -n 128 -r 3`, then three 512-token chat answers through +`stream_generate` and llama-server `/v1/chat/completions`. + +The search rounds below used MLX, which was ahead of llama.cpp on both +prefill and decode in this one comparison. The first MLX request of a process +is slow to first token because Metal kernels compile on first use; time warm +requests separately from the first one. + +## RSI round + +Scripts, candidates and per-round records: +[examples/inkling2_mini_mlx](../examples/inkling2_mini_mlx/README.md). + +**Plane and why.** Decode on this model is launch-bound rather than +bandwidth-bound: a token visits 48 layers (6 + 12 × 3 + 6), each with a MoE +block, two depth-attention residuals and, in loop passes 2 and 3, four +message gates, so many small kernels and Python graph-building per step. +The searched plane is therefore *model-side graph compilation* (`mx.compile` +around pure sub-blocks), not scheduling: one user, one stream, no queue to +order. Loop count was not searched because the variable-loop model has no +Apple-format weights and changing `num_passes` on the released model changes +its output. + +**Gates frozen before the run** (inherited from the +[scheduling RSI lessons](scheduling-rsi-lessons.md)): + +- greedy decoding; a candidate whose token ids differ from the default path on + any prompt is rejected; +- each candidate run is paired with the default run immediately before it, 3 + pairs per candidate, score = median paired ratio of decode tok/s; +- below 5% difference: no conclusion; +- candidates are selected on two "evolve" prompts and reported on two + different "holdout" prompts (TypeScript and Rust code generation); +- a "better" verdict needs a mechanism signal. `rsi_mlx.py` performs only the + speed screen and prints "speed screen passed, pending mechanism check"; + the mechanism is measured separately with `mechanism.py` and `count_ops2.py` + (reported per round below). A candidate with an error, fewer pairs than + `--reps` or a failed revert is INCONCLUSIVE. + +The verdict strings stored in the committed `score-*.json` records ("keep", +and "留" in round 1) were written by earlier versions of the scorer and are +superseded by the rules above; the raw paired ratios in the records are kept +unchanged. + +**Selection and holdout.** Within each round, every candidate was run on the +evolve and the holdout prompts in the same session. There is no record that +the shipped candidate was fixed from the evolve scores and the 5% rule before +the holdout results were read, so both splits are reported as exploratory +data, not as a held-out confirmation. + +**What each round tried and learned.** The per-candidate paired ratios, +noise ranges and mechanism timings are in the committed `rounds/r*/` +records; they are not summarised here. + +- Round 1: `mx.compile` around single sub-blocks (message gates, MoE, depth + residual, gates + MoE). Every candidate was token-identical and below the + 5% rule: no conclusion. The default path drifted upward during the round; + the first scorer compared each candidate with the median of all default + runs, which inflated one candidate, so the scorer was changed to adjacent + pairing before round 2. The "留" verdict and ratio stored in + `rounds/r1/score-evolve.json` come from that superseded scorer. +- Round 2 (proposed from round 1): combined compile points and a fused MLP + half per layer. Lesson: per-module compiles move little on a launch-bound + model; the mechanism probe attributed part of the saving to CPU graph + construction and part to fewer GPU kernels. +- Round 3 (proposed from a per-module profile at decode shape, where the + attention half of a layer visit was dominated by small ops): each layer + visit for B = 1, L = 1 split into a compiled graph before the KV-cache write, + the cache write and attention eager (their length changes every step), and a + compiled graph after; conv states become explicit inputs and outputs. + Options on top: the attention mask/bias as one Metal kernel; q/r/gate/sink + projections in one quantized matmul; message-gate projections in one + matmul. Several options were within 5% of each other: no conclusion between + them. +- Round 4 (proposed from the op count after round 3: each 4-tap short + convolution was a chain of small ops): one-kernel short convolutions on top + of the round-3 options. The mechanism probe showed fewer non-view ops and + less Python graph building per step, and the step stayed launch-bound + rather than bandwidth-bound. A batched message-gate variant was within 5% + of it: no conclusion, not shipped. +- Across rounds the default path's own speed moved with temperature, so only + adjacent-paired ratios are compared, never absolute speeds across rounds. + +**How candidates were produced.** The candidates were written by the +assistant running the campaign (Claude Code, `claude-opus-5-5`) from reading +the model source; no automatic mutation chain was used. Each round was +proposed after reading the previous round's scores and, for rounds 3–4, a +per-module profile and an op count of one decode step. + +## What we did not establish + +- No acceptance issue, frozen campaign evidence bundle or independent rerun; + no performance figure is established, and none is published here. +- One machine, one macOS build, 128 GB; nothing on 32–64 GB Macs, which the + model card lists as supported. +- Single stream only; no batching, concurrency or long-context runs (the + advertised 1M context was not exercised). Prompts are short (61 tokens); + prefill-heavy work was measured only by the 512-token benchmark. +- Quality was checked only as greedy-token identity on four prompts and one + passing generated test file; no benchmark accuracy. +- The patch was not measured inside `mlx_lm.server` and was not proposed to + mlx-lm. "Identical" means identical greedy tokens on the prompts tested; the + compiled graphs are not proven bit-exact for every input. +- llama.cpp was run with default settings; its flags were not searched, so + the route comparison is default against default. diff --git a/docs/scheduling-rsi-lessons.md b/docs/scheduling-rsi-lessons.md new file mode 100644 index 00000000..d4ff1f23 --- /dev/null +++ b/docs/scheduling-rsi-lessons.md @@ -0,0 +1,90 @@ +# Scheduling RSI lessons: Ascend queue ordering and agent KV eviction + +This is an unverified exploration record. The two campaigns below have no +acceptance issue, their measurements are in the maintainers' unpublished +campaign ledger, Campaign 1 is a simulation that was not run on NPU hardware, +and nothing here is an InferRSI PASS. The page therefore publishes no +capacity, latency, hit-rate or speedup figures; it keeps only the method and +the failure modes. + +This page records two earlier campaigns of the same RSI line (model proposes a +policy → a frozen scorer measures it → keep the better, drop the worse) so that +later campaigns inherit the measurement rules and the failure modes, not only +the headline. Apple-silicon lessons from later work are in the +[Apple silicon playbook](apple-silicon-playbook.md) and +[Inkling 2 Mini on Apple silicon](inkling2-mini-apple-silicon.md). + +## Sources and labels + +| Label | Meaning | +| --- | --- | +| **public** | In a public repository or dataset at the link given in the row. | +| **ledger** | From the maintainers' unpublished campaign report or run log; not reproducible from this page; unverified. | +| **inferred** | Analysis made during the work and not measured. | + +## Campaign 1: vLLM-Ascend waiting-queue order (simulation) + +Setup (ledger): real vLLM engine with only the device operators replaced by +stand-ins; a virtual clock calibrated from another team's operator timing +table (step time as a linear function of tokens in the step); multi-turn +sessions in which almost every request contains the whole previous request. +Several candidates ran in parallel until the CPU saturated. Simulator wall time +was never used as the metric. + +| Observation | Cause | Action | Source | +| --- | --- | --- | --- | +| Ordering by the *share* of the prompt already cached, highest first, ranked above ordering by *absolute* tokens saved (either direction). | Share favours requests whose remaining prefill is small relative to what is already resident, so cache pages are reused before eviction. | Try the normalised form of a key as well as the absolute one. | ledger | +| The capacity at which the *mean* TTFT meets the bound was clearly higher than the capacity at which *every* run stays under it. | Mean-at-bound and every-run-under-bound are different operating points. | Publish the conservative point next to any headline and name which one is quoted. | ledger | +| At the same mean TTFT, median TTFT and the shortest prompts' TTFT got worse while the longest-to-mean ratio fell. | The policy equalises latency: long prompts gain, short cold prompts wait. | Report latency by prompt-length band, not only capacity. State the precondition: prefill-dominated load and no hard sub-second SLO for short requests. | ledger | +| Several successive "guaranteed floor" values were each inside the run-to-run noise and were all withdrawn. | Single-seed differences below the noise are not results. | Below 5% difference: no conclusion. A "better" verdict also needs a mechanism signal (hit rate up, recompute down, queue shorter). | ledger | +| Split validation (sessions halved into an evolution half and an exam half) gave similar results on both halves, but under a larger cache pool than the headline run. | Checks the policy is not fitted to the sessions it was found on. | Always hold out sessions, and never combine figures measured under different cache pools. | ledger | +| Parameter-level search improved for a few generations and then stopped moving; a late round produced almost only duplicates of known candidates. | The parameter plane was exhausted; the proposer kept regenerating known candidates. | De-duplicate before scoring. When a plane stops moving, switch planes rather than spend generations. | ledger | +| In the first structure-level round only gate tuning helped; later structural ideas (placing by free slots, a starvation quota for the longest waiter, reserving slots, ordering by waiting time) varied widely. | Structural changes reach gains the parameter plane cannot. | Expect most first-round structural ideas to fail; keep the failures in the trial index. | ledger | +| The champion came from a human-written requirement route, not from the automatic mutation chain. | — | Attribute candidates to their actual origin; write "measured on the same scorer", not "found by the model", unless the chain produced it. | ledger | +| Most human effort went into making candidates run on both the NPU path and the simulator (interfaces, parameter adaptation, debugging). | Two execution targets. | Budget the adaptation work explicitly; it is the cost driver, not proposal generation. | ledger | +| At the highest load, ordering alone came close to perfect reuse, and cold short requests remained slower. | The ordering plane is close to its ceiling. | Next plane: cache residency and admission. | ledger, inferred | + +Not established: NPU hardware confirmation of the simulated ranking; paired +per-session statistics instead of the 5% rule; a separate held-out audit split +for the champion. + +## Campaign 2: agent-aware KV eviction (CPU replay) + +Setup: CPU replay of public agent traces, policy proposed by a model, scored by +hit rate. Code was contributed to +[JiusiServe/vllm-evolve#1](https://github.com/JiusiServe/vllm-evolve/pull/1) +(public). Primary trace: Weka cc-traces with sub-agents +([HF dataset](https://huggingface.co/datasets/semianalysisai/cc-traces-weka-with-subagents-051926), public); +219 traces. + +| Observation | Cause | Action | Source | +| --- | --- | --- | --- | +| Trace chains are idle most of the time; sub-agents return sooner than root agents. | Tool calls and model turns alternate. | Expected return time is a usable eviction feature; sub-agent KV should stay resident before root-agent KV. | ledger, inferred | +| With device memory small relative to the active working set, LRU left a large gap to the offline upper bound, and the gap came from eviction order, not prefetch; with more memory the gap shrank. | The headroom comes from eviction order by expected return, not from prefetch. | Agent-aware eviction is worth searching only when memory is tight; measure the gap to Belady before searching. | ledger | +| After pinning the blocks of running requests, the gap between LRU and Belady almost disappeared; without pinning it looked large. | The unpinned replay credited evictions a real engine cannot make. | Model engine constraints (pinning, admission queueing) in the scorer before quoting headroom. | ledger | +| An exploratory search after pinning moved mean hit rate only inside the noise. | Small remaining headroom after pinning. | Compute the reachable headroom first; do not spend search budget on a plane with almost nothing to gain. | ledger | +| On a real vLLM 0.20 engine with Qwen2.5-7B, the evolved eviction function loaded through `--scheduler-cls`, evicted by score, and kept outputs identical under batch-invariant mode. | — | The hardware test proves the plug-in runs and keeps outputs; it does not score the policy. Gains come from the RSI scorer only. | ledger | +| Locally collected agent sessions were rejected as a dataset. | They have no external standing. | Use public traces already used by papers or benchmarks (Weka cc-traces, FreeInference, TraceLab). | ledger | + +## Campaign gates inherited from both + +These are the pre-run checks the maintainers now apply before any search +(ledger, from a closed research campaign that spent most of its time on its +harness): + +1. **Mechanism first.** Write the hypothesis as "variant X beats Y because part + P makes them differ", and name P in code. No P, no run. +2. **Power before running.** Baseline rate, samples per arm and the smallest + difference that would be visible against the measured run-to-run noise. +3. **Harness time box.** Harness and framework acceptance together at most one + working day or one quarter of the budget. +4. **Smallest discriminating run first.** Two arms, one seed. A tie sends the + work back to step 1, not to more seeds. +5. **Daily split.** Record how much of the day went to the hypothesis and how + much to the harness. +6. **Kernel gain is not end-to-end gain.** In a sibling InferMatrixLab engine + a kernel that was clearly faster in isolation barely moved end-to-end time + (public, `InferMatrixLab/system1-omni`). Score candidates on the full + generation path only. +7. **Freeze the rival.** Thresholds written before the run, rival + configuration frozen, inputs byte-identical, failed candidates kept. diff --git a/examples/inkling2_mini_mlx/README.md b/examples/inkling2_mini_mlx/README.md new file mode 100644 index 00000000..a1890431 --- /dev/null +++ b/examples/inkling2_mini_mlx/README.md @@ -0,0 +1,35 @@ +# Inkling 2 Mini on MLX: benchmark and RSI round scripts + +Scripts and records behind [Inkling 2 Mini on Apple silicon](../../docs/inkling2-mini-apple-silicon.md). +They are maintainer tools, not part of the `inferrsi` package, and need an +Apple-silicon Mac with mlx-lm from ml-explore/mlx-lm#2086 (`0c4c991`). + +| File | Purpose | +| --- | --- | +| `bench_mlx.py` | One code-generation request on MLX: TTFT, prefill/decode tok/s, peak memory, full output | +| `chat_bench.py` | The same request through llama-server's streaming OpenAI API | +| `bench_llama.sh` | `llama-bench` 512/128 × 3 with `/usr/bin/time -l` | +| `rsi_mlx.py` | One search round: load once, alternate each candidate with the default path, paired scoring, token-identity gate, evolve/holdout split | +| `mechanism.py` | Per-step graph-building and step time, default vs one candidate | +| `count_ops2.py` | Ops in one decode step's graph (total and non-view), default or one candidate | +| `route/` | Raw outputs of the MLX / llama.cpp route comparison (`mlx_lm.benchmark` log, `llama-bench` JSON, per-run chat timings in `mlx-chat.jsonl` / `llama-chat.jsonl`, the MLX model output in `mlx-chat-output.txt`) | +| `rounds/r1` … `rounds/r4` | Candidates (`cands/*.py`) and the JSONL/JSON records each round wrote | + +```bash +python rsi_mlx.py rounds/r2 --split evolve --max-tokens 256 --reps 3 +python rsi_mlx.py rounds/r2 --split holdout --max-tokens 256 --reps 3 +python mechanism.py rounds/r4/cands/r4a_conv_kernel.py +python count_ops2.py rounds/r4/cands/r4a_conv_kernel.py +``` + +`rsi_mlx.py` prints "speed screen passed, pending mechanism check" for a +complete, token-identical candidate at least 5% faster, and INCONCLUSIVE for +one with an error, fewer pairs than `--reps` or a failed revert. The verdict +strings in the committed `score-*.json` files ("keep", "留") come from earlier +scorer versions and are superseded. Round 1 records predate adjacent pairing; its paired ratios in the doc were +recomputed from the logged run order (default run `1 + 3k + i` precedes +repetition `i` of candidate `k`). Workload prompts are kept verbatim, +including the Chinese ones. Run one runtime at a time; another GPU job on the +machine invalidates the round. These records are exploratory and +unverified (no acceptance issue). The round-4 candidate is packaged as +[zuiho-kai/inkling2-mlx-fastpath](https://github.com/zuiho-kai/inkling2-mlx-fastpath). diff --git a/examples/inkling2_mini_mlx/bench_llama.sh b/examples/inkling2_mini_mlx/bench_llama.sh new file mode 100755 index 00000000..26d9099a --- /dev/null +++ b/examples/inkling2_mini_mlx/bench_llama.sh @@ -0,0 +1,8 @@ +#!/bin/bash +# llama.cpp route: llama-bench prefill/decode with the peak memory reported by /usr/bin/time -l +# Usage: bench_llama.sh (LLAMA_BIN = llama.cpp build/bin) +set -e; G=$1; O=$2; mkdir -p $O; B=${LLAMA_BIN:?set LLAMA_BIN to llama.cpp build/bin} +P='用 Python 写一个线程安全的 LRU 缓存类,支持 get/put/容量上限和过期时间(TTL),附 3 个 pytest 用例。只输出代码。' +/usr/bin/time -l $B/llama-bench -m $G -p 512 -n 128 -r 3 -o json > $O/bench.json 2> $O/bench.err # 512/128, 3 repetitions: llama-bench defaults +cat $O/bench.json | python3 -c "import sys,json;[print(x['n_prompt'],x['n_gen'],round(x['avg_ts'],2),'tok/s') for x in json.load(sys.stdin)]" +grep -E "peak memory footprint" $O/bench.err diff --git a/examples/inkling2_mini_mlx/bench_mlx.py b/examples/inkling2_mini_mlx/bench_mlx.py new file mode 100644 index 00000000..36dc5af5 --- /dev/null +++ b/examples/inkling2_mini_mlx/bench_mlx.py @@ -0,0 +1,45 @@ +# Usage: python bench_mlx.py --max-tokens N [--effort none] [--out result.json] [stream_generate key=value ...] +# Measures TTFT, prefill tok/s, decode tok/s and peak memory; the full output goes to .txt +import argparse, json, time +import mlx.core as mx +from mlx_lm import load, stream_generate +from mlx_lm.sample_utils import make_sampler + +PROMPT = ("用 Python 写一个线程安全的 LRU 缓存类,支持 get/put/容量上限和过期时间(TTL)," + "附 3 个 pytest 用例。只输出代码。") + +ap = argparse.ArgumentParser() +ap.add_argument("model") +ap.add_argument("--max-tokens", type=int, required=True) # always given on the command line +ap.add_argument("--effort", default="none") # a key of effort_map in the model's chat template +ap.add_argument("--out", default=None) +ap.add_argument("--runs", type=int, default=1) # single run unless asked +ap.add_argument("--prompt", default=PROMPT) +ap.add_argument("kw", nargs="*") +a = ap.parse_args() +kw = {} +for s in a.kw: + k, v = s.split("=", 1) + kw[k] = int(v) if v.lstrip("-").isdigit() else v + +t0 = time.time(); model, tok = load(a.model); load_s = time.time() - t0 +msgs = [{"role": "user", "content": a.prompt}] +prompt = tok.apply_chat_template(msgs, add_generation_prompt=True, reasoning_effort=a.effort) +sampler = make_sampler(temp=0.0) # greedy decoding: reproducible output, so changed results are detectable +res = [] +for r in range(a.runs): + mx.reset_peak_memory() + t0 = time.time(); ttft = None; text = ""; last = None + for resp in stream_generate(model, tok, prompt, max_tokens=a.max_tokens, sampler=sampler, **kw): + if ttft is None: ttft = time.time() - t0 + text += resp.text; last = resp + wall = time.time() - t0 + row = dict(run=r, load_s=load_s, ttft_s=ttft, prompt_tokens=last.prompt_tokens, + prefill_tps=last.prompt_tps, gen_tokens=last.generation_tokens, + decode_tps=last.generation_tps, peak_gb=mx.get_peak_memory() / 1e9, + wall_s=wall, kw=kw) + print(json.dumps(row, ensure_ascii=False), flush=True); res.append(row) +if a.out: + json.dump(res, open(a.out, "w"), ensure_ascii=False, indent=1) + open(a.out.rsplit(".", 1)[0] + ".txt", "w").write(text) +print("----- output -----\n" + text) diff --git a/examples/inkling2_mini_mlx/chat_bench.py b/examples/inkling2_mini_mlx/chat_bench.py new file mode 100644 index 00000000..c3230b68 --- /dev/null +++ b/examples/inkling2_mini_mlx/chat_bench.py @@ -0,0 +1,19 @@ +# One code-generation request through llama-server's streaming OpenAI API; records time to first token and server timings. +import sys, json, time, urllib.request +url, maxtok = sys.argv[1], int(sys.argv[2]) +P = "用 Python 写一个线程安全的 LRU 缓存类,支持 get/put/容量上限和过期时间(TTL),附 3 个 pytest 用例。只输出代码。" +body = dict(messages=[{"role": "user", "content": P}], max_tokens=maxtok, temperature=0, stream=True, + chat_template_kwargs={"reasoning_effort": "none"}, timings_per_token=False, cache_prompt=False) # no prompt-cache reuse, so time to first token is comparable across runs +req = urllib.request.Request(url + "/v1/chat/completions", json.dumps(body).encode(), {"Content-Type": "application/json"}) +t0 = time.time(); ttft = None; text = ""; think = ""; timings = None +for line in urllib.request.urlopen(req): + line = line.decode().strip() + if not line.startswith("data: ") or line == "data: [DONE]": continue + d = json.loads(line[6:]) + if d.get("timings"): timings = d["timings"] + for c in d.get("choices", []): + dl = c.get("delta") or {} + t = dl.get("content") or ""; r = dl.get("reasoning_content") or "" + if (t or r) and ttft is None: ttft = time.time() - t0 + text += t; think += r +print(json.dumps(dict(ttft_s=ttft, wall_s=time.time() - t0, timings=timings, text=text, reasoning=think), ensure_ascii=False)) diff --git a/examples/inkling2_mini_mlx/count_ops2.py b/examples/inkling2_mini_mlx/count_ops2.py new file mode 100644 index 00000000..6954cc5a --- /dev/null +++ b/examples/inkling2_mini_mlx/count_ops2.py @@ -0,0 +1,19 @@ +# Count op nodes in one decode-step graph (total and non-view, the latter ~ GPU kernels); an optional candidate file is applied first. +import sys, collections, re, importlib.util +import mlx.core as mx +from mlx_lm import load +from mlx_lm.models.cache import make_prompt_cache +model, tok = load(sys.argv[1]) +if len(sys.argv) > 2: + spec = importlib.util.spec_from_file_location("c", sys.argv[2]); m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m); m.apply(model) +ids = mx.array(tok.encode("def lru_cache(capacity):"))[None] +cache = make_prompt_cache(model); out = model(ids, cache=cache); mx.eval(out) +for _ in range(3): + y = mx.argmax(out[:, -1:], -1); out = model(y, cache=cache); mx.eval(out) +out = model(y, cache=cache) +mx.export_to_dot("step.dot", out) +txt = open("step.dot").read() +ops = re.findall(r'label ="?([A-Za-z_]+)', txt) +VIEW = {"Broadcast","Transpose","Reshape","Squeeze","Flatten","Unflatten","ExpandDims","StopGradient","Slice","Split"} +c = collections.Counter(o if not o.startswith("Compiled") else "Compiled*" for o in ops) +print("total", sum(c.values()), "non-view", sum(v for k, v in c.items() if k not in VIEW)); print(c.most_common(30)) diff --git a/examples/inkling2_mini_mlx/mechanism.py b/examples/inkling2_mini_mlx/mechanism.py new file mode 100644 index 00000000..b2066170 --- /dev/null +++ b/examples/inkling2_mini_mlx/mechanism.py @@ -0,0 +1,21 @@ +# Mechanism check: per decode step, time spent building the lazy graph in Python vs the whole step. +# A compile candidate should cut graph-building time (fewer small ops) and GPU time (fused kernels). +# Usage: python mechanism.py [candidate.py] +import sys, time, importlib.util, statistics +import mlx.core as mx +from mlx_lm import load +from mlx_lm.models.cache import make_prompt_cache + +model, tok = load(sys.argv[1]); cand = sys.argv[2] if len(sys.argv) > 2 else None +if cand: + spec = importlib.util.spec_from_file_location("c", cand); m = importlib.util.module_from_spec(spec) + spec.loader.exec_module(m); m.apply(model) +ids = mx.array(tok.encode("def lru_cache(capacity):"))[None] +cache = make_prompt_cache(model); out = model(ids, cache=cache); mx.eval(out) +y = mx.argmax(out[:, -1:], -1) +build, total = [], [] +for i in range(60): # 60 steps; the first 10 are warm-up (the first includes compilation) + t0 = time.perf_counter(); out = model(y, cache=cache); t1 = time.perf_counter() + y = mx.argmax(out[:, -1:], -1); mx.eval(y); t2 = time.perf_counter() + if i >= 10: build.append(t1 - t0); total.append(t2 - t0) +print(f"{'cand' if cand else 'base'} build_ms={statistics.median(build)*1e3:.2f} step_ms={statistics.median(total)*1e3:.2f}") diff --git a/examples/inkling2_mini_mlx/rounds/r1/cands/_patch.py b/examples/inkling2_mini_mlx/rounds/r1/cands/_patch.py new file mode 100644 index 00000000..150ea04e --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r1/cands/_patch.py @@ -0,0 +1,14 @@ +# Shared helper: replace a module class __call__ with one mx.compile per instance; revert_all restores it. +import mlx.core as mx +_saved = {} +def compile_class(cls): + orig = cls.__call__; _saved[cls] = orig; cache = {} + def call(self, *a, **k): + f = cache.get(id(self)) + if f is None: + f = cache[id(self)] = mx.compile(lambda *aa: orig(self, *aa)) + return f(*a) if not k else orig(self, *a, **k) + cls.__call__ = call +def revert_all(): + for cls, orig in _saved.items(): cls.__call__ = orig + _saved.clear() diff --git a/examples/inkling2_mini_mlx/rounds/r1/cands/c1_compile_gates.py b/examples/inkling2_mini_mlx/rounds/r1/cands/c1_compile_gates.py new file mode 100644 index 00000000..5652dd52 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r1/cands/c1_compile_gates.py @@ -0,0 +1,5 @@ +"""Compile each loop message gate (MessageGate: two rms_norms, concat, three projections, two tanh) into one graph. Passes 2 and 3 of the loop call 4 gates per layer: about 96 calls per token.""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _patch +from mlx_lm.models.inkling2 import layers as L +def apply(model): _patch.compile_class(L.MessageGate); return {} +def revert(model): _patch.revert_all() diff --git a/examples/inkling2_mini_mlx/rounds/r1/cands/c2_compile_moe.py b/examples/inkling2_mini_mlx/rounds/r1/cands/c2_compile_moe.py new file mode 100644 index 00000000..01bac2b2 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r1/cands/c2_compile_moe.py @@ -0,0 +1,5 @@ +"""Compile the whole MoE block (router, top-k, two expert matmuls, soft-capped SwiGLU, weighted sum). 48 layer visits per token, one MoE block each.""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _patch +from mlx_lm.models.inkling2 import layers as L +def apply(model): _patch.compile_class(L.SparseMoeBlock); return {} +def revert(model): _patch.revert_all() diff --git a/examples/inkling2_mini_mlx/rounds/r1/cands/c3_compile_gates_moe.py b/examples/inkling2_mini_mlx/rounds/r1/cands/c3_compile_gates_moe.py new file mode 100644 index 00000000..5863770e --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r1/cands/c3_compile_gates_moe.py @@ -0,0 +1,5 @@ +"""c1 + c2 together.""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _patch +from mlx_lm.models.inkling2 import layers as L +def apply(model): _patch.compile_class(L.MessageGate); _patch.compile_class(L.SparseMoeBlock); return {} +def revert(model): _patch.revert_all() diff --git a/examples/inkling2_mini_mlx/rounds/r1/cands/c4_compile_residual.py b/examples/inkling2_mini_mlx/rounds/r1/cands/c4_compile_residual.py new file mode 100644 index 00000000..d510f1db --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r1/cands/c4_compile_residual.py @@ -0,0 +1,5 @@ +"""Compile the depth-attention residual (BlockAttentionResidual, twice per layer).""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _patch +from mlx_lm.models.inkling2 import layers as L +def apply(model): _patch.compile_class(L.BlockAttentionResidual); return {} +def revert(model): _patch.revert_all() diff --git a/examples/inkling2_mini_mlx/rounds/r1/log-evolve.jsonl b/examples/inkling2_mini_mlx/rounds/r1/log-evolve.jsonl new file mode 100644 index 00000000..5ea4b25f --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r1/log-evolve.jsonl @@ -0,0 +1,5 @@ +{"cand": "c1_compile_gates", "dec_med": 49.40115411275421, "dec_all": [49.40115411275421, 49.3791565718586, 49.51521835056296], "ttft_med": 0.21953141689300537, "peak": 17.569098932, "same_output": true, "err": null, "doc": "把循环层的消息门(MessageGate:两次 rms_norm、拼接、三次投影、两次 tanh)整段编译成一张图,减少解码时每步的小算子发射。循环段后两遍每层 4 个门,每 token 约 96 次调用。"} +{"cand": "c2_compile_moe", "dec_med": 49.02118691839245, "dec_all": [48.85107143881726, 49.09644688548954, 49.02118691839245], "ttft_med": 0.22214102745056152, "peak": 17.560419744, "same_output": true, "err": null, "doc": "把 MoE 块(路由打分、top-k、两次专家矩阵乘、软截断 SwiGLU、加权求和)整段编译。每 token 走 48 次层访问,每次一个 MoE 块。"} +{"cand": "c3_compile_gates_moe", "dec_med": 51.02677239497174, "dec_all": [51.108004003752164, 50.98362410796302, 51.02677239497174], "ttft_med": 0.22019445896148682, "peak": 17.567906632, "same_output": true, "err": null, "doc": "c1 + c2 同时上。"} +{"cand": "c4_compile_residual", "dec_med": 49.07735030987658, "dec_all": [49.112221671001905, 49.07735030987658, 49.05263728135563], "ttft_med": 0.2178339958190918, "peak": 17.559941956, "same_output": true, "err": null, "doc": "把深度注意力残差(BlockAttentionResidual,每层两次)编译。"} +{"cand": "baseline", "dec_med": 48.44602831890364, "dec_all": [47.446792327900155, 47.84514640626699, 48.09577184676405, 48.003700957422566, 48.013395006973006, 48.38342647762863, 48.468703240708905, 48.76026873686101, 48.64142996610397, 48.743378020755145, 48.44602831890364, 48.871949743256266, 48.47134355045931], "ttft_med": 0.21846795082092285, "peak": 17.5599129, "noise_range": 0.02941742522162574} diff --git a/examples/inkling2_mini_mlx/rounds/r1/score-evolve.json b/examples/inkling2_mini_mlx/rounds/r1/score-evolve.json new file mode 100644 index 00000000..46958f0b --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r1/score-evolve.json @@ -0,0 +1,85 @@ +{ + "baseline": { + "dec_med": 48.44602831890364, + "dec_all": [ + 47.446792327900155, + 47.84514640626699, + 48.09577184676405, + 48.003700957422566, + 48.013395006973006, + 48.38342647762863, + 48.468703240708905, + 48.76026873686101, + 48.64142996610397, + 48.743378020755145, + 48.44602831890364, + 48.871949743256266, + 48.47134355045931 + ], + "ttft_med": 0.21846795082092285, + "peak": 17.5599129 + }, + "noise_range": 0.02941742522162574, + "cands": { + "c1_compile_gates": { + "dec_med": 49.40115411275421, + "dec_all": [ + 49.40115411275421, + 49.3791565718586, + 49.51521835056296 + ], + "ttft_med": 0.21953141689300537, + "peak": 17.569098932, + "same_output": true, + "err": null, + "doc": "把循环层的消息门(MessageGate:两次 rms_norm、拼接、三次投影、两次 tanh)整段编译成一张图,减少解码时每步的小算子发射。循环段后两遍每层 4 个门,每 token 约 96 次调用。", + "speedup": 1.0197152548308668, + "verdict": "差距<5%,不下结论" + }, + "c2_compile_moe": { + "dec_med": 49.02118691839245, + "dec_all": [ + 48.85107143881726, + 49.09644688548954, + 49.02118691839245 + ], + "ttft_med": 0.22214102745056152, + "peak": 17.560419744, + "same_output": true, + "err": null, + "doc": "把 MoE 块(路由打分、top-k、两次专家矩阵乘、软截断 SwiGLU、加权求和)整段编译。每 token 走 48 次层访问,每次一个 MoE 块。", + "speedup": 1.0118721517417844, + "verdict": "差距<5%,不下结论" + }, + "c3_compile_gates_moe": { + "dec_med": 51.02677239497174, + "dec_all": [ + 51.108004003752164, + 50.98362410796302, + 51.02677239497174 + ], + "ttft_med": 0.22019445896148682, + "peak": 17.567906632, + "same_output": true, + "err": null, + "doc": "c1 + c2 同时上。", + "speedup": 1.0532704984416874, + "verdict": "留" + }, + "c4_compile_residual": { + "dec_med": 49.07735030987658, + "dec_all": [ + 49.112221671001905, + 49.07735030987658, + 49.05263728135563 + ], + "ttft_med": 0.2178339958190918, + "peak": 17.559941956, + "same_output": true, + "err": null, + "doc": "把深度注意力残差(BlockAttentionResidual,每层两次)编译。", + "speedup": 1.0130314499016753, + "verdict": "差距<5%,不下结论" + } + } +} \ No newline at end of file diff --git a/examples/inkling2_mini_mlx/rounds/r2/cands/_patch.py b/examples/inkling2_mini_mlx/rounds/r2/cands/_patch.py new file mode 100644 index 00000000..150ea04e --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/cands/_patch.py @@ -0,0 +1,14 @@ +# Shared helper: replace a module class __call__ with one mx.compile per instance; revert_all restores it. +import mlx.core as mx +_saved = {} +def compile_class(cls): + orig = cls.__call__; _saved[cls] = orig; cache = {} + def call(self, *a, **k): + f = cache.get(id(self)) + if f is None: + f = cache[id(self)] = mx.compile(lambda *aa: orig(self, *aa)) + return f(*a) if not k else orig(self, *a, **k) + cls.__call__ = call +def revert_all(): + for cls, orig in _saved.items(): cls.__call__ = orig + _saved.clear() diff --git a/examples/inkling2_mini_mlx/rounds/r2/cands/r2a_gates_moe.py b/examples/inkling2_mini_mlx/rounds/r2/cands/r2a_gates_moe.py new file mode 100644 index 00000000..5863770e --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/cands/r2a_gates_moe.py @@ -0,0 +1,5 @@ +"""c1 + c2 together.""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _patch +from mlx_lm.models.inkling2 import layers as L +def apply(model): _patch.compile_class(L.MessageGate); _patch.compile_class(L.SparseMoeBlock); return {} +def revert(model): _patch.revert_all() diff --git a/examples/inkling2_mini_mlx/rounds/r2/cands/r2b_gates_moe_residual.py b/examples/inkling2_mini_mlx/rounds/r2/cands/r2b_gates_moe_residual.py new file mode 100644 index 00000000..551b5632 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/cands/r2b_gates_moe_residual.py @@ -0,0 +1,7 @@ +"""All three round-1 compile points (message gate, MoE, depth residual).""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _patch +from mlx_lm.models.inkling2 import layers as L +def apply(model): + for c in (L.MessageGate, L.SparseMoeBlock, L.BlockAttentionResidual): _patch.compile_class(c) + return {} +def revert(model): _patch.revert_all() diff --git a/examples/inkling2_mini_mlx/rounds/r2/cands/r2c_fused_mlp_half.py b/examples/inkling2_mini_mlx/rounds/r2/cands/r2c_fused_mlp_half.py new file mode 100644 index 00000000..ac1c2a43 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/cands/r2c_fused_mlp_half.py @@ -0,0 +1,56 @@ +"""Fuse the second half of every layer (depth residual for the MLP input -> message gate -> RMSNorm -> MoE) +into one compiled graph per layer instance. The attention half and the convolution state (cache writes) are untouched.""" +import mlx.core as mx +from mlx_lm.models.inkling2 import layers as L +_orig = L.DecoderLayer.__call__ +_cache = {} + +def _half(self, has_msg): + key = (id(self), has_msg) + f = _cache.get(key) + if f is None: + def body(hidden_states, block_states, *msg): + dtype = self._dtype + normed = L.unweighted_rms_norm(hidden_states) + h = self.mlp_input(block_states, normed)[..., 0, :] + if msg: + h = self.mlp_message_gate(h, msg[0][..., 3, :]) + return self.mlp(self.post_attention_layernorm(h).astype(dtype)) + f = _cache[key] = mx.compile(body) + return f + +def call(self, hidden_states, block_states, *, cache=None, conv_mask=None, messages=None): + dtype = hidden_states.dtype + self._dtype = dtype + conv = cache[1] if cache is not None else None + conv_states = list(conv.cache) if conv is not None else [None] * 4 + block_residual = hidden_states + normed_residual = L.unweighted_rms_norm(block_residual.astype(mx.float32)) + attention_inputs = None + if self.attention_input is not None: + attention_inputs = self.attention_input(block_states, normed_residual) + hidden_states = self._self_attention(hidden_states, attention_inputs, cache, conv_states, conv_mask, messages) + hidden_states, conv_states[2] = self.post_attention_convolution(hidden_states, conv_states[2], conv_mask) + residual = hidden_states = hidden_states.astype(mx.float32) + if self.is_block_boundary: + block = normed_residual[None] + block_states = block if block_states is None else mx.concatenate([block_states, block]) + else: + residual = hidden_states = block_residual.astype(mx.float32) + hidden_states + if messages is not None: + hidden_states = _half(self, True)(hidden_states, block_states, messages) + else: + hidden_states = _half(self, False)(hidden_states, block_states) + hidden_states, conv_states[3] = self.post_mlp_convolution(hidden_states, conv_states[3], conv_mask) + if conv is not None: + n = self.post_mlp_convolution.kernel_size - 1 + for i, window in enumerate(conv_states): + conv[i] = L._conv_state(window, conv.lengths, n) + conv.advance(block_residual.shape[1]) + hidden_states = residual + hidden_states.astype(mx.float32) + return hidden_states.astype(dtype), block_states + +def apply(model): + L.DecoderLayer.__call__ = call; return {} +def revert(model): + L.DecoderLayer.__call__ = _orig diff --git a/examples/inkling2_mini_mlx/rounds/r2/cands/r2d_fused_half_gates.py b/examples/inkling2_mini_mlx/rounds/r2/cands/r2d_fused_half_gates.py new file mode 100644 index 00000000..a398af4a --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/cands/r2d_fused_half_gates.py @@ -0,0 +1,6 @@ +"""r2c plus compiled q/k/v message gates in the attention half.""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _patch +import r2c_fused_mlp_half as H +from mlx_lm.models.inkling2 import layers as L +def apply(model): H.apply(model); _patch.compile_class(L.MessageGate); return {} +def revert(model): H.revert(model); _patch.revert_all() diff --git a/examples/inkling2_mini_mlx/rounds/r2/log-evolve.jsonl b/examples/inkling2_mini_mlx/rounds/r2/log-evolve.jsonl new file mode 100644 index 00000000..8bcf51a1 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/log-evolve.jsonl @@ -0,0 +1,5 @@ +{"cand": "r2a_gates_moe", "dec_med": 50.424029905177704, "dec_all": [50.424029905177704, 50.04613092519887, 50.62708168991173], "ttft_med": 0.220139741897583, "peak": 17.567414856, "paired": [1.0428356184966039, 1.0320574751381653, 1.0474635424160863], "same_output": true, "err": null, "doc": "c1 + c2 同时上。"} +{"cand": "r2b_gates_moe_residual", "dec_med": 51.21983893528104, "dec_all": [51.43588232496421, 51.21983893528104, 51.17689779327035], "ttft_med": 0.22014844417572021, "peak": 17.566595896, "paired": [1.0569113877913525, 1.0565390347402557, 1.055181811728994], "same_output": true, "err": null, "doc": "第 1 轮三个编译点(消息门、MoE、深度残差)全上。"} +{"cand": "r2c_fused_mlp_half", "dec_med": 49.797729329488135, "dec_all": [49.797729329488135, 49.55705031401985, 50.197880166144515], "ttft_med": 0.21745789051055908, "peak": 17.56299792, "paired": [1.0473919742965152, 1.035651301526283, 1.0442818871213315], "same_output": true, "err": null, "doc": "把每层后半段(深度残差取 MLP 输入 → 消息门 → RMSNorm → MoE)合成一张编译图,每层实例一份;\n前半段注意力和卷积状态(带缓存写入)不动。比单独编译各模块多合掉模块之间的接缝。"} +{"cand": "r2d_fused_half_gates", "dec_med": 51.4059422007786, "dec_all": [51.54030730625611, 51.4059422007786, 51.042283946977946], "ttft_med": 0.21700644493103027, "peak": 17.569096812, "paired": [1.0677181572871433, 1.072562396095457, 1.0630550931504787], "same_output": true, "err": null, "doc": "r2c 再加上前半段的消息门编译(q/k/v 三个门)。"} +{"cand": "baseline", "dec_med": 48.27145343037871, "dec_all": [48.17797630886772, 48.35280749028416, 48.491612270430004, 48.333025102845085, 48.666220195101445, 48.47888932742855, 48.500549596673956, 47.5445015348098, 47.8510964462513, 48.06928166160197, 48.27145343037871, 47.9281600659469, 48.014711820540384], "ttft_med": 0.21836340427398682, "peak": 17.5599953, "noise_range": 0.023237722930996654} diff --git a/examples/inkling2_mini_mlx/rounds/r2/log-holdout.jsonl b/examples/inkling2_mini_mlx/rounds/r2/log-holdout.jsonl new file mode 100644 index 00000000..a7073f9d --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/log-holdout.jsonl @@ -0,0 +1,5 @@ +{"cand": "r2a_gates_moe", "dec_med": 51.04474074076323, "dec_all": [50.85660088211513, 51.04474074076323, 51.08489211463147], "ttft_med": 0.19790959358215332, "peak": 17.521604464, "paired": [1.054492129303139, 1.0870758620919294, 1.048187714673263], "same_output": true, "err": null, "doc": "c1 + c2 同时上。"} +{"cand": "r2b_gates_moe_residual", "dec_med": 51.562731237754896, "dec_all": [51.562731237754896, 51.5694744945997, 51.031208850454746], "ttft_med": 0.198167085647583, "peak": 17.52170266, "paired": [1.0562378572332667, 1.0557891451819792, 1.0479394689598494], "same_output": true, "err": null, "doc": "第 1 轮三个编译点(消息门、MoE、深度残差)全上。"} +{"cand": "r2c_fused_mlp_half", "dec_med": 50.094175012824834, "dec_all": [50.13723459977536, 50.094175012824834, 49.96048055644977], "ttft_med": 0.19697952270507812, "peak": 17.516487316, "paired": [1.0280730946681598, 1.043505733952328, 1.0386180183034315], "same_output": true, "err": null, "doc": "把每层后半段(深度残差取 MLP 输入 → 消息门 → RMSNorm → MoE)合成一张编译图,每层实例一份;\n前半段注意力和卷积状态(带缓存写入)不动。比单独编译各模块多合掉模块之间的接缝。"} +{"cand": "r2d_fused_half_gates", "dec_med": 51.2338418260509, "dec_all": [51.49597712697293, 51.2338418260509, 50.81507864179749], "ttft_med": 0.19498789310455322, "peak": 17.521638616, "paired": [1.0679889472833788, 1.0639873985016102, 1.059518895826631], "same_output": true, "err": null, "doc": "r2c 再加上前半段的消息门编译(q/k/v 三个门)。"} +{"cand": "baseline", "dec_med": 48.228525817185286, "dec_all": [48.47356339461423, 48.228525817185286, 46.95600603488204, 48.73639654377695, 48.817348180285336, 48.8444825654189, 48.69671423017082, 48.768161388328714, 48.00565380996111, 48.102844044685014, 48.21770605207308, 48.1526772762557, 47.96052136677736], "ttft_med": 0.19630658626556396, "peak": 17.514715684, "noise_range": 0.039156837131935344} diff --git a/examples/inkling2_mini_mlx/rounds/r2/mechanism.txt b/examples/inkling2_mini_mlx/rounds/r2/mechanism.txt new file mode 100644 index 00000000..03ea6e71 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/mechanism.txt @@ -0,0 +1,4 @@ +base build_ms=3.92 step_ms=22.09 +cand build_ms=3.31 step_ms=20.75 +base build_ms=3.86 step_ms=22.05 +cand build_ms=3.37 step_ms=21.13 diff --git a/examples/inkling2_mini_mlx/rounds/r2/score-evolve.json b/examples/inkling2_mini_mlx/rounds/r2/score-evolve.json new file mode 100644 index 00000000..6b08a1bd --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/score-evolve.json @@ -0,0 +1,105 @@ +{ + "baseline": { + "dec_med": 48.27145343037871, + "dec_all": [ + 48.17797630886772, + 48.35280749028416, + 48.491612270430004, + 48.333025102845085, + 48.666220195101445, + 48.47888932742855, + 48.500549596673956, + 47.5445015348098, + 47.8510964462513, + 48.06928166160197, + 48.27145343037871, + 47.9281600659469, + 48.014711820540384 + ], + "ttft_med": 0.21836340427398682, + "peak": 17.5599953 + }, + "noise_range": 0.023237722930996654, + "cands": { + "r2a_gates_moe": { + "dec_med": 50.424029905177704, + "dec_all": [ + 50.424029905177704, + 50.04613092519887, + 50.62708168991173 + ], + "ttft_med": 0.220139741897583, + "peak": 17.567414856, + "paired": [ + 1.0428356184966039, + 1.0320574751381653, + 1.0474635424160863 + ], + "same_output": true, + "err": null, + "doc": "c1 + c2 同时上。", + "speedup": 1.0428356184966039, + "verdict": "差距<5%,不下结论" + }, + "r2b_gates_moe_residual": { + "dec_med": 51.21983893528104, + "dec_all": [ + 51.43588232496421, + 51.21983893528104, + 51.17689779327035 + ], + "ttft_med": 0.22014844417572021, + "peak": 17.566595896, + "paired": [ + 1.0569113877913525, + 1.0565390347402557, + 1.055181811728994 + ], + "same_output": true, + "err": null, + "doc": "第 1 轮三个编译点(消息门、MoE、深度残差)全上。", + "speedup": 1.0565390347402557, + "verdict": "留" + }, + "r2c_fused_mlp_half": { + "dec_med": 49.797729329488135, + "dec_all": [ + 49.797729329488135, + 49.55705031401985, + 50.197880166144515 + ], + "ttft_med": 0.21745789051055908, + "peak": 17.56299792, + "paired": [ + 1.0473919742965152, + 1.035651301526283, + 1.0442818871213315 + ], + "same_output": true, + "err": null, + "doc": "把每层后半段(深度残差取 MLP 输入 → 消息门 → RMSNorm → MoE)合成一张编译图,每层实例一份;\n前半段注意力和卷积状态(带缓存写入)不动。比单独编译各模块多合掉模块之间的接缝。", + "speedup": 1.0442818871213315, + "verdict": "差距<5%,不下结论" + }, + "r2d_fused_half_gates": { + "dec_med": 51.4059422007786, + "dec_all": [ + 51.54030730625611, + 51.4059422007786, + 51.042283946977946 + ], + "ttft_med": 0.21700644493103027, + "peak": 17.569096812, + "paired": [ + 1.0677181572871433, + 1.072562396095457, + 1.0630550931504787 + ], + "same_output": true, + "err": null, + "doc": "r2c 再加上前半段的消息门编译(q/k/v 三个门)。", + "speedup": 1.0677181572871433, + "verdict": "留" + } + } +} \ No newline at end of file diff --git a/examples/inkling2_mini_mlx/rounds/r2/score-holdout.json b/examples/inkling2_mini_mlx/rounds/r2/score-holdout.json new file mode 100644 index 00000000..201a0807 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r2/score-holdout.json @@ -0,0 +1,105 @@ +{ + "baseline": { + "dec_med": 48.228525817185286, + "dec_all": [ + 48.47356339461423, + 48.228525817185286, + 46.95600603488204, + 48.73639654377695, + 48.817348180285336, + 48.8444825654189, + 48.69671423017082, + 48.768161388328714, + 48.00565380996111, + 48.102844044685014, + 48.21770605207308, + 48.1526772762557, + 47.96052136677736 + ], + "ttft_med": 0.19630658626556396, + "peak": 17.514715684 + }, + "noise_range": 0.039156837131935344, + "cands": { + "r2a_gates_moe": { + "dec_med": 51.04474074076323, + "dec_all": [ + 50.85660088211513, + 51.04474074076323, + 51.08489211463147 + ], + "ttft_med": 0.19790959358215332, + "peak": 17.521604464, + "paired": [ + 1.054492129303139, + 1.0870758620919294, + 1.048187714673263 + ], + "same_output": true, + "err": null, + "doc": "c1 + c2 同时上。", + "speedup": 1.054492129303139, + "verdict": "留" + }, + "r2b_gates_moe_residual": { + "dec_med": 51.562731237754896, + "dec_all": [ + 51.562731237754896, + 51.5694744945997, + 51.031208850454746 + ], + "ttft_med": 0.198167085647583, + "peak": 17.52170266, + "paired": [ + 1.0562378572332667, + 1.0557891451819792, + 1.0479394689598494 + ], + "same_output": true, + "err": null, + "doc": "第 1 轮三个编译点(消息门、MoE、深度残差)全上。", + "speedup": 1.0557891451819792, + "verdict": "留" + }, + "r2c_fused_mlp_half": { + "dec_med": 50.094175012824834, + "dec_all": [ + 50.13723459977536, + 50.094175012824834, + 49.96048055644977 + ], + "ttft_med": 0.19697952270507812, + "peak": 17.516487316, + "paired": [ + 1.0280730946681598, + 1.043505733952328, + 1.0386180183034315 + ], + "same_output": true, + "err": null, + "doc": "把每层后半段(深度残差取 MLP 输入 → 消息门 → RMSNorm → MoE)合成一张编译图,每层实例一份;\n前半段注意力和卷积状态(带缓存写入)不动。比单独编译各模块多合掉模块之间的接缝。", + "speedup": 1.0386180183034315, + "verdict": "差距<5%,不下结论" + }, + "r2d_fused_half_gates": { + "dec_med": 51.2338418260509, + "dec_all": [ + 51.49597712697293, + 51.2338418260509, + 50.81507864179749 + ], + "ttft_med": 0.19498789310455322, + "peak": 17.521638616, + "paired": [ + 1.0679889472833788, + 1.0639873985016102, + 1.059518895826631 + ], + "same_output": true, + "err": null, + "doc": "r2c 再加上前半段的消息门编译(q/k/v 三个门)。", + "speedup": 1.0639873985016102, + "verdict": "留" + } + } +} \ No newline at end of file diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/_fast.py b/examples/inkling2_mini_mlx/rounds/r3/cands/_fast.py new file mode 100644 index 00000000..8fbad890 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/_fast.py @@ -0,0 +1,256 @@ +# Shared by rounds 3-4: each layer visit of a decode step (B=1, L=1) runs as pre graph -> cache write + attention -> post graph. +# pre : residual read-in, message gates, norms, q/k/v/r/gate/sink projections, short convs, clips, relative bias; one compiled graph; +# mid : KV-cache write and attention (length changes every step, not compiled); +# post: output gate, o_proj, convs, residual, MoE, convs; one compiled graph. Conv states are explicit inputs/outputs. +# Options: mask_kernel = attention mask in one Metal kernel; fuse_proj = projections sharing an input in one quantized matmul. +import mlx.core as mx +from mlx import nn +from mlx_lm.models.inkling2 import layers as L +from mlx_lm.models.cache import RotatingKVCache + +OPTS = dict(mask_kernel=False, fuse_proj=False, fuse_all6=False, fuse_msg=False, conv_ew=False) +_DEF = dict(OPTS) +_orig = L.DecoderLayer.__call__ +_cache = {} +_fused = {} + + +def _sliding(self): + return isinstance(self, L.SlidingDecoderLayer) + + +def _proj(self, attn, qi, ki, vi, same_kv): + if OPTS["fuse_proj"]: + f = _fused.get(id(attn)) + if f is not None: + outs = f["qrgs"](qi) + qp, rp, gate, sinks = mx.split(outs, f["qrgs_split"], axis=-1) + if same_kv and OPTS["fuse_all6"]: # sliding layer, no messages: q/k/v share the input, six projections at once + qp, kp, vp, rp, gate, sinks = mx.split(f["all6"](qi), f["all6_split"], axis=-1) + return qp, kp, vp, rp, gate, sinks + kp = attn.k_proj(ki); vp = attn.v_proj(vi) + return qp, kp, vp, rp, gate, sinks + return attn.q_proj(qi), attn.k_proj(ki), attn.v_proj(vi), attn.r_proj(qi), attn.gate_proj(qi), attn.sink_proj(qi) + + +def _pre(self, has_bs, has_msg): + sliding = _sliding(self); attn = self.self_attn + + def body(*args): + it = iter(args) + h = next(it); bs = next(it) if has_bs else None + cs0 = next(it); cs1 = next(it) + msg = next(it) if has_msg else None + pos = None if sliding else next(it) + dtype = h.dtype + nr = L.unweighted_rms_norm(h.astype(mx.float32)) + ai = self.attention_input(bs, nr) if self.attention_input is not None else None + if sliding: + hs = ai[..., 0, :] if ai is not None else h + if msg is None: + qi = ki = vi = self.input_layernorm(hs).astype(dtype) + else: + st = self._mix_messages((hs,) * 3, msg) + qi, ki, vi = [self.input_layernorm(s).astype(dtype) for s in st] + else: + q, k, v = (ai[..., i, :] for i in range(3)) + if msg is not None: + q, k, v = self._mix_messages((q, k, v), msg) + qi = self.q_input_layernorm(q).astype(dtype) + ki = self.k_input_layernorm(k).astype(dtype) + vi = self.v_input_layernorm(v).astype(dtype) + qp, kp, vp, rp, gate, sinks = _proj(self, attn, qi, ki, vi, sliding and msg is None) + if sliding: + ks, w0 = attn.k_sconv(kp, cs0, None); vs, w1 = attn.v_sconv(vp, cs1, None) + else: + ks, w0 = attn.k_sconv(kp, ki, cs0, None); vs, w1 = attn.v_sconv(vp, vi, cs1, None) + B, Ln = 1, 1 + qs = qp.reshape(B, Ln, attn.num_heads, -1) + ks = ks.reshape(B, Ln, attn.num_key_value_heads, -1) + vs = vs.reshape(B, Ln, attn.num_key_value_heads, -1) + rs = rp.reshape(B, Ln, attn.num_heads, -1) + qs = attn.q_l2_clip(qs); ks = attn.k_l2_clip(ks) + vs = L.unweighted_rms_norm(vs.astype(mx.float32), attn.eps).astype(vs.dtype) + rs = attn.rel_l2_clip(rs) + if not sliding: + tau = 1.0 + attn.log_scaling_alpha * mx.log(mx.maximum((pos + 1) / attn.log_scaling_n_floor, 1.0)) + tau = tau[..., None, None] + qs = (qs.astype(mx.float32) * tau).astype(qs.dtype) + rs = (rs.astype(mx.float32) * tau).astype(rs.dtype) + pb = attn.rel_logits_proj(rs) + return (qs.transpose(0, 2, 1, 3), ks.transpose(0, 2, 1, 3), vs.transpose(0, 2, 1, 3), + pb, sinks, gate, nr, w0[:, 1:], w1[:, 1:]) + return mx.compile(body) + + +def _post(self, has_bs, has_msg): + attn = self.self_attn + + def body(*args): + it = iter(args) + out = next(it); gate = next(it); h = next(it); nr = next(it) + bs = next(it) if has_bs else None + cs2 = next(it); cs3 = next(it) + msg = next(it) if has_msg else None + dtype = h.dtype + ao = out.reshape(1, 1, -1).astype(mx.float32) * mx.sigmoid(gate.astype(mx.float32)) + hs = attn.o_proj(ao.astype(gate.dtype)) + hs, w2 = self.post_attention_convolution(hs, cs2, None) + residual = hs = hs.astype(mx.float32) + if self.is_block_boundary: + block = nr[None] + bs = block if bs is None else mx.concatenate([bs, block]) + else: + residual = hs = h.astype(mx.float32) + hs + ns = L.unweighted_rms_norm(hs) + hs = self.mlp_input(bs, ns)[..., 0, :] + if msg is not None: + hs = self.mlp_message_gate(hs, msg[..., 3, :]) + hs = self.mlp(self.post_attention_layernorm(hs).astype(dtype)) + hs, w3 = self.post_mlp_convolution(hs, cs3, None) + hs = residual + hs.astype(mx.float32) + return hs.astype(dtype), bs, w2[:, 1:], w3[:, 1:] + return mx.compile(body) + + +def _get(kind, self, has_bs, has_msg): + key = (kind, id(self), has_bs, has_msg) + f = _cache.get(key) + if f is None: + f = _cache[key] = (_pre if kind == "pre" else _post)(self, has_bs, has_msg) + return f + + +# ---- mask kernel: mask[h, s] = visible ? (dist < R ? pb[h, dist] : 0) : -inf, same values as stock ---- +_MASK_SRC = """ + uint s = thread_position_in_grid.x; + uint h = thread_position_in_grid.y; + int S = params[0]; int end = params[1]; int idx = params[2]; int rot = params[3]; + int window = params[4]; int R = params[5]; + if ((int)s >= S) return; + int k = (int)s; + if (rot) { k = ((k - idx) % S + S) % S; } + int kpos = k + (end - S); + int dist = (end - 1) - kpos; + bool vis = (dist >= 0) && (kpos >= 0); + if (window > 0) vis = vis && (dist < window); + T v; + if (!vis) v = T(-INFINITY); + else if (dist < R) v = pb[h * R + dist]; + else v = T(0); + out[h * S + s] = v; +""" +_mask_k = None + + +def _attention_fast(attn, q, keys, values, pb, sinks, kvc): + global _mask_k + if _mask_k is None: + _mask_k = mx.fast.metal_kernel(name="ink_mask", input_names=["pb", "params"], output_names=["out"], source=_MASK_SRC) + S = keys.shape[2]; end = kvc.offset; H = pb.shape[2]; R = pb.shape[-1] + rot = isinstance(kvc, RotatingKVCache) + params = mx.array([S, end, kvc._idx if rot else 0, int(rot), attn.sliding_window, R], mx.int32) + mask = _mask_k(inputs=[pb[0, 0], params], template=[("T", q.dtype)], grid=(S, H, 1), threadgroup=(min(S, 256), 1, 1), + output_shapes=[(H, S)], output_dtypes=[q.dtype])[0] + o = mx.fast.scaled_dot_product_attention(q, keys, values, scale=attn.scaling, mask=mask[None, :, None, :], + sinks=sinks[0, 0].astype(q.dtype)) + return o.transpose(0, 2, 1, 3) + + +def call(self, hidden_states, block_states, *, cache=None, conv_mask=None, messages=None): + conv = cache[1] if cache is not None else None + if conv is None or hidden_states.shape[:2] != (1, 1) or conv.cache[0] is None or conv_mask is not None: + return _orig(self, hidden_states, block_states, cache=cache, conv_mask=conv_mask, messages=messages) + kvc = cache[0]; attn = self.self_attn + has_bs = block_states is not None; has_msg = messages is not None + args = [hidden_states] + ([block_states] if has_bs else []) + [conv[0], conv[1]] + ([messages] if has_msg else []) + if not _sliding(self): + args.append(mx.array([kvc.offset], mx.float32)) + q, k, v, pb, sinks, gate, nr, c0, c1 = _get("pre", self, has_bs, has_msg)(*args) + keys, values = kvc.update_and_fetch(k, v) + if OPTS["mask_kernel"]: + out = _attention_fast(attn, q, keys, values, pb, sinks, kvc) + else: + out = attn._attention(q, keys, values, pb, sinks, kvc) + args = [out, gate, hidden_states, nr] + ([block_states] if has_bs else []) + [conv[2], conv[3]] + ([messages] if has_msg else []) + hs, bs, c2, c3 = _get("post", self, has_bs, has_msg)(*args) + conv[0], conv[1], conv[2], conv[3] = c0, c1, c2, c3 + return hs, bs + + +def _cat_q(ps): + f = nn.QuantizedLinear.__new__(nn.QuantizedLinear); nn.Module.__init__(f) + f.group_size, f.bits, f.mode = ps[0].group_size, ps[0].bits, getattr(ps[0], "mode", "affine") + f.weight = mx.concatenate([p.weight for p in ps]); f.scales = mx.concatenate([p.scales for p in ps]) + f.biases = mx.concatenate([p.biases for p in ps]) + f.freeze(); mx.eval(f.parameters()) + sizes = [p.weight.shape[0] for p in ps] + return f, [sum(sizes[:i + 1]) for i in range(len(ps) - 1)] + + +def _fuse(model): + for layer in model.model.language_model.layers: + a = layer.self_attn + if id(a) in _fused: continue + ps = [a.q_proj, a.r_proj, a.gate_proj, a.sink_proj] + if not all(isinstance(p, nn.QuantizedLinear) for p in ps): continue + d = _fused[id(a)] = {} + d["qrgs"], d["qrgs_split"] = _cat_q(ps) + d["all6"], d["all6_split"] = _cat_q([a.q_proj, a.k_proj, a.v_proj, a.r_proj, a.gate_proj, a.sink_proj]) + + +# ---- message gate: forget/message projections share the input -> one matmul ---- +_orig_msg = L.MessageGate.__call__ +_msgw = {} + + +def _fuse_msg(model): + for layer in model.model.language_model.layers: + for n in ("q_message_gate", "k_message_gate", "v_message_gate", "mlp_message_gate"): + g = getattr(layer, n, None) + if g is None or id(g) in _msgw: continue + assert not isinstance(g.forget_gate_proj, nn.QuantizedLinear) + w = mx.concatenate([g.forget_gate_proj.weight, g.message_gate_proj.weight]); mx.eval(w) + _msgw[id(g)] = w + + +def _msg_call(self, hidden_states, message): + states = mx.concatenate([L.unweighted_rms_norm(hidden_states), L.unweighted_rms_norm(message)], axis=-1) + states = self.down_proj(states.astype(self.down_proj.weight.dtype)).astype(mx.float32) + w = _msgw.get(id(self)) + if w is None: + return _orig_msg(self, hidden_states, message) + fm = mx.tanh(states @ w.T) + forget_gate, message_gate = mx.split(fm, 2, axis=-1) + return (1 - forget_gate) * hidden_states + message_gate * message + + +# ---- short conv as elementwise multiply-add for L=1 ---- +_orig_conv = L.ShortConvolution.__call__ + + +def _conv_call(self, hidden_states, conv_state, conv_mask): + if hidden_states.shape[1] != 1 or conv_state is None or conv_mask is not None: + return _orig_conv(self, hidden_states, conv_state, conv_mask) + K = self.kernel_size + residual = hidden_states.astype(mx.float32) + window = L._conv_input(hidden_states, conv_state, conv_mask, K) + w = self.conv1d.weight # (C, K, 1) + x = window.astype(w.dtype) + out = (x.astype(mx.float32) * w[:, :, 0].T[None].astype(mx.float32)).sum(axis=1, keepdims=True).astype(w.dtype) + out = (out.astype(mx.float32) + residual) / (K + 1) ** 0.5 + return out.astype(hidden_states.dtype), window + + +def apply(model, **opts): + OPTS.update(_DEF); OPTS.update(opts); _cache.clear() + if OPTS["fuse_proj"]: _fuse(model) + L.DecoderLayer.__call__ = call + if OPTS["fuse_msg"]: _fuse_msg(model); L.MessageGate.__call__ = _msg_call + if OPTS["conv_ew"]: L.ShortConvolution.__call__ = _conv_call + return {} + + +def revert(model): + L.DecoderLayer.__call__ = _orig; _cache.clear() + L.MessageGate.__call__ = _orig_msg; L.ShortConvolution.__call__ = _orig_conv diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3a_split.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3a_split.py new file mode 100644 index 00000000..fc833844 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3a_split.py @@ -0,0 +1,4 @@ +"""Round 3 a_split: three-part layer decode path (pre graph / cache + attention / post graph), options [].""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, ) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3b_mask.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3b_mask.py new file mode 100644 index 00000000..d0ead909 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3b_mask.py @@ -0,0 +1,4 @@ +"""Round 3 b_mask: three-part layer decode path (pre graph / cache + attention / post graph), options [mask_kernel=True].""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, mask_kernel=True) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3c_mask_fuse.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3c_mask_fuse.py new file mode 100644 index 00000000..e2132854 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3c_mask_fuse.py @@ -0,0 +1,4 @@ +"""Round 3 c_mask_fuse: three-part layer decode path (pre graph / cache + attention / post graph), options [mask_kernel=True, fuse_proj=True].""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3d_all6.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3d_all6.py new file mode 100644 index 00000000..67212375 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3d_all6.py @@ -0,0 +1,4 @@ +"""Round 3 d_all6: three-part layer decode path (pre graph / cache + attention / post graph), options [mask_kernel=True, fuse_proj=True, fuse_all6=True].""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r3/cands/r3e_msg.py b/examples/inkling2_mini_mlx/rounds/r3/cands/r3e_msg.py new file mode 100644 index 00000000..a9232e4f --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/cands/r3e_msg.py @@ -0,0 +1,4 @@ +"""Round 3 e_msg: three-part layer decode path (pre graph / cache + attention / post graph), options [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True].""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r3/log-evolve.jsonl b/examples/inkling2_mini_mlx/rounds/r3/log-evolve.jsonl new file mode 100644 index 00000000..fd610258 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/log-evolve.jsonl @@ -0,0 +1,6 @@ +{"cand": "r3a_split", "dec_med": 58.597822881124515, "dec_all": [58.597822881124515, 59.36159852090833, 57.87520297279241], "ttft_med": 0.22028803825378418, "peak": 17.559962404, "paired": [1.2901202935255751, 1.328994706191247, 1.2751990680281606], "same_output": true, "err": null, "doc": "第 3 轮 a_split:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 []。"} +{"cand": "r3b_mask", "dec_med": 64.39439576221922, "dec_all": [64.39439576221922, 64.50176422427727, 62.196659874623435], "ttft_med": 0.21064341068267822, "peak": 17.55992794, "paired": [1.3974349092167846, 1.3538424803419962, 1.337859365648511], "same_output": true, "err": null, "doc": "第 3 轮 b_mask:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True]。"} +{"cand": "r3c_mask_fuse", "dec_med": 65.87582522354171, "dec_all": [66.85689733335586, 60.07978068716797, 65.87582522354171], "ttft_med": 0.21292757987976074, "peak": 18.031822244, "paired": [1.4295186687269676, 1.3107599987142853, 1.475380798263453], "same_output": true, "err": null, "doc": "第 3 轮 c_mask_fuse:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True]。"} +{"cand": "r3d_all6", "dec_med": 62.6047822368255, "dec_all": [62.6047822368255, 59.526029134274545, 65.52385727844654], "ttft_med": 0.21091711521148682, "peak": 18.031755524, "paired": [1.4405800103914217, 1.2814983090199807, 1.4239628466388194], "same_output": true, "err": null, "doc": "第 3 轮 d_all6:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True]。"} +{"cand": "r3e_msg", "dec_med": 63.991903394997216, "dec_all": [63.92982974662121, 65.6092751424128, 63.991903394997216], "ttft_med": 0.21320867538452148, "peak": 18.097880132, "paired": [1.3747761961567637, 1.4176718066732137, 1.37302375595508], "same_output": true, "err": null, "doc": "第 3 轮 e_msg:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。"} +{"cand": "baseline", "dec_med": 46.047785588104894, "dec_all": [44.99154466912073, 45.42043340857104, 44.666542495893125, 45.38522998004131, 46.08042588424395, 47.64347784978898, 46.48968454503763, 46.76881722215917, 45.83583626758505, 44.65004919481033, 43.458038974048435, 46.450337636260144, 46.01514529196584, 46.50199059697087, 46.27959365036335, 46.60655223003351], "ttft_med": 0.2206212282180786, "peak": 18.0883913, "noise_range": 0.09089338004609131} diff --git a/examples/inkling2_mini_mlx/rounds/r3/log-holdout.jsonl b/examples/inkling2_mini_mlx/rounds/r3/log-holdout.jsonl new file mode 100644 index 00000000..84524c79 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/log-holdout.jsonl @@ -0,0 +1,6 @@ +{"cand": "r3a_split", "dec_med": 59.815185922194104, "dec_all": [59.815185922194104, 58.116759316609006, 60.00709217739403], "ttft_med": 0.1961524486541748, "peak": 17.51477316, "paired": [1.2901830930466067, 1.2212808959664423, 1.2992045891748238], "same_output": true, "err": null, "doc": "第 3 轮 a_split:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 []。"} +{"cand": "r3b_mask", "dec_med": 64.60535174358287, "dec_all": [64.47242975670096, 64.88611319582448, 64.60535174358287], "ttft_med": 0.18941152095794678, "peak": 17.514705052, "paired": [1.3532531059417017, 1.355208418973997, 1.349479906786514], "same_output": true, "err": null, "doc": "第 3 轮 b_mask:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True]。"} +{"cand": "r3c_mask_fuse", "dec_med": 66.82440670116867, "dec_all": [66.61798282077788, 66.86022128070815, 66.82440670116867], "ttft_med": 0.18910002708435059, "peak": 17.98661582, "paired": [1.4237253603067872, 1.4040673680084226, 1.419207536882739], "same_output": true, "err": null, "doc": "第 3 轮 c_mask_fuse:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True]。"} +{"cand": "r3d_all6", "dec_med": 66.66493160403824, "dec_all": [66.66493160403824, 66.80098053708868, 64.81861677574682], "ttft_med": 0.18864834308624268, "peak": 17.98495168, "paired": [1.4005889493981019, 1.3960647918871552, 1.3468629042926332], "same_output": true, "err": null, "doc": "第 3 轮 d_all6:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True]。"} +{"cand": "r3e_msg", "dec_med": 66.20176969252506, "dec_all": [66.20176969252506, 66.30925957405773, 64.2510970652737], "ttft_med": 0.19022655487060547, "peak": 18.048848016, "paired": [1.4179043613368552, 1.3922370512388482, 1.3556241642347557], "same_output": true, "err": null, "doc": "第 3 轮 e_msg:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。"} +{"cand": "baseline", "dec_med": 47.59225545047404, "dec_all": [44.875001491678844, 46.36178093215281, 47.58672596005785, 46.1875617415321, 47.642550734687504, 47.87906589670432, 47.87425986758568, 46.7913157116362, 47.618955332282226, 47.085718589084685, 47.597784940890236, 47.84948444032406, 48.12562330520884, 46.68986956927579, 47.627851532218635, 47.395951444656625], "ttft_med": 0.19784337282180786, "peak": 18.043229216, "noise_range": 0.0683014869281136} diff --git a/examples/inkling2_mini_mlx/rounds/r3/mechanism.txt b/examples/inkling2_mini_mlx/rounds/r3/mechanism.txt new file mode 100644 index 00000000..2d32a483 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/mechanism.txt @@ -0,0 +1,6 @@ +base build_ms=3.95 step_ms=22.01 +cand build_ms=2.13 step_ms=16.43 +base build_ms=3.99 step_ms=22.55 +cand build_ms=1.87 step_ms=15.95 +total 12184 non-view 7624 +total 7176 non-view 4224 diff --git a/examples/inkling2_mini_mlx/rounds/r3/score-evolve.json b/examples/inkling2_mini_mlx/rounds/r3/score-evolve.json new file mode 100644 index 00000000..9de69c82 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/score-evolve.json @@ -0,0 +1,128 @@ +{ + "baseline": { + "dec_med": 46.047785588104894, + "dec_all": [ + 44.99154466912073, + 45.42043340857104, + 44.666542495893125, + 45.38522998004131, + 46.08042588424395, + 47.64347784978898, + 46.48968454503763, + 46.76881722215917, + 45.83583626758505, + 44.65004919481033, + 43.458038974048435, + 46.450337636260144, + 46.01514529196584, + 46.50199059697087, + 46.27959365036335, + 46.60655223003351 + ], + "ttft_med": 0.2206212282180786, + "peak": 18.0883913 + }, + "noise_range": 0.09089338004609131, + "cands": { + "r3a_split": { + "dec_med": 58.597822881124515, + "dec_all": [ + 58.597822881124515, + 59.36159852090833, + 57.87520297279241 + ], + "ttft_med": 0.22028803825378418, + "peak": 17.559962404, + "paired": [ + 1.2901202935255751, + 1.328994706191247, + 1.2751990680281606 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 a_split:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 []。", + "speedup": 1.2901202935255751, + "verdict": "留" + }, + "r3b_mask": { + "dec_med": 64.39439576221922, + "dec_all": [ + 64.39439576221922, + 64.50176422427727, + 62.196659874623435 + ], + "ttft_med": 0.21064341068267822, + "peak": 17.55992794, + "paired": [ + 1.3974349092167846, + 1.3538424803419962, + 1.337859365648511 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 b_mask:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True]。", + "speedup": 1.3538424803419962, + "verdict": "留" + }, + "r3c_mask_fuse": { + "dec_med": 65.87582522354171, + "dec_all": [ + 66.85689733335586, + 60.07978068716797, + 65.87582522354171 + ], + "ttft_med": 0.21292757987976074, + "peak": 18.031822244, + "paired": [ + 1.4295186687269676, + 1.3107599987142853, + 1.475380798263453 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 c_mask_fuse:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True]。", + "speedup": 1.4295186687269676, + "verdict": "留" + }, + "r3d_all6": { + "dec_med": 62.6047822368255, + "dec_all": [ + 62.6047822368255, + 59.526029134274545, + 65.52385727844654 + ], + "ttft_med": 0.21091711521148682, + "peak": 18.031755524, + "paired": [ + 1.4405800103914217, + 1.2814983090199807, + 1.4239628466388194 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 d_all6:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True]。", + "speedup": 1.4239628466388194, + "verdict": "留" + }, + "r3e_msg": { + "dec_med": 63.991903394997216, + "dec_all": [ + 63.92982974662121, + 65.6092751424128, + 63.991903394997216 + ], + "ttft_med": 0.21320867538452148, + "peak": 18.097880132, + "paired": [ + 1.3747761961567637, + 1.4176718066732137, + 1.37302375595508 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 e_msg:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。", + "speedup": 1.3747761961567637, + "verdict": "留" + } + } +} \ No newline at end of file diff --git a/examples/inkling2_mini_mlx/rounds/r3/score-holdout.json b/examples/inkling2_mini_mlx/rounds/r3/score-holdout.json new file mode 100644 index 00000000..3ff64c97 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r3/score-holdout.json @@ -0,0 +1,128 @@ +{ + "baseline": { + "dec_med": 47.59225545047404, + "dec_all": [ + 44.875001491678844, + 46.36178093215281, + 47.58672596005785, + 46.1875617415321, + 47.642550734687504, + 47.87906589670432, + 47.87425986758568, + 46.7913157116362, + 47.618955332282226, + 47.085718589084685, + 47.597784940890236, + 47.84948444032406, + 48.12562330520884, + 46.68986956927579, + 47.627851532218635, + 47.395951444656625 + ], + "ttft_med": 0.19784337282180786, + "peak": 18.043229216 + }, + "noise_range": 0.0683014869281136, + "cands": { + "r3a_split": { + "dec_med": 59.815185922194104, + "dec_all": [ + 59.815185922194104, + 58.116759316609006, + 60.00709217739403 + ], + "ttft_med": 0.1961524486541748, + "peak": 17.51477316, + "paired": [ + 1.2901830930466067, + 1.2212808959664423, + 1.2992045891748238 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 a_split:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 []。", + "speedup": 1.2901830930466067, + "verdict": "留" + }, + "r3b_mask": { + "dec_med": 64.60535174358287, + "dec_all": [ + 64.47242975670096, + 64.88611319582448, + 64.60535174358287 + ], + "ttft_med": 0.18941152095794678, + "peak": 17.514705052, + "paired": [ + 1.3532531059417017, + 1.355208418973997, + 1.349479906786514 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 b_mask:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True]。", + "speedup": 1.3532531059417017, + "verdict": "留" + }, + "r3c_mask_fuse": { + "dec_med": 66.82440670116867, + "dec_all": [ + 66.61798282077788, + 66.86022128070815, + 66.82440670116867 + ], + "ttft_med": 0.18910002708435059, + "peak": 17.98661582, + "paired": [ + 1.4237253603067872, + 1.4040673680084226, + 1.419207536882739 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 c_mask_fuse:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True]。", + "speedup": 1.419207536882739, + "verdict": "留" + }, + "r3d_all6": { + "dec_med": 66.66493160403824, + "dec_all": [ + 66.66493160403824, + 66.80098053708868, + 64.81861677574682 + ], + "ttft_med": 0.18864834308624268, + "peak": 17.98495168, + "paired": [ + 1.4005889493981019, + 1.3960647918871552, + 1.3468629042926332 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 d_all6:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True]。", + "speedup": 1.3960647918871552, + "verdict": "留" + }, + "r3e_msg": { + "dec_med": 66.20176969252506, + "dec_all": [ + 66.20176969252506, + 66.30925957405773, + 64.2510970652737 + ], + "ttft_med": 0.19022655487060547, + "peak": 18.048848016, + "paired": [ + 1.4179043613368552, + 1.3922370512388482, + 1.3556241642347557 + ], + "same_output": true, + "err": null, + "doc": "第 3 轮 e_msg:每层解码拆三段(前段合图/缓存+注意力/后段合图),选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。", + "speedup": 1.3922370512388482, + "verdict": "留" + } + } +} \ No newline at end of file diff --git a/examples/inkling2_mini_mlx/rounds/r4/cands/_fast.py b/examples/inkling2_mini_mlx/rounds/r4/cands/_fast.py new file mode 100644 index 00000000..dd64db47 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/cands/_fast.py @@ -0,0 +1,315 @@ +# Shared by rounds 3-4: each layer visit of a decode step (B=1, L=1) runs as pre graph -> cache write + attention -> post graph. +# pre : residual read-in, message gates, norms, q/k/v/r/gate/sink projections, short convs, clips, relative bias; one compiled graph; +# mid : KV-cache write and attention (length changes every step, not compiled); +# post: output gate, o_proj, convs, residual, MoE, convs; one compiled graph. Conv states are explicit inputs/outputs. +# Options: mask_kernel = attention mask in one Metal kernel; fuse_proj = projections sharing an input in one quantized matmul. +import mlx.core as mx +from mlx import nn +from mlx_lm.models.inkling2 import layers as L +from mlx_lm.models.cache import RotatingKVCache + +OPTS = dict(mask_kernel=False, fuse_proj=False, fuse_all6=False, fuse_msg=False, conv_ew=False, conv_kernel=False, batch_msg=False) +_DEF = dict(OPTS) +_orig = L.DecoderLayer.__call__ +_cache = {} +_fused = {} + + +def _sliding(self): + return isinstance(self, L.SlidingDecoderLayer) + + +def _proj(self, attn, qi, ki, vi, same_kv): + if OPTS["fuse_proj"]: + f = _fused.get(id(attn)) + if f is not None: + outs = f["qrgs"](qi) + qp, rp, gate, sinks = mx.split(outs, f["qrgs_split"], axis=-1) + if same_kv and OPTS["fuse_all6"]: # sliding layer, no messages: q/k/v share the input, six projections at once + qp, kp, vp, rp, gate, sinks = mx.split(f["all6"](qi), f["all6_split"], axis=-1) + return qp, kp, vp, rp, gate, sinks + kp = attn.k_proj(ki); vp = attn.v_proj(vi) + return qp, kp, vp, rp, gate, sinks + return attn.q_proj(qi), attn.k_proj(ki), attn.v_proj(vi), attn.r_proj(qi), attn.gate_proj(qi), attn.sink_proj(qi) + + +# ---- short conv kernel (round 4): window, cast to weight dtype, 4-tap depthwise conv, residual, / sqrt(K+1), new state: one kernel ---- +_CONV_SRC = """ + uint c = thread_position_in_grid.x; + if (c >= (uint)C) return; + float xs[4]; + for (int k = 0; k < 3; ++k) xs[k] = state[k * C + c]; + float xr = static_cast(x[c]); + xs[3] = xr; + float acc = 0.0f; + for (int k = 0; k < 4; ++k) acc += static_cast(static_cast(xs[k])) * static_cast(w[c * 4 + k]); + WT o = static_cast(acc); + float r = (static_cast(o) + xr) / 2.2360679774997896f; + out[c] = static_cast(r); + for (int k = 0; k < 3; ++k) new_state[k * C + c] = xs[k + 1]; +""" +_conv_k = None + + +def _sconv(conv, x, st): + """Return (output, new conv state).""" + global _conv_k + if OPTS["conv_kernel"] and conv.kernel_size == 4: + if _conv_k is None: + _conv_k = mx.fast.metal_kernel(name="ink_sconv", input_names=["x", "state", "w"], + output_names=["out", "new_state"], source=_CONV_SRC) + C = x.shape[-1]; w = conv.conv1d.weight + o, ns = _conv_k(inputs=[x, st, w], template=[("C", C), ("WT", w.dtype), ("OT", x.dtype)], grid=(C, 1, 1), + threadgroup=(256, 1, 1), output_shapes=[x.shape, st.shape], output_dtypes=[x.dtype, mx.float32]) + return o, ns + o, win = conv(x, st, None) + return o, win[:, 1:] + + +# ---- q/k/v message gates as one batched matmul (round 4) ---- +_bw = {} + + +def _mix(self, states, msg): + if not OPTS["batch_msg"]: + return self._mix_messages(states, msg) + gs = (self.q_message_gate, self.k_message_gate, self.v_message_gate) + w = _bw.get(id(self)) + if w is None: + wd = mx.stack([g.down_proj.weight for g in gs]) # (3, r, 2D) + wf = mx.stack([mx.concatenate([g.forget_gate_proj.weight, g.message_gate_proj.weight]) for g in gs]) # (3, 2D, r) + mx.eval(wd, wf); w = _bw[id(self)] = (wd, wf) + wd, wf = w + h = mx.stack(list(states)) # (3, B, L, D) + m = mx.stack([msg[..., i, :] for i in range(3)]) + x = mx.concatenate([L.unweighted_rms_norm(h), L.unweighted_rms_norm(m)], axis=-1) + x = x.astype(wd.dtype).reshape(3, -1, x.shape[-1]) + st = (x @ wd.swapaxes(-1, -2)).astype(mx.float32) + fm = mx.tanh(st @ wf.swapaxes(-1, -2)).reshape(*h.shape[:-1], -1) + fg, mg = mx.split(fm, 2, axis=-1) + out = (1 - fg) * h + mg * m + return out[0], out[1], out[2] + + +def _pre(self, has_bs, has_msg): + sliding = _sliding(self); attn = self.self_attn + + def body(*args): + it = iter(args) + h = next(it); bs = next(it) if has_bs else None + cs0 = next(it); cs1 = next(it) + msg = next(it) if has_msg else None + pos = None if sliding else next(it) + dtype = h.dtype + nr = L.unweighted_rms_norm(h.astype(mx.float32)) + ai = self.attention_input(bs, nr) if self.attention_input is not None else None + if sliding: + hs = ai[..., 0, :] if ai is not None else h + if msg is None: + qi = ki = vi = self.input_layernorm(hs).astype(dtype) + else: + st = _mix(self, (hs,) * 3, msg) + qi, ki, vi = [self.input_layernorm(s).astype(dtype) for s in st] + else: + q, k, v = (ai[..., i, :] for i in range(3)) + if msg is not None: + q, k, v = _mix(self, (q, k, v), msg) + qi = self.q_input_layernorm(q).astype(dtype) + ki = self.k_input_layernorm(k).astype(dtype) + vi = self.v_input_layernorm(v).astype(dtype) + qp, kp, vp, rp, gate, sinks = _proj(self, attn, qi, ki, vi, sliding and msg is None) + if sliding: + ks, n0 = _sconv(attn.k_sconv, kp, cs0); vs, n1 = _sconv(attn.v_sconv, vp, cs1) + else: + ks, w0 = attn.k_sconv(kp, ki, cs0, None); vs, w1 = attn.v_sconv(vp, vi, cs1, None) + n0, n1 = w0[:, 1:], w1[:, 1:] + B, Ln = 1, 1 + qs = qp.reshape(B, Ln, attn.num_heads, -1) + ks = ks.reshape(B, Ln, attn.num_key_value_heads, -1) + vs = vs.reshape(B, Ln, attn.num_key_value_heads, -1) + rs = rp.reshape(B, Ln, attn.num_heads, -1) + qs = attn.q_l2_clip(qs); ks = attn.k_l2_clip(ks) + vs = L.unweighted_rms_norm(vs.astype(mx.float32), attn.eps).astype(vs.dtype) + rs = attn.rel_l2_clip(rs) + if not sliding: + tau = 1.0 + attn.log_scaling_alpha * mx.log(mx.maximum((pos + 1) / attn.log_scaling_n_floor, 1.0)) + tau = tau[..., None, None] + qs = (qs.astype(mx.float32) * tau).astype(qs.dtype) + rs = (rs.astype(mx.float32) * tau).astype(rs.dtype) + pb = attn.rel_logits_proj(rs) + return (qs.transpose(0, 2, 1, 3), ks.transpose(0, 2, 1, 3), vs.transpose(0, 2, 1, 3), + pb, sinks, gate, nr, n0, n1) + return mx.compile(body) + + +def _post(self, has_bs, has_msg): + attn = self.self_attn + + def body(*args): + it = iter(args) + out = next(it); gate = next(it); h = next(it); nr = next(it) + bs = next(it) if has_bs else None + cs2 = next(it); cs3 = next(it) + msg = next(it) if has_msg else None + dtype = h.dtype + ao = out.reshape(1, 1, -1).astype(mx.float32) * mx.sigmoid(gate.astype(mx.float32)) + hs = attn.o_proj(ao.astype(gate.dtype)) + hs, n2 = _sconv(self.post_attention_convolution, hs, cs2) + residual = hs = hs.astype(mx.float32) + if self.is_block_boundary: + block = nr[None] + bs = block if bs is None else mx.concatenate([bs, block]) + else: + residual = hs = h.astype(mx.float32) + hs + ns = L.unweighted_rms_norm(hs) + hs = self.mlp_input(bs, ns)[..., 0, :] + if msg is not None: + hs = self.mlp_message_gate(hs, msg[..., 3, :]) + hs = self.mlp(self.post_attention_layernorm(hs).astype(dtype)) + hs, n3 = _sconv(self.post_mlp_convolution, hs, cs3) + hs = residual + hs.astype(mx.float32) + return hs.astype(dtype), bs, n2, n3 + return mx.compile(body) + + +def _get(kind, self, has_bs, has_msg): + key = (kind, id(self), has_bs, has_msg) + f = _cache.get(key) + if f is None: + f = _cache[key] = (_pre if kind == "pre" else _post)(self, has_bs, has_msg) + return f + + +# ---- mask kernel: mask[h, s] = visible ? (dist < R ? pb[h, dist] : 0) : -inf, same values as stock ---- +_MASK_SRC = """ + uint s = thread_position_in_grid.x; + uint h = thread_position_in_grid.y; + int S = params[0]; int end = params[1]; int idx = params[2]; int rot = params[3]; + int window = params[4]; int R = params[5]; + if ((int)s >= S) return; + int k = (int)s; + if (rot) { k = ((k - idx) % S + S) % S; } + int kpos = k + (end - S); + int dist = (end - 1) - kpos; + bool vis = (dist >= 0) && (kpos >= 0); + if (window > 0) vis = vis && (dist < window); + T v; + if (!vis) v = T(-INFINITY); + else if (dist < R) v = pb[h * R + dist]; + else v = T(0); + out[h * S + s] = v; +""" +_mask_k = None + + +def _attention_fast(attn, q, keys, values, pb, sinks, kvc): + global _mask_k + if _mask_k is None: + _mask_k = mx.fast.metal_kernel(name="ink_mask", input_names=["pb", "params"], output_names=["out"], source=_MASK_SRC) + S = keys.shape[2]; end = kvc.offset; H = pb.shape[2]; R = pb.shape[-1] + rot = isinstance(kvc, RotatingKVCache) + params = mx.array([S, end, kvc._idx if rot else 0, int(rot), attn.sliding_window, R], mx.int32) + mask = _mask_k(inputs=[pb[0, 0], params], template=[("T", q.dtype)], grid=(S, H, 1), threadgroup=(min(S, 256), 1, 1), + output_shapes=[(H, S)], output_dtypes=[q.dtype])[0] + o = mx.fast.scaled_dot_product_attention(q, keys, values, scale=attn.scaling, mask=mask[None, :, None, :], + sinks=sinks[0, 0].astype(q.dtype)) + return o.transpose(0, 2, 1, 3) + + +def call(self, hidden_states, block_states, *, cache=None, conv_mask=None, messages=None): + conv = cache[1] if cache is not None else None + if conv is None or hidden_states.shape[:2] != (1, 1) or conv.cache[0] is None or conv_mask is not None: + return _orig(self, hidden_states, block_states, cache=cache, conv_mask=conv_mask, messages=messages) + kvc = cache[0]; attn = self.self_attn + has_bs = block_states is not None; has_msg = messages is not None + args = [hidden_states] + ([block_states] if has_bs else []) + [conv[0], conv[1]] + ([messages] if has_msg else []) + if not _sliding(self): + args.append(mx.array([kvc.offset], mx.float32)) + q, k, v, pb, sinks, gate, nr, c0, c1 = _get("pre", self, has_bs, has_msg)(*args) + keys, values = kvc.update_and_fetch(k, v) + if OPTS["mask_kernel"]: + out = _attention_fast(attn, q, keys, values, pb, sinks, kvc) + else: + out = attn._attention(q, keys, values, pb, sinks, kvc) + args = [out, gate, hidden_states, nr] + ([block_states] if has_bs else []) + [conv[2], conv[3]] + ([messages] if has_msg else []) + hs, bs, c2, c3 = _get("post", self, has_bs, has_msg)(*args) + conv[0], conv[1], conv[2], conv[3] = c0, c1, c2, c3 + return hs, bs + + +def _cat_q(ps): + f = nn.QuantizedLinear.__new__(nn.QuantizedLinear); nn.Module.__init__(f) + f.group_size, f.bits, f.mode = ps[0].group_size, ps[0].bits, getattr(ps[0], "mode", "affine") + f.weight = mx.concatenate([p.weight for p in ps]); f.scales = mx.concatenate([p.scales for p in ps]) + f.biases = mx.concatenate([p.biases for p in ps]) + f.freeze(); mx.eval(f.parameters()) + sizes = [p.weight.shape[0] for p in ps] + return f, [sum(sizes[:i + 1]) for i in range(len(ps) - 1)] + + +def _fuse(model): + for layer in model.model.language_model.layers: + a = layer.self_attn + if id(a) in _fused: continue + ps = [a.q_proj, a.r_proj, a.gate_proj, a.sink_proj] + if not all(isinstance(p, nn.QuantizedLinear) for p in ps): continue + d = _fused[id(a)] = {} + d["qrgs"], d["qrgs_split"] = _cat_q(ps) + d["all6"], d["all6_split"] = _cat_q([a.q_proj, a.k_proj, a.v_proj, a.r_proj, a.gate_proj, a.sink_proj]) + + +# ---- message gate: forget/message projections share the input -> one matmul ---- +_orig_msg = L.MessageGate.__call__ +_msgw = {} + + +def _fuse_msg(model): + for layer in model.model.language_model.layers: + for n in ("q_message_gate", "k_message_gate", "v_message_gate", "mlp_message_gate"): + g = getattr(layer, n, None) + if g is None or id(g) in _msgw: continue + assert not isinstance(g.forget_gate_proj, nn.QuantizedLinear) + w = mx.concatenate([g.forget_gate_proj.weight, g.message_gate_proj.weight]); mx.eval(w) + _msgw[id(g)] = w + + +def _msg_call(self, hidden_states, message): + states = mx.concatenate([L.unweighted_rms_norm(hidden_states), L.unweighted_rms_norm(message)], axis=-1) + states = self.down_proj(states.astype(self.down_proj.weight.dtype)).astype(mx.float32) + w = _msgw.get(id(self)) + if w is None: + return _orig_msg(self, hidden_states, message) + fm = mx.tanh(states @ w.T) + forget_gate, message_gate = mx.split(fm, 2, axis=-1) + return (1 - forget_gate) * hidden_states + message_gate * message + + +# ---- short conv as elementwise multiply-add for L=1 ---- +_orig_conv = L.ShortConvolution.__call__ + + +def _conv_call(self, hidden_states, conv_state, conv_mask): + if hidden_states.shape[1] != 1 or conv_state is None or conv_mask is not None: + return _orig_conv(self, hidden_states, conv_state, conv_mask) + K = self.kernel_size + residual = hidden_states.astype(mx.float32) + window = L._conv_input(hidden_states, conv_state, conv_mask, K) + w = self.conv1d.weight # (C, K, 1) + x = window.astype(w.dtype) + out = (x.astype(mx.float32) * w[:, :, 0].T[None].astype(mx.float32)).sum(axis=1, keepdims=True).astype(w.dtype) + out = (out.astype(mx.float32) + residual) / (K + 1) ** 0.5 + return out.astype(hidden_states.dtype), window + + +def apply(model, **opts): + OPTS.update(_DEF); OPTS.update(opts); _cache.clear() + if OPTS["fuse_proj"]: _fuse(model) + L.DecoderLayer.__call__ = call + if OPTS["fuse_msg"]: _fuse_msg(model); L.MessageGate.__call__ = _msg_call + if OPTS["conv_ew"]: L.ShortConvolution.__call__ = _conv_call + return {} + + +def revert(model): + L.DecoderLayer.__call__ = _orig; _cache.clear() + L.MessageGate.__call__ = _orig_msg; L.ShortConvolution.__call__ = _orig_conv diff --git a/examples/inkling2_mini_mlx/rounds/r4/cands/r4a_conv_kernel.py b/examples/inkling2_mini_mlx/rounds/r4/cands/r4a_conv_kernel.py new file mode 100644 index 00000000..1c4f1f00 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/cands/r4a_conv_kernel.py @@ -0,0 +1,4 @@ +"""Round 4 a_conv_kernel: round-3 three-part decode path, options [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True].""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r4/cands/r4b_ref_r3e.py b/examples/inkling2_mini_mlx/rounds/r4/cands/r4b_ref_r3e.py new file mode 100644 index 00000000..c14bb7ba --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/cands/r4b_ref_r3e.py @@ -0,0 +1,4 @@ +"""Round 4 b_ref_r3e: round-3 three-part decode path, options [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True].""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r4/cands/r4c_conv_batchmsg.py b/examples/inkling2_mini_mlx/rounds/r4/cands/r4c_conv_batchmsg.py new file mode 100644 index 00000000..709f4cfb --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/cands/r4c_conv_batchmsg.py @@ -0,0 +1,4 @@ +"""Round 4 c_conv_batchmsg: round-3 three-part decode path, options [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True].""" +import sys, os; sys.path.insert(0, os.path.dirname(__file__)); import _fast +def apply(model): return _fast.apply(model, mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True) +def revert(model): _fast.revert(model) diff --git a/examples/inkling2_mini_mlx/rounds/r4/log-evolve.jsonl b/examples/inkling2_mini_mlx/rounds/r4/log-evolve.jsonl new file mode 100644 index 00000000..bd643dbd --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/log-evolve.jsonl @@ -0,0 +1,4 @@ +{"cand": "r4a_conv_kernel", "dec_med": 70.37613505324236, "dec_all": [70.37613505324236, 63.407270571352086, 70.53740472243632], "ttft_med": 0.21026098728179932, "peak": 18.097880836, "paired": [1.4781394375224184, 1.4864271727446174, 1.5782824063347054], "same_output": true, "err": null, "doc": "第 4 轮 a_conv_kernel:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True]。"} +{"cand": "r4b_ref_r3e", "dec_med": 64.05252272468896, "dec_all": [64.05252272468896, 66.85197717882195, 62.98588519987612], "ttft_med": 0.2118755578994751, "peak": 18.097897604, "paired": [1.3271082612873688, 1.388945637421096, 1.3089572241154328], "same_output": true, "err": null, "doc": "第 4 轮 b_ref_r3e:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。"} +{"cand": "r4c_conv_batchmsg", "dec_med": 67.27773522180502, "dec_all": [67.27773522180502, 64.80468996476651, 67.576769573957], "ttft_med": 0.21914458274841309, "peak": 18.161532996, "paired": [1.516376817631209, 1.5187029693126763, 1.4963789746016443], "same_output": true, "err": null, "doc": "第 4 轮 c_conv_batchmsg:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True]。"} +{"cand": "baseline", "dec_med": 45.971434473146694, "dec_all": [46.78267174962613, 47.611296516926195, 42.65750232100074, 44.69251158051463, 48.26473061252324, 48.13145696828593, 48.11913180924665, 44.36742532565367, 42.67107609205199, 45.160197196667255], "ttft_med": 0.2195042371749878, "peak": 18.152008372, "noise_range": 0.12197201057099595} diff --git a/examples/inkling2_mini_mlx/rounds/r4/log-holdout.jsonl b/examples/inkling2_mini_mlx/rounds/r4/log-holdout.jsonl new file mode 100644 index 00000000..bf284570 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/log-holdout.jsonl @@ -0,0 +1,4 @@ +{"cand": "r4a_conv_kernel", "dec_med": 66.5639386058893, "dec_all": [66.5639386058893, 64.83010099014217, 66.91319649713606], "ttft_med": 0.192732572555542, "peak": 18.048849136, "paired": [1.4621318225938569, 1.4912991034038232, 1.5152283342410546], "same_output": true, "err": null, "doc": "第 4 轮 a_conv_kernel:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True]。"} +{"cand": "r4b_ref_r3e", "dec_med": 63.21071599250109, "dec_all": [63.48963303775635, 60.665830819532324, 63.21071599250109], "ttft_med": 0.19359838962554932, "peak": 18.049061552, "paired": [1.4089705684626987, 1.3472671620694185, 1.4668349117184647], "same_output": true, "err": null, "doc": "第 4 轮 b_ref_r3e:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。"} +{"cand": "r4c_conv_batchmsg", "dec_med": 70.28057320251098, "dec_all": [70.28057320251098, 69.37463871972089, 70.67514817849448], "ttft_med": 0.18816149234771729, "peak": 18.11250072, "paired": [1.507655637687247, 1.5007724118719972, 1.4964355952280801], "same_output": true, "err": null, "doc": "第 4 轮 c_conv_batchmsg:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True]。"} +{"cand": "baseline", "dec_med": 45.04490632221295, "dec_all": [43.06871846686166, 45.5252649434873, 43.472232258552545, 44.16047072578763, 45.06100727641791, 45.02880536800799, 43.09327211093362, 46.615799686407044, 46.225955495267954, 47.22899428740365], "ttft_med": 0.20153599977493286, "peak": 18.105733216, "noise_range": 0.09235840764731344} diff --git a/examples/inkling2_mini_mlx/rounds/r4/mechanism.txt b/examples/inkling2_mini_mlx/rounds/r4/mechanism.txt new file mode 100644 index 00000000..f92e9883 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/mechanism.txt @@ -0,0 +1,6 @@ +base build_ms=3.94 step_ms=22.17 +cand build_ms=1.75 step_ms=15.28 +base build_ms=3.91 step_ms=22.05 +cand build_ms=1.79 step_ms=15.28 +total 12184 non-view 7624 +total 6440 non-view 3664 diff --git a/examples/inkling2_mini_mlx/rounds/r4/score-evolve.json b/examples/inkling2_mini_mlx/rounds/r4/score-evolve.json new file mode 100644 index 00000000..677bb8e8 --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/score-evolve.json @@ -0,0 +1,82 @@ +{ + "baseline": { + "dec_med": 45.971434473146694, + "dec_all": [ + 46.78267174962613, + 47.611296516926195, + 42.65750232100074, + 44.69251158051463, + 48.26473061252324, + 48.13145696828593, + 48.11913180924665, + 44.36742532565367, + 42.67107609205199, + 45.160197196667255 + ], + "ttft_med": 0.2195042371749878, + "peak": 18.152008372 + }, + "noise_range": 0.12197201057099595, + "cands": { + "r4a_conv_kernel": { + "dec_med": 70.37613505324236, + "dec_all": [ + 70.37613505324236, + 63.407270571352086, + 70.53740472243632 + ], + "ttft_med": 0.21026098728179932, + "peak": 18.097880836, + "paired": [ + 1.4781394375224184, + 1.4864271727446174, + 1.5782824063347054 + ], + "same_output": true, + "err": null, + "doc": "第 4 轮 a_conv_kernel:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True]。", + "speedup": 1.4864271727446174, + "verdict": "留" + }, + "r4b_ref_r3e": { + "dec_med": 64.05252272468896, + "dec_all": [ + 64.05252272468896, + 66.85197717882195, + 62.98588519987612 + ], + "ttft_med": 0.2118755578994751, + "peak": 18.097897604, + "paired": [ + 1.3271082612873688, + 1.388945637421096, + 1.3089572241154328 + ], + "same_output": true, + "err": null, + "doc": "第 4 轮 b_ref_r3e:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。", + "speedup": 1.3271082612873688, + "verdict": "留" + }, + "r4c_conv_batchmsg": { + "dec_med": 67.27773522180502, + "dec_all": [ + 67.27773522180502, + 64.80468996476651, + 67.576769573957 + ], + "ttft_med": 0.21914458274841309, + "peak": 18.161532996, + "paired": [ + 1.516376817631209, + 1.5187029693126763, + 1.4963789746016443 + ], + "same_output": true, + "err": null, + "doc": "第 4 轮 c_conv_batchmsg:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True]。", + "speedup": 1.516376817631209, + "verdict": "留" + } + } +} \ No newline at end of file diff --git a/examples/inkling2_mini_mlx/rounds/r4/score-holdout.json b/examples/inkling2_mini_mlx/rounds/r4/score-holdout.json new file mode 100644 index 00000000..6df13d2a --- /dev/null +++ b/examples/inkling2_mini_mlx/rounds/r4/score-holdout.json @@ -0,0 +1,82 @@ +{ + "baseline": { + "dec_med": 45.04490632221295, + "dec_all": [ + 43.06871846686166, + 45.5252649434873, + 43.472232258552545, + 44.16047072578763, + 45.06100727641791, + 45.02880536800799, + 43.09327211093362, + 46.615799686407044, + 46.225955495267954, + 47.22899428740365 + ], + "ttft_med": 0.20153599977493286, + "peak": 18.105733216 + }, + "noise_range": 0.09235840764731344, + "cands": { + "r4a_conv_kernel": { + "dec_med": 66.5639386058893, + "dec_all": [ + 66.5639386058893, + 64.83010099014217, + 66.91319649713606 + ], + "ttft_med": 0.192732572555542, + "peak": 18.048849136, + "paired": [ + 1.4621318225938569, + 1.4912991034038232, + 1.5152283342410546 + ], + "same_output": true, + "err": null, + "doc": "第 4 轮 a_conv_kernel:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True]。", + "speedup": 1.4912991034038232, + "verdict": "留" + }, + "r4b_ref_r3e": { + "dec_med": 63.21071599250109, + "dec_all": [ + 63.48963303775635, + 60.665830819532324, + 63.21071599250109 + ], + "ttft_med": 0.19359838962554932, + "peak": 18.049061552, + "paired": [ + 1.4089705684626987, + 1.3472671620694185, + 1.4668349117184647 + ], + "same_output": true, + "err": null, + "doc": "第 4 轮 b_ref_r3e:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True]。", + "speedup": 1.4089705684626987, + "verdict": "留" + }, + "r4c_conv_batchmsg": { + "dec_med": 70.28057320251098, + "dec_all": [ + 70.28057320251098, + 69.37463871972089, + 70.67514817849448 + ], + "ttft_med": 0.18816149234771729, + "peak": 18.11250072, + "paired": [ + 1.507655637687247, + 1.5007724118719972, + 1.4964355952280801 + ], + "same_output": true, + "err": null, + "doc": "第 4 轮 c_conv_batchmsg:第 3 轮三段解码路径,选项 [mask_kernel=True, fuse_proj=True, fuse_all6=True, fuse_msg=True, conv_kernel=True, batch_msg=True]。", + "speedup": 1.5007724118719972, + "verdict": "留" + } + } +} \ No newline at end of file diff --git a/examples/inkling2_mini_mlx/route/llama-bench-summary.txt b/examples/inkling2_mini_mlx/route/llama-bench-summary.txt new file mode 100644 index 00000000..60cdbc15 --- /dev/null +++ b/examples/inkling2_mini_mlx/route/llama-bench-summary.txt @@ -0,0 +1,3 @@ +512 0 997.58 tok/s +0 128 41.65 tok/s + 459666632 peak memory footprint diff --git a/examples/inkling2_mini_mlx/route/llama-bench.json b/examples/inkling2_mini_mlx/route/llama-bench.json new file mode 100644 index 00000000..777551ef --- /dev/null +++ b/examples/inkling2_mini_mlx/route/llama-bench.json @@ -0,0 +1,94 @@ +[ + { + "build_commit": "418fe97", + "build_number": 1, + "cpu_info": "Accelerate, Apple M5 Max", + "gpu_info": "Apple M5 Max", + "backends": "MTL,BLAS", + "model_filename": "/lmstudio-community/Inkling-2-Mini-GGUF/Inkling-2-Mini-Q4_K_M.gguf", + "model_type": "inkling2_loop ?B Q4_K - Medium", + "model_size": 17466116736, + "model_n_params": 26070385152, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 6, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "repack": true, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-11-21T06:20:11Z", + "avg_ns": 513241972, + "stddev_ns": 426397, + "avg_ts": 997.580602, + "stddev_ts": 0.828454, + "samples_ns": [ 513724666, 513084708, 512916542 ], + "samples_ts": [ 996.643, 997.886, 998.213 ] + }, + { + "build_commit": "418fe97", + "build_number": 1, + "cpu_info": "Accelerate, Apple M5 Max", + "gpu_info": "Apple M5 Max", + "backends": "MTL,BLAS", + "model_filename": "/lmstudio-community/Inkling-2-Mini-GGUF/Inkling-2-Mini-Q4_K_M.gguf", + "model_type": "inkling2_loop ?B Q4_K - Medium", + "model_size": 17466116736, + "model_n_params": 26070385152, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 6, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "repack": true, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-11-21T06:20:14Z", + "avg_ns": 3073455430, + "stddev_ns": 6380650, + "avg_ts": 41.647055, + "stddev_ts": 0.086542, + "samples_ns": [ 3066224375, 3075850125, 3078291792 ], + "samples_ts": [ 41.7452, 41.6145, 41.5815 ] + } +] diff --git a/examples/inkling2_mini_mlx/route/llama-chat.jsonl b/examples/inkling2_mini_mlx/route/llama-chat.jsonl new file mode 100644 index 00000000..a8e5420c --- /dev/null +++ b/examples/inkling2_mini_mlx/route/llama-chat.jsonl @@ -0,0 +1,3 @@ +{"ttft_s": 0.3104572296142578, "wall_s": 12.131134986877441, "timings": {"cache_n": 0, "prompt_n": 61, "prompt_ms": 253.471, "prompt_per_token_ms": 4.155262295081967, "prompt_per_second": 240.65869468302094, "predicted_n": 512, "predicted_ms": 11870.061, "predicted_per_token_ms": 23.229082191780822, "predicted_per_second": 43.04948390745423}, "text": "", "reasoning": "The user asks in Chinese: write a thread-safe LRU cache class in Python supporting get/put/capacity limit and TTL, with 3 pytest cases. Only output code. Need provide code only, no explanation. Need ensure correctness.\n\nDesign: use OrderedDict for LRU, Lock (RLock), time.time. put(key,value,ttl=None): if ttl is None use default_ttl? Class init capacity, default_ttl maybe. get(key): check expiry, move to end, return value or None. put: if key exists delete then insert end; if len>capacity popitem(last=False). TTL: store (value, expire_at). get if expire_at <= now: delete and return None. Also maybe put with ttl=0? expire immediately. Need thread-safe.\n\nTests: 3 pytest cases: capacity eviction LRU order; TTL expiry; thread safety maybe concurrent puts/gets no exceptions and size <= capacity. Need only code. Include imports. Use pytest.\n\nNeed be careful: only output code, no markdown? User says 只输出代码. So final should be code only, no prose. Could include code block? \"只输出代码\" likely no explanation; code block is okay but maybe raw code safer. Provide raw Python code.\n\nImplementation details:\nfrom collections import OrderedDict\nimport threading, time\nimport pytest\n\nclass LRUCache:\n def __init__(self, capacity, default_ttl=None):\n if capacity <=0: raise ValueError\n self.capacity=capacity\n self.default_ttl=default_ttl\n self._data=OrderedDict()\n self._lock=threading.RLock()\n def get(self,key,default=None):\n with self._lock:\n item=self._data.get(key)\n if item is None: return default\n value, expires_at = item\n if expires_at is not None and expires_at <= time.monotonic():\n del self._data[key]\n return default\n self._data.move_to_end(key)\n return value\n"} +{"ttft_s": 0.29821109771728516, "wall_s": 12.032907009124756, "timings": {"cache_n": 0, "prompt_n": 61, "prompt_ms": 237.324, "prompt_per_token_ms": 3.8905573770491806, "prompt_per_second": 257.0325799329187, "predicted_n": 512, "predicted_ms": 11784.265, "predicted_per_token_ms": 23.061183953033268, "predicted_per_second": 43.36290808124224}, "text": "", "reasoning": "The user asks in Chinese: write a thread-safe LRU cache class in Python supporting get/put/capacity limit and TTL, with 3 pytest cases. Only output code. Need provide code only, no explanation. Need ensure correctness.\n\nDesign: use OrderedDict for LRU, Lock (RLock), time.time. put(key,value,ttl=None): if ttl is None use default_ttl? Class init capacity, default_ttl maybe. get(key): check expiry, move to end, return value or None. put: if key exists delete then insert end; if len>capacity popitem(last=False). TTL: store (value, expire_at). get if expire_at <= now: delete and return None. Also maybe put with ttl=0? expire immediately. Need thread-safe.\n\nTests: 3 pytest cases: capacity eviction LRU order; TTL expiry; thread safety maybe concurrent puts/gets no exceptions and size <= capacity. Need only code. Include imports. Use pytest.\n\nNeed be careful: only output code, no markdown? User says 只输出代码. So final should be code only, no prose. Could include code block? \"只输出代码\" likely no explanation; code block is okay but maybe raw code safer. Provide raw Python code.\n\nImplementation details:\nfrom collections import OrderedDict\nimport threading, time\nimport pytest\n\nclass LRUCache:\n def __init__(self, capacity, default_ttl=None):\n if capacity <=0: raise ValueError\n self.capacity=capacity\n self.default_ttl=default_ttl\n self._data=OrderedDict()\n self._lock=threading.RLock()\n def get(self,key,default=None):\n with self._lock:\n item=self._data.get(key)\n if item is None: return default\n value, expires_at = item\n if expires_at is not None and expires_at <= time.monotonic():\n del self._data[key]\n return default\n self._data.move_to_end(key)\n return value\n"} +{"ttft_s": 0.29129791259765625, "wall_s": 12.558490991592407, "timings": {"cache_n": 0, "prompt_n": 61, "prompt_ms": 236.752, "prompt_per_token_ms": 3.8811803278688526, "prompt_per_second": 257.65357842805975, "predicted_n": 512, "predicted_ms": 12316.508, "predicted_per_token_ms": 24.102755381604695, "predicted_per_second": 41.48903244328668}, "text": "", "reasoning": "The user asks in Chinese: write a thread-safe LRU cache class in Python supporting get/put/capacity limit and TTL, with 3 pytest cases. Only output code. Need provide code only, no explanation. Need ensure correctness.\n\nDesign: use OrderedDict for LRU, Lock (RLock), time.time. put(key,value,ttl=None): if ttl is None use default_ttl? Class init capacity, default_ttl maybe. get(key): check expiry, move to end, return value or None. put: if key exists delete then insert end; if len>capacity popitem(last=False). TTL: store (value, expire_at). get if expire_at <= now: delete and return None. Also maybe put with ttl=0? expire immediately. Need thread-safe.\n\nTests: 3 pytest cases: capacity eviction LRU order; TTL expiry; thread safety maybe concurrent puts/gets no exceptions and size <= capacity. Need only code. Include imports. Use pytest.\n\nNeed be careful: only output code, no markdown? User says 只输出代码. So final should be code only, no prose. Could include code block? \"只输出代码\" likely no explanation; code block is okay but maybe raw code safer. Provide raw Python code.\n\nImplementation details:\nfrom collections import OrderedDict\nimport threading, time\nimport pytest\n\nclass LRUCache:\n def __init__(self, capacity, default_ttl=None):\n if capacity <=0: raise ValueError\n self.capacity=capacity\n self.default_ttl=default_ttl\n self._data=OrderedDict()\n self._lock=threading.RLock()\n def get(self,key,default=None):\n with self._lock:\n item=self._data.get(key)\n if item is None: return default\n value, expires_at = item\n if expires_at is not None and expires_at <= time.monotonic():\n del self._data[key]\n return default\n self._data.move_to_end(key)\n return value\n"} diff --git a/examples/inkling2_mini_mlx/route/mlx-benchmark.log b/examples/inkling2_mini_mlx/route/mlx-benchmark.log new file mode 100644 index 00000000..94169643 --- /dev/null +++ b/examples/inkling2_mini_mlx/route/mlx-benchmark.log @@ -0,0 +1,7 @@ +[transformers] You are using a model of type `inkling2_loop` to instantiate a model of type ``. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. +Running warmup.. +Timing with prompt_tokens=512, generation_tokens=128, batch_size=1. +Trial 1: prompt_tps=1701.590, generation_tps=47.270, peak_memory=19.044, total_time=3.011 +Trial 2: prompt_tps=1704.370, generation_tps=46.170, peak_memory=19.044, total_time=3.075 +Trial 3: prompt_tps=1625.348, generation_tps=47.467, peak_memory=19.045, total_time=3.014 +Averages: prompt_tps=1677.103, generation_tps=46.969, peak_memory=19.044 diff --git a/examples/inkling2_mini_mlx/route/mlx-chat-output.txt b/examples/inkling2_mini_mlx/route/mlx-chat-output.txt new file mode 100644 index 00000000..7a58887c --- /dev/null +++ b/examples/inkling2_mini_mlx/route/mlx-chat-output.txt @@ -0,0 +1,12 @@ +----- 输出 ----- +<|message_model|><|content_thinking|>The user asks in Chinese: write a thread-safe LRU cache class in Python supporting get/put/capacity limit and TTL expiration, with 3 pytest cases. Only output code. Need provide code only. Need ensure correctness. + +Design: class LRUCache with OrderedDict, Lock. get(key): with lock, if key in cache: move_to_end, check expiry: if expired delete and return None (or default). else return value. put(key,value,ttl=None): with lock, if key exists delete; set expiry = time.time()+ttl if ttl else None; cache[key]=(value,expiry); move_to_end; while len>capacity: popitem(last=False). Need capacity maybe maxsize. TTL default from constructor? Support per-put ttl and default ttl. get returns default maybe None. Thread-safe. + +Tests: 3 pytest cases: capacity eviction LRU order; TTL expiration (use small ttl and monkeypatch time or sleep? Better avoid sleep by injecting clock? Could use time.sleep(0.05) with ttl=0.01; acceptable but flaky. Better design class accepts time_func for tests? But user asked class; can include optional clock param. Tests can use fake clock. Simpler: class uses time.time; tests use ttl=0.05 and time.sleep(0.06). 3 cases: 1) LRU eviction order; 2) TTL expiry; 3) thread safety? Need 3 pytest cases. Could do: get/put basic, capacity eviction, TTL expiry. Thread safety test maybe with threads incrementing? But only 3 cases. Include thread-safety in one? Need 3 pytest用例. We can write 3 tests: test_lru_eviction, test_ttl_expiry, test_thread_safety. Thread safety test: create cache capacity large, spawn threads putting/getting, assert no exception and final size <= capacity. Deterministic enough. + +Need only output code, no explanation. Ensure code block only? User says 只输出代码. So final should be code only, likely a single code block. No prose. + +Need be careful with OrderedDict move_to_end and expiry. Implementation: + + diff --git a/examples/inkling2_mini_mlx/route/mlx-chat.jsonl b/examples/inkling2_mini_mlx/route/mlx-chat.jsonl new file mode 100644 index 00000000..99ff0da6 --- /dev/null +++ b/examples/inkling2_mini_mlx/route/mlx-chat.jsonl @@ -0,0 +1,3 @@ +{"run": 0, "load_s": 2.161633014678955, "ttft_s": 2.1704137325286865, "prompt_tokens": 61, "prefill_tps": 29.1318650445503, "gen_tokens": 512, "decode_tps": 47.640906850511996, "peak_gb": 17.559900508, "wall_s": 12.918859958648682, "kw": {}} +{"run": 1, "load_s": 2.161633014678955, "ttft_s": 0.245164155960083, "prompt_tokens": 61, "prefill_tps": 249.52940491322087, "gen_tokens": 512, "decode_tps": 45.252409289809215, "peak_gb": 17.56069738, "wall_s": 11.560818910598755, "kw": {}} +{"run": 2, "load_s": 2.161633014678955, "ttft_s": 0.24292707443237305, "prompt_tokens": 61, "prefill_tps": 251.78264390018035, "gen_tokens": 512, "decode_tps": 47.34382293487728, "peak_gb": 17.560628916, "wall_s": 11.058743953704834, "kw": {}} diff --git a/examples/inkling2_mini_mlx/rsi_mlx.py b/examples/inkling2_mini_mlx/rsi_mlx.py new file mode 100644 index 00000000..4dc951d6 --- /dev/null +++ b/examples/inkling2_mini_mlx/rsi_mlx.py @@ -0,0 +1,144 @@ +# One InferRSI-style round on a Mac: load the model once, alternate each candidate with the default path, score by measured decode speed. +# Usage: python rsi_mlx.py --split evolve|holdout --max-tokens N --reps R +# Each /cands/*.py (not starting with "_") is a candidate defining apply(model) -> dict of stream_generate kwargs; +# it may patch the model in place (swap ops, wrap mx.compile) and must define revert(model). +# Score: median paired ratio candidate/adjacent-default decode tok/s. Gate: greedy token ids identical to the default path +# (same length and same ids on every prompt). +# Noise: range of the default path's own runs. Below 5% difference: no conclusion. +# A candidate with an error, fewer than --reps pairs, or a failed revert is INCONCLUSIVE; its error and samples are kept. +# A speed win is only a screen: "speed screen passed, pending mechanism check". The mechanism gate +# (mechanism.py, count_ops2.py) is checked separately and is not read by this script. +import argparse, glob, importlib.util, json, os, statistics, time + +# Code-generation workload. Split validation, as in earlier InferRSI campaigns: +# candidates are selected on "evolve"; the reported number is the "holdout" split. +SPLITS = { + "evolve": [ + "用 Python 写一个线程安全的 LRU 缓存类,支持 get/put/容量上限和过期时间(TTL),附 3 个 pytest 用例。只输出代码。", + "Write a Go HTTP server with a /healthz endpoint and a rate-limited /api/echo endpoint using a token bucket. Code only.", + ], + "holdout": [ + "用 TypeScript 写一个带重试和指数退避的 fetch 封装,支持超时取消,附用法示例。只输出代码。", + "Write a Rust function that parses a CSV line with quoted fields and escaped quotes, with unit tests. Code only.", + ], +} + +def same_ids(run, ref_ids): + """Quality gate: every prompt's token ids equal the default path's, including length.""" + return len(run) == len(ref_ids) and all(x["ids"] == ref for x, ref in zip(run, ref_ids)) + + +def summary(runs): + dec = [statistics.mean(x["dec"] for x in r) for r in runs] + ttft = [statistics.mean(x["ttft"] for x in r) for r in runs] + return dict(dec_med=statistics.median(dec), dec_all=dec, ttft_med=statistics.median(ttft), + peak=max(x["peak"] for r in runs for x in r)) + + +def pair_candidate(mod, model, run_once, base_runs, ref_ids, reps): + """Alternate the default path with the candidate `reps` times. Keeps every sample and the error, if any.""" + runs = []; pairs = []; same = True; err = None; reverted = True + try: + for i in range(reps): + base_runs.append(run_once({})) # default path, alternated with the candidate + reverted = False # apply() may change the model before it raises + kw = mod.apply(model) + r = run_once(kw); runs.append(r) + pairs.append(summary([r])["dec_med"] / summary([base_runs[-1]])["dec_med"]) # paired with the adjacent default run, cancelling thermal drift + mod.revert(model); reverted = True + same = same and same_ids(r, ref_ids) + except Exception as e: + err = repr(e) + if not reverted: + try: + mod.revert(model); reverted = True + except Exception as e2: + err += f"; revert failed: {e2!r}" + s = summary(runs) if runs else {} + return dict(s, paired=pairs, same_output=same, err=err, reverted=reverted, doc=(mod.__doc__ or "").strip()) + + +def run_round(cands, model, run_once, base_runs, ref_ids, reps, emit): + """Pair each (name, module) in turn. If a candidate could not be reverted, the model may no longer be + the default path, so later default runs would be a polluted baseline: stop the round there.""" + results = {} + for name, mod in cands: + results[name] = pair_candidate(mod, model, run_once, base_runs, ref_ids, reps) + emit(dict(cand=name, **results[name])) + if not results[name]["reverted"]: + emit(dict(stopped=f"{name}: revert not confirmed; round stopped, reload the model before continuing")) + break + return results + + +def score(r, reps): + """Verdict for one candidate. Only complete, error-free, reverted runs that pass the token gate are scored.""" + if r["err"] or not r["reverted"] or len(r["paired"]) != reps: + return dict(verdict="INCONCLUSIVE") + speedup = statistics.median(r["paired"]) + if not r["same_output"]: + return dict(speedup=speedup, verdict="output changed: rejected") + if abs(speedup - 1) < 0.05: + return dict(speedup=speedup, verdict="<5%: no conclusion") + return dict(speedup=speedup, verdict="speed screen passed, pending mechanism check" if speedup > 1 else "reject") + + +def main(): + import mlx.core as mx + from mlx_lm import load, stream_generate + from mlx_lm.sample_utils import make_sampler + + ap = argparse.ArgumentParser() + ap.add_argument("model"); ap.add_argument("round_dir") + ap.add_argument("--max-tokens", type=int, required=True) + ap.add_argument("--reps", type=int, required=True) + ap.add_argument("--only", default=None) + ap.add_argument("--split", choices=list(SPLITS), required=True) + a = ap.parse_args() + + model, tok = load(a.model) + sampler = make_sampler(temp=0.0) + prompts = [tok.apply_chat_template([{"role": "user", "content": p}], add_generation_prompt=True, + reasoning_effort="none") for p in SPLITS[a.split]] + + def run_once(kw): + out = [] + for p in prompts: + mx.reset_peak_memory(); t0 = time.time(); ttft = None; ids = []; last = None + for r in stream_generate(model, tok, p, max_tokens=a.max_tokens, sampler=sampler, **kw): + if ttft is None: ttft = time.time() - t0 + ids.append(r.token); last = r + out.append(dict(ttft=ttft, dec=last.generation_tps, pre=last.prompt_tps, n=last.generation_tokens, + peak=mx.get_peak_memory() / 1e9, ids=ids)) + return out + + log = open(os.path.join(a.round_dir, f"log-{a.split}.jsonl"), "a") + def emit(d): + print(json.dumps(d, ensure_ascii=False), flush=True); log.write(json.dumps(d, ensure_ascii=False) + "\n"); log.flush() + + run_once({}) # warm-up: the first pass compiles Metal kernels; not scored + base_runs = [run_once({})] + ref_ids = [x["ids"] for x in base_runs[0]] + cands = sorted(p for p in glob.glob(os.path.join(a.round_dir, "cands", "*.py")) if not os.path.basename(p).startswith("_")) + if a.only: cands = [c for c in cands if a.only in c] + def load_cand(path): + name = os.path.basename(path)[:-3] + spec = importlib.util.spec_from_file_location(name, path); mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return name, mod + results = run_round((load_cand(p) for p in cands), model, run_once, base_runs, ref_ids, a.reps, emit) + + b = summary(base_runs) + noise = (max(b["dec_all"]) - min(b["dec_all"])) / b["dec_med"] + emit(dict(cand="baseline", **b, noise_range=noise)) + for n, r in results.items(): + r.update(score(r, a.reps)) + json.dump(dict(baseline=b, noise_range=noise, cands=results), open(os.path.join(a.round_dir, f"score-{a.split}.json"), "w"), + ensure_ascii=False, indent=1) + for n, r in sorted(results.items(), key=lambda kv: -kv[1].get("dec_med", 0)): + print(f"{n:24s} {r.get('dec_med', 0):7.2f} tok/s x{r.get('speedup', 0):.3f} same_output={r['same_output']} {r.get('verdict')} {r['err'] or ''}") + print(f"baseline {b['dec_med']:.2f} tok/s noise range {noise:.1%}") + + +if __name__ == "__main__": + main() diff --git a/tests/test_inkling2_mlx_round_scoring.py b/tests/test_inkling2_mlx_round_scoring.py new file mode 100644 index 00000000..59e0727c --- /dev/null +++ b/tests/test_inkling2_mlx_round_scoring.py @@ -0,0 +1,106 @@ +"""CPU checks for the Inkling 2 Mini MLX round scorer; no MLX import needed.""" + +import importlib.util +from pathlib import Path + +path = Path(__file__).resolve().parents[1] / "examples/inkling2_mini_mlx/rsi_mlx.py" +spec = importlib.util.spec_from_file_location("inkling2_rsi_mlx", path) +rsi = importlib.util.module_from_spec(spec) +spec.loader.exec_module(rsi) + +REF = [[1, 2, 3], [4, 5]] + + +def sample(dec, ids=REF): + return [dict(ttft=0.1, dec=dec, pre=100.0, n=len(x), peak=1.0, ids=list(x)) for x in ids] + + +class Cand: + """Candidate whose second pass raises; records apply/revert calls.""" + + def __init__(self, fail_on=None, cand_ids=REF): + self.fail_on, self.cand_ids, self.applied, self.reverted = fail_on, cand_ids, 0, 0 + + def apply(self, model): + self.applied += 1 + return {"cand": True} + + def revert(self, model): + self.reverted += 1 + + +def runner(cand): + def run_once(kw): + if not kw: + return sample(40.0) + if cand.applied == cand.fail_on: + raise RuntimeError("Metal command buffer aborted") + return sample(44.0, cand.cand_ids) # 10% faster than the default path + + return run_once + + +def test_first_pair_ok_second_pair_raises_is_inconclusive_and_keeps_samples(): + cand = Cand(fail_on=2) + r = rsi.pair_candidate(cand, None, runner(cand), [sample(40.0)], REF, reps=3) + assert r["err"] and "Metal command buffer aborted" in r["err"] + assert r["paired"] == [1.1] and r["dec_all"] == [44.0] + assert r["reverted"] and cand.reverted == cand.applied == 2 + assert rsi.score(r, reps=3) == {"verdict": "INCONCLUSIVE"} + + +def test_complete_faster_identical_run_is_only_a_speed_screen(): + cand = Cand() + r = rsi.pair_candidate(cand, None, runner(cand), [sample(40.0)], REF, reps=3) + assert r["err"] is None and len(r["paired"]) == 3 and r["same_output"] + s = rsi.score(r, reps=3) + assert s["verdict"] == "speed screen passed, pending mechanism check" + assert "keep" not in s["verdict"] + + +def test_shorter_output_with_equal_prefix_fails_token_gate(): + cand = Cand(cand_ids=[[1, 2], [4, 5]]) + r = rsi.pair_candidate(cand, None, runner(cand), [sample(40.0)], REF, reps=3) + assert not r["same_output"] + assert rsi.score(r, reps=3)["verdict"] == "output changed: rejected" + + +class ApplyRaises(Cand): + """apply() changes the model, then raises.""" + + def apply(self, model): + self.applied += 1 + raise RuntimeError("apply half done") + + +class RevertFails(Cand): + def revert(self, model): + self.reverted += 1 + raise RuntimeError("revert failed") + + +def test_apply_that_raises_after_changing_the_model_is_reverted(): + cand = ApplyRaises() + r = rsi.pair_candidate(cand, None, runner(cand), [sample(40.0)], REF, reps=3) + assert "apply half done" in r["err"] + assert cand.reverted == 1 and r["reverted"] + assert rsi.score(r, reps=3) == {"verdict": "INCONCLUSIVE"} + + +def test_unconfirmed_revert_stops_the_round(): + bad, good = RevertFails(), Cand() + calls = [] + + def run_once(kw): + calls.append(kw) + return sample(44.0 if kw else 40.0) + + events = [] + results = rsi.run_round( + [("bad", bad), ("good", good)], None, run_once, [sample(40.0)], REF, 3, events.append + ) + assert list(results) == ["bad"] and not results["bad"]["reverted"] + assert "revert failed" in results["bad"]["err"] + assert good.applied == 0 and len(calls) == 2 # no default run after the failed revert + assert "round stopped" in events[-1]["stopped"] + assert rsi.score(results["bad"], reps=3) == {"verdict": "INCONCLUSIVE"}