From e69ff58d0f7dc4dc3f00fb4c6d9e810849e1c919 Mon Sep 17 00:00:00 2001 From: bjf-frz Date: Fri, 20 Nov 2026 10:03:34 +0000 Subject: [PATCH 1/6] docs: clarify model platform support and Ascend setup --- README.md | 33 ++++++++++++++++++++------------- docs/README.md | 9 +++++---- docs/launching.md | 3 +++ docs/recipes/ouro_ascend.md | 7 +++++-- docs/serving.md | 12 ++++++++++-- 5 files changed, 43 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index b0ffc7bc..b7230883 100644 --- a/README.md +++ b/README.md @@ -104,8 +104,15 @@ in mixed prefill/decode batches. ## Getting Started -Requires **Python 3.10+** and **PyTorch 2.5+**. For GPU inference, use Linux and -an NVIDIA GPU with a compatible CUDA-enabled PyTorch installation. +Requires **Python 3.10+** and **PyTorch 2.5+**. Choose the setup for your device. + +| Platform | Setup and support | +| --- | --- | +| CPU | Torch reference execution. See [CPU serving options](docs/serving.md#device-setup). | +| NVIDIA CUDA | Linux, an NVIDIA GPU, and compatible CUDA-enabled PyTorch. Follow the [first-run guide](docs/launching.md). | +| Huawei Ascend NPU | Ouro-1.4B on one NPU, with BF16, four fixed loops, and greedy decoding. Use `main` with compatible Torch, Torch-NPU, and CANN. Support landed after `v0.2.0`. Follow the [Ascend setup](docs/recipes/ouro_ascend.md). | + +These CPU and CUDA installation commands pin `v0.2.0`. ```bash # Activate your Python environment first. @@ -141,18 +148,18 @@ a fixed scale at startup. See the ## Model Support -| Checkpoint | Implementation | Documented validation scope | Guide | -| --- | --- | --- | --- | -| Ouro-1.4B | Native Ouro | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | [Runtime](docs/runtime.md), [Ascend recipe](docs/recipes/ouro_ascend.md#6-fixed-depth-asynchronous-engine-plan) | -| Ouro-2.6B | Shared Ouro | Shares the Ouro implementation; not separately qualified, and 1.4B quality/performance results are not inherited | [Scope notes](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) | -| Ouro-1.4B-Thinking | Reuses Ouro | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | -| Ouro-2.6B-Thinking | Reuses Ouro | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | -| Nanbeige4.2-3B | Native Nanbeige | Fixed two-loop inference | [Runtime](docs/runtime.md) | -| Huginn-0125 | Native Huginn | Synchronous fixed-depth, LAST_EXITED KV, CUDA Graph decode core and coda | [Runtime](docs/runtime.md) | -| NVIDIA-Nemotron-Labs-Dory1.0-4B | Native Dory | Synchronous fixed-loop execution, boundary KV, per-loop and last-loop KV modes, Triton/Torch sliding-window masking, BF16 real-checkpoint agreement at 1/2/4 loops on A100; long context, Graphs/async and reclamation not qualified | [Dory baseline](docs/dory.md) | +| Checkpoint | Implementation | Platforms | Documented validation scope | Guide | +| --- | --- | --- | --- | --- | +| Ouro-1.4B | Native Ouro | CPU, NVIDIA CUDA, Huawei Ascend NPU (`main`) | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | [Runtime](docs/runtime.md), [Ascend recipe](docs/recipes/ouro_ascend.md) | +| Ouro-2.6B | Shared Ouro | CPU and NVIDIA CUDA via the shared runtime | Shares the Ouro implementation; not separately qualified, and 1.4B quality/performance results are not inherited | [Scope notes](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) | +| Ouro-1.4B-Thinking | Reuses Ouro | CPU (reference), NVIDIA CUDA | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | +| Ouro-2.6B-Thinking | Reuses Ouro | CPU (reference), NVIDIA CUDA | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | +| Nanbeige4.2-3B | Native Nanbeige | CPU, NVIDIA CUDA | Fixed two-loop inference | [Runtime](docs/runtime.md), [CUDA PD validation](https://github.com/InferMatrixLab/vllm-rlt/pull/77) | +| Huginn-0125 | Native Huginn | CPU, NVIDIA CUDA | Synchronous fixed-depth, LAST_EXITED KV, CUDA Graph decode core and coda | [Runtime](docs/runtime.md) | +| NVIDIA-Nemotron-Labs-Dory1.0-4B | Native Dory | CPU, NVIDIA CUDA | Synchronous fixed-loop execution, boundary KV, per-loop and last-loop KV modes, Triton/Torch sliding-window masking, BF16 real-checkpoint agreement at 1/2/4 loops on A100; long context, Graphs/async and reclamation not qualified | [Dory baseline](docs/dory.md) | -Shared implementation does not qualify every checkpoint/feature combination; -see the linked guides for validation scope and limitations. See +CPU entries refer to the Torch reference path. See the linked guides for each +checkpoint's validation scope and feature limits. See [RFC #69](https://github.com/InferMatrixLab/vllm-rlt/issues/69) for the broader roadmap. Huginn uses native checkpoint loading and depth-aware KV for its recurrent diff --git a/docs/README.md b/docs/README.md index 47adbd1d..1b933288 100644 --- a/docs/README.md +++ b/docs/README.md @@ -5,9 +5,9 @@ execution; optional GPU paths and historical measurements have narrower qualification scopes than the feature list. See the [changelog and migration notes](../CHANGELOG.md) before upgrading from v0.1.0. -New users should follow the [first-run user guide](launching.md), from creating -an environment to receiving the first generated response. The -[README](../README.md#getting-started) provides a brief installation reference. +The [Getting Started section](../README.md#getting-started) lists the platform +options. The [first-run user guide](launching.md) walks through the NVIDIA CUDA +setup on `v0.2.0`. The Ascend guide below uses `main`. | Guide | Topics | | --- | --- | @@ -26,7 +26,8 @@ an environment to receiving the first generated response. The | [FlashAttention](https://github.com/InferMatrixLab/vllm-rlt/pull/30) | FA2/FA3/FA4 installation, hardware selection, and constraints | | [Cache and scheduling features](https://github.com/InferMatrixLab/vllm-rlt/pull/31) | Prefix reuse, incremental KV, priorities, and preemption | | [Prefill/decode disaggregation](https://github.com/InferMatrixLab/vllm-rlt/pull/31) | Single-host GPU worker pools and NIXL transfer | -| [HTTP serving](serving.md) | Completions API, streaming, and service lifecycle | +| [HTTP serving](serving.md) | Device setup, completions API, streaming, and service lifecycle | +| [Ouro on Ascend](recipes/ouro_ascend.md) | Huawei Ascend NPU setup on `main`, serving commands, and validation scope | | [Profiling](profiling.md) | Core profiler options, Python/HTTP controls, PD collection, and per-rank archives | | [Ouro-Thinking recipe](recipes/ouro_thinking.md) | Shared 1.4B/2.6B setup, JSON/SSE client, budgets and validation limits | | [Accuracy evaluation](accuracy.md) | GSM8K regression setup and comparison methodology | diff --git a/docs/launching.md b/docs/launching.md index 604800ca..de6f208c 100644 --- a/docs/launching.md +++ b/docs/launching.md @@ -10,6 +10,9 @@ For migration from v0.1.0, read the [changelog](../CHANGELOG.md); Python scripts now construct a spawned stage worker and need the guarded example below. The release is for research/evaluation, not a production deployment guarantee. +For Huawei Ascend NPU, follow the [Ascend setup](recipes/ouro_ascend.md) on `main`. +For CPU reference execution, see [device setup](serving.md#device-setup). + ## 1. Check the machine Use Linux with Git, curl, [uv](https://docs.astral.sh/uv/getting-started/installation/), diff --git a/docs/recipes/ouro_ascend.md b/docs/recipes/ouro_ascend.md index 82c3887a..a2773ff5 100644 --- a/docs/recipes/ouro_ascend.md +++ b/docs/recipes/ouro_ascend.md @@ -8,10 +8,13 @@ The example enables the optional asynchronous engine plan. ## Prepare the environment and model -Install the project dependencies in a compatible Torch, Torch-NPU and CANN -environment. Use the checkpoint's config and byte-level tokenizer. +Use `main`, which includes Ascend support added after `v0.2.0`. Install the +project dependencies in a compatible Torch, Torch-NPU and CANN environment. +Use the checkpoint's config and byte-level tokenizer. ```bash +git clone --branch main https://github.com/InferMatrixLab/vllm-rlt.git +cd vllm-rlt source /path/to/cann/set_env.sh source /path/to/npu-environment/bin/activate python -m pip install --no-deps --no-build-isolation -e . diff --git a/docs/serving.md b/docs/serving.md index 81cb6999..ee061e6a 100644 --- a/docs/serving.md +++ b/docs/serving.md @@ -5,9 +5,14 @@ environment. The `vllm-rlt-serve` entrypoint (or `python -m vllm_rlt.entrypoints.serve`) serves one resident model on one device. Bind defaults to `127.0.0.1:8000`; this benchmark frontend has no authentication or TLS. +## Device setup + +For Huawei Ascend NPU on `main`, follow the [Ascend serving setup](recipes/ouro_ascend.md) +for the Torch-NPU and CANN environment and the launch command. + Prepare the Ouro checkpoint and tokenizer separately. The official model ID pins revision `574fa66cb8bf5abdc979642d01cf2b79b16bfab1`; local paths are supported. -Select an available GPU explicitly; a scheduler is optional: +For NVIDIA CUDA, select an available GPU explicitly. ```bash nvidia-smi @@ -18,12 +23,15 @@ CUDA_VISIBLE_DEVICES= \ --num-blocks 512 --max-num-seqs 8 ``` +For CPU reference execution, use `--device cpu --attention-backend torch` in the +command above and omit `CUDA_VISIBLE_DEVICES`. + BF16 weights, ordinary activations and KV are this frontend's primary target, as described in the [precision policy](precision-policy.md). The current model uses FP32 RMSNorm/RoPE intermediates and gate/sampling probabilities; Triton attention uses FP32 internal accumulation. Record the matrix multiplication backend/reduction flags separately. `--dtype float32` -is available for diagnostics. CPU execution requires `--attention-backend torch`. +is available for diagnostics. `GET /health` returns 503 during initialization, shutdown or engine failure, and 200 only after model, tokenizer and engine initialization finish. Tokenization From ecc3557bfbb745a503d3d1636676efc5cb2b0d36 Mon Sep 17 00:00:00 2001 From: bjf-frz Date: Fri, 20 Nov 2026 10:11:58 +0000 Subject: [PATCH 2/6] docs: specify the Ascend PyTorch and Torch-NPU version pair --- docs/recipes/ouro_ascend.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/recipes/ouro_ascend.md b/docs/recipes/ouro_ascend.md index a2773ff5..b379de48 100644 --- a/docs/recipes/ouro_ascend.md +++ b/docs/recipes/ouro_ascend.md @@ -8,9 +8,14 @@ The example enables the optional asynchronous engine plan. ## Prepare the environment and model -Use `main`, which includes Ascend support added after `v0.2.0`. Install the -project dependencies in a compatible Torch, Torch-NPU and CANN environment. -Use the checkpoint's config and byte-level tokenizer. +Use `main`, which includes Ascend support added after `v0.2.0`. Match the PyTorch +package (`torch`) and the Ascend extension package (`torch-npu`, imported as +`torch_npu`) to your CANN version using the +[Ascend compatibility table](https://github.com/Ascend/pytorch/blob/master/COMPATIBILITY.en.md#torchnpu-version-mapping-table). +For example, CANN 9.0.0 pairs `torch==2.10.0` with `torch-npu==2.10.0`. + +Install the project dependencies in that environment. Use the checkpoint's +config and byte-level tokenizer. ```bash git clone --branch main https://github.com/InferMatrixLab/vllm-rlt.git From d25c61ea3014be5069a5cd1a8eb10c8e978734f7 Mon Sep 17 00:00:00 2001 From: bjf-frz Date: Fri, 20 Nov 2026 10:14:53 +0000 Subject: [PATCH 3/6] docs: simplify platform links in Getting Started --- README.md | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index b7230883..abf88570 100644 --- a/README.md +++ b/README.md @@ -104,15 +104,11 @@ in mixed prefill/decode batches. ## Getting Started -Requires **Python 3.10+** and **PyTorch 2.5+**. Choose the setup for your device. +Requires **Python 3.10+** and **PyTorch 2.5+**. The commands below install +`v0.2.0` for CPU or NVIDIA CUDA. CUDA inference requires Linux, an NVIDIA GPU, +and compatible CUDA-enabled PyTorch. -| Platform | Setup and support | -| --- | --- | -| CPU | Torch reference execution. See [CPU serving options](docs/serving.md#device-setup). | -| NVIDIA CUDA | Linux, an NVIDIA GPU, and compatible CUDA-enabled PyTorch. Follow the [first-run guide](docs/launching.md). | -| Huawei Ascend NPU | Ouro-1.4B on one NPU, with BF16, four fixed loops, and greedy decoding. Use `main` with compatible Torch, Torch-NPU, and CANN. Support landed after `v0.2.0`. Follow the [Ascend setup](docs/recipes/ouro_ascend.md). | - -These CPU and CUDA installation commands pin `v0.2.0`. +For Ascend NPU on `main`, follow the [Ascend setup guide](docs/recipes/ouro_ascend.md). ```bash # Activate your Python environment first. @@ -127,10 +123,9 @@ builds use the Torch activation fallback; optional native CUDA compilation requires a compatible toolchain. Model weights are downloaded separately. No PyPI publication is implied. -- [Installation and first request](docs/launching.md) -- [Python API and CLI](docs/launching.md) -- [HTTP serving and streaming](docs/serving.md) — use a trusted network or secure - gateway; the server has no built-in authentication or TLS. +- [Installation, first request, Python API, and CLI](docs/launching.md) +- [HTTP serving and streaming](docs/serving.md), including [CPU setup](docs/serving.md#device-setup). + Use a trusted network or secure gateway. The server has no built-in authentication or TLS. - [Supported models](#model-support) and [Ouro-Thinking recipe](docs/recipes/ouro_thinking.md) - [Configuration and compatibility](docs/runtime.md) - [Native CUDA build](docs/launching.md#automatic-native-cuda-build) @@ -147,7 +142,6 @@ a fixed scale at startup. See the ## Model Support - | Checkpoint | Implementation | Platforms | Documented validation scope | Guide | | --- | --- | --- | --- | --- | | Ouro-1.4B | Native Ouro | CPU, NVIDIA CUDA, Huawei Ascend NPU (`main`) | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | [Runtime](docs/runtime.md), [Ascend recipe](docs/recipes/ouro_ascend.md) | From 703eadde24c559348efe37f471bd20d34ba32337 Mon Sep 17 00:00:00 2001 From: bjf-frz Date: Fri, 20 Nov 2026 10:18:26 +0000 Subject: [PATCH 4/6] docs: format platform entries and show Torch-NPU installation --- README.md | 16 +++++++-------- docs/recipes/ouro_ascend.md | 41 +++++++++++++++++++++++++++++++------ 2 files changed, 43 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index abf88570..b85c0d02 100644 --- a/README.md +++ b/README.md @@ -108,7 +108,7 @@ Requires **Python 3.10+** and **PyTorch 2.5+**. The commands below install `v0.2.0` for CPU or NVIDIA CUDA. CUDA inference requires Linux, an NVIDIA GPU, and compatible CUDA-enabled PyTorch. -For Ascend NPU on `main`, follow the [Ascend setup guide](docs/recipes/ouro_ascend.md). +For Ascend NPU on `main`, follow the [PyTorch and Torch-NPU installation steps](docs/recipes/ouro_ascend.md#install-pytorch-and-torch-npu). ```bash # Activate your Python environment first. @@ -144,13 +144,13 @@ a fixed scale at startup. See the | Checkpoint | Implementation | Platforms | Documented validation scope | Guide | | --- | --- | --- | --- | --- | -| Ouro-1.4B | Native Ouro | CPU, NVIDIA CUDA, Huawei Ascend NPU (`main`) | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | [Runtime](docs/runtime.md), [Ascend recipe](docs/recipes/ouro_ascend.md) | -| Ouro-2.6B | Shared Ouro | CPU and NVIDIA CUDA via the shared runtime | Shares the Ouro implementation; not separately qualified, and 1.4B quality/performance results are not inherited | [Scope notes](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) | -| Ouro-1.4B-Thinking | Reuses Ouro | CPU (reference), NVIDIA CUDA | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | -| Ouro-2.6B-Thinking | Reuses Ouro | CPU (reference), NVIDIA CUDA | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | -| Nanbeige4.2-3B | Native Nanbeige | CPU, NVIDIA CUDA | Fixed two-loop inference | [Runtime](docs/runtime.md), [CUDA PD validation](https://github.com/InferMatrixLab/vllm-rlt/pull/77) | -| Huginn-0125 | Native Huginn | CPU, NVIDIA CUDA | Synchronous fixed-depth, LAST_EXITED KV, CUDA Graph decode core and coda | [Runtime](docs/runtime.md) | -| NVIDIA-Nemotron-Labs-Dory1.0-4B | Native Dory | CPU, NVIDIA CUDA | Synchronous fixed-loop execution, boundary KV, per-loop and last-loop KV modes, Triton/Torch sliding-window masking, BF16 real-checkpoint agreement at 1/2/4 loops on A100; long context, Graphs/async and reclamation not qualified | [Dory baseline](docs/dory.md) | +| Ouro-1.4B | Native Ouro | CPU
NVIDIA CUDA
Huawei Ascend NPU (`main`) | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | [Runtime](docs/runtime.md), [Ascend recipe](docs/recipes/ouro_ascend.md) | +| Ouro-2.6B | Shared Ouro | CPU
NVIDIA CUDA | Shares the Ouro implementation; not separately qualified, and 1.4B quality/performance results are not inherited | [Scope notes](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) | +| Ouro-1.4B-Thinking | Reuses Ouro | CPU
NVIDIA CUDA | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | +| Ouro-2.6B-Thinking | Reuses Ouro | CPU
NVIDIA CUDA | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | +| Nanbeige4.2-3B | Native Nanbeige | CPU
NVIDIA CUDA | Fixed two-loop inference | [Runtime](docs/runtime.md), [CUDA PD validation](https://github.com/InferMatrixLab/vllm-rlt/pull/77) | +| Huginn-0125 | Native Huginn | CPU
NVIDIA CUDA | Synchronous fixed-depth, LAST_EXITED KV, CUDA Graph decode core and coda | [Runtime](docs/runtime.md) | +| NVIDIA-Nemotron-Labs-Dory1.0-4B | Native Dory | CPU
NVIDIA CUDA | Synchronous fixed-loop execution, boundary KV, per-loop and last-loop KV modes, Triton/Torch sliding-window masking, BF16 real-checkpoint agreement at 1/2/4 loops on A100; long context, Graphs/async and reclamation not qualified | [Dory baseline](docs/dory.md) | CPU entries refer to the Torch reference path. See the linked guides for each checkpoint's validation scope and feature limits. See diff --git a/docs/recipes/ouro_ascend.md b/docs/recipes/ouro_ascend.md index b379de48..3a14d870 100644 --- a/docs/recipes/ouro_ascend.md +++ b/docs/recipes/ouro_ascend.md @@ -8,11 +8,42 @@ The example enables the optional asynchronous engine plan. ## Prepare the environment and model -Use `main`, which includes Ascend support added after `v0.2.0`. Match the PyTorch -package (`torch`) and the Ascend extension package (`torch-npu`, imported as -`torch_npu`) to your CANN version using the +Use `main`, which includes Ascend support added after `v0.2.0`. Install CANN and +the matching driver and firmware for your Ascend hardware first. + +### Install PyTorch and Torch-NPU + +The following commands use CANN 9.0.0 with PyTorch 2.10.0 and Torch-NPU 2.10.0. +For other versions, consult the [Ascend compatibility table](https://github.com/Ascend/pytorch/blob/master/COMPATIBILITY.en.md#torchnpu-version-mapping-table). -For example, CANN 9.0.0 pairs `torch==2.10.0` with `torch-npu==2.10.0`. +Activate your Python environment and load the CANN environment. + +```bash +source /path/to/npu-environment/bin/activate +source /path/to/cann/set_env.sh +``` + +On x86-64, install PyTorch from the CPU wheel index. + +```bash +python -m pip install 'torch==2.10.0+cpu' --index-url https://download.pytorch.org/whl/cpu +``` + +On Arm64 (`aarch64`), use the following PyTorch command. + +```bash +python -m pip install 'torch==2.10.0' +``` + +On either architecture, install `torch-npu` and check that PyTorch can see the NPU. +The package is imported as `torch_npu` in Python. + +```bash +python -m pip install 'torch-npu==2.10.0' pyyaml setuptools +python -c 'import torch, torch_npu; assert torch.npu.is_available(), "Ascend NPU unavailable"' +``` + +### Install vllm-rlt and set the checkpoint path Install the project dependencies in that environment. Use the checkpoint's config and byte-level tokenizer. @@ -20,8 +51,6 @@ config and byte-level tokenizer. ```bash git clone --branch main https://github.com/InferMatrixLab/vllm-rlt.git cd vllm-rlt -source /path/to/cann/set_env.sh -source /path/to/npu-environment/bin/activate python -m pip install --no-deps --no-build-isolation -e . export MODEL=/path/to/prepared/Ouro-1.4B export ASCEND_RT_VISIBLE_DEVICES=0 # Replace 0 with your assigned NPU. From 364ee4c641f263d7723d9463dd49859e5b46e32f Mon Sep 17 00:00:00 2001 From: bjf-frz Date: Fri, 20 Nov 2026 10:22:18 +0000 Subject: [PATCH 5/6] docs: focus platform guidance on supported accelerators --- README.md | 26 ++++++++++++-------------- docs/launching.md | 1 - docs/serving.md | 5 +---- 3 files changed, 13 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index b85c0d02..eb36f5f6 100644 --- a/README.md +++ b/README.md @@ -105,8 +105,7 @@ in mixed prefill/decode batches. ## Getting Started Requires **Python 3.10+** and **PyTorch 2.5+**. The commands below install -`v0.2.0` for CPU or NVIDIA CUDA. CUDA inference requires Linux, an NVIDIA GPU, -and compatible CUDA-enabled PyTorch. +`v0.2.0` for NVIDIA CUDA on Linux with compatible CUDA-enabled PyTorch. For Ascend NPU on `main`, follow the [PyTorch and Torch-NPU installation steps](docs/recipes/ouro_ascend.md#install-pytorch-and-torch-npu). @@ -124,7 +123,7 @@ requires a compatible toolchain. Model weights are downloaded separately. No PyPI publication is implied. - [Installation, first request, Python API, and CLI](docs/launching.md) -- [HTTP serving and streaming](docs/serving.md), including [CPU setup](docs/serving.md#device-setup). +- [HTTP serving and streaming](docs/serving.md). Use a trusted network or secure gateway. The server has no built-in authentication or TLS. - [Supported models](#model-support) and [Ouro-Thinking recipe](docs/recipes/ouro_thinking.md) - [Configuration and compatibility](docs/runtime.md) @@ -144,17 +143,16 @@ a fixed scale at startup. See the | Checkpoint | Implementation | Platforms | Documented validation scope | Guide | | --- | --- | --- | --- | --- | -| Ouro-1.4B | Native Ouro | CPU
NVIDIA CUDA
Huawei Ascend NPU (`main`) | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | [Runtime](docs/runtime.md), [Ascend recipe](docs/recipes/ouro_ascend.md) | -| Ouro-2.6B | Shared Ouro | CPU
NVIDIA CUDA | Shares the Ouro implementation; not separately qualified, and 1.4B quality/performance results are not inherited | [Scope notes](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) | -| Ouro-1.4B-Thinking | Reuses Ouro | CPU
NVIDIA CUDA | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | -| Ouro-2.6B-Thinking | Reuses Ouro | CPU
NVIDIA CUDA | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | -| Nanbeige4.2-3B | Native Nanbeige | CPU
NVIDIA CUDA | Fixed two-loop inference | [Runtime](docs/runtime.md), [CUDA PD validation](https://github.com/InferMatrixLab/vllm-rlt/pull/77) | -| Huginn-0125 | Native Huginn | CPU
NVIDIA CUDA | Synchronous fixed-depth, LAST_EXITED KV, CUDA Graph decode core and coda | [Runtime](docs/runtime.md) | -| NVIDIA-Nemotron-Labs-Dory1.0-4B | Native Dory | CPU
NVIDIA CUDA | Synchronous fixed-loop execution, boundary KV, per-loop and last-loop KV modes, Triton/Torch sliding-window masking, BF16 real-checkpoint agreement at 1/2/4 loops on A100; long context, Graphs/async and reclamation not qualified | [Dory baseline](docs/dory.md) | - -CPU entries refer to the Torch reference path. See the linked guides for each -checkpoint's validation scope and feature limits. See -[RFC #69](https://github.com/InferMatrixLab/vllm-rlt/issues/69) for the broader roadmap. +| Ouro-1.4B | Native Ouro | NVIDIA CUDA
Huawei Ascend NPU (`main`) | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | [Runtime](docs/runtime.md), [Ascend recipe](docs/recipes/ouro_ascend.md) | +| Ouro-2.6B | Shared Ouro | NVIDIA CUDA | Shares the Ouro implementation; not separately qualified, and 1.4B quality/performance results are not inherited | [Scope notes](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) | +| Ouro-1.4B-Thinking | Reuses Ouro | NVIDIA CUDA | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | +| Ouro-2.6B-Thinking | Reuses Ouro | NVIDIA CUDA | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | +| Nanbeige4.2-3B | Native Nanbeige | NVIDIA CUDA | Fixed two-loop inference | [Runtime](docs/runtime.md), [CUDA PD validation](https://github.com/InferMatrixLab/vllm-rlt/pull/77) | +| Huginn-0125 | Native Huginn | NVIDIA CUDA | Synchronous fixed-depth, LAST_EXITED KV, CUDA Graph decode core and coda | [Runtime](docs/runtime.md) | +| NVIDIA-Nemotron-Labs-Dory1.0-4B | Native Dory | NVIDIA CUDA | Synchronous fixed-loop execution, boundary KV, per-loop and last-loop KV modes, Triton/Torch sliding-window masking, BF16 real-checkpoint agreement at 1/2/4 loops on A100; long context, Graphs/async and reclamation not qualified | [Dory baseline](docs/dory.md) | + +The linked guides cover each checkpoint's validation scope and feature limits. +The broader model roadmap is in [RFC #69](https://github.com/InferMatrixLab/vllm-rlt/issues/69). Huginn uses native checkpoint loading and depth-aware KV for its recurrent core, with separate prelude and coda KV. It supports synchronous fixed-depth diff --git a/docs/launching.md b/docs/launching.md index de6f208c..2aad2973 100644 --- a/docs/launching.md +++ b/docs/launching.md @@ -11,7 +11,6 @@ now construct a spawned stage worker and need the guarded example below. The release is for research/evaluation, not a production deployment guarantee. For Huawei Ascend NPU, follow the [Ascend setup](recipes/ouro_ascend.md) on `main`. -For CPU reference execution, see [device setup](serving.md#device-setup). ## 1. Check the machine diff --git a/docs/serving.md b/docs/serving.md index ee061e6a..9b645a7e 100644 --- a/docs/serving.md +++ b/docs/serving.md @@ -23,15 +23,12 @@ CUDA_VISIBLE_DEVICES= \ --num-blocks 512 --max-num-seqs 8 ``` -For CPU reference execution, use `--device cpu --attention-backend torch` in the -command above and omit `CUDA_VISIBLE_DEVICES`. - BF16 weights, ordinary activations and KV are this frontend's primary target, as described in the [precision policy](precision-policy.md). The current model uses FP32 RMSNorm/RoPE intermediates and gate/sampling probabilities; Triton attention uses FP32 internal accumulation. Record the matrix multiplication backend/reduction flags separately. `--dtype float32` -is available for diagnostics. +is available for diagnostics. CPU execution requires `--attention-backend torch`. `GET /health` returns 503 during initialization, shutdown or engine failure, and 200 only after model, tokenizer and engine initialization finish. Tokenization From c7dbbc0803ad66baf5f693789139e0a1b9205f48 Mon Sep 17 00:00:00 2001 From: bjf-frz Date: Fri, 20 Nov 2026 10:25:08 +0000 Subject: [PATCH 6/6] docs: link compact platform labels in the model table --- README.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index eb36f5f6..cde38840 100644 --- a/README.md +++ b/README.md @@ -141,17 +141,16 @@ a fixed scale at startup. See the ## Model Support -| Checkpoint | Implementation | Platforms | Documented validation scope | Guide | -| --- | --- | --- | --- | --- | -| Ouro-1.4B | Native Ouro | NVIDIA CUDA
Huawei Ascend NPU (`main`) | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | [Runtime](docs/runtime.md), [Ascend recipe](docs/recipes/ouro_ascend.md) | -| Ouro-2.6B | Shared Ouro | NVIDIA CUDA | Shares the Ouro implementation; not separately qualified, and 1.4B quality/performance results are not inherited | [Scope notes](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) | -| Ouro-1.4B-Thinking | Reuses Ouro | NVIDIA CUDA | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | -| Ouro-2.6B-Thinking | Reuses Ouro | NVIDIA CUDA | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | -| Nanbeige4.2-3B | Native Nanbeige | NVIDIA CUDA | Fixed two-loop inference | [Runtime](docs/runtime.md), [CUDA PD validation](https://github.com/InferMatrixLab/vllm-rlt/pull/77) | -| Huginn-0125 | Native Huginn | NVIDIA CUDA | Synchronous fixed-depth, LAST_EXITED KV, CUDA Graph decode core and coda | [Runtime](docs/runtime.md) | -| NVIDIA-Nemotron-Labs-Dory1.0-4B | Native Dory | NVIDIA CUDA | Synchronous fixed-loop execution, boundary KV, per-loop and last-loop KV modes, Triton/Torch sliding-window masking, BF16 real-checkpoint agreement at 1/2/4 loops on A100; long context, Graphs/async and reclamation not qualified | [Dory baseline](docs/dory.md) | - -The linked guides cover each checkpoint's validation scope and feature limits. +| Checkpoint | Implementation | Platforms | Documented validation scope | +| --- | --- | --- | --- | +| Ouro-1.4B | Native Ouro | [CUDA](docs/launching.md)
[Ascend](docs/recipes/ouro_ascend.md) | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | +| Ouro-2.6B | Shared Ouro | [CUDA](docs/launching.md) | Shares the Ouro implementation. [Separate quality and performance qualification](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) remains open. | +| [Ouro-1.4B-Thinking](docs/recipes/ouro_thinking.md) | Reuses Ouro | [CUDA](docs/launching.md) | Bounded BF16, fixed-four-loop synchronous basic integration | +| [Ouro-2.6B-Thinking](docs/recipes/ouro_thinking.md) | Reuses Ouro | [CUDA](docs/launching.md) | Bounded BF16, fixed-four-loop synchronous basic integration | +| Nanbeige4.2-3B | Native Nanbeige | [CUDA](docs/launching.md) | Fixed two-loop inference. [CUDA PD validation](https://github.com/InferMatrixLab/vllm-rlt/pull/77) | +| Huginn-0125 | Native Huginn | [CUDA](docs/launching.md) | Synchronous fixed-depth, LAST_EXITED KV, CUDA Graph decode core and coda | +| [NVIDIA-Nemotron-Labs-Dory1.0-4B](docs/dory.md) | Native Dory | [CUDA](docs/launching.md) | Synchronous fixed-loop execution, boundary KV, per-loop and last-loop KV modes, Triton/Torch sliding-window masking, BF16 real-checkpoint agreement at 1/2/4 loops on A100; long context, Graphs/async and reclamation not qualified | + The broader model roadmap is in [RFC #69](https://github.com/InferMatrixLab/vllm-rlt/issues/69). Huginn uses native checkpoint loading and depth-aware KV for its recurrent