diff --git a/README.md b/README.md index b0ffc7bc..cde38840 100644 --- a/README.md +++ b/README.md @@ -104,8 +104,10 @@ in mixed prefill/decode batches. ## Getting Started -Requires **Python 3.10+** and **PyTorch 2.5+**. For GPU inference, use Linux and -an NVIDIA GPU with a compatible CUDA-enabled PyTorch installation. +Requires **Python 3.10+** and **PyTorch 2.5+**. The commands below install +`v0.2.0` for NVIDIA CUDA on Linux with compatible CUDA-enabled PyTorch. + +For Ascend NPU on `main`, follow the [PyTorch and Torch-NPU installation steps](docs/recipes/ouro_ascend.md#install-pytorch-and-torch-npu). ```bash # Activate your Python environment first. @@ -120,10 +122,9 @@ builds use the Torch activation fallback; optional native CUDA compilation requires a compatible toolchain. Model weights are downloaded separately. No PyPI publication is implied. -- [Installation and first request](docs/launching.md) -- [Python API and CLI](docs/launching.md) -- [HTTP serving and streaming](docs/serving.md) — use a trusted network or secure - gateway; the server has no built-in authentication or TLS. +- [Installation, first request, Python API, and CLI](docs/launching.md) +- [HTTP serving and streaming](docs/serving.md). + Use a trusted network or secure gateway. The server has no built-in authentication or TLS. - [Supported models](#model-support) and [Ouro-Thinking recipe](docs/recipes/ouro_thinking.md) - [Configuration and compatibility](docs/runtime.md) - [Native CUDA build](docs/launching.md#automatic-native-cuda-build) @@ -140,20 +141,17 @@ a fixed scale at startup. See the ## Model Support - -| Checkpoint | Implementation | Documented validation scope | Guide | +| Checkpoint | Implementation | Platforms | Documented validation scope | | --- | --- | --- | --- | -| Ouro-1.4B | Native Ouro | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | [Runtime](docs/runtime.md), [Ascend recipe](docs/recipes/ouro_ascend.md#6-fixed-depth-asynchronous-engine-plan) | -| Ouro-2.6B | Shared Ouro | Shares the Ouro implementation; not separately qualified, and 1.4B quality/performance results are not inherited | [Scope notes](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) | -| Ouro-1.4B-Thinking | Reuses Ouro | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | -| Ouro-2.6B-Thinking | Reuses Ouro | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) | -| Nanbeige4.2-3B | Native Nanbeige | Fixed two-loop inference | [Runtime](docs/runtime.md) | -| Huginn-0125 | Native Huginn | Synchronous fixed-depth, LAST_EXITED KV, CUDA Graph decode core and coda | [Runtime](docs/runtime.md) | -| NVIDIA-Nemotron-Labs-Dory1.0-4B | Native Dory | Synchronous fixed-loop execution, boundary KV, per-loop and last-loop KV modes, Triton/Torch sliding-window masking, BF16 real-checkpoint agreement at 1/2/4 loops on A100; long context, Graphs/async and reclamation not qualified | [Dory baseline](docs/dory.md) | - -Shared implementation does not qualify every checkpoint/feature combination; -see the linked guides for validation scope and limitations. See -[RFC #69](https://github.com/InferMatrixLab/vllm-rlt/issues/69) for the broader roadmap. +| Ouro-1.4B | Native Ouro | [CUDA](docs/launching.md)
[Ascend](docs/recipes/ouro_ascend.md) | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | +| Ouro-2.6B | Shared Ouro | [CUDA](docs/launching.md) | Shares the Ouro implementation. [Separate quality and performance qualification](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) remains open. | +| [Ouro-1.4B-Thinking](docs/recipes/ouro_thinking.md) | Reuses Ouro | [CUDA](docs/launching.md) | Bounded BF16, fixed-four-loop synchronous basic integration | +| [Ouro-2.6B-Thinking](docs/recipes/ouro_thinking.md) | Reuses Ouro | [CUDA](docs/launching.md) | Bounded BF16, fixed-four-loop synchronous basic integration | +| Nanbeige4.2-3B | Native Nanbeige | [CUDA](docs/launching.md) | Fixed two-loop inference. [CUDA PD validation](https://github.com/InferMatrixLab/vllm-rlt/pull/77) | +| Huginn-0125 | Native Huginn | [CUDA](docs/launching.md) | Synchronous fixed-depth, LAST_EXITED KV, CUDA Graph decode core and coda | +| [NVIDIA-Nemotron-Labs-Dory1.0-4B](docs/dory.md) | Native Dory | [CUDA](docs/launching.md) | Synchronous fixed-loop execution, boundary KV, per-loop and last-loop KV modes, Triton/Torch sliding-window masking, BF16 real-checkpoint agreement at 1/2/4 loops on A100; long context, Graphs/async and reclamation not qualified | + +The broader model roadmap is in [RFC #69](https://github.com/InferMatrixLab/vllm-rlt/issues/69). Huginn uses native checkpoint loading and depth-aware KV for its recurrent core, with separate prelude and coda KV. It supports synchronous fixed-depth diff --git a/docs/README.md b/docs/README.md index 47adbd1d..1b933288 100644 --- a/docs/README.md +++ b/docs/README.md @@ -5,9 +5,9 @@ execution; optional GPU paths and historical measurements have narrower qualification scopes than the feature list. See the [changelog and migration notes](../CHANGELOG.md) before upgrading from v0.1.0. -New users should follow the [first-run user guide](launching.md), from creating -an environment to receiving the first generated response. The -[README](../README.md#getting-started) provides a brief installation reference. +The [Getting Started section](../README.md#getting-started) lists the platform +options. The [first-run user guide](launching.md) walks through the NVIDIA CUDA +setup on `v0.2.0`. The Ascend guide below uses `main`. | Guide | Topics | | --- | --- | @@ -26,7 +26,8 @@ an environment to receiving the first generated response. The | [FlashAttention](https://github.com/InferMatrixLab/vllm-rlt/pull/30) | FA2/FA3/FA4 installation, hardware selection, and constraints | | [Cache and scheduling features](https://github.com/InferMatrixLab/vllm-rlt/pull/31) | Prefix reuse, incremental KV, priorities, and preemption | | [Prefill/decode disaggregation](https://github.com/InferMatrixLab/vllm-rlt/pull/31) | Single-host GPU worker pools and NIXL transfer | -| [HTTP serving](serving.md) | Completions API, streaming, and service lifecycle | +| [HTTP serving](serving.md) | Device setup, completions API, streaming, and service lifecycle | +| [Ouro on Ascend](recipes/ouro_ascend.md) | Huawei Ascend NPU setup on `main`, serving commands, and validation scope | | [Profiling](profiling.md) | Core profiler options, Python/HTTP controls, PD collection, and per-rank archives | | [Ouro-Thinking recipe](recipes/ouro_thinking.md) | Shared 1.4B/2.6B setup, JSON/SSE client, budgets and validation limits | | [Accuracy evaluation](accuracy.md) | GSM8K regression setup and comparison methodology | diff --git a/docs/launching.md b/docs/launching.md index 604800ca..2aad2973 100644 --- a/docs/launching.md +++ b/docs/launching.md @@ -10,6 +10,8 @@ For migration from v0.1.0, read the [changelog](../CHANGELOG.md); Python scripts now construct a spawned stage worker and need the guarded example below. The release is for research/evaluation, not a production deployment guarantee. +For Huawei Ascend NPU, follow the [Ascend setup](recipes/ouro_ascend.md) on `main`. + ## 1. Check the machine Use Linux with Git, curl, [uv](https://docs.astral.sh/uv/getting-started/installation/), diff --git a/docs/recipes/ouro_ascend.md b/docs/recipes/ouro_ascend.md index 82c3887a..3a14d870 100644 --- a/docs/recipes/ouro_ascend.md +++ b/docs/recipes/ouro_ascend.md @@ -8,12 +8,49 @@ The example enables the optional asynchronous engine plan. ## Prepare the environment and model -Install the project dependencies in a compatible Torch, Torch-NPU and CANN -environment. Use the checkpoint's config and byte-level tokenizer. +Use `main`, which includes Ascend support added after `v0.2.0`. Install CANN and +the matching driver and firmware for your Ascend hardware first. + +### Install PyTorch and Torch-NPU + +The following commands use CANN 9.0.0 with PyTorch 2.10.0 and Torch-NPU 2.10.0. +For other versions, consult the +[Ascend compatibility table](https://github.com/Ascend/pytorch/blob/master/COMPATIBILITY.en.md#torchnpu-version-mapping-table). +Activate your Python environment and load the CANN environment. ```bash -source /path/to/cann/set_env.sh source /path/to/npu-environment/bin/activate +source /path/to/cann/set_env.sh +``` + +On x86-64, install PyTorch from the CPU wheel index. + +```bash +python -m pip install 'torch==2.10.0+cpu' --index-url https://download.pytorch.org/whl/cpu +``` + +On Arm64 (`aarch64`), use the following PyTorch command. + +```bash +python -m pip install 'torch==2.10.0' +``` + +On either architecture, install `torch-npu` and check that PyTorch can see the NPU. +The package is imported as `torch_npu` in Python. + +```bash +python -m pip install 'torch-npu==2.10.0' pyyaml setuptools +python -c 'import torch, torch_npu; assert torch.npu.is_available(), "Ascend NPU unavailable"' +``` + +### Install vllm-rlt and set the checkpoint path + +Install the project dependencies in that environment. Use the checkpoint's +config and byte-level tokenizer. + +```bash +git clone --branch main https://github.com/InferMatrixLab/vllm-rlt.git +cd vllm-rlt python -m pip install --no-deps --no-build-isolation -e . export MODEL=/path/to/prepared/Ouro-1.4B export ASCEND_RT_VISIBLE_DEVICES=0 # Replace 0 with your assigned NPU. diff --git a/docs/serving.md b/docs/serving.md index 81cb6999..9b645a7e 100644 --- a/docs/serving.md +++ b/docs/serving.md @@ -5,9 +5,14 @@ environment. The `vllm-rlt-serve` entrypoint (or `python -m vllm_rlt.entrypoints.serve`) serves one resident model on one device. Bind defaults to `127.0.0.1:8000`; this benchmark frontend has no authentication or TLS. +## Device setup + +For Huawei Ascend NPU on `main`, follow the [Ascend serving setup](recipes/ouro_ascend.md) +for the Torch-NPU and CANN environment and the launch command. + Prepare the Ouro checkpoint and tokenizer separately. The official model ID pins revision `574fa66cb8bf5abdc979642d01cf2b79b16bfab1`; local paths are supported. -Select an available GPU explicitly; a scheduler is optional: +For NVIDIA CUDA, select an available GPU explicitly. ```bash nvidia-smi