Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 17 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,10 @@ in mixed prefill/decode batches.

## Getting Started

Requires **Python 3.10+** and **PyTorch 2.5+**. For GPU inference, use Linux and
an NVIDIA GPU with a compatible CUDA-enabled PyTorch installation.
Requires **Python 3.10+** and **PyTorch 2.5+**. The commands below install
`v0.2.0` for NVIDIA CUDA on Linux with compatible CUDA-enabled PyTorch.

For Ascend NPU on `main`, follow the [PyTorch and Torch-NPU installation steps](docs/recipes/ouro_ascend.md#install-pytorch-and-torch-npu).

```bash
# Activate your Python environment first.
Expand All @@ -120,10 +122,9 @@ builds use the Torch activation fallback; optional native CUDA compilation
requires a compatible toolchain. Model weights are downloaded separately.
No PyPI publication is implied.

- [Installation and first request](docs/launching.md)
- [Python API and CLI](docs/launching.md)
- [HTTP serving and streaming](docs/serving.md) — use a trusted network or secure
gateway; the server has no built-in authentication or TLS.
- [Installation, first request, Python API, and CLI](docs/launching.md)
- [HTTP serving and streaming](docs/serving.md).
Use a trusted network or secure gateway. The server has no built-in authentication or TLS.
- [Supported models](#model-support) and [Ouro-Thinking recipe](docs/recipes/ouro_thinking.md)
- [Configuration and compatibility](docs/runtime.md)
- [Native CUDA build](docs/launching.md#automatic-native-cuda-build)
Expand All @@ -140,20 +141,17 @@ a fixed scale at startup. See the

## Model Support


| Checkpoint | Implementation | Documented validation scope | Guide |
| Checkpoint | Implementation | Platforms | Documented validation scope |
| --- | --- | --- | --- |
| Ouro-1.4B | Native Ouro | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. | [Runtime](docs/runtime.md), [Ascend recipe](docs/recipes/ouro_ascend.md#6-fixed-depth-asynchronous-engine-plan) |
| Ouro-2.6B | Shared Ouro | Shares the Ouro implementation; not separately qualified, and 1.4B quality/performance results are not inherited | [Scope notes](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) |
| Ouro-1.4B-Thinking | Reuses Ouro | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) |
| Ouro-2.6B-Thinking | Reuses Ouro | Bounded BF16, fixed-four-loop synchronous basic integration | [Shared recipe](docs/recipes/ouro_thinking.md) |
| Nanbeige4.2-3B | Native Nanbeige | Fixed two-loop inference | [Runtime](docs/runtime.md) |
| Huginn-0125 | Native Huginn | Synchronous fixed-depth, LAST_EXITED KV, CUDA Graph decode core and coda | [Runtime](docs/runtime.md) |
| NVIDIA-Nemotron-Labs-Dory1.0-4B | Native Dory | Synchronous fixed-loop execution, boundary KV, per-loop and last-loop KV modes, Triton/Torch sliding-window masking, BF16 real-checkpoint agreement at 1/2/4 loops on A100; long context, Graphs/async and reclamation not qualified | [Dory baseline](docs/dory.md) |

Shared implementation does not qualify every checkpoint/feature combination;
see the linked guides for validation scope and limitations. See
[RFC #69](https://github.com/InferMatrixLab/vllm-rlt/issues/69) for the broader roadmap.
| Ouro-1.4B | Native Ouro | [CUDA](docs/launching.md)<br>[Ascend](docs/recipes/ouro_ascend.md) | Ascend BF16 full prefill with eager Python execution, paged KV and optional fixed-depth async scheduling. Arbitrary chunked prefill remains unresolved. |
| Ouro-2.6B | Shared Ouro | [CUDA](docs/launching.md) | Shares the Ouro implementation. [Separate quality and performance qualification](docs/recipes/ouro_thinking.md#6-evidence-and-limitations) remains open. |
| [Ouro-1.4B-Thinking](docs/recipes/ouro_thinking.md) | Reuses Ouro | [CUDA](docs/launching.md) | Bounded BF16, fixed-four-loop synchronous basic integration |
| [Ouro-2.6B-Thinking](docs/recipes/ouro_thinking.md) | Reuses Ouro | [CUDA](docs/launching.md) | Bounded BF16, fixed-four-loop synchronous basic integration |
| Nanbeige4.2-3B | Native Nanbeige | [CUDA](docs/launching.md) | Fixed two-loop inference. [CUDA PD validation](https://github.com/InferMatrixLab/vllm-rlt/pull/77) |
| Huginn-0125 | Native Huginn | [CUDA](docs/launching.md) | Synchronous fixed-depth, LAST_EXITED KV, CUDA Graph decode core and coda |
| [NVIDIA-Nemotron-Labs-Dory1.0-4B](docs/dory.md) | Native Dory | [CUDA](docs/launching.md) | Synchronous fixed-loop execution, boundary KV, per-loop and last-loop KV modes, Triton/Torch sliding-window masking, BF16 real-checkpoint agreement at 1/2/4 loops on A100; long context, Graphs/async and reclamation not qualified |

The broader model roadmap is in [RFC #69](https://github.com/InferMatrixLab/vllm-rlt/issues/69).

Huginn uses native checkpoint loading and depth-aware KV for its recurrent
core, with separate prelude and coda KV. It supports synchronous fixed-depth
Expand Down
9 changes: 5 additions & 4 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ execution; optional GPU paths and historical measurements have narrower
qualification scopes than the feature list. See the [changelog and migration
notes](../CHANGELOG.md) before upgrading from v0.1.0.

New users should follow the [first-run user guide](launching.md), from creating
an environment to receiving the first generated response. The
[README](../README.md#getting-started) provides a brief installation reference.
The [Getting Started section](../README.md#getting-started) lists the platform
options. The [first-run user guide](launching.md) walks through the NVIDIA CUDA
setup on `v0.2.0`. The Ascend guide below uses `main`.

| Guide | Topics |
| --- | --- |
Expand All @@ -26,7 +26,8 @@ an environment to receiving the first generated response. The
| [FlashAttention](https://github.com/InferMatrixLab/vllm-rlt/pull/30) | FA2/FA3/FA4 installation, hardware selection, and constraints |
| [Cache and scheduling features](https://github.com/InferMatrixLab/vllm-rlt/pull/31) | Prefix reuse, incremental KV, priorities, and preemption |
| [Prefill/decode disaggregation](https://github.com/InferMatrixLab/vllm-rlt/pull/31) | Single-host GPU worker pools and NIXL transfer |
| [HTTP serving](serving.md) | Completions API, streaming, and service lifecycle |
| [HTTP serving](serving.md) | Device setup, completions API, streaming, and service lifecycle |
| [Ouro on Ascend](recipes/ouro_ascend.md) | Huawei Ascend NPU setup on `main`, serving commands, and validation scope |
| [Profiling](profiling.md) | Core profiler options, Python/HTTP controls, PD collection, and per-rank archives |
| [Ouro-Thinking recipe](recipes/ouro_thinking.md) | Shared 1.4B/2.6B setup, JSON/SSE client, budgets and validation limits |
| [Accuracy evaluation](accuracy.md) | GSM8K regression setup and comparison methodology |
Expand Down
2 changes: 2 additions & 0 deletions docs/launching.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ For migration from v0.1.0, read the [changelog](../CHANGELOG.md); Python scripts
now construct a spawned stage worker and need the guarded example below.
The release is for research/evaluation, not a production deployment guarantee.

For Huawei Ascend NPU, follow the [Ascend setup](recipes/ouro_ascend.md) on `main`.

## 1. Check the machine

Use Linux with Git, curl, [uv](https://docs.astral.sh/uv/getting-started/installation/),
Expand Down
43 changes: 40 additions & 3 deletions docs/recipes/ouro_ascend.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,49 @@ The example enables the optional asynchronous engine plan.

## Prepare the environment and model

Install the project dependencies in a compatible Torch, Torch-NPU and CANN
environment. Use the checkpoint's config and byte-level tokenizer.
Use `main`, which includes Ascend support added after `v0.2.0`. Install CANN and
the matching driver and firmware for your Ascend hardware first.

### Install PyTorch and Torch-NPU

The following commands use CANN 9.0.0 with PyTorch 2.10.0 and Torch-NPU 2.10.0.
For other versions, consult the
[Ascend compatibility table](https://github.com/Ascend/pytorch/blob/master/COMPATIBILITY.en.md#torchnpu-version-mapping-table).
Activate your Python environment and load the CANN environment.

```bash
source /path/to/cann/set_env.sh
source /path/to/npu-environment/bin/activate
source /path/to/cann/set_env.sh
```

On x86-64, install PyTorch from the CPU wheel index.

```bash
python -m pip install 'torch==2.10.0+cpu' --index-url https://download.pytorch.org/whl/cpu
```

On Arm64 (`aarch64`), use the following PyTorch command.

```bash
python -m pip install 'torch==2.10.0'
```

On either architecture, install `torch-npu` and check that PyTorch can see the NPU.
The package is imported as `torch_npu` in Python.

```bash
python -m pip install 'torch-npu==2.10.0' pyyaml setuptools
python -c 'import torch, torch_npu; assert torch.npu.is_available(), "Ascend NPU unavailable"'
```

### Install vllm-rlt and set the checkpoint path

Install the project dependencies in that environment. Use the checkpoint's
config and byte-level tokenizer.

```bash
git clone --branch main https://github.com/InferMatrixLab/vllm-rlt.git
cd vllm-rlt
python -m pip install --no-deps --no-build-isolation -e .
export MODEL=/path/to/prepared/Ouro-1.4B
export ASCEND_RT_VISIBLE_DEVICES=0 # Replace 0 with your assigned NPU.
Expand Down
7 changes: 6 additions & 1 deletion docs/serving.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,14 @@ environment. The `vllm-rlt-serve` entrypoint (or `python -m
vllm_rlt.entrypoints.serve`) serves one resident model on one device. Bind defaults
to `127.0.0.1:8000`; this benchmark frontend has no authentication or TLS.

## Device setup

For Huawei Ascend NPU on `main`, follow the [Ascend serving setup](recipes/ouro_ascend.md)
for the Torch-NPU and CANN environment and the launch command.

Prepare the Ouro checkpoint and tokenizer separately. The official model ID
pins revision `574fa66cb8bf5abdc979642d01cf2b79b16bfab1`; local paths are supported.
Select an available GPU explicitly; a scheduler is optional:
For NVIDIA CUDA, select an available GPU explicitly.

```bash
nvidia-smi
Expand Down
Loading