Skip to content

feat(kv): add CUDA VMM KV storage with fixed addresses - #194

Open
XinhaoTheo wants to merge 3 commits into
InferMatrixLab:mainfrom
XinhaoTheo:feat/rfc192-vmm-kv-storage
Open

XinhaoTheo wants to merge 3 commits into
InferMatrixLab:mainfrom
XinhaoTheo:feat/rfc192-vmm-kv-storage

Conversation

@XinhaoTheo

@XinhaoTheo XinhaoTheo commented Nov 17, 2026 •

Copy link
Copy Markdown

Required human contributor answers

1. What outcome are you trying to achieve?

Size the KV pool from the memory that is free after warmup and CUDA Graph capture (RFC #192). The captured graphs must stay valid.

2. What is the highest-leverage variable for that outcome?

The KV tensor addresses. A captured graph keeps the KV pointers, the strides and the size of the first dimension (cuda_graph.py#L121). If the addresses stay fixed, the engine can add KV capacity after capture.

3. How will this work move that variable rather than create busywork?

StableKVStorage reserves the full virtual address range at startup. It adds physical memory later with CUDA VMM. The addresses, strides and stored values do not change. The tests show that a graph that was captured before a commit writes correctly to the new blocks.

4. Who is the customer or user, and what do they actually need?

PR necessity

A PR is necessary. No current storage path keeps the KV addresses fixed when the capacity increases. This is not a bugfix.

Modification

This PR adds vllm_rlt/core/kv_storage.py. The engine does not use it yet, so no behavior changes.

  • PoolSpec(block_shape, dtype) gives the shape and dtype of one block in one pool. A pool is not specific to K or V, so other page layouts ([KV] Pack recurrent depth planes for partially recurrent models #147, feat(kv): store INT4/NVFP4 KV pages in the cache manager and read them with Triton #174) can add pools.
  • StableKVStorage(specs, max_blocks=..., initial_blocks=..., device=...):
    • It reserves one virtual address range for each pool. Each pool has one tensor with the shape [max_blocks, *block_shape].
    • commit(blocks) adds physical memory from the start of each range, in multiples of the granule (2 MiB on H100). The capacity can only increase.
    • committed_bytes gives the physical bytes. The PyTorch allocator does not count them.
    • As with torch.empty, the memory is released when no reference to the storage or to its tensor views remains. The release synchronizes the device first. There is no explicit release, because it would leave the remaining views on unmapped memory.
  • vmm_unavailable_reason(device) gives the reason for a fallback: not a CUDA device, a ROCm build, not Linux, or no VMM support.
  • The driver binding uses ctypes and libcuda.so.1. It does not need a build step or a new dependency.

Out of scope: the KVCacheManager integration (#195), the startup order change and the measured final capacity (other RFC #192 contributors), HIP VMM for ROCm, and VMM memory for NIXL RDMA.

Related PR

Accuracy Test

This PR does not change an execution path, so the model outputs do not change.

The tests in tests/test_kv_storage.py check three things:

  • The addresses stay fixed after commit().
  • A graph that was captured before a commit writes correctly to the new blocks.
  • The last reference releases the memory, and the PyTorch allocator does not count it.

All tests passed on H100 (Modal) at 594dc63. The CPU suite also passed, except 12 tests in test_async_speculative_sampling_plan.py that need Triton. Triton is not available on macOS. The token equality test against torch.empty comes with the integration in #195.

Benchmark and Profiling

Micro-benchmark on one H100 80GB HBM3, measured at 7aaa7c6. 594dc63 changes only when the release starts, not the release code. The KV geometry comes from the ByteDance/Ouro-1.4B config.json: 24 layers, 16 KV heads, head dim 128, block size 16, BF16. One block of one pool is 1.5 MiB.

Lifecycle cost (K and V pools together, median of 3 runs):

Size of each pool VMM construct VMM commit() VMM release torch.empty allocate torch.empty free
1 GiB 2.7 ms 2.0 ms 1.3 ms 1.8 ms 1.3 ms
8 GiB 2.4 ms 3.2 ms 8.8 ms 5.2 ms 7.6 ms
32 GiB 2.6 ms 143 ms 37 ms 180 ms 41 ms

Rounding: the granule is 2 MiB. The unused physical memory is 1 MiB to 2 MiB for K and V together. The usable block count is equal to the requested block count.

Bandwidth (both storages in memory, 6 interleaved rounds, median):

Workload VMM torch.empty Difference
Triton paged_attention, batch 32, context 4096, 24 layers 2932.9 GB/s 2933.3 GB/s −0.02% (noise)
Device copy, 12 GiB 3033.0 GB/s 3033.9 GB/s −0.03% (all rounds)

The attention outputs of the two storages are bitwise equal.

Profiling: not applicable, because no execution path changes. The end-to-end A/B and the profiles come with the integration in #195.

Conclusion

This PR contains only the KV storage. The KVCacheManager integration, the token equality test and the end-to-end A/B are in #195.

  • Add StableKVStorage: fixed addresses, commit(), release on the last reference, and the fallback reason.
  • Test it on GPU: fixed addresses, graph replay on new blocks, and memory release.
  • Measure the lifecycle cost, the rounding and the bandwidth on H100.
  • Apply the review: remove close(), make the release test independent of other processes, and rebase onto main.
  • Run CI (it waits for maintainer approval).

@kunkunblueberry kunkunblueberry left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — this is a useful piece. I read the diff and ran your two files on a clean upstream/main checkout on an A100 (torch 2.14+cu130), so the observations below are reproduced, not inferred. I am the author of #195, the end-to-end variant of the same RFC #192 step, so the overlap is discussed at the end.

Reviewed head: 7aaa7c6 (base c338aaa, 11 commits behind main).

Blocking

1. commit() has no upper bound — vllm_rlt/core/kv_storage.py:320

The docstring says blocks must not exceed max_blocks, but nothing enforces it. With 3 MiB blocks and max_blocks=1 (4 MiB reserved), commit(2) asks for 6 MiB and fails inside the driver:

commit(2) raised: RuntimeError : CUDA driver error 1: invalid argument

An integration mistake would surface as an opaque driver error during startup. if not 1 <= blocks <= self.max_blocks: raise ValueError(...) turns it into a usable message. Note that when the block is smaller than the granule the round-up hides it: max_blocks=4 with 64 KiB blocks, commit(5) silently returns 4.

2. close() leaves the object looking alive — :336

After close() on a 64-block / 2 MiB store:

before close: committed_blocks = 64 committed_bytes = 2097152
after  close: committed_blocks = 64 committed_bytes = 2097152
after  close: tensors still exposed = 1
commit after close -> 64
write through a stale view after close: no error

The VA has already been freed at that point, so this is a silent-corruption window rather than a crash: commit() is a no-op that reports success, and a surviving view still writes. A _closed flag that raises from commit() and from tensors, plus zeroed counters, makes the misuse loud.

3. ruff format --check . fails, which is what cpu-tests.yml runs (pinned ruff==0.13.2)

Would reformat: tests\test_kv_storage.py
Would reformat: vllm_rlt\core\kv_storage.py

Reproduced with 0.13.2 itself, so it is not a version artifact. ruff check . passes.

Non-blocking

4. initial_blocks=0 fails deep inside torch — :308: RuntimeError: The specified pointer resides on host memory and is not registered with any CUDA device. The constraint comes from building the tensors out of the pool before any mapping exists, not from CUDA, so an explicit validation would be clearer.

5. No capacity gate for the consumer. tensors spans max_blocks, and "do not access the blocks after committed_blocks" is documentation only. Once KVCacheManager is wired (your unchecked item), allocating past committed_blocks is a sticky illegal address that poisons the CUDA context for the whole process. A published <= committed <= max split, where publishing requires committed_blocks, makes that unreachable; your TODO already lists the free-pool update, and the invariant is what keeps it safe.

6. No CI evidence attached — statusCheckRollup is empty and mergeStateStatus is UNSTABLE, so the H200/H100 runs are not visible from the PR. Rebasing onto current main would also help, since the base is 11 commits behind.

7. Test oracle depends on absolute driver memory — test_close_returns_memory_the_torch_allocator_never_counted compares mem_get_info() values. That is fine on a dedicated Modal box, but another tenant's allocations move the same counter, so it flakes on a shared device (I hit exactly this in my own suite). A sizeable mapping plus release accounting from the storage itself is more stable.

Verified as correct, for the record

  • The lifetime story holds: torch.as_tensor keeps the __cuda_array_interface__ provider alive, so views keep the pool alive. Reproduced: after del storage the finalizer did not run and the surviving view stayed writable.
  • weakref.finalize receives a module-level function rather than a bound method, which avoids the self-reference keepalive trap.
  • close() is idempotent, a failed cuMemMap releases its handle, and memory_reserved() stays untouched, so "the PyTorch allocator does not count this memory" is accurate.
  • Driving the real write_graph_kv kernel in the graph test is a stronger check than a torch-only write.

Overlap with #195

#195 implements the same RFC #192 step end-to-end (storage + KVCacheManager capacity states + startup growth + CLI flag) and also adds tests/test_kv_storage.py, so the two will conflict. Two storage modules for one mechanism is the part worth avoiding. My suggestion:

  • keep a single storage layer — your PoolSpec API is more general (it can host the #147/#174 layouts) and your vmm_unavailable_reason probe is better than mine;
  • port items 1, 2 and 5 into it;
  • then I rebase #195 onto your module, drop my vmm_pool.py and rename my test file.

I am happy to do that rebase once 1-3 are addressed, if you prefer to keep the storage module where it is.

Add StableKVStorage for RFC InferMatrixLab#192. It reserves one virtual address range
for each KV pool. The tensor of each pool has the maximum block count from
the start. commit() adds physical memory to more blocks. It does not change
a tensor address, a stride or a stored value. Because of this, a CUDA Graph
that was captured before a commit also writes correctly to the new blocks.

The driver binding uses ctypes and libcuda.so.1. It does not need a build
step or a new dependency. vmm_unavailable_reason() gives the reason when a
device cannot use this storage.

The engine does not use this storage yet.
@XinhaoTheo

Copy link
Copy Markdown
Author

Thanks for the review. I rebased #194 onto main (33d950d). For each item:

  1. Not done. A request above max_blocks already stops with a driver error, so it does not fail silently. commit(5) returns 4 on a 4-block storage, because the return value is the number of committed blocks. The caller must use this value. I do not add a second check for the same contract.
  2. Done, with a different fix. I removed close(). The memory is released only when no reference to the storage or to its views remains, as with torch.empty. Thus a stale view cannot occur, and a _closed flag is not necessary. For an earlier release, remove all references to the storage and its views.
  3. Not done. With the CI version (ruff 0.13.2) on Linux and macOS, ruff format --check . gives "139 files already formatted". Your output shows Windows paths. I think that your checkout changed the line endings.
  4. Not done. The error occurs in the constructor, before the storage is used. The comment on that line gives the cause.
  5. Not done in feat(kv): add CUDA VMM KV storage with fixed addresses #194. The published/committed gate is part of the KVCacheManager integration, which is your part of the split. committed_blocks gives the value for the gate.
  6. Done. I rebased onto main. The CI workflow waits for maintainer approval (action_required), so it has not run yet.
  7. Done. The release test does not read the free memory of the device. It checks that the views keep the pools alive, that the last reference releases them, and that torch.cuda.memory_reserved() does not change. A failed release in the finalizer now fails the test.

GPU: tests/test_kv_storage.py passes on H100 (5 passed).

Overlap: I agree. Please rebase #195 onto vllm_rlt/core/kv_storage.py, remove vmm_pool.py and rename your test file.

close() released the memory also when tensor views remained. A remaining
view then wrote to unmapped memory without an error. Now the memory is
released only when no reference to the storage or to its views remains,
as with torch.empty.

The release test does not read the free memory of the device, because
other processes on a shared device change it. The test checks the pool
lifetime, and a failed release in the finalizer fails the test.
@XinhaoTheo
XinhaoTheo force-pushed the feat/rfc192-vmm-kv-storage branch from 7aaa7c6 to 594dc63 Compare November 17, 2026 16:50
@XinhaoTheo
XinhaoTheo marked this pull request as ready for review November 17, 2026 18:38
kunkunblueberry added a commit to kunkunblueberry/vllm-rlt that referenced this pull request Nov 19, 2026
Turn the RFC InferMatrixLab#192 step-1 preparation report into a KV block count, and let
KVCacheManager grow its published range without reallocating tensors.

- measured_kv_capacity() reads the final drained phase of the report: the
  driver's remaining memory is the physical budget, so weights, execution
  buffers, captured graphs and any live mapping are never subtracted twice.
  The utilization cap still bounds the instance, and the transient allowance
  and safety reserve are explicit inputs that are recorded with the result.
  Coverage gaps reported by the preparation (long contexts, random
  speculation, unreported batch/history combinations) travel with the plan,
  so an estimated margin is never presented as a measured one.
- KVCacheManager accepts a capacity source whose tensors span the full
  reservation and whose physical backing grows by commit(). Requests are only
  ever handed published blocks, publishing requires the storage to report the
  blocks as backed, and published <= committed <= max is enforced, so no
  allocation can address unmapped memory. Release stays reference-based, as
  agreed for the storage contract: the owner drops references, the manager
  never unmaps.
- With no capacity source nothing changes: tensors come from torch.empty and
  max == committed == published.

Engine wiring (prepare_runtime -> plan -> commit -> publish -> replay and
representative execution -> ready) needs InferMatrixLab#198 for the report and InferMatrixLab#194 for the
storage, so it is the next step rather than part of this change.

Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>

@ycjcx123 ycjcx123 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The implementation matches the scope and design described in the PR body, with no changes to current engine execution paths. I found one P2 exception-handling issue in _Pool.grow() and listed two non-blocking robustness suggestions below. Recommend merging once the P2 rollback path and a regression test are added.

  • Reviewed commit: 594dc63 (base main at 33d950d)

Design & PR Description Alignment

  • Scope & isolation: This PR only adds vllm_rlt/core/kv_storage.py and tests/test_kv_storage.py without wiring them into the engine yet, matching the stated no-behavior-change scope.
  • Approach & fallback: Reserving the full virtual address range upfront and mapping physical memory in allocation-granule multiples preserves tensor base pointers, strides, and first-dimension bounds for captured CUDA Graphs across commit(), whereas reallocating ordinary tensors would require recapturing graphs. Retaining the existing torch.empty path as the baseline and fallback is an appropriate boundary.
  • Lifetime management: Removing explicit close() in 594dc63 in favor of reference-based cleanup (weakref.finalize) resolves the earlier stale-view hazard.
  • Follow-up boundaries: Capacity publication in KVCacheManager, startup order changes, token-equality checks against torch.empty, and end-to-end A/B benchmarks are explicitly scoped to follow-up integration PRs.

Finding

_Pool.grow() leaves mapping state and committed_bytes inconsistent if set_access() fails, preventing retries

  • Location: vllm_rlt/core/kv_storage.py:267-269

_Pool.grow() currently rolls back only when self._driver.map() fails:

If map() succeeds and set_access() then raises:

  1. The new mapping and physical handle remain active in _chunks, while self.mapped_bytes is not updated, so committed_bytes underreports actual physical memory usage while the storage object is alive. (Finalizer cleanup still iterates _chunks on destruction, so this is not a permanent leak after the object is collected.)
  2. Retrying the same expansion target uses the old offset and attempts to map an already-mapped range, causing the retry to fail.

Verification: Reproduced on 594dc63 via CPU fault injection with a mock driver (not triggered on a live GPU): after a failed set_access(), mapped_bytes stayed at the old value while _chunks and active mappings grew; clearing the fault and retrying grow() failed due to the already-mapped range.

Suggested fix:

  • Keep resource tracking as soon as map() succeeds.
  • If set_access() fails, unmap the range and release the handle, removing the record from _chunks only after rollback cleanup succeeds (if rollback itself fails, keep the unreleased entry tracked and prevent further growth from reusing that offset).
  • Add a unit test covering set_access() failure $\rightarrow$ rollback cleanup $\rightarrow$ successful retry.

Non-blocking Suggestions

  1. Explicit boundary checks on constructor and commit() arguments :
    blocks <= max_blocks is a documented precondition, while the current tensor-construction path requires initial_blocks >= 1. Missing checks are not a P1 defect on their own, but validating 1 <= initial_blocks <= max_blocks and 1 <= blocks <= self.max_blocks upfront to raise a clear ValueError would make integration mistakes easier to diagnose than relying on granule-dependent clamping or a downstream PyTorch/driver error.
  2. Driver-missing behavior in vmm_unavailable_reason(device) (kv_storage.py:188-200):
    On Linux with device.type == "cuda", calling vmm_unavailable_reason() when libcuda.so.1 is absent or CUDA driver initialization fails currently raises OSError / RuntimeError. Please confirm whether callers are expected to check torch.cuda.is_available() first, or whether vmm_unavailable_reason() should catch driver/initialization errors and return a fallback reason string (with a corresponding unit test).

Validation Scope

  • Tests on frozen head (594dc63): tests/test_kv_storage.py ran 1 passed, 4 skipped (GPU tests skipped on non-GPU host), plus the CPU fault-injection check on _Pool.grow().
  • Lint & formatting: ruff check and ruff format --check (local ruff 0.16.8) passed on the added files.
  • CI & benchmarks: At review time, the CPU workflow was awaiting maintainer approval (action_required); no passing CI result was available. The H100 lifecycle and bandwidth numbers in the PR description are author-reported measurements and were not independently re-run in this review.

@XinhaoTheo

Copy link
Copy Markdown
Author

@ycjcx123 Thanks for the review.

  1. set_access() failure in _Pool.grow(): not done. The CUDA documentation gives only three errors for cuMemSetAccess: an invalid value, an invalid device and an unsupported operation. It has no out-of-memory error. grow() gives it the device and range that cuMemCreate and cuMemMap accepted immediately before.

    I tested this on H100 (594dc63). I filled the device memory with a PyTorch tensor, then called commit() until it failed:

    Case Result
    Large commits (64 blocks of 1.5 MiB each step) Out of memory at 2688 committed blocks
    Small commits (1 block each step, two pools) Out of memory after 510 steps
    Call that gave the out-of-memory error cuMemCreate, in both cases
    set_access() calls 1068 calls, 0 failures
    map() calls 1068 calls, 0 failures
    Retry after the tensor was freed Successful in both cases

    Each out-of-memory error occurred before any map or record, so the pool state stayed correct and the retry was successful.

  2. Boundary checks: not done. This is the same as items 1 and 4 in the first review. A request above max_blocks stops with a driver error, and initial_blocks=0 stops in the constructor. Both fail before the storage is used.

  3. Missing driver: not done. The caller gives a CUDA device that PyTorch already uses. PyTorch loads libcuda.so.1 for CUDA, so the driver is present when vmm_unavailable_reason() runs. Without a driver, PyTorch fails earlier on the same device.

kunkunblueberry added a commit to kunkunblueberry/vllm-rlt that referenced this pull request Nov 21, 2026
Turn the RFC InferMatrixLab#192 step-1 preparation report into a KV block count, and let
KVCacheManager grow its published range without reallocating tensors.

- measured_kv_capacity() reads the final drained phase of the report: the
  driver's remaining memory is the physical budget, so weights, execution
  buffers, captured graphs and any live mapping are never subtracted twice.
  The utilization cap still bounds the instance, and the transient allowance
  and safety reserve are explicit inputs that are recorded with the result.
  Coverage gaps reported by the preparation (long contexts, random
  speculation, unreported batch/history combinations) travel with the plan,
  so an estimated margin is never presented as a measured one.
- KVCacheManager accepts a capacity source whose tensors span the full
  reservation and whose physical backing grows by commit(). Requests are only
  ever handed published blocks, publishing requires the storage to report the
  blocks as backed, and published <= committed <= max is enforced, so no
  allocation can address unmapped memory. Release stays reference-based, as
  agreed for the storage contract: the owner drops references, the manager
  never unmaps.
- With no capacity source nothing changes: tensors come from torch.empty and
  max == committed == published.

Engine wiring (prepare_runtime -> plan -> commit -> publish -> replay and
representative execution -> ready) needs InferMatrixLab#198 for the report and InferMatrixLab#194 for the
storage, so it is the next step rather than part of this change.

Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>

@ycjcx123 ycjcx123 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the additional tests and clarification. I’m withdrawing the blocking P2 finding.

I still recommend explicit capacity validation in the storage layer and documenting the initialized-CUDA-device precondition in vmm_unavailable_reason(). These are non-blocking suggestions.

No remaining blocking findings from my side.

@XinhaoTheo

Copy link
Copy Markdown
Author

Thanks for the additional tests and clarification. I’m withdrawing the blocking P2 finding.

I still recommend explicit capacity validation in the storage layer and documenting the initialized-CUDA-device precondition in vmm_unavailable_reason(). These are non-blocking suggestions.

No remaining blocking findings from my side.

Thanks. Both suggestions are done in bed5d33:

  • StableKVStorage now raises ValueError before any driver call if initial_blocks is not in 1..max_blocks, or if commit() asks for more than max_blocks.
  • The vmm_unavailable_reason() docstring now says that the device must be one that PyTorch can already use.

All 7 tests in tests/test_kv_storage.py pass on H100.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants