Skip to content

feat(memory): size KV capacity from the post-warmup measurement - #200

Draft
kunkunblueberry wants to merge 5 commits into
InferMatrixLab:mainfrom
kunkunblueberry:kunkun/rfc192-measured-capacity
Draft

kunkunblueberry wants to merge 5 commits into
InferMatrixLab:mainfrom
kunkunblueberry:kunkun/rfc192-measured-capacity

Conversation

@kunkunblueberry

@kunkunblueberry kunkunblueberry commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Implements the capacity-planning part of RFC #192 assigned to me in the issue, on the terms agreed there: the planner consumes the step-1 preparation report, and the logical allocator only ever receives fully backed blocks. Rebased onto ce200f0.

Required human contributor answers

TODO(contributor): the four answers and the PR-necessity decision are reserved for the human contributor; the recorded split in RFC #192 describes the agreed scope but is not a substitute for them.

1. What outcome are you trying to achieve?

TODO(contributor)

2. What is the highest-leverage variable for that outcome?

TODO(contributor)

3. How will this work move that variable rather than create busywork?

TODO(contributor)

4. Who is the customer or user, and what do they actually need?

TODO(contributor)

PR necessity and bugfix evidence

TODO(contributor)

Decisions recorded with the maintainer

The four open questions in this thread were settled as follows, and the code now states them explicitly instead of leaving them to the caller:

  1. The trial pool covers the expected maximum concurrency plus a medium-length prompt. Paths the preparation does not exercise are recorded as coverage gaps and the configured estimate stays in force for them; nothing unmeasured is sized as if measured.
  2. The reservation is sized from the useful cap (useful_blocks), not from the pre-warmup estimate. This is the point of the change: a reservation derived from the estimate can never grow, so the measurement would be truncated back to the number it was meant to replace. A reservation that truncates the measured capacity is now an error rather than a silent clamp; allow_reservation_truncation=True opts back in when that is genuinely intended. Reserving the useful cap costs address space, not physical memory.
  3. FP8 over a mapped pool is a hook, not a hard rejection. A capacity source declares the format it carries and either exposes scale_tensors (K first) for per-vector scales or is used with a fixed fp8_scale, which needs no per-vector storage. A source that declares FP8 and supplies neither is still rejected, but the message names the hook.
  4. gpu_memory_utilization keeps its existing meaning: a fraction of the whole device, not of the free memory left after the baseline. Only the docstring changes.

Purpose

Three pieces the measured-sizing path needs. None of them depends on the unmerged orchestration in #198 or the storage in #194, and each is usable on its own.

measured_kv_capacity() (vllm_rlt/core/memory.py) turns the final drained phase of the preparation report into a block count.

  • The physical budget is the driver's remaining memory, which already excludes weights, execution buffers, captured graphs and anything mapped, so none of it is subtracted a second time. total - free only bounds the remaining instance budget under gpu_memory_utilization.
  • existing_blocks is the already-mapped bootstrap prefix, taken from the report's num_blocks. It extends the measured delta and is never sized from free memory again, so the returned count is the total the pool may hold and a cumulative commit(total) maps only the new part.
  • reservation_blocks clamps the result to what the virtual reservation can back; truncation is recorded as a coverage gap, and limited_by names the binding bound (budget, useful_blocks, reservation).
  • A zero transient allowance is rejected when the preparation reported coverage gaps: an unexercised path cannot be sized as if it were measured.

transient_allowance() derives that margin from the report instead of leaving it to the caller: each phase's peak_allocated_bytes - allocated_bytes is what the phase needed above what it kept, the largest delta is measured, the configured estimate stays in force for paths the preparation did not reach, and an overlap factor covers supported concurrency. The larger value wins.

commit_measured_capacity() is the seam the engine will call, so the ordering preconditions live in one tested place: the report must describe the pool the storage already backs (report num_blocks equal to committed_blocks), the plan cannot exceed the reservation, and the storage must report at least the planned blocks as backed. It returns that count and the caller publishes exactly it. Two caller policies are enforced here rather than assumed:

  • the trial pool already backs at least one block, because a mapped pool cannot be built with no mapping at all (the storage raises from inside the driver otherwise);
  • a measured capacity that the reservation would truncate back to the pre-warmup estimate is an error, per decision 2 above.

This also contains the missing commit() bound in the storage implementation: an overshoot is rejected with the numbers rather than reaching cuMemMap.

Publish gate (vllm_rlt/core/kv_cache_manager.py): KVCacheManager accepts a capacity source whose tensors span the full reservation and whose backing grows through commit(blocks) -> backed_blocks.

  • Requests only receive published blocks, publish_capacity() re-reads the storage's committed count before growing the free pool, and published <= committed <= max is enforced.
  • A source that claims more committed blocks than it reserves is rejected.
  • FP8 over a mapped pool goes through _bind_scale_planes(): the pool must declare the format it carries, and the scales come from scale_tensors (K first, float32, shaped (max_blocks, layers, block_size, kv_heads)) or from a fixed fp8_scale. committed_bytes is expected to cover the data and the scale planes together.
  • Release stays reference-based, as agreed for feat(kv): add CUDA VMM KV storage with fixed addresses #194: the owner drops references, the manager never unmaps.
  • Without a capacity source the existing behaviour is unchanged (torch.empty, max == committed == published).

The branch also separates the two plane-setup paths (_open_torch_planes / _open_mapped_planes). No line from the FP8 work merged in #129 was dropped: the FP8-related lines were diffed against main, and the only differences are the mapped-path guard and the local storage -> storage_dtype rename that the new parameter required.

Test Plan

vllm-rlt Version / Commit: ab3a946 (based on ce200f0)

  • python -m ruff check . and python -m ruff format --check . (the cpu-tests.yml commands), run with the ruff version CI pins (0.13.2) as well as the locally installed one.
  • python -m pytest -q in a worktree of this head, after each commit and again after the rebase.
  • Device check with real weights (Ouro-1.4B BF16 snapshot, A100-SXM4-80GB): sample mem_get_info() at a drained boundary, take bytes_per_block and max_useful_blocks from the engine's own plan_cache, and run the planner with existing_blocks set to the pool that estimate already mapped.

Test Result

  • Ruff 0.13.2 (the pinned version): All checks passed!, 156 files already formatted.
  • CPU suite: 1307 passed, 32 failed. The same 32 tests fail on a clean checkout of ce200f0 on this host (test_async_speculative_sampling_plan.py 12, test_serving.py 18, test_pd.py 1, test_ouro.py 1), so this change adds no failures.
  • New tests: tests/test_measured_capacity.py, 43 cases, all expectations hand-computed from the KV layout and the stated margins (resident 20 GiB, cap 72 GiB, remaining 52 GiB, transient 4 GiB, safety 256 MiB, 3 MiB blocks, 16298 additional blocks). They cover the existing-prefix accounting, a bootstrap-only pool, reservation versus useful-block binding, the zero-transient-with-gaps rule, transient derivation from peaks, the commit seam's report/pool consistency check, a storage that silently under-commits, one that reports beyond its reservation, an oversubscribed or mismatched pool, the trial-pool requirement, and the FP8 hook (per-vector scales bound, fixed scale without scale planes, a pool missing both, and a pool that declares the wrong format).
  • Real weights: plan_cache maps 21852 blocks (3.0 MiB each). The planner returns 22894 additional blocks, 44746 total, and only the additional 67.07 GiB spends the 67.07 GiB budget. This shows the arithmetic on real numbers; it is not a capacity claim, because the sample is taken after model load rather than after warmup and capture.

Not run: any GPU execution path, because none changes here. The end-to-end A/B belongs to the integration.

Corrections from self-review

An earlier revision returned the measured delta as if it were the total; with the Ouro checkpoint that under-counts the pool by the 21852 already-mapped blocks. Review also closed the smaller holes listed above: no reservation clamp, a bootstrap-only pool failing although usable, a zero transient accepted while gaps were reported, a capacity source allowed to claim more committed blocks than it reserves, a trial pool with no backing at all, and a reservation that silently clamps the measurement back to the estimate. Each is covered by a test named after the behaviour.

Known and deliberately unchanged: the storage parameter shadows the name main uses for the FP8 tensor dtype (hence the local storage_dtype rename), and self.device is re-derived from the bound tensors, which would hide a backend/pool device mismatch rather than reject it.

Coordination

Limitations and next step

  • The report is not consumed by StageEngine yet: prepare_runtime() arrives with feat(runtime): prepare runners and CUDA Graphs before engine readiness #198 and the VMM storage with feat(kv): add CUDA VMM KV storage with fixed addresses #194. The agreed ordering (prepare_runtime -> plan -> commit -> publish -> replay and representative execution -> ready) and recording the storage kind plus any fallback reason in memory_plan are the next change, stacked on those two.
  • Because of that, the three helpers have no production caller yet: their evidence is the unit suite plus one real-number check, not an end-to-end run. That is why this PR stays draft until the integration lands.
  • PD/NIXL keeps torch.empty for the first integration and will record that reason.
  • No measured claim is made for long contexts, random block speculation or the full concurrent memory envelope; those stay estimated and are reported as gaps.

Turn the RFC InferMatrixLab#192 step-1 preparation report into a KV block count, and let
KVCacheManager grow its published range without reallocating tensors.

- measured_kv_capacity() reads the final drained phase of the report: the
  driver's remaining memory is the physical budget, so weights, execution
  buffers, captured graphs and any live mapping are never subtracted twice.
  The utilization cap still bounds the instance, and the transient allowance
  and safety reserve are explicit inputs that are recorded with the result.
  Coverage gaps reported by the preparation (long contexts, random
  speculation, unreported batch/history combinations) travel with the plan,
  so an estimated margin is never presented as a measured one.
- KVCacheManager accepts a capacity source whose tensors span the full
  reservation and whose physical backing grows by commit(). Requests are only
  ever handed published blocks, publishing requires the storage to report the
  blocks as backed, and published <= committed <= max is enforced, so no
  allocation can address unmapped memory. Release stays reference-based, as
  agreed for the storage contract: the owner drops references, the manager
  never unmaps.
- With no capacity source nothing changes: tensors come from torch.empty and
  max == committed == published.

Engine wiring (prepare_runtime -> plan -> commit -> publish -> replay and
representative execution -> ready) needs InferMatrixLab#198 for the report and InferMatrixLab#194 for the
storage, so it is the next step rather than part of this change.

Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
Move the tensor/scale allocation and the mapped-pool binding out of
KVCacheManager.__init__ into _open_torch_planes and _open_mapped_planes.

The FP8 capability rules, the allocation and the per-vector scale lifecycle now
sit in one method, and the mapped path states its own FP8 limitation instead of
being special-cased at the top of __init__. The capacity fields keep the same
meaning: both paths set max_blocks and committed_blocks, and __init__ publishes
only afterwards.

Behaviour is unchanged: the FP8 KV, backend-selection, Dory, prepared-KV and
KV-cache suites pass as before, the full CPU suite keeps the same 32
pre-existing failures, and no line from the FP8 work merged in InferMatrixLab#129 was dropped
(diffed the fp8-related lines against main; the only differences are the new
mapped-path guard and the local `storage` -> `storage_dtype` rename that the
new parameter required).

Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
measured_kv_capacity() reported the measured delta as if it were the total block
count. Driver free memory excludes the bootstrap prefix that is already mapped,
so an integration would have committed only the delta and under-counted the pool
by the bootstrap size: with the Ouro-1.4B configuration, 22894 instead of 44746
blocks, about 64 GiB of KV capacity.

- existing_blocks adds the already-mapped prefix to the measured delta and keeps
  it out of the free-memory sizing, so it is never counted twice.
- reservation_blocks clamps the result to what the pool can actually grow to,
  and the truncation is recorded as a coverage gap.
- limited_by reports which bound applied (budget, useful_blocks, reservation).
- A bootstrap-only pool is now accepted when no additional capacity fits,
  instead of failing although the existing prefix is usable.
- A zero transient allowance is rejected when the preparation reported coverage
  gaps, because an unexercised path cannot be sized as if it were measured.
- KVCacheManager rejects a capacity source that claims more committed blocks
  than it reserves.

Verified on the real checkpoint: existing 21852 blocks, additional 22894, total
44746, and only the additional part spends the 67.07 GiB budget.

Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
…ommit

Two pieces the agreed startup ordering needs before the engine can call it.

- transient_allowance() derives the margin from each phase's own peak minus its
  end state, keeps the larger of that and the configured estimate, and scales it
  by an overlap factor for supported concurrency. An unexercised path keeps the
  estimate instead of being sized as though it were measured.
- commit_measured_capacity() enforces the preconditions in one place: the report
  must describe the pool the storage already backs, the plan is clamped to the
  storage's reservation, and the storage has to report at least the planned
  blocks as backed. It returns that count, and the caller publishes exactly it.

The second one is also what keeps the missing commit() bound in a storage
implementation from turning into an overshoot: the plan cannot exceed the
reservation, and a storage that under- or over-commits is rejected with the
numbers in the message.

Covered by tests with hand-computed expectations, including a storage that
silently ignores the commit and one that reports beyond its reservation.

Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
The mapped-pool path rejected FP8 because a pool carried no per-vector
scales. Make the pool declare the format it carries and bind FP8 through an
explicit hook: a source that exposes scale_tensors (K first) supplies the
per-vector scales, and a fixed fp8_scale needs no per-vector storage. A pool
that declares neither is still rejected, but the error names the hook.

The measured commit now also enforces the two caller policies the ordering
depends on: the trial pool must already back at least one block, and a
measured capacity that the reservation would truncate back to the pre-warmup
estimate is an error instead of a silent clamp, since that clamp defeats the
purpose of measuring. allow_reservation_truncation opts back in.

Document that gpu_memory_utilization keeps capping a fraction of the whole
device rather than of post-baseline free memory.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant