Repository navigation
feat(memory): size KV capacity from the post-warmup measurement - #200
Draft
kunkunblueberry wants to merge 5 commits into
Draft
kunkunblueberry wants to merge 5 commits into
kunkunblueberry wants to merge 5 commits into
Conversation
kunkunblueberry
force-pushed
the
kunkun/rfc192-measured-capacity
branch
from
November 19, 2026 15:20
5386a34 to
c63df73
Compare
Turn the RFC InferMatrixLab#192 step-1 preparation report into a KV block count, and let KVCacheManager grow its published range without reallocating tensors. - measured_kv_capacity() reads the final drained phase of the report: the driver's remaining memory is the physical budget, so weights, execution buffers, captured graphs and any live mapping are never subtracted twice. The utilization cap still bounds the instance, and the transient allowance and safety reserve are explicit inputs that are recorded with the result. Coverage gaps reported by the preparation (long contexts, random speculation, unreported batch/history combinations) travel with the plan, so an estimated margin is never presented as a measured one. - KVCacheManager accepts a capacity source whose tensors span the full reservation and whose physical backing grows by commit(). Requests are only ever handed published blocks, publishing requires the storage to report the blocks as backed, and published <= committed <= max is enforced, so no allocation can address unmapped memory. Release stays reference-based, as agreed for the storage contract: the owner drops references, the manager never unmaps. - With no capacity source nothing changes: tensors come from torch.empty and max == committed == published. Engine wiring (prepare_runtime -> plan -> commit -> publish -> replay and representative execution -> ready) needs InferMatrixLab#198 for the report and InferMatrixLab#194 for the storage, so it is the next step rather than part of this change. Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
Move the tensor/scale allocation and the mapped-pool binding out of KVCacheManager.__init__ into _open_torch_planes and _open_mapped_planes. The FP8 capability rules, the allocation and the per-vector scale lifecycle now sit in one method, and the mapped path states its own FP8 limitation instead of being special-cased at the top of __init__. The capacity fields keep the same meaning: both paths set max_blocks and committed_blocks, and __init__ publishes only afterwards. Behaviour is unchanged: the FP8 KV, backend-selection, Dory, prepared-KV and KV-cache suites pass as before, the full CPU suite keeps the same 32 pre-existing failures, and no line from the FP8 work merged in InferMatrixLab#129 was dropped (diffed the fp8-related lines against main; the only differences are the new mapped-path guard and the local `storage` -> `storage_dtype` rename that the new parameter required). Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
measured_kv_capacity() reported the measured delta as if it were the total block count. Driver free memory excludes the bootstrap prefix that is already mapped, so an integration would have committed only the delta and under-counted the pool by the bootstrap size: with the Ouro-1.4B configuration, 22894 instead of 44746 blocks, about 64 GiB of KV capacity. - existing_blocks adds the already-mapped prefix to the measured delta and keeps it out of the free-memory sizing, so it is never counted twice. - reservation_blocks clamps the result to what the pool can actually grow to, and the truncation is recorded as a coverage gap. - limited_by reports which bound applied (budget, useful_blocks, reservation). - A bootstrap-only pool is now accepted when no additional capacity fits, instead of failing although the existing prefix is usable. - A zero transient allowance is rejected when the preparation reported coverage gaps, because an unexercised path cannot be sized as if it were measured. - KVCacheManager rejects a capacity source that claims more committed blocks than it reserves. Verified on the real checkpoint: existing 21852 blocks, additional 22894, total 44746, and only the additional part spends the 67.07 GiB budget. Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
…ommit Two pieces the agreed startup ordering needs before the engine can call it. - transient_allowance() derives the margin from each phase's own peak minus its end state, keeps the larger of that and the configured estimate, and scales it by an overlap factor for supported concurrency. An unexercised path keeps the estimate instead of being sized as though it were measured. - commit_measured_capacity() enforces the preconditions in one place: the report must describe the pool the storage already backs, the plan is clamped to the storage's reservation, and the storage has to report at least the planned blocks as backed. It returns that count, and the caller publishes exactly it. The second one is also what keeps the missing commit() bound in a storage implementation from turning into an overshoot: the plan cannot exceed the reservation, and a storage that under- or over-commits is rejected with the numbers in the message. Covered by tests with hand-computed expectations, including a storage that silently ignores the commit and one that reports beyond its reservation. Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
kunkunblueberry
force-pushed
the
kunkun/rfc192-measured-capacity
branch
from
November 21, 2026 08:36
61987aa to
32a4a79
Compare
The mapped-pool path rejected FP8 because a pool carried no per-vector scales. Make the pool declare the format it carries and bind FP8 through an explicit hook: a source that exposes scale_tensors (K first) supplies the per-vector scales, and a fixed fp8_scale needs no per-vector storage. A pool that declares neither is still rejected, but the error names the hook. The measured commit now also enforces the two caller policies the ordering depends on: the trial pool must already back at least one block, and a measured capacity that the reservation would truncate back to the pre-warmup estimate is an error instead of a silent clamp, since that clamp defeats the purpose of measuring. allow_reservation_truncation opts back in. Document that gpu_memory_utilization keeps capping a fraction of the whole device rather than of post-baseline free memory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the capacity-planning part of RFC #192 assigned to me in the issue, on the terms agreed there: the planner consumes the step-1 preparation report, and the logical allocator only ever receives fully backed blocks. Rebased onto
ce200f0.Required human contributor answers
TODO(contributor): the four answers and the PR-necessity decision are reserved for the human contributor; the recorded split in RFC #192 describes the agreed scope but is not a substitute for them.
1. What outcome are you trying to achieve?
TODO(contributor)
2. What is the highest-leverage variable for that outcome?
TODO(contributor)
3. How will this work move that variable rather than create busywork?
TODO(contributor)
4. Who is the customer or user, and what do they actually need?
TODO(contributor)
PR necessity and bugfix evidence
TODO(contributor)
Decisions recorded with the maintainer
The four open questions in this thread were settled as follows, and the code now states them explicitly instead of leaving them to the caller:
useful_blocks), not from the pre-warmup estimate. This is the point of the change: a reservation derived from the estimate can never grow, so the measurement would be truncated back to the number it was meant to replace. A reservation that truncates the measured capacity is now an error rather than a silent clamp;allow_reservation_truncation=Trueopts back in when that is genuinely intended. Reserving the useful cap costs address space, not physical memory.scale_tensors(K first) for per-vector scales or is used with a fixedfp8_scale, which needs no per-vector storage. A source that declares FP8 and supplies neither is still rejected, but the message names the hook.gpu_memory_utilizationkeeps its existing meaning: a fraction of the whole device, not of the free memory left after the baseline. Only the docstring changes.Purpose
Three pieces the measured-sizing path needs. None of them depends on the unmerged orchestration in #198 or the storage in #194, and each is usable on its own.
measured_kv_capacity()(vllm_rlt/core/memory.py) turns the final drained phase of the preparation report into a block count.total - freeonly bounds the remaining instance budget undergpu_memory_utilization.existing_blocksis the already-mapped bootstrap prefix, taken from the report'snum_blocks. It extends the measured delta and is never sized from free memory again, so the returned count is the total the pool may hold and a cumulativecommit(total)maps only the new part.reservation_blocksclamps the result to what the virtual reservation can back; truncation is recorded as a coverage gap, andlimited_bynames the binding bound (budget,useful_blocks,reservation).transient_allowance()derives that margin from the report instead of leaving it to the caller: each phase'speak_allocated_bytes - allocated_bytesis what the phase needed above what it kept, the largest delta is measured, the configured estimate stays in force for paths the preparation did not reach, and an overlap factor covers supported concurrency. The larger value wins.commit_measured_capacity()is the seam the engine will call, so the ordering preconditions live in one tested place: the report must describe the pool the storage already backs (reportnum_blocksequal tocommitted_blocks), the plan cannot exceed the reservation, and the storage must report at least the planned blocks as backed. It returns that count and the caller publishes exactly it. Two caller policies are enforced here rather than assumed:This also contains the missing
commit()bound in the storage implementation: an overshoot is rejected with the numbers rather than reachingcuMemMap.Publish gate (
vllm_rlt/core/kv_cache_manager.py):KVCacheManageraccepts a capacity source whosetensorsspan the full reservation and whose backing grows throughcommit(blocks) -> backed_blocks.publish_capacity()re-reads the storage's committed count before growing the free pool, andpublished <= committed <= maxis enforced._bind_scale_planes(): the pool must declare the format it carries, and the scales come fromscale_tensors(K first,float32, shaped(max_blocks, layers, block_size, kv_heads)) or from a fixedfp8_scale.committed_bytesis expected to cover the data and the scale planes together.torch.empty,max == committed == published).The branch also separates the two plane-setup paths (
_open_torch_planes/_open_mapped_planes). No line from the FP8 work merged in #129 was dropped: the FP8-related lines were diffed againstmain, and the only differences are the mapped-path guard and the localstorage->storage_dtyperename that the new parameter required.Test Plan
vllm-rlt Version / Commit:
ab3a946(based once200f0)python -m ruff check .andpython -m ruff format --check .(thecpu-tests.ymlcommands), run with the ruff version CI pins (0.13.2) as well as the locally installed one.python -m pytest -qin a worktree of this head, after each commit and again after the rebase.mem_get_info()at a drained boundary, takebytes_per_blockandmax_useful_blocksfrom the engine's ownplan_cache, and run the planner withexisting_blocksset to the pool that estimate already mapped.Test Result
All checks passed!,156 files already formatted.1307 passed, 32 failed. The same 32 tests fail on a clean checkout ofce200f0on this host (test_async_speculative_sampling_plan.py12,test_serving.py18,test_pd.py1,test_ouro.py1), so this change adds no failures.tests/test_measured_capacity.py, 43 cases, all expectations hand-computed from the KV layout and the stated margins (resident 20 GiB, cap 72 GiB, remaining 52 GiB, transient 4 GiB, safety 256 MiB, 3 MiB blocks, 16298 additional blocks). They cover the existing-prefix accounting, a bootstrap-only pool, reservation versus useful-block binding, the zero-transient-with-gaps rule, transient derivation from peaks, the commit seam's report/pool consistency check, a storage that silently under-commits, one that reports beyond its reservation, an oversubscribed or mismatched pool, the trial-pool requirement, and the FP8 hook (per-vector scales bound, fixed scale without scale planes, a pool missing both, and a pool that declares the wrong format).plan_cachemaps 21852 blocks (3.0 MiB each). The planner returns 22894 additional blocks, 44746 total, and only the additional 67.07 GiB spends the 67.07 GiB budget. This shows the arithmetic on real numbers; it is not a capacity claim, because the sample is taken after model load rather than after warmup and capture.Not run: any GPU execution path, because none changes here. The end-to-end A/B belongs to the integration.
Corrections from self-review
An earlier revision returned the measured delta as if it were the total; with the Ouro checkpoint that under-counts the pool by the 21852 already-mapped blocks. Review also closed the smaller holes listed above: no reservation clamp, a bootstrap-only pool failing although usable, a zero transient accepted while gaps were reported, a capacity source allowed to claim more committed blocks than it reserves, a trial pool with no backing at all, and a reservation that silently clamps the measurement back to the estimate. Each is covered by a test named after the behaviour.
Known and deliberately unchanged: the
storageparameter shadows the namemainuses for the FP8 tensor dtype (hence the localstorage_dtyperename), andself.deviceis re-derived from the bound tensors, which would hide a backend/pool device mismatch rather than reject it.Coordination
measured_kv_capacity().Limitations and next step
StageEngineyet:prepare_runtime()arrives with feat(runtime): prepare runners and CUDA Graphs before engine readiness #198 and the VMM storage with feat(kv): add CUDA VMM KV storage with fixed addresses #194. The agreed ordering (prepare_runtime -> plan -> commit -> publish -> replay and representative execution -> ready) and recording the storage kind plus any fallback reason inmemory_planare the next change, stacked on those two.torch.emptyfor the first integration and will record that reason.