Repository navigation
feat(memory): factor final KV capacity planning and record committed bytes - #193
kunkunblueberry wants to merge 1 commit into
Conversation
Move the startup KV budget arithmetic in plan_cache into kv_capacity(), a pure function over measured free memory, the utilization cap, and each non-KV overhead. The numbers are unchanged; the plan now also reports metadata_bytes and kv_committed_bytes so the committed capacity is observable. Covered by CPU unit tests. Part of RFC InferMatrixLab#192 (capacity planning). Storage and warmup/capture ordering are not changed.
ycjcx123
left a comment
There was a problem hiding this comment.
LGTM for this capacity-computation refactor, reviewed at 0b1daac against c338aaa.
The changes match the preparatory scope described in the PR. kv_capacity() preserves the existing budget formula, overhead accounting, block rounding and useful-capacity limit. Explicit capacity overrides and the CPU default remain unchanged.
An arithmetic equivalence check across 20,008 input cases produced matching capacities and error messages. CI reports 1057 passed and 744 skipped. Real CUDA auto-sizing remains unvalidated, as documented.
One non-blocking suggestion: please extend the existing plan_cache() tests to cover metadata_bytes and kv_committed_bytes. The new CPU tests exercise only the arithmetic helpers, so missing or incorrect output fields would go undetected. Check that metadata_bytes matches the deducted metadata overhead and that kv_committed_bytes matches the total bytes of the allocated K/V tensors.
|
Closing in favour of #200, which carries this piece forward. #193 factored the startup budget arithmetic out of No objection if a reviewer prefers to keep a separate "retire the fixed estimates" step; that would be a new PR on top of #200 rather than this one. |
Required human contributor answers
Answered and confirmed by the human contributor (@kunkunblueberry) before submission.
1. What outcome are you trying to achieve?
Size the KV pool from the memory actually available after runtime initialization, warmup and CUDA Graph capture, instead of relying on fixed per-pool estimates. This is the direction of RFC #192.
2. What is the highest-leverage variable for that outcome?
The number of committed KV blocks that remains once the measured free memory has been reduced by the utilization cap, the existing reservation and every non-KV overhead (probe peak, concurrent/async peak, graph reserve, static buffers, metadata).
3. How will this work move that variable rather than create busywork?
The startup budget arithmetic in
plan_cache()becomes a single pure function,kv_capacity(), over measured free memory, the utilization cap, the reservation and an explicit overhead map. A later measured-sizing change then has one place to change and a CPU-testable contract, and the plan now reportskv_committed_bytesandmetadata_bytes, making committed capacity observable. This PR is a preparatory step for RFC #192 step 1; it deliberately does not change the number of allocated blocks.4. Who is the customer or user, and what do they actually need?
Operators and maintainers deploying vllm-rlt on GPUs. They need capacity to be predictable and not to leave usable VRAM unused, or to run out of it, because of estimates that vary with the enabled execution paths.
PR necessity and bugfix evidence
A PR is needed: RFC #192 is an open, direction-defining RFC, the human contributor claimed the runtime KV capacity planning part in #192, and the outcome requires a code change. This PR is not a bugfix; there is no observed failure to link. It is the first, preparatory slice of the claimed scope.
Purpose
Part of RFC #192 (runtime KV cache capacity planning), split into several PRs. This PR covers capacity computation only:
plan_cache()intokv_capacity(), a pure function takingfree_bytes,cap_bytes,reserved_bytes,per_block,overheadsanduseful_blocks.min(free_bytes, cap - reserved) - sum(overheads), floored to whole blocks and capped byuseful_blocks. No capacity value changes for any existing configuration.plan_cache().metadata_bytesandkv_committed_bytesinmemory_plan, so the committed capacity (not only the budget before block rounding) is observable.Explicitly out of scope
gpu_memory_utilization,num_blocks/kv_cache_memory_bytesoverride semantics, orgraph_memory_reserve().These remain for follow-up PRs under the same RFC claim.
Test Plan
New
tests/test_kv_capacity.py(CPU-only, no GPU required):useful_blockscaps the block count without altering the reported budget;budget_blocks()floors partial blocks.Pre-existing suites are unchanged; all call sites still go through
plan_cache()with the same numbers.vllm-rlt Version / Commit: 0.2.0 /
0b1daacon top ofupstream/mainc338aaaTest Result
Environment: Windows, CPU-only
torch 2.12.1+cpu. No CUDA device was available, so thecuda_profilebranch ofplan_cache()was not executed. The extracted function is pure and fully covered on CPU; the CUDA path is a call-site refactor that passes the same values.python -m pytest tests/test_kv_capacity.py -q-> 5 passedpython -m pre_commit run ruff-check --all-files-> Passedpython -m pre_commit run ruff-format --all-files-> Passedpython -m pytest tests -q-> 1037 passed, 744 skipped, 20 failedThe 20 failures are pre-existing in this environment and unrelated to this change: the same 20 tests fail on a clean
upstream/main(c338aaa) checkout on the same host, and they are HTTP serving / control-channel tests affected by the Windows + CPU-only environment, not by memory planning. Thepre-commitpytesthook therefore fails here for the same environmental reason; the two ruff hooks pass.GPU validation of the end-to-end
cuda_profilepath (auto-sizing under a real CUDA device, and the existing GPU-marked tests) is still pending and will be done before this path is extended with measured post-capture sizing.