biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Optimize KV cache distribution for asymmetric pipeline parallelism (#25164)

Browse Source 
Signed-off-by: gholmes829 <g.holmes429@gmail.com>
This commit is contained in:
Grant Holmes (Ren)
2025-10-07 04:20:30 -05:00
committed by GitHub
parent 7e4cd070b0
commit d100d78eb3
5 changed files with 64 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
vllm/entrypoints/llm.py
View File

@@ -143,7 +143,7 @@ class LLM:
size based on gpu_memory_utilization. However, users may want to
manually specify the kv cache memory size. kv_cache_memory_bytes
allows more fine-grain control of how much memory gets used when
compared with using gpu_memory_memory_utilization. Note that
compared with using gpu_memory_utilization. Note that
kv_cache_memory_bytes (when not-None) ignores
gpu_memory_utilization
swap_space: The size (GiB) of CPU memory per GPU to use as swap space.