biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

[Nixl][P/D] Add cuda2cpu support (HD->DH transfer) (#24690)

Browse Source 
Signed-off-by: Chenxi Yang <cxyang@fb.com>
Co-authored-by: Chenxi Yang <cxyang@fb.com>
This commit is contained in:
Chenxi Yang
2025-09-29 07:31:51 -07:00
committed by GitHub
parent 43227236ec
commit d0d138bc55
6 changed files with 96 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
vllm/config/kv_transfer.py
View File

@@ -28,8 +28,8 @@ class KVTransferConfig:
"""The engine id for KV transfers."""
kv_buffer_device: Optional[str] = "cuda"
"""The device used by kv connector to buffer the KV cache.
Currently only support 'cuda'."""
"""The device used by kv connector to buffer the KV cache. Choices are
'cuda' and 'cpu'."""
kv_buffer_size: float = 1e9
"""The buffer size for TorchDistributedConnector. Measured in number of