biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

[Core] Run garbage collector after CUDA graph capture to fix throughput regression (#24128)

Browse Source 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Co-authored-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
This commit is contained in:
Micah Williamson
2025-09-09 09:38:10 -05:00
committed by GitHub
parent 922d3b401b
commit 1c63a16b65
1 changed files with 1 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
vllm/v1/worker/gpu_model_runner.py
View File

@@ -2885,6 +2885,7 @@ class GPUModelRunner(LoRAModelRunnerMixin, KVConnectorModelRunnerMixin):
finally: finally:
if should_freeze: if should_freeze:
gc.unfreeze() gc.unfreeze()
gc.collect()
# Trigger CUDA graph capture for specific shapes. # Trigger CUDA graph capture for specific shapes.
# Capture the large shapes first so that the smaller shapes # Capture the large shapes first so that the smaller shapes