biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

[BUG] Make 'binary' default option for saving torch compile artifacts when using standalone_compile (#27616)

Browse Source 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
This commit is contained in:
ahao-anyscale
2025-11-03 08:13:51 -08:00
committed by GitHub
parent f7d2946e99
commit cac4c10ef0
5 changed files with 43 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/design/torch_compile.md
View File

@@ -27,6 +27,8 @@ With all these factors taken into consideration, usually we can guarantee that t
A unique aspect of vLLM's `torch.compile` integration, is that we guarantee all the compilation finishes before we serve any requests. No requests will trigger new compilations. Otherwise, the engine would be blocked on that request, and the response time will have unexpected spikes.
By default, the cache saves compiled artifacts as binary files. If you would like to interact with the generated code for debugging purposes, set the field `compile_cache_save_format=unpacked` in the compilation config, or omit this and set the env variable `VLLM_COMPILE_CACHE_SAVE_FORMAT=unpacked`.
## Python Code Compilation
In the very verbose logs, we can see: