biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
14,549 Commits 2 Branches 140 Tags
00bd08edeee5dd4d4c13277c0114a464011acf72
Commit Graph

1054 Commits

Author SHA1 Message Date
Mark McLoughlin
27066d1b2b [Frontend][Core] Add shutdown timeout - allowing in-flight requests to finish (#34730) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-03-05 22:04:31 -08:00
Shiyan Deng
03a49bb8f0 [Feature] Add --distributed-timeout-seconds CLI option (#36047) 
Signed-off-by: Shiyan Deng <dsy842974287@meta.com>
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
 2026-03-05 20:57:51 -08:00
Nicolò Lucchesi
18e01a0a10 [Misc] Add --attention-backend auto option (#35738) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-03-04 15:12:27 +00:00
sungsoo ha
6cb901093f [Core] Add All-to-All communication backend for DCP (#34883) 
Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
Signed-off-by: sungsoo ha <hasungsoo@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-04 10:01:57 -05:00
Cyrus Leung
792a74b973 [Doc] Improve UX of --enable-log-requests (#35723) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-03-02 08:24:09 -08:00
Itay Alroy
dea268336f [1/N] Elastic EP Milestone 2 (#34861) 
Signed-off-by: Yongji Wu <wuyongji317@gmail.com>
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Signed-off-by: Ron Tourgeman <rtourgeman@nvidia.com>
Co-authored-by: Yongji Wu <wuyongji317@gmail.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Ron Tourgeman <rtourgeman@nvidia.com>
 2026-02-28 04:46:42 +00:00
Akash kaothalkar
e03ddcfbd4 [Hardware][Powerpc]Enable prefix caching and chunked prefill for ppc64le (#35081) 
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
Co-authored-by: Akash kaothalkar <akash.kaothalkar@ibm.com>
 2026-02-26 10:21:24 +00:00
Michael Goin
de527e1cec [UX] Add --moe-backend arg for explicit kernel selection (#33807) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2026-02-25 17:44:44 -08:00
Michael Goin
cbf8f7028c [UX] Add --performance-mode {balanced,interactivity,throughput} (#34936) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-02-25 17:28:31 -08:00
Ming Yang
6831650c40 [offloader] v2: Hide weight onloading latency via prefetching (#29941) 
Signed-off-by: Ming Yang <minos.future@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2026-02-25 17:20:59 -08:00
Nick Hill
dbf0da817a [Core] Cleanup engine pause/sleep logic (#34528) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-24 19:33:34 -08:00
R3hankhan
34ce0ffd1f [CPU][Perf] Accelerate Attention head for s390x using vector intrinsics (#34434) 
Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
 2026-02-24 07:25:39 -08:00
Lucas Wilkinson
aaefc58ee0 [CI] Revert PRs 34818 and 33600 (#34979) 2026-02-20 13:25:50 -08:00
Matthew Bonanni
7743152957 [Attention] Refactor check_and_update_config (#33600) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-02-17 17:06:54 -08:00
Cyrus Leung
574fe75245 [Renderer] Move InputPreprocessor into Renderer (2/2) (#34560) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-17 05:29:01 -08:00
Lucas Kabela
a3205beffb [CI] Enable mypy coverage for individual excluded files (#34292) 
Signed-off-by: Lucas Kabela <lucaskabela@meta.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-02-16 07:34:29 -08:00
Cyrus Leung
73391a1baa [Renderer] Move InputPreprocessor into Renderer (1/2) (#34510) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2026-02-14 10:14:21 -08:00
Wei Zhao
b37b679770 [Feature][Perf] Support Selective CPU Weight Offloading (#34535) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
 2026-02-13 20:02:24 -08:00
Cyrus Leung
fb455ed547 [V0 Deprecation] Remove code related to per-request logits processors (#34400) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-12 20:44:28 +08:00
Cyrus Leung
b96f7314b4 [Refactor] Pass Renderer to Input Processor (#34329) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-11 19:38:11 -08:00
Harry Mellor
40b8f55358 [Docs] Reduce time spent generating API docs (#34255) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-02-11 02:56:02 -08:00
Gregory Shtrasberg
f0ca0671c7 [Feature] Warn about unrecognized environment variables (#33581) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
 2026-02-10 15:45:38 -06:00
Roger Wang
64a9c2528b [UX] Add --language-model-only for hybrid models (#34120) 
Signed-off-by: Roger Wang <hey@rogerw.io>
 2026-02-09 14:57:33 +00:00
Mohammad Miadh Angkad
dd6a6e1190 [Kernel] Add KernelConfig flag to enable/disable FlashInfer autotune (#34006) 
Signed-off-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
 2026-02-07 05:24:44 -08:00
Aaron Hao
89a385d79f [Feat][RL] Pause and Resume with keep requests for single engine (#32351) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
Signed-off-by: Aaron Hao <ahao@anyscale.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2026-02-07 00:08:58 +00:00
Cyrus Leung
cd8b405bd0 [Refactor] Consolidate sequence normalization and enc-dec parsing (#33928) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-06 15:43:47 +00:00
Harry Mellor
51a7bda625 Update WeightTransferConfig to be more standard like the others (#33989) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-02-06 13:15:00 +00:00
Aaron Hao
c1858b7ec8 [Feat][RL][1/2] Native Weight Syncing API: NCCL (#31943) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
Signed-off-by: Aaron Hao <ahao@anyscale.com>
Co-authored-by: SumanthRH <sumanthrh99@gmail.com>
 2026-02-05 12:13:23 -05:00
yugong333
ffe1fc7a28 Reduce the kernel overhead when num of active loras is smaller than max loras. Multiple cuda graphs are captured for each num of active-loras. (#32005) 
Signed-off-by: Yu Gong <yu3.gong@gmail.com>
 2026-02-02 12:30:06 -05:00
Cyrus Leung
a358e4dffe [Refactor] Make Renderer an abstract class (#33479) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-01 10:36:30 +08:00
jma99_2333
22d9a056d5 Support clear mm and encoder cache (#33452) 
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-01-31 15:22:25 +00:00
Cyrus Leung
f0a1c8453a [Frontend] Use new Renderer for Completions and Tokenize API (#32863) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-31 04:51:15 -08:00
Patrick von Platen
10152d2194 [Realtime API] Adds minimal realtime API based on websockets (#33187) 
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-01-30 18:41:29 +08:00
Harry Mellor
d334dd26c4 Move decode context parallel validationn to ParallelConfig (#33239) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-01-30 06:18:41 +00:00
linhaifeng
f5d7049cc1 [Bugfix] Fix display error (inconsistent with context) (#33020) 
Signed-off-by: linhaifeng <1371675203@qq.com>
 2026-01-27 20:33:29 +00:00
Harry Huang
5206e5e28c [V1][Hybrid] Mamba Prefix Caching with align mode (#30877) 
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
 2026-01-23 09:56:48 -08:00
Cyrus Leung
d117a4d1a9 [Frontend] Introduce Renderer for processing chat messages (using ModelConfig) (#30200) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-22 12:44:22 +00:00
Cyrus Leung
4753f3bf69 [Model] Use context managers for encoder- and LM-only mode (#32605) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-20 11:43:38 +08:00
Tomas Ruiz
4a5299c93f feat: spec decode with draft models (#24322) 
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
 2026-01-19 16:05:46 -05:00
Michael Goin
1be5a73571 [UX] Use kv_offloading_backend=native by default (#32421) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-01-15 18:55:11 +00:00
Yi Liu
50632adc58 Consolidate Intel Quantization Toolkit Integration in vLLM (#31716) 
Signed-off-by: yiliu30 <yi4.liu@intel.com>
 2026-01-14 07:11:30 +00:00
Nick Hill
9273a427b5 [Misc] Allow enabling NCCL for DP sync when async scheduling (#32197) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-13 02:03:08 +00:00
Hongxin Xu
49e6b86c91 [Feature] Support recording expert indices for rollout router replay (#28284) 
Signed-off-by: xhx1022 <1737006628@qq.com>
Signed-off-by: Hongxin Xu <70438206+xhx1022@users.noreply.github.com>
Signed-off-by: arlenxu <arlenxu@tencent.com>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: arlenxu <arlenxu@tencent.com>
 2026-01-12 06:23:04 -08:00
Matthew Bonanni
2612ba9285 [1/N][Attention] Restructure attention: move files (#31916) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-01-09 13:10:24 -08:00
Max Hu
6ebe34d6fa [Feature] Add iteration level logging and enhance nvtx marker (#31193) 
Signed-off-by: Max Hu <maxhu@nvidia.com>
Signed-off-by: Max Hu <hyoung2991@gmail.com>
Co-authored-by: Max Hu <maxhu@nvidia.com>
 2026-01-09 00:13:39 +00:00
Robert Shaw
5825bbc1f7 [Quantization] Deprecate Long Tail of Schemes (#31688) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-01-08 19:07:45 -05:00
omer-dayan
04a49669d1 RayLLM Bugfix - Preserve obj store URL for multi engine_config creation (#30803) 
Signed-off-by: Omer Dayan <omdayan@nvidia.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-01-08 10:00:25 +00:00
prashanth058
d3235cb503 [Fix] Enable mm_processor_cache with vision LoRA (#31927) 
Signed-off-by: prashanth058 <prashanth.dannamaneni@uipath.com>
 2026-01-08 15:31:51 +08:00
Michael Goin
a68e703c32 [UX] Add -ep shorthand for --enable-expert-parallel (#30890) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-01-05 19:13:36 -08:00
Nick Hill
bd877162eb [BugFix] Support online dense model DP without overhead (#30739) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: njhill <nickhill123@gmail.com>
 2026-01-02 23:36:38 +08:00