biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
8,158 Commits 2 Branches 140 Tags
ec02e536dfa46c7e8785cb5aaf5dd4eaad88f405
Commit Graph

417 Commits

Author SHA1 Message Date
Yong Hoon Shin
ad510309ee Override attention metadata for fast prefill in some KV sharing setups (#21590) 
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
 2025-07-30 08:54:15 -07:00
Reza Barazesh
37efc63b64 [V0 deprecation] Guided decoding (#21347) 
Signed-off-by: Reza Barazesh <rezabarazesh@meta.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-07-29 03:15:30 -07:00
Harry Mellor
94b71ae106 Use metavar to list the choices for a CLI arg when custom values are also accepted (#21760) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-07-28 19:31:10 +00:00
Cyrus Leung
86ae693f20 [Deprecation][2/N] Replace --task with --runner and --convert (#21470) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-07-27 19:42:40 -07:00
Maximilien de Bayser
1cd6eaba54 Support encoder-only models without KV-Cache (#21270) 
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
 2025-07-26 21:09:52 +08:00
22quinn
610852a423 [Core] Support model loader plugins (#21067) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-07-24 01:49:44 -07:00
Robert Shaw
d5b981f8b1 [DP] Internal Load Balancing Per Node [one-pod-per-node] (#21238) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2025-07-23 20:57:32 -07:00
Michael Goin
82ec66f514 [V0 Deprecation] Remove Prompt Adapters (#20588) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-07-23 16:36:48 -07:00
Lu Fang
accac82928 [Sampler] Introduce logprobs mode for logging (#21398) 
Signed-off-by: Lu Fang <lufang@fb.com>
 2025-07-23 01:39:25 -07:00
Kebe
bc8a8ce5ec [Misc] Remove deprecated args in v0.10 (#21349) 
Signed-off-by: Kebe <mail@kebe7jun.com>
 2025-07-22 05:26:39 -07:00
Konrad Zawora
c17231e827 Fix kv_cache_dtype handling for out-of-tree HPU plugin (#21302) 
Signed-off-by: Konrad Zawora <kzawora@habana.ai>
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
Co-authored-by: Chendi.Xue <chendi.xue@intel.com>
 2025-07-21 23:35:14 -07:00
Li, Jiang
a15a50fc17 [CPU] Enable shared-memory based pipeline parallel for CPU backend (#21289) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2025-07-21 09:07:08 -07:00
Chengji Yao
3a1d8940ae [TPU] support fp8 kv cache quantization (#19292) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
 2025-07-20 03:01:00 +00:00
Sungjae Lee
da6579bf41 [CI/CD][bugfix]fix: error argument to loads has incompatible type (#21223) 
Signed-off-by: Sungjae Lee <33976427+llsj14@users.noreply.github.com>
Signed-off-by: Sungjae Lee <sung-jae.lee@navercorp.com>
 2025-07-19 05:16:48 -07:00
Jee Jee Li
1eaff27815 [V0 deprecation] Remove long context LoRA (#21169) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-07-19 02:15:41 -07:00
Woosuk Kwon
dd572c0ab3 [V0 Deprecation] Remove V0 Spec Decode workers (#21152) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-07-18 21:47:50 -07:00
Woosuk Kwon
4de7146351 [V0 deprecation] Remove V0 HPU backend (#21131) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-07-17 16:37:36 -07:00
Nir David
01513a334a Support FP8 Quantization and Inference Run on Intel Gaudi (HPU) using INC (Intel Neural Compressor) (#12010) 
Signed-off-by: Nir David <ndavid@habana.ai>
Signed-off-by: Uri Livne <ulivne@habana.ai>
Co-authored-by: Uri Livne <ulivne@habana.ai>
 2025-07-16 15:33:41 -04:00
Harry Mellor
1e36c8687e [Deprecation] Remove nullable_kvs (#20969) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-07-15 17:21:50 +00:00
Harry Mellor
313ae8c16a [Deprecation] Remove everything scheduled for removal in v0.10.0 (#20979) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-07-15 15:57:53 +00:00
Harry Mellor
56fe4bedd6 [Deprecation] Remove TokenizerPoolConfig (#20968) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-07-15 14:00:50 +00:00
Woosuk Kwon
d4d309409f Implement Async Scheduling (#19970) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-07-14 23:01:46 -07:00
Pavani Majety
9ad0a4588b [Bugfix] Switch bailout logic for kv-cache-dtype with SM100 Flashinfer (#20934) 
Signed-off-by: Pavani Majety <pmajety@nvidia.com>
 2025-07-15 03:27:50 +00:00
Pavani Majety
7bd4c37ae7 [Core] Add Flashinfer TRTLLM Backend for Flashinfer decode path (SM100). (#19825) 
Signed-off-by: Pavani Majety <pmajety@nvidia.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: shuw <shuw@nvidia.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-07-11 09:23:23 +00:00
Alex Brooks
41060c6e08 [Core] Add Support for Default Modality Specific LoRAs [generate / chat completions] (#19126) 
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
 2025-07-10 21:09:37 +01:00
Harry Mellor
3482fd7e4e [Doc] Add engine args back in to the docs (#20674) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-07-10 08:02:40 -07:00
Sanger Steel
4ac9c33f78 [Bugfix] Fix handling of Tensorizer arguments for LoadConfig (#20643) 
Signed-off-by: Sanger Steel <sangersteel@gmail.com>
 2025-07-09 15:36:37 +00:00
Akash kaothalkar
6db31e7a27 [Hardware][PPC64LE] Enable V1 for ppc64le and ARM (#20554) 
Signed-off-by: Akash Kaothalkar <akash.kaothalkar@ibm.com>
Co-authored-by: Akash Kaothalkar <akash.kaothalkar@ibm.com>
Co-authored-by: Nikhil Gupta <nikhil.gupta2@arm.com>
 2025-07-08 20:00:41 -07:00
Sanger Steel
72d14d0eed [Frontend] [Core] Integrate Tensorizer in to S3 loading machinery, allow passing arbitrary arguments during save/load (#19619) 
Signed-off-by: Sanger Steel <sangersteel@gmail.com>
Co-authored-by: Eta <esyra@coreweave.com>
 2025-07-07 22:47:43 -07:00
Anton
e601efcb10 [Misc] Add fully interleaved support for multimodal 'string' content format (#14047) 
Signed-off-by: drobyshev.anton <drobyshev.anton@wb.ru>
Co-authored-by: drobyshev.anton <drobyshev.anton@wb.ru>
 2025-07-07 19:43:08 +00:00
Isotr0py
32c9be2200 [v1] Re-add fp32 support to v1 engine through FlexAttention (#19754) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-07-05 09:41:10 +00:00
Cyrus Leung
b024a42e93 [Core] Move multimodal placeholder from chat utils to model definition (#20355) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-07-03 08:18:30 +00:00
Nick Hill
657f2f301a [DP] Support external DP Load Balancer mode (#19790) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-07-02 10:21:52 -07:00
Chenheli Hua
2e7cbf2d7d [Frontend] Support configurable mm placeholder strings & flexible video sampling policies via CLI flags. (#20105) 
Signed-off-by: Chenheli Hua <huachenheli@outlook.com>
 2025-07-01 23:34:03 -07:00
Luka Govedič
6d42ce8315 [CLI] Improve CLI arg parsing for -O/--compilation-config (#20156) 
Signed-off-by: luka <luka@neuralmagic.com>
 2025-07-01 01:03:13 +00:00
Bowen Wang
e9fd658a73 [Feature] Expert Parallelism Load Balancer (EPLB) (#18343) 
Signed-off-by: Bowen Wang <abmfy@icloud.com>
 2025-06-26 15:30:21 -07:00
amit
981eeca41a [Fix][V1] Remove --scheduling-policy oracle (#20010) 
Signed-off-by: amit <amit.man@gmail.com>
 2025-06-24 09:52:15 -07:00
Aaron Pham
c4cf260677 [Perf][CLI] Improve overall startup time (#19941) 2025-06-22 23:11:22 +00:00
Li, Jiang
79f2f1c2a1 [CPU][CI] Fallback sliding window to v0 and fix CPU pooling model tests (#19901) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2025-06-20 15:30:36 +00:00
Maximilien de Bayser
799397ee4f Support embedding models in V1 (#16188) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-06-18 21:36:33 -07:00
Chen Zhang
a89209b78d [v1] Support mamba2 (#19327) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-06-18 20:34:15 +00:00
wangxiyuan
257ab95439 [Platform] Allow platform use V1 Engine by default (#19792) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2025-06-18 13:03:36 +00:00
Ning Xie
6e9cc73f67 [MISC] correct DeviceConfig device field static type analysis (#19699) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-06-17 17:21:50 -07:00
Li, Jiang
6458721108 [CPU] Refine default config for the CPU backend (#19539) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2025-06-13 13:27:39 +08:00
rasmith
c7ea0b56cd [AMD] [Quantization] Add override flag for attention dtype instead of using kv_cache_dtype trigger (#17331) 
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
 2025-06-11 15:53:28 -04:00
Cyrus Leung
68b4a26149 [Doc] Update V1 User Guide for Hardware and Models (#19474) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-06-11 00:49:06 -07:00
Gregory Shtrasberg
5241ca50d6 [ROCm][V1] Adding ROCm to the list of plaforms using V1 by default (#19440) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
 2025-06-10 22:06:15 +00:00
Isotr0py
5f1ac1e1d1 Revert "[v1] Add fp32 support to v1 engine through flex attn" (#19404) 2025-06-10 01:30:20 -07:00
Isotr0py
b8089195b4 [v1] Add fp32 support to v1 engine through flex attn (#19319) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-06-09 22:10:44 +08:00
Driss Guessous
cf02f9b283 Add FlexAttention to V1 (#16078) 
Signed-off-by: drisspg <drisspguessous@gmail.com>
 2025-06-06 21:58:55 -07:00