biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
6,623 Commits 2 Branches 140 Tags
20bd6f4d2ee10d6187d93e4bc6a64f78195243a2
Commit Graph

189 Commits

Author SHA1 Message Date
Satyajith Chilappagari
dc1440cf9f Neuron up mistral (#18222) 
Signed-off-by: Satyajith Chilappagari <satchill@amazon.com>
 2025-05-19 09:54:47 -07:00
Jee Jee Li
6781af5608 [Quantization] Pool model support bitsandbytes (#18087) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-05-19 09:03:43 -07:00
Shaoyu Yang
d637b96099 [BugFix] [Vul] Add missing usedforsecurity=False in MD5 hashing to enable FIPS (#18319) 
Signed-off-by: cascade812 <cascade812@outlook.com>
Signed-off-by: shaoyuyoung <shaoyuyoung@gmail.com>
Co-authored-by: cascade <cascade812@outlook.com>
 2025-05-19 01:31:23 -07:00
CYJiang
275c5daeb0 fix: Add type specifications for CLI arguments in tensorizer options (#18314) 2025-05-18 23:42:17 -07:00
Harry Mellor
07ad27121f Update deprecated type hinting in model_loader (#18130) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-15 04:00:21 -07:00
Tao He
60f7624334 Implements dual-chunk-flash-attn backend for dual chunk attention with sparse attention support (#11844) 2025-05-12 19:52:47 -07:00
Bowen Bao
db593aa67f [Quantization] Quark MXFP4 format loading (#16943) 2025-05-07 15:05:05 -04:00
Satyajith Chilappagari
043e4c4955 Add NeuronxDistributedInference support, Speculative Decoding, Dynamic on-device sampling (#16357) 
Signed-off-by: Satyajith Chilappagari <satchill@amazon.com>
Co-authored-by: Aaron Dou <yzdou@amazon.com>
Co-authored-by: Shashwat Srijan <sssrijan@amazon.com>
Co-authored-by: Chongming Ni <chongmni@amazon.com>
Co-authored-by: Amulya Ballakur <amulyaab@amazon.com>
Co-authored-by: Patrick Lange <patlange@amazon.com>
Co-authored-by: Elaine Zhao <elaineyz@amazon.com>
Co-authored-by: Lin Lin Pan <tailinpa@amazon.com>
Co-authored-by: Navyadhara Gogineni <navyadha@amazon.com>
Co-authored-by: Yishan McNabb <yishanm@amazon.com>
Co-authored-by: Mrinal Shukla <181322398+mrinalks@users.noreply.github.com>
 2025-05-07 00:07:30 -07:00
Jee Jee Li
ba7703e659 [Misc] Remove qlora_adapter_name_or_path (#17699) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-05-06 23:10:37 -07:00
Jee Jee Li
822de7fb94 [Misc] Split model loader (#17712) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-05-07 12:42:26 +08:00
Jerry Zhang
109e15a335 Add pt_load_map_location to allow loading to cuda (#16869) 
Signed-off-by: Jerry Zhang <jerryzh168@gmail.com>
 2025-05-01 23:23:42 -07:00
Aaron Pham
da4e7687b5 [Fix] Support passing args to logger (#17425) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
 2025-04-30 08:06:58 -07:00
Harry Mellor
13698db634 Improve configs - ModelConfig (#17130) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-04-30 10:38:22 +08:00
Harry Mellor
2c8ed8ee48 More informative error when using Transformers backend (#16988) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-04-23 19:54:03 -07:00
Harry Mellor
8e630d680e Improve Transformers backend model loading QoL (#17039) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-04-23 07:33:51 -07:00
Cyrus Leung
d6da9322c8 [Bugfix] Fix f-string for Python 3.9-3.11 (#16962) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-04-21 21:45:55 -07:00
omer-dayan
71ce44047f Support S3 Sharded loading with RunAI Model Streamer (#16317) 
Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-04-21 21:21:49 -07:00
Lily Liu
e8224f3dca [V1][Spec Decode] Eagle Model loading (#16035) 
Signed-off-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
 2025-04-10 11:21:48 -07:00
Chengji Yao
1621b25288 [TPU] Fix dummy loading OOM (#16372) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
 2025-04-10 04:06:16 +00:00
ajayvohra2005
24834f4894 update neuron config (#16289) 
Signed-off-by: Ajay Vohra <ajayvohr@amazon.com>
 2025-04-09 03:43:22 -07:00
Isotr0py
40b4284fe3 [Bugfix] Handle process_weights_after_loading for QKVCrossParallelLinear (#15328) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-04-08 10:02:23 -07:00
Lu Fang
55dcce91df Upstream Llama4 Support to Main (#16113) 
Signed-off-by: Aston Zhang <22279212+astonzhang@users.noreply.github.com>
Signed-off-by: Chris Thi <chris.c.thi@gmail.com>
Signed-off-by: drisspg <drisspguessous@gmail.com>
Signed-off-by: Jon Swenson <jmswen@gmail.com>
Signed-off-by: Keyun Tong <tongkeyun@gmail.com>
Signed-off-by: Lu Fang <fanglu@meta.com>
Signed-off-by: Xiaodong Wang <xdwang@meta.com>
Signed-off-by: Yang Chen <yangche@fb.com>
Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Signed-off-by: Lu Fang <lufang@fb.com>
Signed-off-by: Lu Fang <fanglu@fb.com>
Signed-off-by: Lucia Fang <fanglu@fb.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Lu Fang <fanglu@fb.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-04-07 08:06:27 -07:00
Harry Mellor
a76f547e11 Rename fallback model and refactor supported models section (#15829) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-03-31 22:49:41 -07:00
Harry Mellor
d4bfc23ef0 Fix Transformers backend compatibility check (#15290) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-03-31 10:27:07 -07:00
Jee Jee Li
726efc6a32 [Quantization][V1] BitsAndBytes support V1 (#15611) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-03-28 10:12:47 +08:00
Harry Mellor
cf5c8f1686 Separate base model from TransformersModel (#15467) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-03-26 18:13:38 +08:00
liuzhenwei
5eeadc2642 [Hardware][Gaudi][Feature] Enable Dynamic MoE for Mixtral (#12303) 
Signed-off-by: zhenwei <zhenweiliu@habana.ai>
 2025-03-24 09:48:40 -07:00
Manish Sethi
761702fd19 [Core] Integrate fastsafetensors loader for loading model weights (#10647) 
Signed-off-by: Manish Sethi <Manish.sethi1@ibm.com>
 2025-03-24 08:08:02 -07:00
ℍ𝕠𝕝𝕝𝕠𝕨 𝕄𝕒𝕟
948ab03e7e [Bugfix][V1] Avoid importing PreTrainedModel (#15366) 
Signed-off-by: Hollow Man <hollowman@opensuse.org>
 2025-03-24 10:33:12 +00:00
Jee Jee Li
3892e58ad7 [Misc] Upgrade BNB version (#15183) 2025-03-24 05:51:42 +00:00
Russell Bryant
b877031d80 Remove openvino support in favor of external plugin (#15339) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-03-22 14:06:39 -07:00
Tristan Leclercq
5eeabc2a44 [Bugfix] Fix bnb quantization for models with both HF-format and Mistral-format weights (#14950) 2025-03-17 23:27:26 +00:00
Robert Shaw
d4d93db2c5 [V1] V1 Enablement Oracle (#13726) 
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2025-03-14 22:02:20 -07:00
TY-AMD
128bf75283 [BugFix][TritonMLA] Process weights after model loading for GGUF (#14555) 
Signed-off-by: TianyuanWu <Tianyuan.Wu@amd.com>
 2025-03-12 20:14:36 -07:00
Aaron Pham
0b7f06b447 [Misc] add use_tqdm_on_load to reduce logs (#14407) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
 2025-03-08 05:57:46 -08:00
Harry Mellor
f7a6bd0fa1 Fix missing kv_caches and attn_metadata in OpenVINOCausalLM (#14271) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-03-07 12:30:42 +00:00
Jun Duan
82fbeae92b [Misc] Accurately capture the time of loading weights (#14063) 
Signed-off-by: Jun Duan <jun.duan.phd@outlook.com>
 2025-03-01 17:20:30 -08:00
Jee Jee Li
6a84164add [Bugfix] Add file lock for ModelScope download (#14060) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-03-01 06:10:28 +00:00
Szymon Ożóg
7f0be2aa24 [Model] Deepseek GGUF support (#13167) 2025-02-27 02:08:35 -08:00
cjackal
51010a1807 [Misc] set single whitespace between log sentences (#13771) 
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
 2025-02-25 10:26:12 +08:00
Mengqing Cao
23eca9cf68 [model][refactor] remove cuda hard code in models and layers (#13658) 2025-02-24 06:10:14 -08:00
Kevin H. Luu
2c5e637b57 [ci] Use env var to control whether to use S3 bucket in CI (#13634) 2025-02-22 19:19:45 -08:00
Helena Kloosterman
382f66fb08 [Bugfix] Fix boolean conversion for OpenVINO env variable (#13615) 2025-02-22 08:04:12 -08:00
Kevin H. Luu
473f51cfd9 [3/n][CI] Load Quantization test models with S3 (#13570) 
Signed-off-by: <>
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-20-117.us-west-2.compute.internal>
 2025-02-20 10:12:30 +08:00
Kevin H. Luu
d5d214ac7f [1/n][CI] Load models in CI from S3 instead of HF (#13205) 
Signed-off-by: <>
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-20-117.us-west-2.compute.internal>
 2025-02-19 07:34:59 +00:00
Isotr0py
8cf97f8661 [Bugfix] Fix failing transformers dynamic module resolving with spawn multiproc method (#13403) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-02-18 10:25:53 +00:00
Nick Hill
9076325677 [BugFix] Don't scan entire cache dir when loading model (#13302) 2025-02-14 21:33:31 -08:00
Michael Goin
f0b2da72a8 Expand MLA to support most types of quantization (#13181) 2025-02-13 22:19:22 -08:00
youkaichao
b2496bb07f [core] fix sleep mode and pytorch checkpoint compatibility (#13001) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-02-10 13:03:43 +08:00
Jun Duan
011e612d92 [Misc] Log time consumption on weight downloading (#12926) 2025-02-08 09:16:42 +00:00