biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
3,782 Commits 2 Branches 140 Tags
1efce686053c15cd6f84361bb0bd1898fbb23a82
Commit Graph

53 Commits

Author SHA1 Message Date
Gene Der Su
82c73fd510 [Bugfix] cuda error running llama 3.2 (#11047) 2024-12-10 07:41:11 +00:00
Tyler Michael Smith
28b3a1c7e5 [V1] Multiprocessing Tensor Parallel Support for v1 (#9856) 
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2024-12-10 06:28:14 +00:00
Gregory Shtrasberg
b63ba84832 [ROCm][bugfix] scpecilative decoding worker class (#11035) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
 2024-12-09 14:00:29 -08:00
wangxiyuan
aea2fc38c3 [Platform] Move async output check to platform (#10768) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2024-12-09 17:24:46 +00:00
Cyrus Leung
aa39a8e175 [Doc] Create a new "Usage" section (#10827) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-05 11:19:35 +08:00
Michael Goin
7090c27bb2 [Bugfix] Only require XGrammar on x86 (#10865) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-03 10:32:21 -08:00
wangxiyuan
661175bc82 [platform] Add verify_quantization in platform. (#10757) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2024-11-29 15:22:21 +00:00
Chendi.Xue
0a71900bc9 Remove hard-dependencies of Speculative decode to CUDA workers (#10587) 
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
 2024-11-26 17:57:11 -08:00
Conroy Cheers
f5792c7c4a [Hardware][NVIDIA] Add non-NVML CUDA mode for Jetson (#9735) 
Signed-off-by: Conroy Cheers <conroy@corncheese.org>
 2024-11-26 10:26:28 -08:00
Isotr0py
04668ebe7a [Bugfix] Avoid import AttentionMetadata explicitly in Mllama (#10593) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2024-11-23 18:12:20 +00:00
JiHuazhong
86a44fb896 [Platforms] Refactor openvino code (#10573) 
Signed-off-by: statelesshz <hzji210@gmail.com>
 2024-11-22 22:23:12 -08:00
youkaichao
eebad39f26 [torch.compile] support all attention backends (#10558) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-22 14:04:42 -08:00
youkaichao
a111d0151f [platforms] absorb worker cls difference into platforms folder (#10555) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2024-11-21 21:00:32 -08:00
youkaichao
cf656f5a02 [misc] improve error message (#10553) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-21 13:13:17 -08:00
youkaichao
aaddce5d26 [platforms] improve error message for unspecified platforms (#10520) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-20 23:07:56 -08:00
Mengqing Cao
9d827170a3 [Platforms] Add device_type in Platform (#10508) 
Signed-off-by: MengqingCao <cmq0113@163.com>
 2024-11-21 04:44:20 +00:00
youkaichao
388ee3de66 [torch.compile] limit inductor threads and lazy import quant (#10482) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-20 18:36:33 -08:00
youkaichao
772a66732d [platforms] restore xpu check for parallel config (#10479) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-20 17:13:28 +00:00
Li, Jiang
63f1fde277 [Hardware][CPU] Support chunked-prefill and prefix-caching on CPU (#10355) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2024-11-20 10:57:39 +00:00
Mengqing Cao
d5b28447e0 [Platforms] Refactor xpu code (#10468) 
Signed-off-by: MengqingCao <cmq0113@163.com>
 2024-11-19 22:52:13 -08:00
youkaichao
803f37eaaa [6/N] torch.compile rollout to users (#10437) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-19 10:09:03 -08:00
Mengqing Cao
8c1fb50705 [Platform][Refactor] Extract func get_default_attn_backend to Platform (#10358) 
Signed-off-by: Mengqing Cao <cmq0113@163.com>
 2024-11-19 11:22:26 +08:00
youkaichao
51bb12d17b [4/N][torch.compile] clean up set_torch_compile_backend (#10401) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-17 23:57:20 -08:00
youkaichao
8d74b5aee9 [platforms] refactor cpu code (#10402) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-16 23:14:23 -08:00
youkaichao
4fd9375028 [2/N][torch.compile] make compilation cfg part of vllm cfg (#10383) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-16 18:02:14 -08:00
Konrad Zawora
a02a50e6e5 [Hardware][Intel-Gaudi] Add Intel Gaudi (HPU) inference backend (#6143) 
Signed-off-by: yuwenzho <yuwen.zhou@intel.com>
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
Signed-off-by: Bob Zhu <bob.zhu@intel.com>
Signed-off-by: zehao-intel <zehao.huang@intel.com>
Signed-off-by: Konrad Zawora <kzawora@habana.ai>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Sanju C Sudhakaran <scsudhakaran@habana.ai>
Co-authored-by: Michal Adamczyk <madamczyk@habana.ai>
Co-authored-by: Marceli Fylcek <mfylcek@habana.ai>
Co-authored-by: Himangshu Lahkar <49579433+hlahkar@users.noreply.github.com>
Co-authored-by: Vivek Goel <vgoel@habana.ai>
Co-authored-by: yuwenzho <yuwen.zhou@intel.com>
Co-authored-by: Dominika Olszewska <dolszewska@habana.ai>
Co-authored-by: barak goldberg <149692267+bgoldberg-habana@users.noreply.github.com>
Co-authored-by: Michal Szutenberg <37601244+szutenberg@users.noreply.github.com>
Co-authored-by: Jan Kaniecki <jkaniecki@habana.ai>
Co-authored-by: Agata Dobrzyniewicz <160237065+adobrzyniewicz-habana@users.noreply.github.com>
Co-authored-by: Krzysztof Wisniewski <kwisniewski@habana.ai>
Co-authored-by: Dudi Lester <160421192+dudilester@users.noreply.github.com>
Co-authored-by: Ilia Taraban <tarabanil@gmail.com>
Co-authored-by: Chendi.Xue <chendi.xue@intel.com>
Co-authored-by: Michał Kuligowski <mkuligowski@habana.ai>
Co-authored-by: Jakub Maksymczuk <jmaksymczuk@habana.ai>
Co-authored-by: Tomasz Zielinski <85164140+tzielinski-habana@users.noreply.github.com>
Co-authored-by: Sun Choi <schoi@habana.ai>
Co-authored-by: Iryna Boiko <iboiko@habana.ai>
Co-authored-by: Bob Zhu <41610754+czhu15@users.noreply.github.com>
Co-authored-by: hlin99 <73271530+hlin99@users.noreply.github.com>
Co-authored-by: Zehao Huang <zehao.huang@intel.com>
Co-authored-by: Andrzej Kotłowski <Andrzej.Kotlowski@intel.com>
Co-authored-by: Yan Tomsinsky <73292515+Yantom1@users.noreply.github.com>
Co-authored-by: Nir David <ndavid@habana.ai>
Co-authored-by: Yu-Zhou <yu.zhou@intel.com>
Co-authored-by: Ruheena Suhani Shaik <rsshaik@habana.ai>
Co-authored-by: Karol Damaszke <kdamaszke@habana.ai>
Co-authored-by: Marcin Swiniarski <mswiniarski@habana.ai>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Jacek Czaja <jacek.czaja@intel.com>
Co-authored-by: Jacek Czaja <jczaja@habana.ai>
Co-authored-by: Yuan <yuan.zhou@outlook.com>
 2024-11-06 01:09:10 -08:00
Mengqing Cao
ccb5376a9a [Bugfix][OpenVINO] Fix circular reference #9939 (#9974) 
Signed-off-by: MengqingCao <cmq0113@163.com>
 2024-11-04 18:14:13 +08:00
youkaichao
ff5ed6e1bc [torch.compile] rework compile control with piecewise cudagraph (#9715) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-10-29 23:03:49 -07:00
Yan Ma
04a3ae0aca [Bugfix] Fix multi nodes TP+PP for XPU (#8884) 
Signed-off-by: YiSheng5 <syhm@mail.ustc.edu.cn>
Signed-off-by: yan ma <yan.ma@intel.com>
Co-authored-by: YiSheng5 <syhm@mail.ustc.edu.cn>
 2024-10-29 21:34:45 -07:00
wangshuai09
622b7ab955 [Hardware] using current_platform.seed_everything (#9785) 
Signed-off-by: wangshuai09 <391746016@qq.com>
 2024-10-29 14:47:44 +00:00
bnellnm
3cb07a36a2 [Misc] Upgrade to pytorch 2.5 (#9588) 
Signed-off-by: Bill Nell <bill@neuralmagic.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
 2024-10-27 09:44:24 +00:00
Mengqing Cao
5cbdccd151 [Hardware][openvino] is_openvino --> current_platform.is_openvino (#9716) 2024-10-26 10:59:06 +00:00
xendo
9dbcce84a7 [Neuron] [Bugfix] Fix neuron startup (#9374) 
Co-authored-by: Jerzy Zagorski <jzagorsk@amazon.com>
 2024-10-22 12:51:41 +00:00
Cyrus Leung
390be74649 [Misc] Print stack trace using logger.exception (#9461) 2024-10-17 13:55:48 +00:00
youkaichao
e4d652ea3e [torch.compile] integration with compilation control (#9058) 2024-10-10 12:39:36 -07:00
Cyrus Leung
26a68d5d7e [CI/Build] Add test decorator for minimum GPU memory (#8925) 2024-09-29 02:50:51 +00:00
Tyler Titsworth
260024a374 [Bugfix][Intel] Fix XPU Dockerfile Build (#7824) 
Signed-off-by: tylertitsworth <tyler.titsworth@intel.com>
Co-authored-by: youkaichao <youkaichao@126.com>
 2024-09-27 23:45:50 -07:00
Cyrus Leung
6ffa3f314c [CI/Build] Avoid CUDA initialization (#8534) 2024-09-18 10:38:11 +00:00
Li, Jiang
0b952af458 [Hardware][Intel] Support compressed-tensor W8A8 for CPU backend (#7257) 2024-09-11 09:46:46 -07:00
youkaichao
bc6e42a9b1 [hardware][rocm] allow rocm to override default env var (#7926) 2024-08-27 19:50:06 -07:00
youkaichao
ed6f002d33 [cuda][misc] error on empty CUDA_VISIBLE_DEVICES (#7924) 2024-08-27 12:06:11 -07:00
youkaichao
70c094ade6 [misc][cuda] improve pynvml warning (#7852) 2024-08-25 14:30:09 -07:00
Woosuk Kwon
eeee1c3b1a [TPU] Avoid initializing TPU runtime in is_tpu (#7763) 2024-08-21 21:31:49 -07:00
youkaichao
ad28a74beb [misc][cuda] add warning for pynvml user (#7675) 2024-08-20 00:35:09 -07:00
youkaichao
e54ebc2f8f [doc] fix doc build error caused by msgspec (#7659) 2024-08-19 17:50:59 -07:00
youkaichao
eed020f673 [misc] use nvml to get consistent device name (#7582) 2024-08-16 21:15:13 -07:00
youkaichao
4d2dc5072b [hardware] unify usage of is_tpu to current_platform.is_tpu() (#7102) 2024-08-13 00:16:42 -07:00
Cyrus Leung
9ba85bc152 [mypy] Misc. typing improvements (#7417) 2024-08-13 09:20:20 +08:00
youkaichao
639159b2a6 [distributed][misc] add specialized method for cuda platform (#7249) 2024-08-07 08:54:52 -07:00
Woosuk Kwon
42de2cefcb [Misc] Add a wrapper for torch.inference_mode (#6618) 2024-07-21 18:43:11 -07:00