biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
9,037 Commits 2 Branches 140 Tags
5b31cb1781e594aae29d878b1acde3e2f900bc41
Commit Graph

76 Commits

Author SHA1 Message Date
dubejf
5b31cb1781 [Bugfix] Fix --config arg expansion called from api_server.py (#23944) 
Signed-off-by: Jean-Francois Dube <dubejf+gh@gmail.com>
Co-authored-by: Jean-Francois Dube <dubejf+gh@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2025-08-29 21:36:39 -07:00
Adit Chawdhary
4f7cde7272 Adds json_count_leaves utility function (#23899) 
Signed-off-by: aditchawdhary <aditxy@hotmail.com>
 2025-08-29 05:28:13 -07:00
Wentao Ye
d3d2aad5a2 [Log] Use Debug Once for DeepGEMM E8M0 When not Enabled (#23858) 2025-08-28 22:18:10 +00:00
Wentao Ye
3af47c3cc6 [Feature] Add Hopper DeepGEMM E8M0 for DeepSeekV3.1 scale_fmt (#23666) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
 2025-08-27 14:09:08 +00:00
rongfu.leng
8dbf6ed7be [Bugfix] fix when config.yaml config value is list parse error (#23528) 
Signed-off-by: rongfu.leng <rongfu.leng@daocloud.io>
 2025-08-27 05:54:39 +00:00
nvjullin
f66673a39d [Kernel] Added flashinfer fp8 per-tensor gemms (#22895) 
Signed-off-by: Julien Lin <jullin@nvidia.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2025-08-26 06:54:04 -07:00
Wentao Ye
56dcf4e7e9 [Bug] Fix DeepGEMM Env Control (#23591) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-08-25 18:41:21 -07:00
Shiyan Deng
da65bec309 add an env var for path to pre-downloaded flashinfer cubin files (#22675) 2025-08-22 19:25:45 +00:00
Didier Durand
22cf679aad [Doc]: fix various typos in multiple files (#23179) 
Signed-off-by: Didier Durand <durand.didier@gmail.com>
 2025-08-22 10:38:46 -07:00
Li, Jiang
88016c372a [Bugfix] Fix pooling models on CPU backend (#23392) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2025-08-22 09:47:17 +00:00
Wentao Ye
394591e343 [Feature] Enable DeepGEMM Linear on B200; 1.5% E2E throughput improvement (#23351) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-08-21 21:01:08 -07:00
Ming Yang
10f535c086 [Bugfix] Fix port conflict by obtaining a list of open ports upfront (#21894) 
Signed-off-by: Ming Yang <minos.future@gmail.com>
 2025-08-21 10:22:18 -07:00
elvischenv
03752dba8f [NVIDIA] Support Flashinfer TRTLLM FP8-q/kv/out Attention Kernel (#21716) 
Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
 2025-08-19 08:22:15 -04:00
afeldman-nm
bf7f470b22 [V1] Logits processors extensibility (#19912) 
Signed-off-by: Andrew Feldman <afeldman@redhat.com>
Signed-off-by: Andrew Feldman <afeld2012@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Andrew Feldman <afeld2012@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-08-16 12:59:17 -07:00
Nicolò Lucchesi
070da660c1 [Kernel] Simplify get_kv_cache_layout and cache use_trtllm_attention env-dependent bit (#22735) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-08-16 00:14:08 +00:00
Nick Hill
ad0297d113 [Misc] Support passing multiple request ids at once to AsyncLLM.abort() (#22944) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-08-15 17:00:36 -07:00
Yichen Yan
236b864e4f [BugFix] Make run_once thread-safe (#22978) 
Signed-off-by: <wenji.yyc@alibaba-inc.com>
Signed-off-by: Yichen Yan <wenji.yyc@alibaba-inc.com>
 2025-08-15 16:56:17 -07:00
Or Ozeri
c280066f9d [v1] Move block_hashes from KVCacheManager to Request.block_hashes (#19728) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2025-08-15 16:52:52 -07:00
Thomas Parnell
75531a6c13 [V1] [Hybrid] Support using float32 for state in Hybrid Models (Mamba2, Mamba1, Minimax) (#22928) 
Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Daniel Afrimi <danielafrimi8@gmail.com>
Co-authored-by: Burkhard Ringlein <ngl@zurich.ibm.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
 2025-08-15 12:57:06 +00:00
nvjullin
279a5f31b3 [Kernel] Add nvfp4 gemm flashinfer backends (#22346) 
Signed-off-by: Julien Lin <jullin@nvidia.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-08-14 16:03:55 -04:00
Nick Hill
eb08487b18 [BugFix] Threadsafe close async zmq sockets (#22877) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-08-14 03:44:29 -07:00
HWH
9bd9294f0e [Bugfix] Fix MiniCPMV Image input inference failed (#22813) 
Signed-off-by: HWH <67449739+jio-H@users.noreply.github.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-08-13 09:41:41 -07:00
Michael Goin
c6b928798e Force TRTLLM attention for gpt-oss on SM100 (#22678) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-08-12 21:22:16 -07:00
Wentao Ye
f7dcce7a4a [Feature] Add VLLM_USE_DEEP_GEMM_E8M0 Env to Control E8M0 Scale (#21968) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-08-11 09:39:08 -07:00
Cyrus Leung
951b038298 [Misc] Move jsontree to utils (#22622) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-11 03:49:32 -07:00
Harry Mellor
bc1d02ac85 [Docs] Add comprehensive CLI reference for all large vllm subcommands (#22601) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-08-11 00:13:33 -07:00
Benji Beck
b4e2916721 Migrate LlavaNextImageInputs to TensorSchema (#21774) 
Signed-off-by: Benji Beck <benjibeck@meta.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-08-10 09:05:21 -07:00
Harry Mellor
56186474f6 [Docs] Reduce noise in docs and --help from the JSON tip (#22567) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-08-09 08:31:32 -07:00
Wentao Ye
3157aebb63 [Log] Add Warning for Deprecation of DeepGEMM old version (#22194) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-08-08 23:07:48 -07:00
Yongye Zhu
e789cad6b8 [gpt-oss] triton kernel mxfp4 (#22421) 
Signed-off-by: <zyy1102000@gmail.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
 2025-08-08 08:24:07 -07:00
Harry Mellor
e5ebeeba53 Remove exception for Python 3.8 typing from linter (#22506) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-08-08 03:06:46 -07:00
Nick Hill
ccdae737a0 [BugFix] Don't cancel asyncio tasks directly from destructors (#22476) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-08-08 01:13:18 -07:00
Shu Wang
a3b9c17b56 Support Tensorrt-LLM MoE fp4 for low-latency (#21331) 
Signed-off-by: Shu Wang <shuw@nvidia.com>
Signed-off-by: Po-Han Huang <pohanh@nvidia.com>
Signed-off-by: Shu Wang. <shuw@nvidia.com>
Signed-off-by: XIn Li <xinli@nvidia.com>
Co-authored-by: XIn Li <xinli@nvidia.com>
 2025-08-07 19:18:22 -07:00
Lain
9a3835aaa9 Fix trtllm-gen attention env and add attention sink (#22378) 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
Signed-off-by: Lain <fusiyuan2000@hotmail.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
 2025-08-06 18:07:41 -07:00
elvischenv
83156c7b89 [NVIDIA] Support Flashinfer TRT-LLM Prefill Attention Kernel (#22095) 
Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
 2025-08-05 02:45:34 -07:00
Roger Wang
067c34a155 docs: remove deprecated disable-log-requests flag (#22113) 
Signed-off-by: Roger Wang <hey@rogerw.me>
 2025-08-02 00:19:48 -07:00
Rui Qiao
4ac8437352 [Misc] Getting and passing ray runtime_env to workers (#22040) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
 2025-08-01 23:54:40 -07:00
Wentao Ye
6e8d8c4afb [Test] Add Unit Test for Batched DeepGEMM (#21559) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-08-02 10:45:46 +08:00
Varun Sundar Rabindranath
a65f46be5e [Misc] DeepGemmExperts : Avoid JIT generation in the hot-path (#21955) 
Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
 2025-08-01 19:42:03 -07:00
Nick Hill
881e1af43a [BugFix] Harden distributed DP startup (#21538) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-08-01 21:40:45 +00:00
Harry Mellor
2d7b09b998 Deprecate --disable-log-requests and replace with --enable-log-requests (#21739) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-08-01 17:16:37 +01:00
Michael Goin
f81c1bb055 [Bugfix] Check NVIDIA artifactory is accessible before using flashinfer cubin kernels (#21893) 2025-08-01 08:28:45 -04:00
wuhang
e6680f9e25 [Bugfix] Add log prefix in non-dp mode engine core (#21889) 
Signed-off-by: wuhang <wuhang6@huawei.com>
 2025-08-01 09:04:16 +00:00
Cyrus Leung
82de9b9d46 [Misc] Automatically resolve HF processor init kwargs (#22005) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-07-31 22:44:10 -07:00
Wentao Ye
3700642013 [Refactor] Remove Duplicate per_block_cast_to_fp8, Remove Dependencies of DeepGEMM (#21787) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-08-01 01:13:27 +00:00
Matthew Bonanni
e360316ab9 Add DeepGEMM to Dockerfile in vllm-base image (#21533) 
Signed-off-by: Matthew Bonanni <mbonanni001@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-07-31 18:01:55 -07:00
amirkl94
207b750e19 [NVIDIA] Add SM100 Flashinfer MoE per tensor scale fp8 backend (#21458) 
Signed-off-by: Amir Klein <203507526+amirkl94@users.noreply.github.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-07-31 06:00:01 -07:00
Nick Hill
7234fe2685 [Misc] Rework process titles (#21780) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-07-29 05:14:47 +00:00
Clayton Coleman
c6f36cfa26 [Bugfix] DeepGEMM is not enabled on B200 due to _lazy_init() (#21472) 
Signed-off-by: Clayton Coleman <smarterclayton@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-07-28 20:51:22 +00:00
Benji Beck
3339cba3ff Migrate FuyuImagePatchInputs to TensorSchema (#21662) 
Signed-off-by: Benji Beck <benjibeck@meta.com>
 2025-07-26 19:34:14 -07:00