biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
8,756 Commits 2 Branches 140 Tags
3663870c72da246d81d8bd8f5c059890fb3f3f5d
Commit Graph

212 Commits

Author SHA1 Message Date
Asaf Joseph Gardin
3663870c72 [V1][Mamba1] - Full CUDA and Piecewise CUDA Graphs Support (#23035) 
Signed-off-by: asafg <asafg@ai21.com>
Signed-off-by: asafg <39553475+Josephasafg@users.noreply.github.com>
Co-authored-by: asafg <asafg@ai21.com>
 2025-08-20 20:08:51 -07:00
Matthew Bonanni
10cc12ba66 Feature/mla tests (#23195) 
Signed-off-by: Matthew Bonanni <mbonanni001@gmail.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2025-08-20 21:46:47 +00:00
Matthew Bonanni
a4fbb32fab Remove chunked_prefill_enabled flag in V1 MLA (#23183) 
Signed-off-by: Matthew Bonanni <mbonanni001@gmail.com>
 2025-08-20 21:43:17 +00:00
Woosuk Kwon
d6d13bd49e [Misc] Add max_seq_len to CommonAttentionMetadata (#23216) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-08-20 09:05:29 -07:00
who who who
d983769c41 fix cuda graph (#22721) 
Signed-off-by: fsx950223 <fsx950223@outlook.com>
 2025-08-20 06:24:37 +00:00
Zebing Lin
a634733f67 [Attention] Optimize make_local_attention_virtual_batches for Flash Attention (#23185) 
Signed-off-by: linzebing <linzebing1995@gmail.com>
 2025-08-20 02:57:47 +00:00
Lucas Wilkinson
14e2b0730b [BugFix] fix CUTLASS MLA full cudagraph (#23200) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2025-08-19 22:17:08 +00:00
Woosuk Kwon
e61bac87ee [Misc] Minor refactoring for FlashInfer backend (#23147) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-08-19 13:11:51 -07:00
Woosuk Kwon
5b5f350d67 [Misc] Enable yapf for FlashInfer backend (#23193) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-08-19 10:33:47 -07:00
elvischenv
03752dba8f [NVIDIA] Support Flashinfer TRTLLM FP8-q/kv/out Attention Kernel (#21716) 
Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
 2025-08-19 08:22:15 -04:00
Wentao Ye
90bbe0a5ad [Log] Warning Once for Cutlass MLA (#23137) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-08-18 23:24:16 -07:00
Chengji Yao
e9d6a3db69 [TPU] make ptxla not imported when using tpu_commons (#23081) 
Signed-off-by: Chengji Yao <chengjiyao@gmail.com>
Signed-off-by: Chengji Yao <chengjiyao@google.com>
Co-authored-by: Chengji Yao <chengjiyao@gmail.com>
 2025-08-19 11:46:42 +08:00
Michael Goin
000cceca8c [Bugfix gpt-oss] Fix float32 convert for flashinfer sink support (#23016) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-08-16 11:16:00 -07:00
Nicolò Lucchesi
070da660c1 [Kernel] Simplify get_kv_cache_layout and cache use_trtllm_attention env-dependent bit (#22735) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-08-16 00:14:08 +00:00
eigen
1723ef1aae minor: zero workspace buffer init for flashinfer trtllm-gen attn (#22603) 2025-08-15 21:38:10 +00:00
fhl2000
74f441f4b5 [Core] Allow full cudagraph with separate attention routines and orthogonal to compilation, add support for FA2 and FlashInfer (#20059) 
Signed-off-by: fhl <2410591650@qq.com>
Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
 2025-08-15 10:01:39 -04:00
amirai21
fe91ce9591 [V1] - Split Prefill and Decode for Mamba1 models (#22653) 
Signed-off-by: amirk <amirk@ai21.com>
Signed-off-by: asafg <asafg@ai21.com>
Co-authored-by: asafg <asafg@ai21.com>
Co-authored-by: Asaf Joseph Gardin <39553475+Josephasafg@users.noreply.github.com>
 2025-08-15 08:59:52 +00:00
Asaf Joseph Gardin
3d232dbd19 [Mamba] - refactor: Renamed mamba_attn to mamba2_attn (#22818) 
Signed-off-by: asafg <asafg@ai21.com>
Co-authored-by: asafg <asafg@ai21.com>
 2025-08-15 06:38:05 +00:00
Wentao Ye
5c3fbfe46b [Feature] Full Cuda Graph Support for Cutlass MLA and 6% E2E Throughput Improvement (#22763) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-08-15 06:27:30 +00:00
Chen Zhang
fceafaf582 [Bugfix][mamba] Fix type annotation of Mamba2Metadata (#22787) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-08-13 06:07:09 -07:00
Giancarlo Delfin
d94e3026de [V1] Add tree drafting tests for eagle spec decoding (#22705) 
Signed-off-by: Giancarlo Delfin <gdelfin@meta.com>
 2025-08-13 04:11:28 -07:00
Michael Goin
c6b928798e Force TRTLLM attention for gpt-oss on SM100 (#22678) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-08-12 21:22:16 -07:00
Xiaozhu Meng
6bd8ebf026 [Kernel][AMD] Avoid D2H copy and cumsum kernel (#22683) 
Signed-off-by: Xiaozhu <mxz297@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-08-12 12:53:36 -07:00
wang.yuqi
6d729c43fb [Bugfix] Fix ModernBert load & Enable sliding window attention for bidirectional attention. (#22637) 
Signed-off-by: wang.yuqi <noooop@126.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Max de Bayser <mbayser@br.ibm.com>
 2025-08-12 00:23:17 -07:00
Thomas Parnell
61f67d8acd [V1] [Hybrid] Enable Full CUDA Graph (decode-only) for Mamba layers (#21401) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2025-08-09 20:16:11 -07:00
Thomas Parnell
6ade99eafa [V1] [Hybrid] Support Minimax-Text-01 in V1 (#22151) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2025-08-08 23:08:48 -07:00
Lucas Wilkinson
cd9b9de1fb [BugFix] Fix IMA FlashMLA full cuda-graph and DP + Update FlashMLA (#21691) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2025-08-08 16:09:42 -07:00
Po-Han Huang (NVIDIA)
af473f0a85 [bugfix] Fix Llama3/4 issues caused by FlashInfer 0.2.10 (#22426) 
Signed-off-by: Po-Han Huang <pohanh@nvidia.com>
 2025-08-07 20:25:01 -07:00
Lucas Wilkinson
1dc8a70b6d [Attention] Support multiple attention metadata builders per kv_cache_spec + proper local attention no hybrid kv cache fix (#21588) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2025-08-06 18:40:52 -07:00
Maximilien de Bayser
f825c6bd22 Support encoder_only attention for FlexAttention (#22273) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
 2025-08-06 18:37:14 -07:00
Lain
9a3835aaa9 Fix trtllm-gen attention env and add attention sink (#22378) 
Signed-off-by: Siyuan Fu <siyuanf@nvidia.com>
Signed-off-by: Lain <fusiyuan2000@hotmail.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
 2025-08-06 18:07:41 -07:00
Asaf Joseph Gardin
46a13949d5 [v1] - Mamba1 Attention Metadata (#21249) 
Signed-off-by: asafg <asafg@ai21.com>
Co-authored-by: asafg <asafg@ai21.com>
 2025-08-06 17:03:42 -07:00
Yongye Zhu
31f5dc5b2a [gpt-oss] Enhance error msg on attention sink init (#22335) 
Signed-off-by: simon-mo <xmo@berkeley.edu>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
 2025-08-06 11:41:42 -07:00
Yongye Zhu
90ec006937 [gpt-oss] flashinfer attention sink init (#22330) 
Signed-off-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com>
Co-authored-by: Minseok Lee <47620120+minseokl@users.noreply.github.com>
 2025-08-05 23:48:19 -07:00
Woosuk Kwon
6e20924350 Add attention sink in attention backends (#22320) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Co-authored-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com>
Co-authored-by: Minseok Lee <47620120+minseokl@users.noreply.github.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
 2025-08-05 22:37:21 -07:00
Giancarlo Delfin
469b3ffaaa [V1] port xformers backend to v1 (#21342) 
Signed-off-by: Giancarlo Delfin <gdelfin@meta.com>
 2025-08-05 10:04:46 -07:00
elvischenv
83156c7b89 [NVIDIA] Support Flashinfer TRT-LLM Prefill Attention Kernel (#22095) 
Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
 2025-08-05 02:45:34 -07:00
Cyrus Leung
cdfd6871a5 [Bugfix] Misaligned params in TreeAttentionImpl (#22226) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-04 22:40:09 -07:00
Giancarlo Delfin
aa7012eb6d Add tree attention backend for v1 (part 1) (#20401) 
Signed-off-by: Giancarlo Delfin <gdelfin@meta.com>
 2025-08-03 22:13:26 -07:00
ZiTian.Zhao
e27d25a0dc [fix] fix correct assertion syntax error in attention utils. (#22154) 
Signed-off-by: zitian.zhao <zitian.zhao@tencentmusic.com>
 2025-08-03 19:24:02 -07:00
Thomas Parnell
4abfd8796f [V1] [Hybrid] Validate compatibility of attention backend batch reordering at init time (#21557) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2025-08-02 05:29:40 -07:00
Sage Moore
0edaf752d7 [Attention][DBO] Add support for "splitting" the CommonAttentionMetadata (#21153) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
 2025-08-01 19:47:53 -07:00
fhl2000
23322431c8 [V1][CUDA] Full cudagraph support for FlashInfer (#21367) 2025-08-01 21:49:34 -04:00
Michael Goin
f81c1bb055 [Bugfix] Check NVIDIA artifactory is accessible before using flashinfer cubin kernels (#21893) 2025-08-01 08:28:45 -04:00
Mickaël Seznec
e1a7fe4af5 [BugFix] fix: aot passes kvcache dtype information (#19750) 
Signed-off-by: Mickael Seznec <mickael@mistral.ai>
 2025-08-01 05:45:02 +00:00
Michael Goin
61445453df [UX] Rename CUTLASS_MLA_VLLM_V1 to CUTLASS_MLA (#21966) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-07-30 20:40:34 -07:00
Yong Hoon Shin
ad510309ee Override attention metadata for fast prefill in some KV sharing setups (#21590) 
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
 2025-07-30 08:54:15 -07:00
Chen Zhang
555e7225bc [v1][attention] Support Hybrid Allocator + FlashInfer (#21412) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-07-30 01:45:29 +00:00
elvischenv
58b11b24a6 [Bugfix] Fix workspace buffer None issue for Flashinfer TRTLLM Backend (#21525) 
Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
 2025-07-29 10:34:00 -04:00
Asaf Joseph Gardin
a6c050286a [v1][mamba] Added mamba_type into MambaSpec (#21715) 
Signed-off-by: asafg <asafg@ai21.com>
Co-authored-by: asafg <asafg@ai21.com>
 2025-07-28 08:15:55 +00:00