biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
5,552 Commits 2 Branches 140 Tags
2b93162fb0bc91c8e1e397cb2a385e8febce11e2
Commit Graph

55 Commits

Author SHA1 Message Date
yarongmu-google
7c1f760024 [Kernel][TPU][ragged-paged-attn] vLLM code change for PR#8896 (#15659) 
Signed-off-by: Yarong Mu <ymu@google.com>
 2025-03-28 21:13:15 -07:00
Lucas Wilkinson
dccf535f8e [V1] Enable V1 Fp8 cache for FA3 in the oracle (#15191) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2025-03-23 15:07:04 -07:00
Lehua Ding
91ca929dc7 [V1] Fix wrong import path of get_flash_attn_version (#15280) 
Signed-off-by: Lehua Ding <lehuading@tencent.com>
 2025-03-21 03:54:11 -07:00
Isotr0py
f8a08cb90d [V1] Enable Triton(ROCm) Attention backend for Nvidia GPUs (#14071) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-03-21 03:14:19 +00:00
Woosuk Kwon
0c6f5023c3 [V1] Scheduler Refactoring [1/N] - Add Scheduler Interface (#15250) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-03-20 17:50:43 -07:00
Mickaël Seznec
a597a57595 [Attention] Flash Attention 3 - fp8 (#14570) 
Signed-off-by: Mickael Seznec <mickael@mistral.ai>
 2025-03-20 01:14:20 -04:00
iefgnoix
b0e96aaebb [V1][TPU] Change kv cache shape. (#15145) 
Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>
 2025-03-19 12:16:42 -07:00
Robert Shaw
d4d93db2c5 [V1] V1 Enablement Oracle (#13726) 
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2025-03-14 22:02:20 -07:00
Lucas Wilkinson
9532c49836 [Attention] MLA get rid of materialization (#14770) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2025-03-13 23:39:02 -07:00
Jeff Daily
a1c8f3796c dynamic distpatch of fp8 kernels (#14245) 
Signed-off-by: Jeff Daily <jeff.daily@amd.com>
 2025-03-11 10:54:56 -04:00
Simon Mo
fb0acb6c72 [Perf] Improve MLA on V1 (#14540) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-03-10 12:06:58 -07:00
iefgnoix
10f7552789 [V1][TPU] Remove unnecessary padding for running on TPU. (#14467) 2025-03-08 21:56:04 -05:00
Lucas Wilkinson
db84f5eb3b [Bugfix] DeepSeek Accuracy (#14476) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2025-03-08 16:47:03 +00:00
Tyler Michael Smith
ca7a2d5f28 Revert "[Perf] Reduce MLA CPU overheads in V1 (#14384)" (#14471) 2025-03-07 22:18:53 -08:00
iefgnoix
1e3598edeb Use the optimized block sizes after tuning the kernel. (#14329) 2025-03-07 13:25:13 +00:00
Luka Govedič
e1744502c2 [FP8] Refactor apply_fp8_linear and apply_fp8_linear_generic into an object (#14390) 
Signed-off-by: luka <luka@neuralmagic.com>
 2025-03-07 05:20:16 +00:00
Lucas Wilkinson
dae6896977 [Perf] Reduce MLA CPU overheads in V1 (#14384) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2025-03-06 19:59:14 -08:00
Chengji Yao
0578e5a462 [Hardware][TPU]Enable ragged paged attention kernel and resolve recompilation issue (#14310) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
 2025-03-06 23:31:05 +00:00
Michael Goin
6832707e90 [V1][Bugfix] Standardize quantized kv cache rejection for attention backends (#14221) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-03-06 14:18:29 -08:00
Ying Zhong
9f1710f1ac Fix mla prefill context performance (#13897) 
Signed-off-by: ZhongYingMatrix <zhongyingmatrix@gmail.com>
 2025-03-06 09:35:49 -08:00
Thomas Parnell
6bd1dd9d26 [Kernel] [V1] Improved performance for V1 Triton (ROCm) backend (#14152) 2025-03-06 07:39:16 -08:00
Lucas Wilkinson
f6bb18fd9a [BugFix] MLA + V1, illegal memory access and accuracy issues (#14253) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2025-03-05 17:10:13 -08:00
iefgnoix
79e4937c65 [v1] Add comments to the new ragged paged attention Pallas kernel (#14155) 
Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2025-03-03 23:00:55 +00:00
Harry Mellor
cf069aa8aa Update deprecated Python 3.8 typing (#13971) 2025-03-02 17:34:51 -08:00
Sage Moore
b28246f6ff [ROCm][V1][Bugfix] Add get_builder_cls method to the ROCmAttentionBackend class (#14065) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
 2025-03-01 07:18:32 +00:00
iefgnoix
c3b6559a10 [V1][TPU] Integrate the new ragged paged attention kernel with vLLM v1 on TPU (#13379) 
Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-02-28 11:01:36 -07:00
Lucas Wilkinson
2e94b9cfbb [Attention] Flash MLA for V1 (#13867) 
Signed-off-by: Yang Chen <yangche@fb.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Yang Chen <yangche@fb.com>
 2025-02-27 23:03:41 +00:00
Yang Chen
58d1b2aa77 [Attention] MLA support for V1 (#13789) 
Signed-off-by: Yang Chen <yangche@fb.com>
 2025-02-27 13:14:17 -05:00
Sage Moore
558db8083c [V1][Kernel] Refactor the prefix_prefill kernel so that the caller no longer has to pass in the context lengths (#13095) 2025-02-22 05:25:41 -08:00
Lucas Wilkinson
288cc6c234 [Attention] MLA with chunked prefill (#12639) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Patrick Horn <patrick.horn@gmail.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2025-02-21 15:30:12 -08:00
Alexander Matveev
45f90bcbba [WIP] TPU V1 Support Refactored (#13049) 2025-02-14 00:21:53 -08:00
Sage Moore
ba59b78a9c [ROCm][V1] Add intial ROCm support to V1 (#12790) 2025-02-13 22:21:50 -08:00
youkaichao
fe743b798d [bugfix] fix early import of flash attention (#12959) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-02-09 00:06:56 +08:00
Lucas Wilkinson
c786e757fa [Attention] Use FA3 for MLA on Hopper (#12807) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
 2025-02-06 11:43:12 +00:00
Russell Bryant
e489ad7a21 [Misc] Add SPDX-License-Identifier headers to python source files (#12628) 
- **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-02 11:58:18 -08:00
Lucas Wilkinson
ab5bbf5ae3 [Bugfix][Kernel] Fix CUDA 11.8 being broken by FA3 build (#12375) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
 2025-01-24 15:27:59 +00:00
Gregory Shtrasberg
e97f802b2d [FP8][Kernel] Dynamic kv cache scaling factors computation (#11906) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
 2025-01-23 18:04:03 +00:00
Lucas Wilkinson
978b45f399 [Kernel] Flash Attention 3 Support (#12093) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
 2025-01-23 06:45:48 -08:00
wangxiyuan
86bfb6dba7 [Misc] Pass attention to impl backend (#12218) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2025-01-20 23:25:28 +08:00
wangxiyuan
2e0e017610 [Platform] Add output for Attention Backend (#11981) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2025-01-14 13:27:04 +00:00
Chen Zhang
e20c92bb61 [Kernel] Move attn_type to Attention.__init__() (#11690) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-01-07 00:11:28 +08:00
Woosuk Kwon
73001445fb [V1] Implement Cascade Attention (#11635) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-01-01 21:56:46 +09:00
Mark McLoughlin
6d917d0eeb Enable mypy checking on V1 code (#11105) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2024-12-14 09:54:04 -08:00
youkaichao
75f89dc44c [torch.compile] add a flag to track batchsize statistics (#11059) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-12-10 12:40:52 -08:00
Woosuk Kwon
3b61cb450d [V1] Further reduce CPU overheads in flash-attn (#10989) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2024-12-09 12:38:46 -08:00
youkaichao
a4c4daf364 [misc] use out argument for flash attention (#10822) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-12-02 10:50:10 +00:00
Woosuk Kwon
073a4bd1c0 [Kernel] Use out arg in flash_attn_varlen_func (#10811) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2024-12-01 17:55:39 -08:00
Woosuk Kwon
98f47f2a40 [V1] Optimize the CPU overheads in FlashAttention custom op (#10733) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2024-11-28 09:01:02 -08:00
Isotr0py
04668ebe7a [Bugfix] Avoid import AttentionMetadata explicitly in Mllama (#10593) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2024-11-23 18:12:20 +00:00
youkaichao
eebad39f26 [torch.compile] support all attention backends (#10558) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-22 14:04:42 -08:00