biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
5,693 Commits 2 Branches 140 Tags
f6b32efb7f08a58e06cfee87c3dd241962bce3e1
Commit Graph

445 Commits

Author SHA1 Message Date
Yong Hoon Shin
05a015d6a5 Add warning for Attention backends that do not support irope yet (#16212) 2025-04-08 03:59:26 +00:00
Roger Wang
f2ebb6f541 [V1] Scatter and gather placeholders in the model runner (#16076) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>
 2025-04-08 10:43:41 +08:00
youkaichao
a865bc1ca6 [core] do not send error across process (#16174) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-04-07 19:09:03 -07:00
Nick Hill
7f6d47c1a2 [V1][BugFix] Exit properly if engine core fails during startup (#16137) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-07 15:30:15 -07:00
Benjamin Chislett
3147586ebd [Bugfix] Fix guidance backend for Qwen models (#16210) 
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
 2025-04-07 22:15:43 +00:00
Lu Fang
55dcce91df Upstream Llama4 Support to Main (#16113) 
Signed-off-by: Aston Zhang <22279212+astonzhang@users.noreply.github.com>
Signed-off-by: Chris Thi <chris.c.thi@gmail.com>
Signed-off-by: drisspg <drisspguessous@gmail.com>
Signed-off-by: Jon Swenson <jmswen@gmail.com>
Signed-off-by: Keyun Tong <tongkeyun@gmail.com>
Signed-off-by: Lu Fang <fanglu@meta.com>
Signed-off-by: Xiaodong Wang <xdwang@meta.com>
Signed-off-by: Yang Chen <yangche@fb.com>
Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Signed-off-by: Lu Fang <lufang@fb.com>
Signed-off-by: Lu Fang <fanglu@fb.com>
Signed-off-by: Lucia Fang <fanglu@fb.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Lu Fang <fanglu@fb.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-04-07 08:06:27 -07:00
Shanshan Shen
e9ba99f296 [V1][Structured Output] Add supports_structured_output() method to Platform (#16148) 
Signed-off-by: shen-shanshan <467638484@qq.com>
 2025-04-07 11:06:24 +00:00
Woosuk Kwon
3749e28774 [V1][Minor] Minor simplification for get_computed_blocks (#16139) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-06 20:38:12 -07:00
Kay Yan
86fc2321ff [Metrics] Add bucket for request_latency, time_to_first_token and time_per_output_token (#15202) 
Signed-off-by: Kay Yan <kay.yan@daocloud.io>
 2025-04-06 20:34:51 -07:00
Woosuk Kwon
b10e519895 [V1][Minor] Optimize get_cached_block (#16135) 2025-04-06 20:48:14 +00:00
Roger Wang
9ca710e525 [CI][V1] Fix passing tokenizer as kwarg to validate_guidance_grammar (#16117) 
Signed-off-by: Roger Wang <ywang@roblox.com>
 2025-04-06 16:18:00 +08:00
Woosuk Kwon
63375f0cdb [V1][Spec Decode] Update N-gram Proposer Interface (#15750)
Some checks failed
Create Release / Create Release (push) Has been cancelled
Details 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-04 16:32:54 -07:00
Michael Goin
70ad3f9e98 [Bugfix][TPU] Fix V1 TPU worker for sliding window (#16059) 
Signed-off-by: Michael Goin <mgoin64@gmail.com>
 2025-04-04 23:31:19 +00:00
Roger Wang
af51d80fa1 Revert "[V1] Scatter and gather placeholders in the model runner" (#16075) 2025-04-04 14:50:57 -07:00
Cyrus Leung
f5722a5052 [V1] Scatter and gather placeholders in the model runner (#15712) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2025-04-04 21:26:44 +00:00
Nick Hill
651cf0fec1 [V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue (#15906) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-04 12:56:43 -07:00
Mark McLoughlin
a35a8a8392 [V1][Spec Decode] Avoid logging useless nan metrics (#16023) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-04-04 08:52:41 -07:00
Chengji Yao
fadc59c0e6 [TPU][V1] Remove ragged attention kernel parameter hard coding (#16041) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
 2025-04-04 07:48:50 -04:00
iefgnoix
b6be6f8d1e [TPU] Support sliding window and logit soft capping in the paged attention kernel for TPU. (#15732) 
Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>
 2025-04-03 14:23:28 -07:00
wwl2755
463bbb1835 [Bugfix][V1] Fix bug from putting llm_engine.model_executor in a background process (#15367) 
Signed-off-by: wwl2755 <wangwenlong2755@gmail.com>
 2025-04-03 07:32:10 +00:00
Aleksandr Malyshev
e73ff24e31 [ROCM][KERNEL] Paged attention for V1 (#15720) 
Signed-off-by: Aleksandr Malyshev <maleksan@amd.com>
Signed-off-by: root <root@banff-cyxtera-s65-4.amd.com>
Co-authored-by: Aleksandr Malyshev <maleksan@amd.com>
Co-authored-by: root <root@banff-cyxtera-s65-4.amd.com>
 2025-04-02 19:48:00 -07:00
Nicolò Lucchesi
bd7599d34a [V1][TPU] Do not compile sampling more than needed (#15883) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-04-03 01:36:01 +00:00
Chengji Yao
01b6113659 [TPU] optimize the all-reduce performance (#15903) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
 2025-04-03 00:25:14 +00:00
Hyesoo Yang
1b84eff03a [V1][TPU] TPU-optimized top-p implementation (avoids scattering). (#15736) 
Signed-off-by: Hyesoo Yang <hyeygit@gmail.com>
Co-authored-by: root <root@t1v-n-822696b7-w-0.us-central2-b.c.tpu-prod-env-large-adhoc.internal>
 2025-04-02 17:18:08 -07:00
Russell Bryant
14e53ed11f [V1] Fix json_object support with xgrammar (#15488) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-04-02 02:00:08 -07:00
Eric Tang
ddb94c2605 [core] Add tags parameter to wake_up() (#15500) 
Signed-off-by: Eric <erictang000@gmail.com>
 2025-04-02 01:59:27 -07:00
Woosuk Kwon
274d8e8818 [V1][Minor] Enhance SpecDecoding Metrics Log in V1 (#15902) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-01 23:38:02 -07:00
Brayden Zhong
6efb195a6e [V1] Fix: make sure k_index is int64 for apply_top_k_only (#15907) 
Signed-off-by: Brayden Zhong <b8zhong@uwaterloo.ca>
 2025-04-01 19:06:44 -07:00
Ekagra Ranjan
24b7fb455a [Spec Decode] Fix input triton kernel for eagle (#15909) 2025-04-01 18:15:14 -07:00
Woosuk Kwon
e75a6301bd [V1][Spec Decode] Implement Eagle Proposer [1/N] (#15729) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-01 12:33:16 -07:00
Mark McLoughlin
a79cc68b3a [V1][Metrics] Initial speculative decoding metrics (#15151) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-04-01 10:45:04 -07:00
Chen Zhang
3a5f0afcd2 [V1] Implement sliding window attention in kv_cache_manager (#14097) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-04-01 00:33:17 -07:00
Mark McLoughlin
f98a4920f9 [V1][Core] Remove unused speculative config from scheduler (#15818) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-03-31 19:15:21 +00:00
Alexander Matveev
9a2160fa55 [V1] TPU CI - Add basic perf regression test (#15414) 
Signed-off-by: Alexander Matveev <amatveev@redhat.com>
 2025-03-31 13:25:20 -04:00
yihong
2de4118243 fix: change GB to GiB in logging close #14979 (#15807) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-03-31 10:00:50 -07:00
Cyrus Leung
09e974d483 [Bugfix] Check dimensions of multimodal embeddings in V1 (#15816) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-03-31 09:01:35 -07:00
youkaichao
555aa21905 [V1] Fully Transparent Implementation of CPU Offloading (#15354) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-03-31 20:22:34 +08:00
Chengyang LIU
18ed3132d2 [Misc] update the comments (#15780) 
Signed-off-by: chengyang liu <lcy4869@gmail.com>
Co-authored-by: chengyang liu <lcy4869@gmail.com>
 2025-03-30 19:39:56 -07:00
Woosuk Kwon
9b459eca88 [V1][Scheduler] Avoid calling _try_schedule_encoder_inputs for every request (#15778) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-03-30 14:10:42 -07:00
Cyrus Leung
803d5c35f3 [V1] Override mm_counts for dummy data creation (#15703) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-03-30 03:20:42 -07:00
Julien Denize
6909a76201 [Bugfix] Fix Mistral guided generation using xgrammar (#15704) 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
 2025-03-29 20:20:19 -07:00
Woosuk Kwon
2bc4be4e32 [V1][Minor] Simplify rejection sampler's parse_output (#15741) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-03-29 09:25:17 -07:00
Roger Wang
c67abd614f [V1] Support interleaved modality items (#15605) 
Signed-off-by: Roger Wang <ywang@roblox.com>
 2025-03-29 06:30:09 -07:00
wwl2755
94744ba41a [V1] [Feature] Collective RPC (#15444) 
Signed-off-by: wwl2755 <wangwenlong2755@gmail.com>
 2025-03-29 03:39:14 -07:00
yarongmu-google
7c1f760024 [Kernel][TPU][ragged-paged-attn] vLLM code change for PR#8896 (#15659) 
Signed-off-by: Yarong Mu <ymu@google.com>
 2025-03-28 21:13:15 -07:00
Nicolò Lucchesi
da461f3cbf [TPU][V1][Bugfix] Fix w8a8 recompiilation with GSM8K (#15714) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-03-28 21:13:06 -07:00
Nick Hill
6d531ad7b8 [Misc][V1] Misc code streamlining (#15723) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-03-28 20:59:47 -07:00
Alexander Matveev
c3f687ac22 [V1] TPU - Fix the chunked prompt bug (#15713) 
Signed-off-by: Alexander Matveev <amatveev@redhat.com>
 2025-03-28 20:19:04 +00:00
Robert Shaw
038bededba [TPU] [Perf] Improve Memory Usage Estimation (#15671) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
 2025-03-28 17:37:52 +00:00
Cyrus Leung
c6bc0034d0 [Misc] Remove unused utils and clean up imports (#15708) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-03-28 09:41:16 -07:00