biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
13,562 Commits 2 Branches 140 Tags
f0d525171557e3fe74e8e6df52257f9d66831d3f
Commit Graph

948 Commits

Author SHA1 Message Date
yugong333
ffe1fc7a28 Reduce the kernel overhead when num of active loras is smaller than max loras. Multiple cuda graphs are captured for each num of active-loras. (#32005) 
Signed-off-by: Yu Gong <yu3.gong@gmail.com>
 2026-02-02 12:30:06 -05:00
shanjiaz
d95b4be47a move spec decode slow test to test_areas.yaml (#33365) 
Signed-off-by: shanjiaz <zsjwpianpian@gmail.com>
 2026-02-02 06:28:36 -08:00
Nicolò Lucchesi
528b3076af [CI][Bugfix] Fix flaky tests/v1/kv_connector/unit/test_multi_connector.py::test_multi_example_connector_consistency (#33555) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-02-02 03:01:29 -08:00
Yifan Qiao
a01ef3fa51 [Fix] prefix cache hit rate == 0 bug with gpt-oss style models (#33524) 
Signed-off-by: Yifan Qiao <yifanqiao@berkeley.edu>
 2026-02-02 01:59:58 +00:00
jma99_2333
22d9a056d5 Support clear mm and encoder cache (#33452) 
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-01-31 15:22:25 +00:00
Nick Hill
876a16f4fb [ModelRunner V2] Fix spec decoding + logprobs (#33391) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-31 03:33:26 +00:00
Matthew Bonanni
aaa901ad55 [Attention] Move MLA forward from backend to layer (#33284) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-01-30 19:30:00 -08:00
Kyle Sayers
f857a03f6b [QeRL] Layerwise Reloading (#32133) 
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
 2026-01-30 08:50:05 -07:00
Frank Wang
8f5d51203b Disable Cascade Attention for Batch Invariance (#32561) 
Signed-off-by: frankwang28 <frank.wbb@hotmail.com>
Signed-off-by: Frank Wang <41319051+frankwang28@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-01-30 10:00:46 -05:00
Patrick von Platen
10152d2194 [Realtime API] Adds minimal realtime API based on websockets (#33187) 
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-01-30 18:41:29 +08:00
Cyrus Leung
c6e7404cc5 [Multimodal] Simplify MM input definitions (#33331) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-29 13:32:04 +00:00
shanjiaz
5eeba80c74 Adding optional speculator tests for larger models (#32943) 
Signed-off-by: shanjiaz <zsjwpianpian@gmail.com>
 2026-01-29 16:54:02 +08:00
Wentao Ye
3e440786af [Feature] Fully support for async scheduling + PP, 30.8% E2E throughput improvement, 31.8% TPOT improvement (#32618) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-01-28 20:30:32 +00:00
Or Ozeri
2e8de86777 Revert "Enable Cross layers KV cache layout at NIXL Connector (#30207)" (#33241) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
 2026-01-28 04:36:00 -08:00
Matthew Bonanni
a608b4c6c2 [5/N][Attention] Finish eliminating vllm/attention folder (#32064) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-01-27 10:02:51 -05:00
omerpaz95
7227d06156 [Metrics] [KVConnector] Add Offloading Connector metrics (#27942) 
Added queries and hits metrics for the Offloading Connector.

Also added timing metrics for store and load operations, which take the
average time it takes to load/store, per-token.

The metrics are available from Prometheus and from the StatLogger.

Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: Omer Paz <Omer.Paz@ibm.com>
 2026-01-27 13:34:49 +00:00
Cyrus Leung
c25dbee40d [Model] Bump transformers version for test registry (#33100) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-26 18:53:22 +00:00
Cyrus Leung
11b556878b [Refactor] Use data parser for matching data items to multi-modal UUIDs (#32955) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-26 15:00:28 +08:00
Joshua Deng
91601ff478 [Feature] add session based streaming input support to v1 (#28973) 
Signed-off-by: Joshua Deng <joshuakdeng@gmail.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-01-24 12:06:28 -08:00
7. Sun
cd775bdbe0 [Tests] Replace flaky sleep with polling in test_background_cancel (#32986) 
Signed-off-by: 7. Sun <jhao.sun@gmail.com>
 2026-01-24 16:39:07 +00:00
7. Sun
0ccecf8833 [Tests] Standardize RNG seed utility across test files (#32982) 
Signed-off-by: 7. Sun <jhao.sun@gmail.com>
 2026-01-24 06:47:14 +00:00
7. Sun
0b9a735e11 [Tests] Clarify pytest skip reasons with actionable context (#32981) 
Signed-off-by: 7. Sun <jhao.sun@gmail.com>
 2026-01-24 06:38:50 +00:00
ElizaWszola
a28b94e6ef [Performance] Split FlashAttn attention and cache update (#25954) 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Luka Govedič <luka.govedic@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Luka Govedič <luka.govedic@gmail.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Luka Govedič <lgovedic@redhat.com>
 2026-01-23 17:28:06 -08:00
Lucas Wilkinson
3a41459501 [cudagraphs] Refactor cudagraph capture loop (#32946) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-01-23 13:22:20 -07:00
Harry Huang
5206e5e28c [V1][Hybrid] Mamba Prefix Caching with align mode (#30877) 
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
 2026-01-23 09:56:48 -08:00
David Ramon Prados
3a63be0faa Support custom URI schemes and trace handlers for profiler (#32393) 2026-01-22 09:45:40 -08:00
Matt
c517d8c934 [Hardware][AMD][CI][Bugfix] Fix regressions from deprecated env vars (#32837) 
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
 2026-01-23 00:59:15 +08:00
Cyrus Leung
d117a4d1a9 [Frontend] Introduce Renderer for processing chat messages (using ModelConfig) (#30200) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-22 12:44:22 +00:00
Or Ozeri
421012b63a OffloadingConnector: Support kernel_block_size != block_size (#30692) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-01-22 12:30:04 +00:00
liranschour
64e3d67ac0 Enable Cross layers KV cache layout at NIXL Connector (#30207) 
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: liranschour <liranschour@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-01-22 10:12:58 +00:00
Andreas Karatzas
a810299838 [ROCm][CI][Docs] Add comment explaining TRITON_ATTN fallback for ROCm (#32835) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-01-21 22:11:09 -08:00
Micah Williamson
019e2c3b7c [ROCm][CI] Lower Acceptance Len Threshold For test_draft_model_quantization (#32731) 
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
 2026-01-22 05:47:33 +00:00
Lucas Wilkinson
889722f3bf [FlashMLA] Update FlashMLA to expose new arguments (#32810) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-01-21 22:02:39 -07:00
Divakar Verma
49d9653852 [ROCm][CI] fix get_valid_backends (#32787) 
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
 2026-01-22 04:27:47 +00:00
knlnguyen1802
378385b90c [EC Connector] Optimize remote cache check in scheduler (#32585) 
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
 2026-01-22 03:30:59 +00:00
Wentao Ye
6437ff1fb9 [Deprecation] Remove deprecated environment variables (#32812) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-01-22 02:25:16 +00:00
elvischenv
808d6fd7b9 Bump Flashinfer to v0.6.1 (#30993) 
Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
 2026-01-21 08:49:50 -08:00
Lucas Wilkinson
b4f64e5b02 Update FlashMLA (#32491) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-01-21 13:03:37 +08:00
Nick Hill
6f067b1fb7 [Cleanup] Remove unused KVConnectorModelRunnerMixin methods (#32077) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-21 11:16:37 +08:00
Or Ozeri
7013e9ac8f OffloadingConnector: Prevent redundant loads (#29087) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-01-21 01:15:42 +00:00
Lucas Wilkinson
2261340806 [Misc] Remove pad_for_cudagraphs from config (#30143) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-01-20 15:05:48 -05:00
Rahul Tuli
f0feb1cf81 Test: added acceptance length tests (#32030) 
Signed-off-by: rahul-tuli <rtuli@redhat.com>
 2026-01-20 18:55:15 +00:00
Tomas Ruiz
4a5299c93f feat: spec decode with draft models (#24322) 
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
 2026-01-19 16:05:46 -05:00
Vadim Gimpelson
0727cc9ecf [BUGFIX] Fix test_mla_backends.py. Scale MLA projection weights to prevent numerical instability (#32529) 
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
 2026-01-19 13:49:29 -05:00
Andreas Karatzas
c0a350ca73 [ROCm][CI] Add ROCm attention backend support for EAGLE DP tests (#32363) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-01-19 09:57:54 +00:00
Michael Goin
1be5a73571 [UX] Use kv_offloading_backend=native by default (#32421) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-01-15 18:55:11 +00:00
Wentao Ye
b34474bf2c [Feature] Support async scheduling + PP (#32359) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-01-15 12:06:23 -05:00
Chauncey
707b44cc28 [Refactor] [11/N] to simplify the mcp architecture (#32396) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-01-15 18:49:31 +08:00
Cyrus Leung
cbbae38f93 [2/N] Move cache factories to MM registry (#32382) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-15 01:02:30 -08:00
dtc
1e584823f8 [Bugfix] Strengthen the check of X-data-parallel-rank in Hybrid LB mode (#32314) 
Signed-off-by: Tianchen Ding <dtcccc@linux.alibaba.com>
 2026-01-15 16:31:16 +08:00