biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
14,386 Commits 2 Branches 140 Tags
fa6a6be51978bd4b49ba0da17039e60f96dc5b13
Commit Graph

1011 Commits

Author SHA1 Message Date
Aaron Hao
89a385d79f [Feat][RL] Pause and Resume with keep requests for single engine (#32351) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
Signed-off-by: Aaron Hao <ahao@anyscale.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2026-02-07 00:08:58 +00:00
Seiji Eicher
aca5967416 [KV Connector] Add missing method overrides to MultiConnector (#33292) 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
 2026-02-06 12:58:21 -05:00
emricksini-h
325ab6b0a8 [Feature] OTEL tracing during loading (#31162) 2026-02-05 16:59:28 -08:00
Benjamin Chislett
af3162d3aa [Spec Decode] Unified Parallel Drafting (#32887) 
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
 2026-02-05 12:37:18 -05:00
liranschour
8322d4e47f Enable Cross layers KV cache layout at NIXL Connector V2 (#33339) 
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: liranschour <liranschour@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-02-05 02:17:02 -08:00
Mark McLoughlin
2abd97592f [KV Connector][Metrics] Do not count local prefix cache hits in connector queries (#30522) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2026-02-05 09:57:27 +02:00
Nick Hill
add9f1fbd9 [Minor] Include StreamingInput in inputs package (#33856) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-05 04:38:20 +00:00
Nick Hill
fa4e0fb028 [Core] Don't schedule spec tokens with prefill chunks (#33652) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-04 23:40:22 +00:00
Or Ozeri
8e32690869 [KV Connector][BugFix] scheduler: Delay freeing blocks of aborted async loads (#32255) 
Fixes a not-yet-reported case where it was possible for blocks to be
freed by an abort before an async transfer completed, resulting
in corrupted KV data.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-02-04 11:16:34 +00:00
zhanqiuhu
4403e3ed4c [Metrics] Add labeled prompt token metrics for P/D disaggregation (#33290) 
Add labeled Prometheus metrics to distinguish where prompt tokens come
from in P/D disaggregated deployments.

In P/D disaggregation, decode instances receive KV cache from prefill instances.
Currently, decode reports inflated prompt throughput because it counts all
prompt tokens as "computed", even though most were transferred.

This PR adds labeled metrics so users can understand actual compute work vs
transferred work:

vllm:prompt_tokens_by_source_total{source="local_compute"}        # Tokens prefilled locally
vllm:prompt_tokens_by_source_total{source="external_kv_transfer"} # Tokens received via KV transfer  
vllm:prompt_tokens_by_source_total{source="local_cache_hit"}      # Tokens from local prefix cache
vllm:prompt_tokens_cached_total                                    # Total cached (local + external, -1 when all 

Signed-off-by: Zhanqiu Hu <zh338@cornell.edu>
 2026-02-04 07:46:48 +00:00
Frank Wang
45f8fd6f97 [Feature] Enable TRITON_ATTN for Batch Invariance (#33688) 
Signed-off-by: frankwang28 <frank.wbb@hotmail.com>
 2026-02-04 13:27:34 +08:00
Nick Hill
52ee21021a [BugFix][Spec Decoding] Fix negative accepted tokens metric crash (#33729) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-03 23:34:41 +00:00
Harry Mellor
61e632aea1 Turn @config into a dataclass_transform (#31541) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-02-03 17:40:59 +00:00
yugong333
ffe1fc7a28 Reduce the kernel overhead when num of active loras is smaller than max loras. Multiple cuda graphs are captured for each num of active-loras. (#32005) 
Signed-off-by: Yu Gong <yu3.gong@gmail.com>
 2026-02-02 12:30:06 -05:00
shanjiaz
d95b4be47a move spec decode slow test to test_areas.yaml (#33365) 
Signed-off-by: shanjiaz <zsjwpianpian@gmail.com>
 2026-02-02 06:28:36 -08:00
Nicolò Lucchesi
528b3076af [CI][Bugfix] Fix flaky tests/v1/kv_connector/unit/test_multi_connector.py::test_multi_example_connector_consistency (#33555) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-02-02 03:01:29 -08:00
Yifan Qiao
a01ef3fa51 [Fix] prefix cache hit rate == 0 bug with gpt-oss style models (#33524) 
Signed-off-by: Yifan Qiao <yifanqiao@berkeley.edu>
 2026-02-02 01:59:58 +00:00
jma99_2333
22d9a056d5 Support clear mm and encoder cache (#33452) 
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-01-31 15:22:25 +00:00
Nick Hill
876a16f4fb [ModelRunner V2] Fix spec decoding + logprobs (#33391) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-31 03:33:26 +00:00
Matthew Bonanni
aaa901ad55 [Attention] Move MLA forward from backend to layer (#33284) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-01-30 19:30:00 -08:00
Kyle Sayers
f857a03f6b [QeRL] Layerwise Reloading (#32133) 
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
 2026-01-30 08:50:05 -07:00
Frank Wang
8f5d51203b Disable Cascade Attention for Batch Invariance (#32561) 
Signed-off-by: frankwang28 <frank.wbb@hotmail.com>
Signed-off-by: Frank Wang <41319051+frankwang28@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-01-30 10:00:46 -05:00
Patrick von Platen
10152d2194 [Realtime API] Adds minimal realtime API based on websockets (#33187) 
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-01-30 18:41:29 +08:00
Cyrus Leung
c6e7404cc5 [Multimodal] Simplify MM input definitions (#33331) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-29 13:32:04 +00:00
shanjiaz
5eeba80c74 Adding optional speculator tests for larger models (#32943) 
Signed-off-by: shanjiaz <zsjwpianpian@gmail.com>
 2026-01-29 16:54:02 +08:00
Wentao Ye
3e440786af [Feature] Fully support for async scheduling + PP, 30.8% E2E throughput improvement, 31.8% TPOT improvement (#32618) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-01-28 20:30:32 +00:00
Or Ozeri
2e8de86777 Revert "Enable Cross layers KV cache layout at NIXL Connector (#30207)" (#33241) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
 2026-01-28 04:36:00 -08:00
Matthew Bonanni
a608b4c6c2 [5/N][Attention] Finish eliminating vllm/attention folder (#32064) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-01-27 10:02:51 -05:00
omerpaz95
7227d06156 [Metrics] [KVConnector] Add Offloading Connector metrics (#27942) 
Added queries and hits metrics for the Offloading Connector.

Also added timing metrics for store and load operations, which take the
average time it takes to load/store, per-token.

The metrics are available from Prometheus and from the StatLogger.

Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: Omer Paz <Omer.Paz@ibm.com>
 2026-01-27 13:34:49 +00:00
Cyrus Leung
c25dbee40d [Model] Bump transformers version for test registry (#33100) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-26 18:53:22 +00:00
Cyrus Leung
11b556878b [Refactor] Use data parser for matching data items to multi-modal UUIDs (#32955) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-26 15:00:28 +08:00
Joshua Deng
91601ff478 [Feature] add session based streaming input support to v1 (#28973) 
Signed-off-by: Joshua Deng <joshuakdeng@gmail.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-01-24 12:06:28 -08:00
7. Sun
cd775bdbe0 [Tests] Replace flaky sleep with polling in test_background_cancel (#32986) 
Signed-off-by: 7. Sun <jhao.sun@gmail.com>
 2026-01-24 16:39:07 +00:00
7. Sun
0ccecf8833 [Tests] Standardize RNG seed utility across test files (#32982) 
Signed-off-by: 7. Sun <jhao.sun@gmail.com>
 2026-01-24 06:47:14 +00:00
7. Sun
0b9a735e11 [Tests] Clarify pytest skip reasons with actionable context (#32981) 
Signed-off-by: 7. Sun <jhao.sun@gmail.com>
 2026-01-24 06:38:50 +00:00
ElizaWszola
a28b94e6ef [Performance] Split FlashAttn attention and cache update (#25954) 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Luka Govedič <luka.govedic@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Luka Govedič <luka.govedic@gmail.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Luka Govedič <lgovedic@redhat.com>
 2026-01-23 17:28:06 -08:00
Lucas Wilkinson
3a41459501 [cudagraphs] Refactor cudagraph capture loop (#32946) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-01-23 13:22:20 -07:00
Harry Huang
5206e5e28c [V1][Hybrid] Mamba Prefix Caching with align mode (#30877) 
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
 2026-01-23 09:56:48 -08:00
David Ramon Prados
3a63be0faa Support custom URI schemes and trace handlers for profiler (#32393) 2026-01-22 09:45:40 -08:00
Matt
c517d8c934 [Hardware][AMD][CI][Bugfix] Fix regressions from deprecated env vars (#32837) 
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
 2026-01-23 00:59:15 +08:00
Cyrus Leung
d117a4d1a9 [Frontend] Introduce Renderer for processing chat messages (using ModelConfig) (#30200) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-22 12:44:22 +00:00
Or Ozeri
421012b63a OffloadingConnector: Support kernel_block_size != block_size (#30692) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-01-22 12:30:04 +00:00
liranschour
64e3d67ac0 Enable Cross layers KV cache layout at NIXL Connector (#30207) 
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: liranschour <liranschour@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-01-22 10:12:58 +00:00
Andreas Karatzas
a810299838 [ROCm][CI][Docs] Add comment explaining TRITON_ATTN fallback for ROCm (#32835) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-01-21 22:11:09 -08:00
Micah Williamson
019e2c3b7c [ROCm][CI] Lower Acceptance Len Threshold For test_draft_model_quantization (#32731) 
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
 2026-01-22 05:47:33 +00:00
Lucas Wilkinson
889722f3bf [FlashMLA] Update FlashMLA to expose new arguments (#32810) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-01-21 22:02:39 -07:00
Divakar Verma
49d9653852 [ROCm][CI] fix get_valid_backends (#32787) 
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
 2026-01-22 04:27:47 +00:00
knlnguyen1802
378385b90c [EC Connector] Optimize remote cache check in scheduler (#32585) 
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
 2026-01-22 03:30:59 +00:00
Wentao Ye
6437ff1fb9 [Deprecation] Remove deprecated environment variables (#32812) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-01-22 02:25:16 +00:00
elvischenv
808d6fd7b9 Bump Flashinfer to v0.6.1 (#30993) 
Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
 2026-01-21 08:49:50 -08:00