biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
10,595 Commits 2 Branches 140 Tags
221bf725772b891db27e1b9200c36a721ba9a304
Commit Graph

141 Commits

Author SHA1 Message Date
Simon Mo
02f0c7b220 [Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
Rui Qiao
bdce64f236 [V1] Support DP with Ray (#18779) 2025-06-02 21:15:13 -07:00
Nick Hill
2dbe8c0774 [Perf] API-server scaleout with many-to-many server-engine comms (#17546) 2025-05-30 08:17:00 -07:00
Will Eaton
64eaf5fe05 [P/D] NixlConnector DP fixes (#18903) 
Signed-off-by: Will Eaton <weaton@redhat.com>
 2025-05-29 18:08:40 +00:00
Nick Hill
d1d61f3351 [BugFix] Make DP work with connector-delayed new requests (#18559) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Will Eaton <weaton@redhat.com>
 2025-05-29 18:04:18 +00:00
CYJiang
d58f9c7f7a [Misc] Remove duplicate init for self.vllm_config (#18896) 
Signed-off-by: googs1025 <googs1025@gmail.com>
 2025-05-29 17:26:07 +00:00
Sanger Steel
c32e249a23 [Frontend] [Core] Add Tensorizer support for V1, LoRA adapter serialization and deserialization (#17926) 
Signed-off-by: Sanger Steel <sangersteel@gmail.com>
 2025-05-22 18:44:18 -07:00
Nick Hill
cf5984b2fe [BugFix][DP] Send DP wave completion only from dp_rank==0 (#18502) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: kourosh hakhamaneshi <kourosh@anyscale.com>
 2025-05-21 20:25:25 -07:00
Yong Hoon Shin
907f935de9 [V1] Fix general plugins not loaded in engine for multiproc (#18326) 
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
 2025-05-21 01:21:49 -07:00
Nick Hill
59dd311cf5 [KVConnector] Keep KVTransferParams as a dict (#18033) 2025-05-14 08:05:57 -07:00
Nick Hill
55aa7af994 [V1] DP scale-out (2/N): Decouple engine process management and comms (#15977) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-05-13 10:48:21 -07:00
Cyrus Leung
b922c2ebd2 [Bugfix] Fix entrypoints metrics tests (#18063) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-13 06:42:43 -07:00
Cyrus Leung
61e0a506a3 [Bugfix] Avoid repeatedly creating dummy data during engine startup (#17935) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-12 22:40:19 -07:00
Jade Zheng
289199feb6 [Core] Use platform-agnostic device control for DP engine core (#17245) 
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com>
 2025-05-12 12:09:16 -07:00
Robert Shaw
d19110204c [P/D] NIXL Integration (#17751) 
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: Robert Shaw <rshaw@neuralmagic.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Brent Salisbury <bsalisbu@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: ApostaC <yihua98@uchicago.edu>
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Brent Salisbury <bsalisbu@redhat.com>
 2025-05-12 09:46:16 -07:00
Wallas Henrique
d43f914d42 [Core][Feature] Input metadata dump on crash (#13407) 
Signed-off-by: Wallas Santos <wallashss@ibm.com>
 2025-05-07 22:15:09 +00:00
Alec
0be6d05b5e [V1][Metrics] add support for kv event publishing (#16750) 
Signed-off-by: alec-flowers <aflowers@nvidia.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2025-04-30 07:44:45 -07:00
rongfu.leng
d803786731 [V1][Bugfix]: vllm v1 verison metric num_gpu_blocks is None (#15755) 
Signed-off-by: rongfu.leng <rongfu.leng@daocloud.io>
 2025-04-30 18:20:39 +08:00
Nick Hill
b07bf83c7d [BugFix] Avoid race conditions in zero-copy tensor transmission (#17203) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-26 06:00:07 +00:00
Shanshan Shen
b724afe343 [V1][Structured Output] Clear xgrammar compiler object when engine core shut down to avoid nanobind leaked warning (#16954) 
Signed-off-by: shen-shanshan <467638484@qq.com>
 2025-04-24 06:15:03 -07:00
Rui Qiao
c0dfd97519 [V1][PP] Optimization: continue scheduling prefill chunks (#17080) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
 2025-04-24 05:27:08 -07:00
Nick Hill
b2f195c429 [V1] Avoid socket errors during shutdown when requests are in in-flight (#16807) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-23 12:36:29 +08:00
Nick Hill
1e013fa388 [V1][DP] More robust DP/EP dummy request coordination (#16277) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-22 19:12:15 -07:00
Yihua Cheng
3408e47159 [P/D][V1] KV Connector API V1 (#15960) 
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: remi <remi@mistral.ai>
Co-authored-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Rémi Delacourt <54138269+Flechman@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
 2025-04-17 13:22:40 -07:00
Robert Shaw
2b05b8ce69 [V1][Frontend] Improve Shutdown And Logs (#11737) 
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Signed-off-by: Andrew Feldman <afeldman@neuralmagic.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Andrew Feldman <afeldman@neuralmagic.com>
Co-authored-by: afeldman-nm <156691304+afeldman-nm@users.noreply.github.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-04-16 19:48:34 -07:00
Lily Liu
f49e5aff11 [V1][Spec Decode] KV cache slots for eagle heads (#16370) 
Signed-off-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
 2025-04-12 19:42:51 -07:00
Nick Hill
dd143ef541 [V1] Zero-copy tensor/ndarray serialization/transmission (#13790) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-10 19:23:14 +00:00
Cyrus Leung
e484e02857 [Bugfix] Avoid transferring cached multi-modal items from P0 to P1 (#16273) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-04-09 00:51:27 -07:00
Nick Hill
651cf0fec1 [V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue (#15906) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-04 12:56:43 -07:00
wwl2755
463bbb1835 [Bugfix][V1] Fix bug from putting llm_engine.model_executor in a background process (#15367) 
Signed-off-by: wwl2755 <wangwenlong2755@gmail.com>
 2025-04-03 07:32:10 +00:00
Eric Tang
ddb94c2605 [core] Add tags parameter to wake_up() (#15500) 
Signed-off-by: Eric <erictang000@gmail.com>
 2025-04-02 01:59:27 -07:00
Chen Zhang
3a5f0afcd2 [V1] Implement sliding window attention in kv_cache_manager (#14097) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-04-01 00:33:17 -07:00
Mark McLoughlin
f98a4920f9 [V1][Core] Remove unused speculative config from scheduler (#15818) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-03-31 19:15:21 +00:00
wwl2755
94744ba41a [V1] [Feature] Collective RPC (#15444) 
Signed-off-by: wwl2755 <wangwenlong2755@gmail.com>
 2025-03-29 03:39:14 -07:00
Nick Hill
15dac210f0 [V1] AsyncLLM data parallel (#13923) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-03-27 16:14:41 -07:00
Nick Hill
6aa196c8dc [V1][Minor] Use SchedulerInterface type for engine scheduler field (#15499) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-03-25 14:21:36 -07:00
Chen Zhang
93a00d7dde [v1] Refactor KVCacheConfig (#14079) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-03-21 04:56:27 -07:00
Woosuk Kwon
0c6f5023c3 [V1] Scheduler Refactoring [1/N] - Add Scheduler Interface (#15250) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-03-20 17:50:43 -07:00
Jason
d8e82bc06d [Bugfix] fix V1 Engine crash while handling requests with duplicate request id (#15043) 
Signed-off-by: Jiahui Sun <jhsun2020@gmail.com>
 2025-03-20 10:01:02 -07:00
Jun Duan
74bc397b0a [Core] Expose API endpoint /is_sleeping (#14312) 
Signed-off-by: Jun Duan <jun.duan.phd@outlook.com>
 2025-03-15 06:28:14 -07:00
Joe Runde
165290d357 [bugfix] fixup warning message for plugged schedulers for v1 (#14700) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2025-03-12 20:12:13 -07:00
Joe Runde
47532cd9f4 [core][V1] pluggable scheduler (#14466) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2025-03-12 01:15:15 +00:00
Russell Bryant
4cbf286794 [V1] Remove cache from StructuredOutputManager (#14622) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-03-11 10:36:07 -07:00
Cody Yu
4290b704ff [V1][PP] Do not block engine core when no requests to schedule (#14585) 
Signed-off-by: Cody Yu <hao.yu.cody@gmail.com>
 2025-03-10 19:48:24 -07:00
Nick Hill
8ed5421aaa [V1] Eagerly remove finished requests from the batch (#14388) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-03-07 10:56:00 -08:00
Aaron Pham
80e9afb5bc [V1][Core] Support for Structured Outputs (#12388) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-03-07 07:19:11 -08:00
Tyler Michael Smith
72c62eae5f [V1] EP/TP MoE + DP Attention (#13931) 2025-03-04 21:27:26 -08:00
Nick Hill
872db2be0e [V1] Simplify stats logging (#14082) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-03-03 10:34:14 -08:00
Harry Mellor
cf069aa8aa Update deprecated Python 3.8 typing (#13971) 2025-03-02 17:34:51 -08:00
Varun Sundar Rabindranath
03f48b3db6 [Core] LoRA V1 - Add add/pin/list/remove_lora functions (#13705) 2025-02-25 00:18:02 -08:00