biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
7,143 Commits 2 Branches 140 Tags
a77aea59fd2f0300160dee6fff2e359f572d7f57
Commit Graph

231 Commits

Author SHA1 Message Date
Nick Hill
bd517eb9fe [BugFix] Fix DP Coordinator incorrect debug log message (#19624) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-14 00:18:03 +00:00
Nick Hill
d5bdf899e4 [BugFix] Work-around incremental detokenization edge case error (#19449) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-12 06:43:20 +00:00
Adolfo Victoria
ca27f0f9c1 [Bugfix][Core] Update cancellation logic in generate() to handle Generator exits (#19225) 
Co-authored-by: Adolfo Victoria <adovi@meta.com>
 2025-06-06 20:17:54 +00:00
jmswen
7353492a47 [Core] Raise when non-multi-instance DP clients target a DP rank (#19227) 
Signed-off-by: Jon Swenson <jmswen@gmail.com>
 2025-06-06 19:03:01 +08:00
CYJiang
23027e2daf [Misc] refactor: simplify EngineCoreClient.make_async_mp_client in AsyncLLM (#18817) 
Signed-off-by: googs1025 <googs1025@gmail.com>
 2025-06-04 15:37:25 -07:00
jmswen
c8dcc15921 Allow AsyncLLMEngine.generate to target a specific DP rank (#19102) 
Signed-off-by: Jon Swenson <jmswen@gmail.com>
 2025-06-04 08:26:47 -07:00
Simon Mo
02f0c7b220 [Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
Raushan Turganbay
d81edded69 [Bugfix] disable processor cache (#19068) 
Signed-off-by: raushan <raushan@huggingface.co>
 2025-06-03 15:06:04 +00:00
Rui Qiao
bdce64f236 [V1] Support DP with Ray (#18779) 2025-06-02 21:15:13 -07:00
Nick Hill
9a1b9b99d7 [BugFix] Fix multi-node offline data-parallel (#18981) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Yizhou Liu <liu_yizhou@outlook.com>
 2025-05-31 08:34:52 -07:00
Nick Hill
2dbe8c0774 [Perf] API-server scaleout with many-to-many server-engine comms (#17546) 2025-05-30 08:17:00 -07:00
Will Eaton
64eaf5fe05 [P/D] NixlConnector DP fixes (#18903) 
Signed-off-by: Will Eaton <weaton@redhat.com>
 2025-05-29 18:08:40 +00:00
Nick Hill
d1d61f3351 [BugFix] Make DP work with connector-delayed new requests (#18559) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Will Eaton <weaton@redhat.com>
 2025-05-29 18:04:18 +00:00
CYJiang
d58f9c7f7a [Misc] Remove duplicate init for self.vllm_config (#18896) 
Signed-off-by: googs1025 <googs1025@gmail.com>
 2025-05-29 17:26:07 +00:00
Mark McLoughlin
06a0338015 [V1][Metrics] Add API for accessing in-memory Prometheus metrics (#17010) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-05-27 09:37:06 +00:00
Seiji Eicher
7891fdf0c6 [V1] Fix _pickle.PicklingError: Can't pickle <class 'transformers_modules.deepseek-ai.DeepSeek-V2-Lite... (#18640) 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
 2025-05-24 20:07:20 -07:00
Chauncey
b046cf792d [Feature][V1]: suupports cached_tokens in response usage (#18149) 
Co-authored-by: simon-mo <xmo@berkeley.edu>
 2025-05-23 01:41:03 -07:00
Sanger Steel
c32e249a23 [Frontend] [Core] Add Tensorizer support for V1, LoRA adapter serialization and deserialization (#17926) 
Signed-off-by: Sanger Steel <sangersteel@gmail.com>
 2025-05-22 18:44:18 -07:00
Nick Hill
cf5984b2fe [BugFix][DP] Send DP wave completion only from dp_rank==0 (#18502) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: kourosh hakhamaneshi <kourosh@anyscale.com>
 2025-05-21 20:25:25 -07:00
Yong Hoon Shin
907f935de9 [V1] Fix general plugins not loaded in engine for multiproc (#18326) 
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
 2025-05-21 01:21:49 -07:00
Nick Hill
59dd311cf5 [KVConnector] Keep KVTransferParams as a dict (#18033) 2025-05-14 08:05:57 -07:00
Nick Hill
55aa7af994 [V1] DP scale-out (2/N): Decouple engine process management and comms (#15977) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-05-13 10:48:21 -07:00
Cyrus Leung
b922c2ebd2 [Bugfix] Fix entrypoints metrics tests (#18063) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-13 06:42:43 -07:00
Cyrus Leung
61e0a506a3 [Bugfix] Avoid repeatedly creating dummy data during engine startup (#17935) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-12 22:40:19 -07:00
Jade Zheng
289199feb6 [Core] Use platform-agnostic device control for DP engine core (#17245) 
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com>
 2025-05-12 12:09:16 -07:00
Robert Shaw
d19110204c [P/D] NIXL Integration (#17751) 
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: Robert Shaw <rshaw@neuralmagic.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Brent Salisbury <bsalisbu@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: ApostaC <yihua98@uchicago.edu>
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Brent Salisbury <bsalisbu@redhat.com>
 2025-05-12 09:46:16 -07:00
Cheng Kuan Yong Jason
08bf784078 [Bugfix] validate grammar and throw 400 error instead of crashing the engine when xgrammar validation fails (#17623) 
Signed-off-by: Jason Cheng <jasoncky96@gmail.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
 2025-05-12 09:06:10 +08:00
Ben Browning
8132365b74 [Bugfix]: v1 engine - consider lora adapters in allowed_token_ids (#17855) 
Signed-off-by: Ben Browning <bbrownin@redhat.com>
 2025-05-11 00:53:58 -07:00
Nick Hill
3d13ca0e24 [BugFix] Fix --disable-log-stats in V1 server mode (#17600) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-05-08 04:08:15 +00:00
Wallas Henrique
d43f914d42 [Core][Feature] Input metadata dump on crash (#13407) 
Signed-off-by: Wallas Santos <wallashss@ibm.com>
 2025-05-07 22:15:09 +00:00
Harry Mellor
d6484ef3c3 Add full API docs and improve the UX of navigating them (#17485) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-03 19:42:43 -07:00
Alec
0be6d05b5e [V1][Metrics] add support for kv event publishing (#16750) 
Signed-off-by: alec-flowers <aflowers@nvidia.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2025-04-30 07:44:45 -07:00
Marko Rosenmueller
77073c77bc [Core] Prevent side-channel attacks via cache salting (#17045) 
Signed-off-by: Marko Rosenmueller <5467316+dr75@users.noreply.github.com>
 2025-04-30 20:27:21 +08:00
rongfu.leng
d803786731 [V1][Bugfix]: vllm v1 verison metric num_gpu_blocks is None (#15755) 
Signed-off-by: rongfu.leng <rongfu.leng@daocloud.io>
 2025-04-30 18:20:39 +08:00
Gabriel Marinho
1c2bc7ead0 Truncation control for embedding models (#14776) 
Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Max de Bayser <mbayser@br.ibm.com>
 2025-04-30 09:24:57 +08:00
Harry Mellor
a6977dbd15 Simplify (and fix) passing of guided decoding backend options (#17008) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-04-29 19:02:23 +00:00
Cyrus Leung
ebb3930d28 [Misc] Move config fields to MultiModalConfig (#17343) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-04-29 06:37:21 +00:00
Nick Hill
df6f3ce883 [Core] Remove prompt string from engine core data structures (#17214) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-25 23:41:05 -07:00
Nick Hill
b07bf83c7d [BugFix] Avoid race conditions in zero-copy tensor transmission (#17203) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-26 06:00:07 +00:00
Zijing Liu
53e8cf53a4 [V1][Metrics] Allow V1 AsyncLLM to use custom logger (#14661) 
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-04-25 22:05:40 -07:00
Daniel Li
48cb2109b6 [V1] Move usage stats to worker and start logging TPU hardware (#16211) 2025-04-25 14:06:01 -06:00
Lu Fang
fc966e9cc6 Only turn on FastIncrementalDetokenizer when tokenizers >= 0.21.1 (#17158) 2025-04-25 17:10:32 +08:00
Yinghai Lu
fe92176321 Add collective_rpc to llm engine (#16999) 
Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai>
 2025-04-24 20:16:52 +00:00
Shanshan Shen
b724afe343 [V1][Structured Output] Clear xgrammar compiler object when engine core shut down to avoid nanobind leaked warning (#16954) 
Signed-off-by: shen-shanshan <467638484@qq.com>
 2025-04-24 06:15:03 -07:00
Rui Qiao
c0dfd97519 [V1][PP] Optimization: continue scheduling prefill chunks (#17080) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
 2025-04-24 05:27:08 -07:00
Harry Mellor
0a05ed57e6 Simplify TokenizerGroup (#16790) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-04-24 04:43:56 -07:00
Michael Goin
ed50f46641 [Bugfix] Enable V1 usage stats (#16986) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-04-23 19:54:00 -07:00
Travis Johnson
3cde34a4a4 [Frontend] Support guidance:no-additional-properties for compatibility with xgrammar (#15949) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2025-04-23 18:34:41 +00:00
Harry Mellor
bdb3660312 Use @property and private field for data_parallel_rank_local (#17053) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-04-23 08:50:08 -07:00
Nick Hill
b2f195c429 [V1] Avoid socket errors during shutdown when requests are in in-flight (#16807) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-23 12:36:29 +08:00