biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
14,900 Commits 2 Branches 140 Tags
bcf2be96120005e9aea171927f85055a6a5c0cf6
Commit Graph

85 Commits

Author SHA1 Message Date
Cyrus Leung
392645454b [Refactor] Decouple TimingContext from InputProcessingContext (#35083) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-23 14:15:50 +00:00
Cyrus Leung
987506bca6 [Refactor] Simplify dummy data generation (#35025) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-22 20:55:27 -08:00
Cyrus Leung
a0d8d944e2 [Renderer] Move MM Hash parsing into Renderer (#34711) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-17 19:18:55 -08:00
Cyrus Leung
73391a1baa [Renderer] Move InputPreprocessor into Renderer (1/2) (#34510) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2026-02-14 10:14:21 -08:00
Isotr0py
0ab06100f4 [Multimodal] Expose mm_processor_kwargs for DummyInputsBuilder (#34330) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-02-11 09:37:40 -08:00
Cyrus Leung
88c3e114d8 [Refactor] Move MM data parsing outside processor (#33408) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-31 16:46:14 +00:00
Cyrus Leung
dcd80206b7 [Chore] Update type annotation of input_ids in model forward (#33063) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-26 06:02:10 -08:00
Cyrus Leung
2b8a38b6d6 [Model] Extend collect_children and no_init_weights contexts (#32757) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-22 08:20:27 +00:00
Cyrus Leung
fda3f03eb2 [4/N] Initialize MM components in context managers (M-P) (#32663) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-20 14:06:32 +00:00
Cyrus Leung
9ea07b41da [1/N] Reorganize multimodal processing code (#32327) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-14 15:25:31 +00:00
Ce Zhao
a051525e07 [Model] Enable LoRA support for PaliGemma (#31656) 
Signed-off-by: 赵策 <alcor@mac.mynetworksettings.com>
Signed-off-by: Alcor <alcor_zhao@outlook.com>
Co-authored-by: 赵策 <alcor@mac.mynetworksettings.com>
 2026-01-07 10:09:32 +08:00
Cyrus Leung
c46b932df2 [Chore] Deprecate SupportsMultiModal.merge_by_field_config (#30170) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-06 07:57:28 +00:00
Cyrus Leung
b286a311c2 [Chore] Deprecate merge_by_field_config arg (#30035) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-04 17:21:24 +00:00
Harry Mellor
97d1c99302 Rename clashing method names for vLLM model protocol (#27583) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-11-12 19:14:33 -08:00
Luciano Martins
e05a6754a8 [Model] Revert PR #26715: Restore custom PaliGemma and Gemma3-MM impl… (#27309) 
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
 2025-10-22 10:05:34 -07:00
Cyrus Leung
8c017b3490 [Model] Always use Transformers backend for PaliGemma and Gemma3-MM (#26715) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-17 05:03:35 +00:00
Harry Mellor
8fcaaf6a16 Update Optional[x] -> x | None and Union[x, y] to x | y (#26633) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-12 09:51:31 -07:00
Harry Mellor
d6953beb91 Convert formatting to use ruff instead of yapf + isort (#26247) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 07:06:22 -07:00
Wenlong Wang
79aa244678 [Multi Modal] Configurable MM Profiling (#25631) 
Signed-off-by: wwl2755 <wangwenlong2755@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-03 03:59:10 -07:00
Cyrus Leung
0ad9951c41 [Input] Remove unused prompt field (#26097) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-03 00:23:21 -07:00
Isotr0py
bd51f78e39 [V0 Deprecation][Models] Remove all V0 condition for mm embeddings merge (#25331) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: isotr0py <2037008807@qq.com>
 2025-09-29 14:09:18 +08:00
Cyrus Leung
27d7638b94 [Bugfix] Merge MM embeddings by index instead of token IDs (#16229) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: NickLucche <nlucches@redhat.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2025-09-27 08:15:12 +00:00
Woosuk Kwon
1c3ffdbecc [V0 Deprecation] Remove V0 sampling metadata (#25345) 
Signed-off-by: Woosuk Kwon <woosuk@thinkingmachines.ai>
 2025-09-21 10:37:11 -07:00
Chenheli Hua
009d689b0c [Core] Simplify and unify mm uuid handling & auto-generated mm hash overrides processing. (#24271) 
Signed-off-by: Chenheli Hua <huachenheli@outlook.com>
 2025-09-09 21:36:09 -07:00
Roger Wang
8bf6266a17 [Multimodal] Generate mm_hash based on request metadata when caching is turned off (#23690) 
Signed-off-by: Roger Wang <hey@rogerw.io>
 2025-08-27 20:24:31 +00:00
Benji Beck
a75277285b Migrate Paligemma inputs to TensorSchema (#23470) 
Signed-off-by: Benji Beck <benjibeck@meta.com>
 2025-08-24 04:56:56 +00:00
Roger Wang
79f05e4436 [Multimodal] Always enable hashing mm data (#23308) 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-21 07:23:28 -07:00
Cyrus Leung
27e8d1ea3e [Refactor] Define MultiModalKwargsItems separate from MultiModalKwargs (#23053) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-18 09:52:00 +00:00
Isotr0py
01cae37713 [CI/Build] Ensure compatability with Transformers v4.53 (#20541) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-07-11 20:53:07 -07:00
Cyrus Leung
b024a42e93 [Core] Move multimodal placeholder from chat utils to model definition (#20355) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-07-03 08:18:30 +00:00
Kyle Sayers
d8cf819a9a [Core] [Bugfix] [Multimodal] Fix multimodal profiling and generation for SFT/PTQed models (#20058) 
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
 2025-06-30 17:26:49 +00:00
Russell Bryant
14fdd21d39 [Core] More fixes to MultiModalEmbeddings type handling (#19715) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-06-18 22:48:29 +00:00
Isotr0py
ca94d7fa00 [Bugfix] Update multimodel models mapping to fit new checkpoint after Transformers v4.52 (#19151) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-06-17 15:58:38 +00:00
Russell Bryant
90f9c2eb5c [V1] Change return type on get_multimodal_embeddings() (#19446) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-06-16 13:32:15 -04:00
Simon Mo
02f0c7b220 [Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
Harry Mellor
26d0419309 Update deprecated type hinting in models (#18132) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-14 22:06:50 -07:00
Woosuk Kwon
b411418ff0 [Chore] Remove Sampler from Model Code (#17084) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-24 02:49:33 -07:00
Cyrus Leung
56d4aefa33 [VLM] Avoid unnecessary dummy multimodal data during processing (#16416) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-04-10 19:32:14 +00:00
Cyrus Leung
83b824c8b4 [VLM] Remove BaseProcessingInfo.get_mm_max_tokens_per_item (#16408) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-04-10 09:06:58 -07:00
Nicolò Lucchesi
d55244df31 [Model] Add SupportsMultiModal.get_language_model interface (#16007) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-04-09 04:12:54 -07:00
Roger Wang
f2ebb6f541 [V1] Scatter and gather placeholders in the model runner (#16076) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>
 2025-04-08 10:43:41 +08:00
Roger Wang
af51d80fa1 Revert "[V1] Scatter and gather placeholders in the model runner" (#16075) 2025-04-04 14:50:57 -07:00
Cyrus Leung
f5722a5052 [V1] Scatter and gather placeholders in the model runner (#15712) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2025-04-04 21:26:44 +00:00
Cyrus Leung
ab656f2c2f [Bugfix] Loosen type check to avoid errors in V1 (#15021) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-03-18 12:54:40 +00:00
Rémi Delacourt
61c6a5a796 [VLM] Merged multi-modal processor for Pixtral (#12211) 
Signed-off-by: remi <remi@mistral.ai>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-03-15 06:28:27 -07:00
Cyrus Leung
601bd3268e [Misc] Clean up type annotation for SupportsMultiModal (#14794) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-03-14 00:59:56 -07:00
Isotr0py
1477ffc381 [VLM] Cleanup siglip legacy code and fix broken paligemma multimodal processor (#14602) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-03-11 11:27:36 +00:00
kYLe
1769928079 [Model] Update Paligemma multimodal processing with PromptUpdate (#14015) 
Signed-off-by: Kyle Huang <kylhuang@nvidia.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2025-03-06 08:31:38 +00:00
lkchen
b3cf368d79 [V1][Molmo] Fix get_multimodal_embeddings() in molmo.py (#14161) 2025-03-04 15:43:59 +00:00
Roger Wang
6c85da3a18 [V1]SupportsV0Only protocol for model definitions (#13959) 
Signed-off-by: Roger Wang <ywang@roblox.com>
 2025-02-27 20:02:15 -05:00