Yang Fan 2c1bd848a6 [Model][VLM] Add Qwen2.5-Omni model support (thinker only) (#15130) ...

Signed-off-by: fyabc <suyang.fy@alibaba-inc.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Xiong Wang <wangxiongts@163.com>

2025-04-18 23:14:36 -07:00

..

__init__.py

[V1]SupportsV0Only protocol for model definitions (#13959)

2025-02-27 20:02:15 -05:00

adapters.py

[Bugfix] Fix missing return value in load_weights method of adapters.py (#15542)

2025-03-31 06:56:42 -07:00

arctic.py

Clean up unused padding_idx variables across many model definitions (#13240)

2025-03-04 21:27:00 +00:00

aria.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

aya_vision.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

baichuan.py

[Model] use AutoWeightsLoader for baichuan, gpt-neox, mpt (#15939)

2025-04-04 09:38:52 -07:00

bamba.py

[Model] Reduce redundant computations in mamba2 blocks for Bamba-9B (#15423)

2025-04-10 19:07:07 +00:00

bart.py

[Quant] BartModel SupportsQuant (#14699)

2025-03-13 04:55:59 +00:00

bert.py

[New Model]: Snowflake Arctic Embed (Family) (#16649)

2025-04-18 08:11:57 -07:00

blip2.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

blip.py

[SupportsQuant] Bert, Blip, Blip2, Bloom (#15573)

2025-04-03 08:23:19 -07:00

bloom.py

[SupportsQuant] Bert, Blip, Blip2, Bloom (#15573)

2025-04-03 08:23:19 -07:00

chameleon.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

chatglm.py

[SupportsQuant] Chameleon, Chatglm, Commandr (#15952)

2025-04-03 08:25:22 -07:00

clip.py

[VLM] Remove BaseProcessingInfo.get_mm_max_tokens_per_item (#16408)

2025-04-10 09:06:58 -07:00

commandr.py

[SupportsQuant] Chameleon, Chatglm, Commandr (#15952)

2025-04-03 08:25:22 -07:00

constant_size_cache.py

[Model][MiniMaxText01] Support MiniMaxText01 model inference (#13454)

2025-04-01 16:23:55 -04:00

dbrx.py

[V1] EP/TP MoE + DP Attention (#13931)

2025-03-04 21:27:26 -08:00

deepseek_mtp.py

[Model][Speculative Decoding] Expand DeepSeek MTP code to support k > n_predict (#13626)

2025-02-27 15:28:08 -08:00

deepseek_v2.py

Revert "[Model] use AutoWeightsLoader for deepseek_v2, internlm2" (#16453)

2025-04-11 06:44:22 +00:00

deepseek_vl2.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

deepseek.py

[Model] use AutoWeightsLoader for phi, gemma, deepseek (#16088)

2025-04-05 20:34:38 -07:00

eagle.py

[Feature] Enhance EAGLE Architecture with Proper RMS Norms (#14990)

2025-03-26 08:24:07 +00:00

exaone.py

[Model] use AutoWeightsLoader in model load_weights (#15770)

2025-04-02 07:47:31 -07:00

fairseq2_llama.py

[Misc] Add SPDX-License-Identifier headers to python source files (#12628)

2025-02-02 11:58:18 -08:00

falcon.py

[Model] use AutoWeightsLoader in model load_weights (#15770)

2025-04-02 07:47:31 -07:00

florence2.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

fuyu.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

gemma2.py

[Misc] Remove misleading message in gemma2 and gemma3 (#14850)

2025-03-14 21:35:12 -07:00

gemma3_mm.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

gemma3.py

[Model] Add module name prefixes to gemma3 (#15889)

2025-04-01 10:13:40 -07:00

gemma.py

[Model] use AutoWeightsLoader for phi, gemma, deepseek (#16088)

2025-04-05 20:34:38 -07:00

glm4.py

[Bugfix] Fix GLM4 model (#16618)

2025-04-17 03:35:07 -07:00

glm4v.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

glm.py

[V1] V1 Enablement Oracle (#13726)

2025-03-14 22:02:20 -07:00

gpt2.py

Remove unused kwargs from model definitions (#13555)

2025-02-24 17:13:52 -08:00

gpt_bigcode.py

[Model] use AutoWeightsLoader for BigCode, GPT-J (#16823)

2025-04-18 10:42:41 +00:00

gpt_j.py

[Model] use AutoWeightsLoader for BigCode, GPT-J (#16823)

2025-04-18 10:42:41 +00:00

gpt_neox.py

[Model] use AutoWeightsLoader for baichuan, gpt-neox, mpt (#15939)

2025-04-04 09:38:52 -07:00

granite.py

[Model] use AutoWeightsLoader for granite, granitemoe, granitemoeshared, grok1, mixtral (#16325)

2025-04-09 20:07:40 -07:00

granitemoe.py

[Model] use AutoWeightsLoader for granite, granitemoe, granitemoeshared, grok1, mixtral (#16325)

2025-04-09 20:07:40 -07:00

granitemoeshared.py

[Model] use AutoWeightsLoader for granite, granitemoe, granitemoeshared, grok1, mixtral (#16325)

2025-04-09 20:07:40 -07:00

gritlm.py

[Bugfix] Fix broken GritLM model and tests (missing pooling_metadata) (#16631)

2025-04-14 23:09:58 -07:00

grok1.py

[Model] use AutoWeightsLoader for granite, granitemoe, granitemoeshared, grok1, mixtral (#16325)

2025-04-09 20:07:40 -07:00

h2ovl.py

[VLM] Remove BaseProcessingInfo.get_mm_max_tokens_per_item (#16408)

2025-04-10 09:06:58 -07:00

idefics2_vision_model.py

[VLM] Clean up Phi-4-MM ViT implementation (#14812)

2025-03-15 18:53:52 -07:00

idefics3.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

interfaces_base.py

Remove unused kwargs from model definitions (#13555)

2025-02-24 17:13:52 -08:00

interfaces.py

[Model] Add SupportsMultiModal.get_language_model interface (#16007)

2025-04-09 04:12:54 -07:00

intern_vit.py

[Misc] Add SPDX-License-Identifier headers to python source files (#12628)

2025-02-02 11:58:18 -08:00

internlm2_ve.py

Remove unused kwargs from model definitions (#13555)

2025-02-24 17:13:52 -08:00

internlm2.py

Revert "[Model] use AutoWeightsLoader for deepseek_v2, internlm2" (#16453)

2025-04-11 06:44:22 +00:00

internvl.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

jais.py

Remove unused kwargs from model definitions (#13555)

2025-02-24 17:13:52 -08:00

jamba.py

[MODEL] Add support for Zamba2 models (#13185)

2025-03-18 08:56:21 -07:00

kimi_vl.py

[Misc] Clean up Kimi-VL (#16833)

2025-04-18 05:15:09 -07:00

llama4.py

[BugFix] Accuracy fix for llama4 int4 - improperly casted scales (#16801)

2025-04-17 22:13:29 -07:00

llama_eagle.py

[V1][Spec Decode] Eagle Model loading (#16035)

2025-04-10 11:21:48 -07:00

llama.py

Upstream Llama4 Support to Main (#16113)

2025-04-07 08:06:27 -07:00

llava_next_video.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

llava_next.py

[Model] Add SupportsMultiModal.get_language_model interface (#16007)

2025-04-09 04:12:54 -07:00

llava_onevision.py

[Model][VLM] Add Qwen2.5-Omni model support (thinker only) (#15130)

2025-04-18 23:14:36 -07:00

llava.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

mamba2.py

[Model] Reduce redundant computations in mamba2 blocks for Bamba-9B (#15423)

2025-04-10 19:07:07 +00:00

mamba_cache.py

[Model][MiniMaxText01] Support MiniMaxText01 model inference (#13454)

2025-04-01 16:23:55 -04:00

mamba.py

Clean up unused padding_idx variables across many model definitions (#13240)

2025-03-04 21:27:00 +00:00

medusa.py

[Misc] Add SPDX-License-Identifier headers to python source files (#12628)

2025-02-02 11:58:18 -08:00

minicpm3.py

Remove unused kwargs from model definitions (#13555)

2025-02-24 17:13:52 -08:00

minicpm.py

Clean up unused padding_idx variables across many model definitions (#13240)

2025-03-04 21:27:00 +00:00

minicpmo.py

[V1] Enable multi-input by default (#15799)

2025-04-12 08:52:39 +00:00

minicpmv.py

[V1] Enable multi-input by default (#15799)

2025-04-12 08:52:39 +00:00

minimax_cache.py

[Model][MiniMaxText01] Support MiniMaxText01 model inference (#13454)

2025-04-01 16:23:55 -04:00

minimax_text_01.py

[Model][MiniMaxText01] Support MiniMaxText01 model inference (#13454)

2025-04-01 16:23:55 -04:00

mistral3.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

mixtral_quant.py

Add support to modelopt quantization of Mixtral model (#15961)

2025-04-09 01:53:31 +00:00

mixtral.py

[Model] use AutoWeightsLoader for granite, granitemoe, granitemoeshared, grok1, mixtral (#16325)

2025-04-09 20:07:40 -07:00

mllama4.py

[Bugfix] fix pp for llama4 (#16746)

2025-04-18 13:51:30 +08:00

mllama.py

[Bugfix] handle alignment of encoder_seq_lens in mllama.py (#14784)

2025-04-11 19:59:50 +00:00

mlp_speculator.py

[Doc] Replace ibm-fms with ibm-ai-platform (#12709)

2025-02-04 07:05:04 +00:00

modernbert.py

[Model] support modernbert (#16648)

2025-04-16 05:30:15 -07:00

module_mapping.py

[Misc] Add SPDX-License-Identifier headers to python source files (#12628)

2025-02-02 11:58:18 -08:00

molmo.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

moonvit.py

[Model][VLM] Add Kimi-VL model support (#16387)

2025-04-14 21:41:48 +00:00

mpt.py

[Model] use AutoWeightsLoader for baichuan, gpt-neox, mpt (#15939)

2025-04-04 09:38:52 -07:00

nemotron_nas.py

[Model] Update support for NemotronNAS models (#15008)

2025-03-31 20:35:14 +08:00

nemotron.py

Apply torchfix (#15532)

2025-03-26 12:09:06 +00:00

nvlm_d.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

olmo2.py

[Bugfix] Fix interface for Olmo2 on V1 (#14976)

2025-03-17 11:26:38 -07:00

olmo.py

Remove unused kwargs from model definitions (#13555)

2025-02-24 17:13:52 -08:00

olmoe.py

[Model] use AutoWeightsLoader for olmoe,opt,orion,persimmon,phi3_small (#16548)

2025-04-17 17:48:31 +00:00

opt.py

[Model] use AutoWeightsLoader for olmoe,opt,orion,persimmon,phi3_small (#16548)

2025-04-17 17:48:31 +00:00

orion.py

[Model] use AutoWeightsLoader for olmoe,opt,orion,persimmon,phi3_small (#16548)

2025-04-17 17:48:31 +00:00

paligemma.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

persimmon.py

[Model] use AutoWeightsLoader for olmoe,opt,orion,persimmon,phi3_small (#16548)

2025-04-17 17:48:31 +00:00

phi3_small.py

[Model] use AutoWeightsLoader for olmoe,opt,orion,persimmon,phi3_small (#16548)

2025-04-17 17:48:31 +00:00

phi3.py

[Misc] Add SPDX-License-Identifier headers to python source files (#12628)

2025-02-02 11:58:18 -08:00

phi3v.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

phi4mm_audio.py

[VLM] Add TP support for Phi-4-MM (#14453)

2025-03-08 05:57:14 -08:00

phi4mm_utils.py

Apply torchfix (#15532)

2025-03-26 12:09:06 +00:00

phi4mm.py

[Model] Add SupportsMultiModal.get_language_model interface (#16007)

2025-04-09 04:12:54 -07:00

phi.py

[Model] use AutoWeightsLoader for phi, gemma, deepseek (#16088)

2025-04-05 20:34:38 -07:00

phimoe.py

[Model] use AutoWeightsLoader for phimoe,qwen2_moe,qwen3_moe (#16203)

2025-04-08 04:05:47 -07:00

pixtral.py

Bugfix for PixtralHF models without spatial_merge_size (#16513)

2025-04-11 23:32:22 +00:00

plamo2.py

[Model] Add PLaMo2 (#14323)

2025-04-15 19:31:30 -07:00

prithvi_geospatial_mae.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

qwen2_5_omni_thinker.py

[Model][VLM] Add Qwen2.5-Omni model support (thinker only) (#15130)

2025-04-18 23:14:36 -07:00

qwen2_5_vl.py

[Model][VLM] Add Qwen2.5-Omni model support (thinker only) (#15130)

2025-04-18 23:14:36 -07:00

qwen2_audio.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

qwen2_moe.py

[Model] use AutoWeightsLoader for phimoe,qwen2_moe,qwen3_moe (#16203)

2025-04-08 04:05:47 -07:00

qwen2_rm.py

[V1]SupportsV0Only protocol for model definitions (#13959)

2025-02-27 20:02:15 -05:00

qwen2_vl.py

[V1] Enable multi-input by default (#15799)

2025-04-12 08:52:39 +00:00

qwen2.py

[Model] Add Qwen3 and Qwen3MoE (#15289)

2025-04-07 04:06:41 -07:00

qwen3_moe.py

[Model] use AutoWeightsLoader for phimoe,qwen2_moe,qwen3_moe (#16203)

2025-04-08 04:05:47 -07:00

qwen3.py

[Model] Add Qwen3 and Qwen3MoE (#15289)

2025-04-07 04:06:41 -07:00

qwen_vl.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

qwen.py

[Bugfix] Fix prompt format of GLM4V (#14539)

2025-03-13 11:37:17 +00:00

registry.py

[Model][VLM] Add Qwen2.5-Omni model support (thinker only) (#15130)

2025-04-18 23:14:36 -07:00

roberta.py

[New Model]: jinaai/jina-embeddings-v3 (#16120)

2025-04-08 08:39:12 -07:00

siglip.py

[VLM] Remove BaseProcessingInfo.get_mm_max_tokens_per_item (#16408)

2025-04-10 09:06:58 -07:00

skyworkr1v.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

smolvlm.py

[Model] Add smolvlm support (#16017)

2025-04-08 19:12:17 -07:00

solar.py

Clean up unused padding_idx variables across many model definitions (#13240)

2025-03-04 21:27:00 +00:00

stablelm.py

[Model] use AutoWeightsLoader for stablelm,starcoder2,zamba2 (#16103)

2025-04-06 05:52:01 -07:00

starcoder2.py

[Model] use AutoWeightsLoader for stablelm,starcoder2,zamba2 (#16103)

2025-04-06 05:52:01 -07:00

telechat2.py

Upstream Llama4 Support to Main (#16113)

2025-04-07 08:06:27 -07:00

teleflm.py

Upstream Llama4 Support to Main (#16113)

2025-04-07 08:06:27 -07:00

transformers.py

[Model][VLM] Add Qwen2.5-Omni model support (thinker only) (#15130)

2025-04-18 23:14:36 -07:00

ultravox.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

utils.py

Optimized topk for topk=1 (Llama-4) (#16512)

2025-04-12 14:21:08 +08:00

vision.py

[VLM] Remove BaseProcessingInfo.get_mm_max_tokens_per_item (#16408)

2025-04-10 09:06:58 -07:00

whisper.py

[VLM] Avoid unnecessary dummy multimodal data during processing (#16416)

2025-04-10 19:32:14 +00:00

zamba2.py

[Model] Reduce redundant computations in mamba2 blocks for Bamba-9B (#15423)

2025-04-10 19:07:07 +00:00