biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

[Doc] Move guide for multimodal model and other improvements (#6168)

Browse Source
This commit is contained in:
Cyrus Leung
2024-07-06 17:18:59 +08:00
committed by GitHub
parent 175c43eca4
commit 9389380015
8 changed files with 61 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
docs/source/dev/multimodal/multimodal_index.rst
View File

@@ -7,25 +7,17 @@ Multi-Modality
vLLM provides experimental support for multi-modal models through the :mod:`vllm.multimodal` package.
:class:`vllm.inputs.PromptStrictInputs` accepts an additional attribute ``multi_modal_data``
which allows you to pass in multi-modal input alongside text and token prompts.
Multi-modal input can be passed alongside text and token prompts to :ref:`supported models <supported_vlms>`
via the ``multi_modal_data`` field in :class:`vllm.inputs.PromptStrictInputs`.
.. note::
``multi_modal_data`` can accept keys and values beyond the builtin ones, as long as a customized plugin is registered through
:class:`vllm.multimodal.MULTIMODAL_REGISTRY`.
the :class:`~vllm.multimodal.MULTIMODAL_REGISTRY`.
By default, vLLM models do not support multi-modal inputs. To enable multi-modal support for a model, please follow :ref:`the guide for adding a new multimodal model. <adding_a_new_multimodal_model>`.
To implement a new multi-modal model in vLLM, please follow :ref:`this guide <enabling_multimodal_inputs>`.
# TODO: Add more instructions on how to do that once embeddings is in.
Guides
++++++
.. toctree::
:maxdepth: 1
adding_multimodal_model
..
TODO: Add more instructions on how to add new plugins once embeddings is in.
Module Contents
+++++++++++++++