[Docs] Convert rST to MyST (Markdown) (#11145)
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
This commit is contained in:
16
docs/source/design/multimodal/adding_multimodal_plugin.md
Normal file
16
docs/source/design/multimodal/adding_multimodal_plugin.md
Normal file
@@ -0,0 +1,16 @@
|
||||
(adding-multimodal-plugin)=
|
||||
|
||||
# Adding a Multimodal Plugin
|
||||
|
||||
This document teaches you how to add a new modality to vLLM.
|
||||
|
||||
Each modality in vLLM is represented by a {class}`~vllm.multimodal.MultiModalPlugin` and registered to {data}`~vllm.multimodal.MULTIMODAL_REGISTRY`.
|
||||
For vLLM to recognize a new modality type, you have to create a new plugin and then pass it to {meth}`~vllm.multimodal.MultiModalRegistry.register_plugin`.
|
||||
|
||||
The remainder of this document details how to define custom {class}`~vllm.multimodal.MultiModalPlugin` s.
|
||||
|
||||
```{note}
|
||||
This article is a work in progress.
|
||||
```
|
||||
|
||||
% TODO: Add more instructions on how to add new plugins once embeddings is in.
|
||||
@@ -1,17 +0,0 @@
|
||||
.. _adding_multimodal_plugin:
|
||||
|
||||
Adding a Multimodal Plugin
|
||||
==========================
|
||||
|
||||
This document teaches you how to add a new modality to vLLM.
|
||||
|
||||
Each modality in vLLM is represented by a :class:`~vllm.multimodal.MultiModalPlugin` and registered to :data:`~vllm.multimodal.MULTIMODAL_REGISTRY`.
|
||||
For vLLM to recognize a new modality type, you have to create a new plugin and then pass it to :meth:`~vllm.multimodal.MultiModalRegistry.register_plugin`.
|
||||
|
||||
The remainder of this document details how to define custom :class:`~vllm.multimodal.MultiModalPlugin` s.
|
||||
|
||||
.. note::
|
||||
This article is a work in progress.
|
||||
|
||||
..
|
||||
TODO: Add more instructions on how to add new plugins once embeddings is in.
|
||||
@@ -1,66 +1,83 @@
|
||||
.. _multi_modality:
|
||||
(multi-modality)=
|
||||
|
||||
Multi-Modality
|
||||
==============
|
||||
# Multi-Modality
|
||||
|
||||
```{eval-rst}
|
||||
.. currentmodule:: vllm.multimodal
|
||||
|
||||
vLLM provides experimental support for multi-modal models through the :mod:`vllm.multimodal` package.
|
||||
```
|
||||
|
||||
Multi-modal inputs can be passed alongside text and token prompts to :ref:`supported models <supported_mm_models>`
|
||||
via the ``multi_modal_data`` field in :class:`vllm.inputs.PromptType`.
|
||||
vLLM provides experimental support for multi-modal models through the {mod}`vllm.multimodal` package.
|
||||
|
||||
Multi-modal inputs can be passed alongside text and token prompts to [supported models](#supported-mm-models)
|
||||
via the `multi_modal_data` field in {class}`vllm.inputs.PromptType`.
|
||||
|
||||
Currently, vLLM only has built-in support for image data. You can extend vLLM to process additional modalities
|
||||
by following :ref:`this guide <adding_multimodal_plugin>`.
|
||||
by following [this guide](#adding-multimodal-plugin).
|
||||
|
||||
Looking to add your own multi-modal model? Please follow the instructions listed :ref:`here <enabling_multimodal_inputs>`.
|
||||
Looking to add your own multi-modal model? Please follow the instructions listed [here](#enabling-multimodal-inputs).
|
||||
|
||||
Guides
|
||||
++++++
|
||||
## Guides
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
```{toctree}
|
||||
:maxdepth: 1
|
||||
|
||||
adding_multimodal_plugin
|
||||
adding_multimodal_plugin
|
||||
```
|
||||
|
||||
Module Contents
|
||||
+++++++++++++++
|
||||
## Module Contents
|
||||
|
||||
```{eval-rst}
|
||||
.. automodule:: vllm.multimodal
|
||||
```
|
||||
|
||||
Registry
|
||||
--------
|
||||
### Registry
|
||||
|
||||
```{eval-rst}
|
||||
.. autodata:: vllm.multimodal.MULTIMODAL_REGISTRY
|
||||
```
|
||||
|
||||
```{eval-rst}
|
||||
.. autoclass:: vllm.multimodal.MultiModalRegistry
|
||||
:members:
|
||||
:show-inheritance:
|
||||
```
|
||||
|
||||
Base Classes
|
||||
------------
|
||||
### Base Classes
|
||||
|
||||
```{eval-rst}
|
||||
.. autodata:: vllm.multimodal.NestedTensors
|
||||
```
|
||||
|
||||
```{eval-rst}
|
||||
.. autodata:: vllm.multimodal.BatchedTensorInputs
|
||||
```
|
||||
|
||||
```{eval-rst}
|
||||
.. autoclass:: vllm.multimodal.MultiModalDataBuiltins
|
||||
:members:
|
||||
:show-inheritance:
|
||||
```
|
||||
|
||||
```{eval-rst}
|
||||
.. autodata:: vllm.multimodal.MultiModalDataDict
|
||||
```
|
||||
|
||||
```{eval-rst}
|
||||
.. autoclass:: vllm.multimodal.MultiModalKwargs
|
||||
:members:
|
||||
:show-inheritance:
|
||||
```
|
||||
|
||||
```{eval-rst}
|
||||
.. autoclass:: vllm.multimodal.MultiModalPlugin
|
||||
:members:
|
||||
:show-inheritance:
|
||||
```
|
||||
|
||||
Image Classes
|
||||
-------------
|
||||
### Image Classes
|
||||
|
||||
```{eval-rst}
|
||||
.. automodule:: vllm.multimodal.image
|
||||
:members:
|
||||
:show-inheritance:
|
||||
```
|
||||
Reference in New Issue
Block a user