[Doc] Guide for adding multi-modal plugins (#6205)

This commit is contained in:
Cyrus Leung
2024-07-10 14:55:34 +08:00
committed by GitHub
parent 5ed3505d82
commit 8a924d2248
7 changed files with 64 additions and 23 deletions

View File

@@ -0,0 +1,17 @@
.. _adding_multimodal_plugin:
Adding a Multimodal Plugin
==========================
This document teaches you how to add a new modality to vLLM.
Each modality in vLLM is represented by a :class:`~vllm.multimodal.MultiModalPlugin` and registered to :data:`~vllm.multimodal.MULTIMODAL_REGISTRY`.
For vLLM to recognize a new modality type, you have to create a new plugin and then pass it to :meth:`~vllm.multimodal.MultiModalRegistry.register_plugin`.
The remainder of this document details how to define custom :class:`~vllm.multimodal.MultiModalPlugin` s.
.. note::
This article is a work in progress.
..
TODO: Add more instructions on how to add new plugins once embeddings is in.