[Doc] Convert docs to use colon fences (#12471)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
This commit is contained in:
@@ -1,8 +1,8 @@
|
||||
# Built-in Extensions
|
||||
|
||||
```{toctree}
|
||||
:::{toctree}
|
||||
:maxdepth: 1
|
||||
|
||||
runai_model_streamer
|
||||
tensorizer
|
||||
```
|
||||
:::
|
||||
|
||||
@@ -48,6 +48,6 @@ You can read further about CPU buffer memory limiting [here](https://github.com/
|
||||
vllm serve /home/meta-llama/Llama-3.2-3B-Instruct --load-format runai_streamer --model-loader-extra-config '{"memory_limit":5368709120}'
|
||||
```
|
||||
|
||||
```{note}
|
||||
:::{note}
|
||||
For further instructions about tunable parameters and additional parameters configurable through environment variables, read the [Environment Variables Documentation](https://github.com/run-ai/runai-model-streamer/blob/master/docs/src/env-vars.md).
|
||||
```
|
||||
:::
|
||||
|
||||
@@ -11,6 +11,6 @@ For more information on CoreWeave's Tensorizer, please refer to
|
||||
[CoreWeave's Tensorizer documentation](https://github.com/coreweave/tensorizer). For more information on serializing a vLLM model, as well a general usage guide to using Tensorizer with vLLM, see
|
||||
the [vLLM example script](https://docs.vllm.ai/en/stable/getting_started/examples/offline_inference/tensorize_vllm_model.html).
|
||||
|
||||
```{note}
|
||||
:::{note}
|
||||
Note that to use this feature you will need to install `tensorizer` by running `pip install vllm[tensorizer]`.
|
||||
```
|
||||
:::
|
||||
|
||||
@@ -70,10 +70,10 @@ The {class}`~vllm.LLM.chat` method implements chat functionality on top of {clas
|
||||
In particular, it accepts input similar to [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat)
|
||||
and automatically applies the model's [chat template](https://huggingface.co/docs/transformers/en/chat_templating) to format the prompt.
|
||||
|
||||
```{important}
|
||||
:::{important}
|
||||
In general, only instruction-tuned models have a chat template.
|
||||
Base models may perform poorly as they are not trained to respond to the chat conversation.
|
||||
```
|
||||
:::
|
||||
|
||||
```python
|
||||
llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct")
|
||||
|
||||
@@ -8,54 +8,54 @@ In vLLM, pooling models implement the {class}`~vllm.model_executor.models.VllmMo
|
||||
These models use a {class}`~vllm.model_executor.layers.Pooler` to extract the final hidden states of the input
|
||||
before returning them.
|
||||
|
||||
```{note}
|
||||
:::{note}
|
||||
We currently support pooling models primarily as a matter of convenience.
|
||||
As shown in the [Compatibility Matrix](#compatibility-matrix), most vLLM features are not applicable to
|
||||
pooling models as they only work on the generation or decode stage, so performance may not improve as much.
|
||||
```
|
||||
:::
|
||||
|
||||
For pooling models, we support the following `--task` options.
|
||||
The selected option sets the default pooler used to extract the final hidden states:
|
||||
|
||||
```{list-table}
|
||||
:::{list-table}
|
||||
:widths: 50 25 25 25
|
||||
:header-rows: 1
|
||||
|
||||
* - Task
|
||||
- Pooling Type
|
||||
- Normalization
|
||||
- Softmax
|
||||
* - Embedding (`embed`)
|
||||
- `LAST`
|
||||
- ✅︎
|
||||
- ✗
|
||||
* - Classification (`classify`)
|
||||
- `LAST`
|
||||
- ✗
|
||||
- ✅︎
|
||||
* - Sentence Pair Scoring (`score`)
|
||||
- \*
|
||||
- \*
|
||||
- \*
|
||||
* - Reward Modeling (`reward`)
|
||||
- `ALL`
|
||||
- ✗
|
||||
- ✗
|
||||
```
|
||||
- * Task
|
||||
* Pooling Type
|
||||
* Normalization
|
||||
* Softmax
|
||||
- * Embedding (`embed`)
|
||||
* `LAST`
|
||||
* ✅︎
|
||||
* ✗
|
||||
- * Classification (`classify`)
|
||||
* `LAST`
|
||||
* ✗
|
||||
* ✅︎
|
||||
- * Sentence Pair Scoring (`score`)
|
||||
* \*
|
||||
* \*
|
||||
* \*
|
||||
- * Reward Modeling (`reward`)
|
||||
* `ALL`
|
||||
* ✗
|
||||
* ✗
|
||||
:::
|
||||
|
||||
\*The default pooler is always defined by the model.
|
||||
|
||||
```{note}
|
||||
:::{note}
|
||||
If the model's implementation in vLLM defines its own pooler, the default pooler is set to that instead of the one specified in this table.
|
||||
```
|
||||
:::
|
||||
|
||||
When loading [Sentence Transformers](https://huggingface.co/sentence-transformers) models,
|
||||
we attempt to override the default pooler based on its Sentence Transformers configuration file (`modules.json`).
|
||||
|
||||
```{tip}
|
||||
:::{tip}
|
||||
You can customize the model's pooling method via the `--override-pooler-config` option,
|
||||
which takes priority over both the model's and Sentence Transformers's defaults.
|
||||
```
|
||||
:::
|
||||
|
||||
## Offline Inference
|
||||
|
||||
@@ -111,10 +111,10 @@ The {class}`~vllm.LLM.score` method outputs similarity scores between sentence p
|
||||
It is primarily designed for [cross-encoder models](https://www.sbert.net/examples/applications/cross-encoder/README.html).
|
||||
These types of models serve as rerankers between candidate query-document pairs in RAG systems.
|
||||
|
||||
```{note}
|
||||
:::{note}
|
||||
vLLM can only perform the model inference component (e.g. embedding, reranking) of RAG.
|
||||
To handle RAG at a higher level, you should use integration frameworks such as [LangChain](https://github.com/langchain-ai/langchain).
|
||||
```
|
||||
:::
|
||||
|
||||
```python
|
||||
llm = LLM(model="BAAI/bge-reranker-v2-m3", task="score")
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user