[Doc] Convert docs to use colon fences (#12471)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
This commit is contained in:
@@ -8,54 +8,54 @@ In vLLM, pooling models implement the {class}`~vllm.model_executor.models.VllmMo
|
||||
These models use a {class}`~vllm.model_executor.layers.Pooler` to extract the final hidden states of the input
|
||||
before returning them.
|
||||
|
||||
```{note}
|
||||
:::{note}
|
||||
We currently support pooling models primarily as a matter of convenience.
|
||||
As shown in the [Compatibility Matrix](#compatibility-matrix), most vLLM features are not applicable to
|
||||
pooling models as they only work on the generation or decode stage, so performance may not improve as much.
|
||||
```
|
||||
:::
|
||||
|
||||
For pooling models, we support the following `--task` options.
|
||||
The selected option sets the default pooler used to extract the final hidden states:
|
||||
|
||||
```{list-table}
|
||||
:::{list-table}
|
||||
:widths: 50 25 25 25
|
||||
:header-rows: 1
|
||||
|
||||
* - Task
|
||||
- Pooling Type
|
||||
- Normalization
|
||||
- Softmax
|
||||
* - Embedding (`embed`)
|
||||
- `LAST`
|
||||
- ✅︎
|
||||
- ✗
|
||||
* - Classification (`classify`)
|
||||
- `LAST`
|
||||
- ✗
|
||||
- ✅︎
|
||||
* - Sentence Pair Scoring (`score`)
|
||||
- \*
|
||||
- \*
|
||||
- \*
|
||||
* - Reward Modeling (`reward`)
|
||||
- `ALL`
|
||||
- ✗
|
||||
- ✗
|
||||
```
|
||||
- * Task
|
||||
* Pooling Type
|
||||
* Normalization
|
||||
* Softmax
|
||||
- * Embedding (`embed`)
|
||||
* `LAST`
|
||||
* ✅︎
|
||||
* ✗
|
||||
- * Classification (`classify`)
|
||||
* `LAST`
|
||||
* ✗
|
||||
* ✅︎
|
||||
- * Sentence Pair Scoring (`score`)
|
||||
* \*
|
||||
* \*
|
||||
* \*
|
||||
- * Reward Modeling (`reward`)
|
||||
* `ALL`
|
||||
* ✗
|
||||
* ✗
|
||||
:::
|
||||
|
||||
\*The default pooler is always defined by the model.
|
||||
|
||||
```{note}
|
||||
:::{note}
|
||||
If the model's implementation in vLLM defines its own pooler, the default pooler is set to that instead of the one specified in this table.
|
||||
```
|
||||
:::
|
||||
|
||||
When loading [Sentence Transformers](https://huggingface.co/sentence-transformers) models,
|
||||
we attempt to override the default pooler based on its Sentence Transformers configuration file (`modules.json`).
|
||||
|
||||
```{tip}
|
||||
:::{tip}
|
||||
You can customize the model's pooling method via the `--override-pooler-config` option,
|
||||
which takes priority over both the model's and Sentence Transformers's defaults.
|
||||
```
|
||||
:::
|
||||
|
||||
## Offline Inference
|
||||
|
||||
@@ -111,10 +111,10 @@ The {class}`~vllm.LLM.score` method outputs similarity scores between sentence p
|
||||
It is primarily designed for [cross-encoder models](https://www.sbert.net/examples/applications/cross-encoder/README.html).
|
||||
These types of models serve as rerankers between candidate query-document pairs in RAG systems.
|
||||
|
||||
```{note}
|
||||
:::{note}
|
||||
vLLM can only perform the model inference component (e.g. embedding, reranking) of RAG.
|
||||
To handle RAG at a higher level, you should use integration frameworks such as [LangChain](https://github.com/langchain-ai/langchain).
|
||||
```
|
||||
:::
|
||||
|
||||
```python
|
||||
llm = LLM(model="BAAI/bge-reranker-v2-m3", task="score")
|
||||
|
||||
Reference in New Issue
Block a user