biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

[Models] Support Qwen model with PP (#6974)

Browse Source 
Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>
This commit is contained in:
Murali Andoorveedu
2024-08-01 12:40:43 -07:00
committed by GitHub
parent f4fd390f5d
commit fc912e0886
3 changed files with 46 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/source/serving/distributed_serving.rst
View File

@@ -50,7 +50,7 @@ You can also additionally specify :code:`--pipeline-parallel-size` to enable pip
$ --pipeline-parallel-size 2
.. note::
Pipeline parallel is a beta feature. It is only supported for online serving as well as LLaMa, GPT2, and Mixtral style models.
Pipeline parallel is a beta feature. It is only supported for online serving as well as LLaMa, GPT2, Mixtral, Qwen, Qwen2, and Nemotron style models.
Multi-Node Inference and Serving
--------------------------------