[doc] improve readability (#18675)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
This commit is contained in:
@@ -11,7 +11,7 @@ vLLM offers an official Docker image for deployment.
|
||||
The image can be used to run OpenAI compatible server and is available on Docker Hub as [vllm/vllm-openai](https://hub.docker.com/r/vllm/vllm-openai/tags).
|
||||
|
||||
```console
|
||||
$ docker run --runtime nvidia --gpus all \
|
||||
docker run --runtime nvidia --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HUGGING_FACE_HUB_TOKEN=<secret>" \
|
||||
-p 8000:8000 \
|
||||
@@ -23,7 +23,7 @@ $ docker run --runtime nvidia --gpus all \
|
||||
This image can also be used with other container engines such as [Podman](https://podman.io/).
|
||||
|
||||
```console
|
||||
$ podman run --gpus all \
|
||||
podman run --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
|
||||
-p 8000:8000 \
|
||||
@@ -73,7 +73,10 @@ You can build and run vLLM from source via the provided <gh-file:docker/Dockerfi
|
||||
|
||||
```console
|
||||
# optionally specifies: --build-arg max_jobs=8 --build-arg nvcc_threads=2
|
||||
DOCKER_BUILDKIT=1 docker build . --target vllm-openai --tag vllm/vllm-openai --file docker/Dockerfile
|
||||
DOCKER_BUILDKIT=1 docker build . \
|
||||
--target vllm-openai \
|
||||
--tag vllm/vllm-openai \
|
||||
--file docker/Dockerfile
|
||||
```
|
||||
|
||||
!!! note
|
||||
@@ -96,8 +99,8 @@ of PyTorch Nightly and should be considered **experimental**. Using the flag `--
|
||||
|
||||
```console
|
||||
# Example of building on Nvidia GH200 server. (Memory usage: ~15GB, Build time: ~1475s / ~25 min, Image size: 6.93GB)
|
||||
$ python3 use_existing_torch.py
|
||||
$ DOCKER_BUILDKIT=1 docker build . \
|
||||
python3 use_existing_torch.py
|
||||
DOCKER_BUILDKIT=1 docker build . \
|
||||
--file docker/Dockerfile \
|
||||
--target vllm-openai \
|
||||
--platform "linux/arm64" \
|
||||
@@ -113,7 +116,7 @@ $ DOCKER_BUILDKIT=1 docker build . \
|
||||
To run vLLM with the custom-built Docker image:
|
||||
|
||||
```console
|
||||
$ docker run --runtime nvidia --gpus all \
|
||||
docker run --runtime nvidia --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
-p 8000:8000 \
|
||||
--env "HUGGING_FACE_HUB_TOKEN=<secret>" \
|
||||
|
||||
@@ -82,7 +82,11 @@ Check the output of the command. There will be a shareable gradio link (like the
|
||||
**Optional**: Serve the 70B model instead of the default 8B and use more GPU:
|
||||
|
||||
```console
|
||||
HF_TOKEN="your-huggingface-token" sky launch serving.yaml --gpus A100:8 --env HF_TOKEN --env MODEL_NAME=meta-llama/Meta-Llama-3-70B-Instruct
|
||||
HF_TOKEN="your-huggingface-token" \
|
||||
sky launch serving.yaml \
|
||||
--gpus A100:8 \
|
||||
--env HF_TOKEN \
|
||||
--env MODEL_NAME=meta-llama/Meta-Llama-3-70B-Instruct
|
||||
```
|
||||
|
||||
## Scale up to multiple replicas
|
||||
@@ -155,7 +159,9 @@ run: |
|
||||
Start the serving the Llama-3 8B model on multiple replicas:
|
||||
|
||||
```console
|
||||
HF_TOKEN="your-huggingface-token" sky serve up -n vllm serving.yaml --env HF_TOKEN
|
||||
HF_TOKEN="your-huggingface-token" \
|
||||
sky serve up -n vllm serving.yaml \
|
||||
--env HF_TOKEN
|
||||
```
|
||||
|
||||
Wait until the service is ready:
|
||||
@@ -318,7 +324,9 @@ run: |
|
||||
1. Start the chat web UI:
|
||||
|
||||
```console
|
||||
sky launch -c gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint vllm)
|
||||
sky launch \
|
||||
-c gui ./gui.yaml \
|
||||
--env ENDPOINT=$(sky serve status --endpoint vllm)
|
||||
```
|
||||
|
||||
2. Then, we can access the GUI at the returned gradio link:
|
||||
|
||||
@@ -33,7 +33,8 @@ pip install streamlit openai
|
||||
streamlit run streamlit_openai_chatbot_webserver.py
|
||||
|
||||
# or specify the VLLM_API_BASE or VLLM_API_KEY
|
||||
VLLM_API_BASE="http://vllm-server-host:vllm-server-port/v1" streamlit run streamlit_openai_chatbot_webserver.py
|
||||
VLLM_API_BASE="http://vllm-server-host:vllm-server-port/v1" \
|
||||
streamlit run streamlit_openai_chatbot_webserver.py
|
||||
|
||||
# start with debug mode to view more details
|
||||
streamlit run streamlit_openai_chatbot_webserver.py --logger.level=debug
|
||||
|
||||
@@ -77,7 +77,11 @@ If you are behind proxy, you can pass the proxy settings to the docker build com
|
||||
|
||||
```console
|
||||
cd $vllm_root
|
||||
docker build -f docker/Dockerfile . --tag vllm --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy
|
||||
docker build \
|
||||
-f docker/Dockerfile . \
|
||||
--tag vllm \
|
||||
--build-arg http_proxy=$http_proxy \
|
||||
--build-arg https_proxy=$https_proxy
|
||||
```
|
||||
|
||||
[](){ #nginxloadbalancer-nginx-docker-network }
|
||||
@@ -102,8 +106,26 @@ Notes:
|
||||
```console
|
||||
mkdir -p ~/.cache/huggingface/hub/
|
||||
hf_cache_dir=~/.cache/huggingface/
|
||||
docker run -itd --ipc host --network vllm_nginx --gpus device=0 --shm-size=10.24gb -v $hf_cache_dir:/root/.cache/huggingface/ -p 8081:8000 --name vllm0 vllm --model meta-llama/Llama-2-7b-chat-hf
|
||||
docker run -itd --ipc host --network vllm_nginx --gpus device=1 --shm-size=10.24gb -v $hf_cache_dir:/root/.cache/huggingface/ -p 8082:8000 --name vllm1 vllm --model meta-llama/Llama-2-7b-chat-hf
|
||||
docker run \
|
||||
-itd \
|
||||
--ipc host \
|
||||
--network vllm_nginx \
|
||||
--gpus device=0 \
|
||||
--shm-size=10.24gb \
|
||||
-v $hf_cache_dir:/root/.cache/huggingface/ \
|
||||
-p 8081:8000 \
|
||||
--name vllm0 vllm \
|
||||
--model meta-llama/Llama-2-7b-chat-hf
|
||||
docker run \
|
||||
-itd \
|
||||
--ipc host \
|
||||
--network vllm_nginx \
|
||||
--gpus device=1 \
|
||||
--shm-size=10.24gb \
|
||||
-v $hf_cache_dir:/root/.cache/huggingface/ \
|
||||
-p 8082:8000 \
|
||||
--name vllm1 vllm \
|
||||
--model meta-llama/Llama-2-7b-chat-hf
|
||||
```
|
||||
|
||||
!!! note
|
||||
@@ -114,7 +136,12 @@ docker run -itd --ipc host --network vllm_nginx --gpus device=1 --shm-size=10.24
|
||||
## Launch Nginx
|
||||
|
||||
```console
|
||||
docker run -itd -p 8000:80 --network vllm_nginx -v ./nginx_conf/:/etc/nginx/conf.d/ --name nginx-lb nginx-lb:latest
|
||||
docker run \
|
||||
-itd \
|
||||
-p 8000:80 \
|
||||
--network vllm_nginx \
|
||||
-v ./nginx_conf/:/etc/nginx/conf.d/ \
|
||||
--name nginx-lb nginx-lb:latest
|
||||
```
|
||||
|
||||
[](){ #nginxloadbalancer-nginx-verify-nginx }
|
||||
|
||||
Reference in New Issue
Block a user