[Feature] specify model in config.yaml (#15798)

Signed-off-by: weizeng <weizeng@roblox.com>
This commit is contained in:
Wei Zeng
2025-04-01 01:20:06 -07:00
committed by GitHub
parent 8af5a5c4e5
commit 30d6a015e0
7 changed files with 109 additions and 32 deletions

View File

@@ -188,6 +188,7 @@ For example:
```yaml
# config.yaml
model: meta-llama/Llama-3.1-8B-Instruct
host: "127.0.0.1"
port: 6379
uvicorn-log-level: "info"
@@ -196,12 +197,13 @@ uvicorn-log-level: "info"
To use the above config file:
```bash
vllm serve SOME_MODEL --config config.yaml
vllm serve --config config.yaml
```
:::{note}
In case an argument is supplied simultaneously using command line and the config file, the value from the command line will take precedence.
The order of priorities is `command line > config file values > defaults`.
e.g. `vllm serve SOME_MODEL --config config.yaml`, SOME_MODEL takes precedence over `model` in config file.
:::
## API Reference