biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

[BugFix] Do not raise a ValueError when tool_choice is set to the supported none option and tools are not defined. (#10000)

Browse Source 
Signed-off-by: Guillaume Calmettes <gcalmettes@scaleway.com>
This commit is contained in:
Guillaume Calmettes
2024-11-12 12:13:46 +01:00
committed by GitHub
parent d201d41973
commit 36c513a076
3 changed files with 19 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/source/serving/openai_compatible_server.md
View File

@@ -215,10 +215,10 @@ The order of priorities is `command line > config file values > defaults`.
---
## Tool calling in the chat completion API
vLLM supports named function calling and `auto` tool choice in the chat completion API. The `tool_choice` options `required` is **not yet supported** but on the roadmap.
vLLM currently supports named function calling, as well as the `auto` and `none` options for the `tool_choice` field in the chat completion API. The `tool_choice` option `required` is **not yet supported** but on the roadmap.
It is the callers responsibility to prompt the model with the tool information, vLLM will not automatically manipulate the prompt.
Please see below for recommended configuration and chat templates to use when function calling is to be used with the different models.
### Named Function Calling