biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

[Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models (#5649)

Browse Source 
Co-authored-by: constellate <constellate@1-ai-appserver-staging.codereach.com>
Co-authored-by: Kyle Mistele <kyle@constellate.ai>
This commit is contained in:
Kyle Mistele
2024-09-04 15:18:13 -05:00
committed by GitHub
parent 561d6f8077
commit e02ce498be
26 changed files with 2591 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
vllm/model_executor/guided_decoding/__init__.py
View File

@@ -59,8 +59,9 @@ def _adapt_request_for_tool_use(request: Union[CompletionRequest,
if type(request) is CompletionRequest:
return request
# user has chosen to not use any tool
if request.tool_choice == "none":
# user has chosen to not use any tool,
# OR is allowing the model to choose a tool.
if request.tool_choice == "none" or request.tool_choice == "auto":
return request
# user has chosen to use a named tool