biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Default model load/config/tokenizer to mistral format if relevant files exist (#28659)

Browse Source 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
Signed-off-by: Julien Denize <40604584+juliendenize@users.noreply.github.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
This commit is contained in:
Julien Denize
2025-11-21 22:58:59 +01:00
committed by GitHub
parent c68c7b403d
commit 57430fc95c
15 changed files with 230 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
docs/features/tool_calling.md
View File

@@ -142,7 +142,7 @@ Flags: `--tool-call-parser hermes`
Supported models:
* `mistralai/Mistral-7B-Instruct-v0.3` (confirmed)
* Additional mistral function-calling models are compatible as well.
* Additional Mistral function-calling models are compatible as well.
Known issues:
@@ -158,12 +158,25 @@ Known issues:
Recommended flags:
1. To use [mistral-common](https://github.com/mistralai/mistral-common) the official Mistral tokenization backend:
1. To use the official Mistral AI's format:
`--tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral`
`--tool-call-parser mistral`
2. To use the default Transformers tokenization backend:
`--tool-call-parser mistral --chat-template examples/tool_chat_template_mistral_parallel.jinja`
2. To use the Transformers format when available:
`--tokenizer_mode hf --config_format hf --load_format hf --tool-call-parser mistral --chat-template examples/tool_chat_template_mistral_parallel.jinja`
!!! note
Models officially released by Mistral AI have two possible formats:
1. The official format that is used by default with `auto` or `mistral` arguments:
`--tokenizer_mode mistral --config_format mistral --load_format mistral`
This format uses [mistral-common](https://github.com/mistralai/mistral-common), the Mistral AI's tokenizer backend.
2. The Transformers format, when available, that is used with `hf` arguments:
`--tokenizer_mode hf --config_format hf --load_format hf --chat-template examples/tool_chat_template_mistral_parallel.jinja`
### Llama Models (`llama3_json`)