81 lines
3.5 KiB
Markdown
81 lines
3.5 KiB
Markdown
|
# SmolLM3-3B Tool Call Fix — Notes
|
||
|
|
|
||
|
|
## Status: SOLVED ✅
|
||
|
|
|
||
|
|
All three template bugs fixed, reasoning parser working, tool calling functional.
|
||
|
|
|
||
|
|
## What Was Fixed
|
||
|
|
|
||
|
|
### Bug 1: Tool responses rendered as plain user messages
|
||
|
|
Tool responses showed up as `<|im_start|>user\n...` — model couldn't distinguish them from new user turns and kept re-calling tools. Fixed by wrapping tool responses with the model's dedicated `tool_response_start`/`tool_response_end` tokens (128013/128014).
|
||
|
|
|
||
|
|
### Bug 2: Assistant tool_calls not rendered in history
|
||
|
|
When assistant message had `tool_calls`, the template only rendered `content` and dropped the tool call array. Model never saw its own prior invocations. Fixed by rendering tool calls using `tool_call_start`/`tool_call_end` tokens (128015/128016).
|
||
|
|
|
||
|
|
### Bug 3: Thinking mode direction swapped
|
||
|
|
`/think` mode produced bare assistant prompt (no think tags), `/no_think` wrapped in think tags. Completely backwards. Fixed: `/think` opens `...` tags, `/no_think` is plain text.
|
||
|
|
|
||
|
|
## Special Tokens
|
||
|
|
|
||
|
|
| Token ID | Text | Purpose |
|
||
|
|
|----------|------|---------|
|
||
|
|
| 128002 | `...` | Tool call start |
|
||
|
|
| 128016 | `...` | Tool call end |
|
||
|
|
|
||
|
|
## Patched Files (in model-files/)
|
||
|
|
|
||
|
|
### `chat_template.jinja` — Fixed template
|
||
|
|
Three fixes applied:
|
||
|
|
1. Tool responses wrapped in `tool_response_start`/`tool_response_end` tokens
|
||
|
|
2. Assistant tool_calls rendered in `tool_call_start`/`tool_call_end` format
|
||
|
|
3. Thinking mode direction corrected
|
||
|
|
|
||
|
|
Uses Jinja2 `~` operator (not `+`) to avoid type errors when `message.content` is None.
|
||
|
|
|
||
|
|
### `gen_template.py` — Template generator
|
||
|
|
Regenerates `chat_template.jinja` inside the container where the tokenizer is available. Required because the special tokens are Unicode private-use-area characters that can't be typed in editors.
|
||
|
|
|
||
|
|
### `smol_tool_parser.py` — Tool call parser is just the unchanged hermes_tool_parser.py in case we need to change it
|
||
|
|
The stock vLLM Hermes parser works as-is for parsing `...` blocks. No patches needed.
|
||
|
|
|
||
|
|
## Reasoning Parser — NOT PATCHED
|
||
|
|
|
||
|
|
The built-in `deepseek_r1` reasoning parser in vLLM works with SmolLM3 out of the box — they share the same `...` tokens. Verified by diffing the container's copy against the vllm source: identical, no patches needed.
|
||
|
|
|
||
|
|
## Deploying
|
||
|
|
|
||
|
|
1. Generate template inside the container:
|
||
|
|
```bash
|
||
|
|
docker cp model-files/gen_template.py smol-vllm-1:/tmp/
|
||
|
|
docker exec smol-vllm-1 python3 /tmp/gen_template.py
|
||
|
|
```
|
||
|
|
|
||
|
|
2. Copy to mounted volume and restart:
|
||
|
|
```bash
|
||
|
|
docker cp smol-vllm-1:/root/chat_template.jinja /root/smol/chat_template.jinja
|
||
|
|
cd /root/smol && docker compose restart
|
||
|
|
```
|
||
|
|
|
||
|
|
3. Required vLLM flags:
|
||
|
|
```
|
||
|
|
--chat-template=/root/chat_template.jinja
|
||
|
|
--enable-auto-tool-choice
|
||
|
|
--tool-call-parser=hermes
|
||
|
|
--reasoning-parser=deepseek_r1
|
||
|
|
--chat-template-content-format=string
|
||
|
|
```
|
||
|
|
|
||
|
|
## Test Results
|
||
|
|
|
||
|
|
- ✅ Tool response tests: All PASS (streaming + non-streaming)
|
||
|
|
- ✅ Streaming tool calls: Incremental, 325+ chunks
|
||
|
|
- ✅ Reasoning parser: Correctly splits thinking/content
|
||
|
|
- ✅ Multi-turn tool use: Model reads results, answers properly
|
||
|
|
- ⚠️ 3B model doesn't reliably choose tools over free-text for complex tasks (writes code as content instead of calling write_file). This is a model capability gap, not a parsing issue. Planned LoRA to address.
|
||
|
|
|
||
|
|
## Next Steps
|
||
|
|
|
||
|
|
- **LoRA training** to make tool calling more reliable (especially forced tool use scenarios)
|
||
|
|
- Candidate dataset: `interstellarninja/tool-calls-multiturn`
|
||
|
|
- Also worth considering: `NousResearch/Hermes-Function-Calling-V1`, `Salesforce/xLAM-function-calling-60k`
|