# SmolLM3-3B Tool Call Fix — Notes

## Problem

The SmolLM3-3B model's chat template has three bugs that break multi-turn tool calling in vLLM.

## Bugs Found

### Bug 1: Tool responses rendered as plain user messages
**Location:** `chat_template.jinja`, main loop, `message.role == "tool"` branch

**Original:**
```jinja2
{%- elif message.role == "tool" -%}
{{ "<|im_start|>" + "user\n"  + content + "<|im_end|>\n" }}
```

Tool responses show up as `<|im_start|>user\n...<|im_end|>` — the model cannot distinguish a tool result from a new user turn. When it sees weather data in a user message, it re-invokes the tool instead of answering.

**Fix:** Use the model's dedicated `tool_response_start`/`tool_response_end` tokens (128013/128014) to wrap tool responses so the model can distinguish them from user messages.

### Bug 2: Assistant tool_calls not rendered in history
**Location:** `chat_template.jinja`, main loop, `message.role == "assistant"` branch

When the assistant message has `tool_calls`, the template only renders `content` (often empty/None) and drops the entire `tool_calls` array. The model never sees its own prior tool invocations.

**Fix:** Render tool calls using the model's native `tool_call_start`/`tool_call_end` tokens (128015/128016) with proper JSON format.

### Bug 3: Thinking mode inverted
**Location:** `chat_template.jinja`, main loop and generation prompt

When `reasoning_mode == "/think"`, the template does NOT wrap content in think tags. When `reasoning_mode == "/no_think"`, it DOES wrap in `...` tags. Completely backwards.

**Fix:** `/think` mode wraps content in `...` tags. `/no_think` renders plain text.

## Special Tokens

The model has these tool-related tokens in its tokenizer (added_tokens_decoder):

| Token ID | Text | Purpose |
|----------|------|---------|
| 128002 | `...` | Think end |
| 128013 | `...` | Tool call start |
| 128016 | `...` | Tool call end |

## How the Fix Works

### Template Changes

1. **Tool responses** now render as:
   ```
   <|im_start|>user
   [tool_response_start]
   {tool result content}
   [tool_response_end]<|im_end|>
   ```
   Instead of a bare user message.

2. **Assistant tool calls** now render as:
   ```
   <|im_start|>assistant
   {"name": "func_name", "arguments": {...}}
   [tool_call_end]<|im_end|>
   ```
   Instead of being dropped entirely.

3. **Thinking mode** is now correctly mapped: `/think` → think tags, `/no_think` → plain text.

### Key Technical Details

- The template uses Jinja2's `~` operator instead of `+` for string concatenation. This avoids type errors when `message.content` is `None` (Jinja2's `~` coerces to string, `+` does not).
- The `tool_call_start`/`tool_call_end` tokens are Unicode private-use-area characters that can't be typed in a text editor. The template must be generated programmatically using `gen_template.py`.
- The `tc.function.name` and `tc.function.arguments` Jinja2 dot notation works correctly because Jinja2 resolves `dict.key` as `dict["key"]`.
- The `{% generation %}` tag is vLLM-specific and marks the assistant output region. It must be preserved.

## Files

- `model-files/chat_template.jinja` — The fixed template (generated, contains Unicode PUA characters)
- `model-files/gen_template.py` — Script to regenerate the template inside the container where the tokenizer is available
- `model-files/hermes_tool_parser.py` — vLLM Hermes tool parser (unchanged, works as-is for parsing `...` format)

## Deploying

1. Run `gen_template.py` inside the vLLM container:
   ```bash
   docker cp model-files/gen_template.py smol-vllm-1:/tmp/
   docker exec smol-vllm-1 python3 /tmp/gen_template.py
   ```

2. Copy the generated template to the mounted volume:
   ```bash
   docker cp smol-vllm-1:/root/chat_template.jinja /root/smol/chat_template.jinja
   ```

3. Restart the container:
   ```bash
   cd /root/smol && docker compose restart
   ```

## Remaining Issues

- The model sometimes re-invokes tools in a loop instead of providing a final text answer. This is likely a training issue with the `/no_think` mode — the model outputs reasoning as content text but still generates tool calls.
- The Hermes tool parser works for parsing `...` blocks but the streaming parser may buffer long argument strings. This is a vLLM-level issue, not a template issue.