70 lines
2.1 KiB
Markdown
70 lines
2.1 KiB
Markdown
# Chat Template Debugger
|
|
|
|
Isolate whether tool-call failures are a **model problem** or a **parser/template problem**.
|
|
|
|
Runs vLLM inside Docker, bypasses all OpenClaw middlewares, and captures raw token output from the model directly.
|
|
|
|
## The Problem
|
|
|
|
90% of models break on streaming tool calls. Is it the model generating garbage, or is something in the middleware stack mangling the output? This debugger lets us answer that definitively.
|
|
|
|
## Plan of Attack
|
|
|
|
### 1. Build & Run the Container
|
|
|
|
```bash
|
|
docker build -t ct-debug .
|
|
docker run --gpus all -v $(pwd)/scripts:/workspace/scripts -v $(pwd)/models:/workspace/models -it ct-debug
|
|
```
|
|
|
|
### 2. Stage 0 — Download Weights (if not mounted)
|
|
|
|
```bash
|
|
# Inside the container:
|
|
python /workspace/scripts/stage0_download.py
|
|
```
|
|
|
|
This downloads `HuggingFaceTB/SmolLM3-3B` to `/workspace/models/SmolLM3-3B` if it doesn't already exist.
|
|
|
|
### 3. Stage 1 — Run the Debugger
|
|
|
|
Edit `scripts/stage1_debug.py` to point at the model path and your test prompt. Then:
|
|
|
|
```bash
|
|
# Inside the container:
|
|
python /workspace/scripts/stage1_debug.py
|
|
```
|
|
|
|
This runs the model with a raw prompt (no chat template applied by vLLM's serving layer — you control the prompt string directly). It dumps:
|
|
|
|
- The raw generated text
|
|
- The actual token IDs
|
|
- A per-token decode so you can see exactly what the model emitted
|
|
|
|
### 4. Analyze
|
|
|
|
- If the model emits correct tool-call tokens → **parser/template problem**
|
|
- If the model emits garbage or broken tokens → **model problem**, go fix the LoRA/chat template
|
|
|
|
## Directory Layout
|
|
|
|
```
|
|
chat-template-debugger/
|
|
├── Dockerfile
|
|
├── README.md
|
|
├── models/ # Downloaded weights (gitignored)
|
|
├── scripts/
|
|
│ ├── stage0_download.py
|
|
│ └── stage1_debug.py
|
|
└── prompts/
|
|
└── smol_tool_call.txt
|
|
```
|
|
|
|
## Swapping Models
|
|
|
|
Change `MODEL_ID` in `stage0_download.py` and `MODEL_PATH` in `stage1_debug.py`. Works with any HF model.
|
|
|
|
## Swapping Prompts
|
|
|
|
Drop a `.txt` file in `prompts/` and update the path in `stage1_debug.py`. The prompt is passed as a raw string — no chat template is applied by vLLM. You control the full context.
|