biondizzle 8cfc1cae58 Canonical encoding: derive special token IDs from official encoding module + tokenizer ...

- Remove hardcoded THINK_START/THINK_END/USER_TOKEN/ASSISTANT_TOKEN IDs
- Import token strings from encoding.deepseek_v4_encoding (official source)
- Resolve IDs via tokenizer.convert_tokens_to_ids() at runtime
- Use parse_message_from_completion_text() for structured output parsing
- No more hand-rolled prompt construction or hardcoded token IDs
- Clean up TEMP: replace old deepseek_v4_ref with dsv4thing.zip reference

2026-06-03 10:23:02 +00:00

..

dsv4thing

Canonical encoding: derive special token IDs from official encoding module + tokenizer

2026-06-03 10:23:02 +00:00

dsv4thing.zip

Canonical encoding: derive special token IDs from official encoding module + tokenizer

2026-06-03 10:23:02 +00:00