Some Kimi K2.5 model variants (nvidia/Kimi-K2.5-NVFP4) omit
<|tool_calls_section_begin|> and go directly to <|tool_call_begin|>.
The tool parser was only looking for section-level markers, so these
tool calls were forwarded as raw content text instead of being parsed.
Fix: _find_section_start and _find_section_start_end now fall back to
<|tool_call_begin|> as a section start when no section-level marker
is found. The section end falls back to <|tool_call_end|>.
When the model runs out of tokens while still reasoning (no think-end
emitted), all text goes to the reasoning field with zero content — the
model appears silent to the client.
Streaming fix: yield an extra content delta with the extracted reasoning
text before the finish chunk, so the client can see the output.
Non-streaming fix: move reasoning to content when finish_reason=length
and content is None.
Also adds the patched serving.py to the Dockerfile.
Instead of always returning False (which broke tool call streaming),
use a heuristic: if think-end appears in the token IDs but is
followed by more than 3 tokens (chat template wrapping like
<|im_end|>, user markers, etc.), it's from a prior turn's prompt
and reasoning hasn't started in the current generation. Return False.
If think-end is at or near the end, it's from generated tokens and
reasoning has ended. Return True.
The vLLM serving layer calls is_reasoning_end() with prompt_token_ids
to pre-compute whether reasoning has ended before streaming starts. On
multi-turn conversations, prompt_token_ids contains think-end tokens
from prior assistant messages in the chat history. This causes a false
positive — the serving layer sets reasoning_end_arr[i] = True, skips
extract_reasoning_streaming entirely, and routes all thinking text to
content.
By returning False, the serving layer always calls
extract_reasoning_streaming, which correctly tracks reasoning state
via _reasoning_ended based only on the model's generated text.
The streaming path was using is_reasoning_end(previous_token_ids) to
check if reasoning had ended. On multi-turn conversations,
previous_token_ids includes the entire chat history, including
think-end tokens from prior assistant messages. This caused the parser
to incorrectly think reasoning was already over before the model
generated anything, routing all thinking text to content instead of
reasoning.
Fix: Replace the token-ID-based check with a text-based state variable
(_reasoning_ended) that tracks reasoning end based solely on what the
model has generated in the current turn. Reset on each new generation.
Also includes the chat template for reference.
Tool parser:
- Case 3/4: return None instead of DeltaMessage(content='') when
inside an open tool section with no parseable content yet.
Empty-string content deltas pollute the response and break the
content=null vs content='' contract with non-streaming.
Reasoning parser:
- Suppress tool-calls section markers from content forwarding.
The tool parser detects them via current_text re-parsing; forwarding
them as content causes double-handling.
- Already-past-reasoning path: strip section markers from content
for the same reason.
