biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

[AsyncScheduling] Don't schedule past request max_tokens (#27922)

Browse Source 
Signed-off-by: Nick Hill <nhill@redhat.com>
This commit is contained in:
Nick Hill
2025-11-04 09:06:28 -08:00
committed by GitHub
parent c9f66da8fd
commit 938a81692e
3 changed files with 14 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
vllm/v1/core/sched/scheduler.py
View File

@@ -217,10 +217,14 @@ class Scheduler(SchedulerInterface):
num_new_tokens = self.scheduler_config.long_prefill_token_threshold
num_new_tokens = min(num_new_tokens, token_budget)
# Make sure the input position does not exceed the max model len.
# This is necessary when using spec decoding.
# Make sure the input position does not exceed the max model len or
# request's max_tokens.
# This is necessary when using spec decoding and/or async scheduling.
max_total_tokens = min(
request.num_prompt_tokens + request.max_tokens, self.max_model_len
)
num_new_tokens = min(
num_new_tokens, self.max_model_len - 1 - request.num_computed_tokens
num_new_tokens, max_total_tokens - 1 - request.num_computed_tokens
)
# Schedule encoder inputs.