biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

[Core] Optimize block_manager_v2 vs block_manager_v1 (to make V2 default) (#5602)

Browse Source
This commit is contained in:
Alexander Matveev
2024-07-01 23:10:37 -04:00
committed by GitHub
parent 54600709b6
commit 3476ed0809
19 changed files with 1182 additions and 525 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
vllm/entrypoints/openai/serving_completion.py
View File

@@ -345,7 +345,7 @@ class OpenAIServingCompletion(OpenAIServing):
out_logprobs = prompt_logprobs
output_text = prompt_text
elif request.echo and request.max_tokens > 0:
token_ids = prompt_token_ids + output.token_ids
token_ids = prompt_token_ids + list(output.token_ids)
out_logprobs = (prompt_logprobs + output.logprobs
if request.logprobs is not None else None)
output_text = prompt_text + output.text