biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

[Bugfix]: Clamp -inf logprob values in prompt_logprobs (#11073)

Browse Source 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
This commit is contained in:
Rafael Vasquez
2024-12-11 04:27:39 -05:00
committed by GitHub
parent 2e32f5d28d
commit 40766ca1b8
1 changed files with 6 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
vllm/entrypoints/openai/serving_completion.py
View File

@@ -392,6 +392,12 @@ class OpenAIServingCompletion(OpenAIServing):
prompt_token_ids = final_res.prompt_token_ids
assert prompt_token_ids is not None
prompt_logprobs = final_res.prompt_logprobs
if prompt_logprobs:
for logprob_dict in prompt_logprobs:
if logprob_dict:
for logprob_values in logprob_dict.values():
if logprob_values.logprob == float('-inf'):
logprob_values.logprob = -9999.0
prompt_text = final_res.prompt
token_ids: GenericSequence[int]