biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Default to generation_config from model (#12622)

Browse Source 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
This commit is contained in:
Harry Mellor
2025-03-08 07:46:15 +01:00
committed by GitHub
parent 3b9c6c6947
commit 47512b3200
7 changed files with 27 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
vllm/entrypoints/openai/serving_chat.py
View File

@@ -109,8 +109,10 @@ class OpenAIServingChat(OpenAIServing):
self.default_sampling_params = (
self.model_config.get_diff_sampling_param())
if self.default_sampling_params:
logger.info("Overwriting default chat sampling param with: %s",
self.default_sampling_params)
source = self.model_config.generation_config
source = "model" if source == "auto" else source
logger.info("Using default chat sampling params from %s: %s",
source, self.default_sampling_params)
async def create_chat_completion(
self,