biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

[Frontend] Kill the server on engine death (#6594)

Browse Source 
Signed-off-by: Joe Runde <joe@joerun.de>
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
This commit is contained in:
Joe Runde
2024-08-08 10:47:48 -06:00
committed by GitHub
parent 5fb4a3f678
commit 21b9c49aa3
8 changed files with 136 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
vllm/engine/async_llm_engine.py
View File

@@ -58,7 +58,7 @@ def _log_task_completion(task: asyncio.Task,
error_callback(exception)
raise AsyncEngineDeadError(
"Task finished unexpectedly. This should never happen! "
"Please open an issue on Github. See stack trace above for the"
"Please open an issue on Github. See stack trace above for the "
"actual cause.") from e
@@ -132,7 +132,9 @@ class RequestTracker:
self._request_streams[request_id].put(exc)
self.abort_request(request_id)
else:
for rid, stream in self._request_streams.items():
# NB: list() used here because self.abort_request pops the stream
# out of self._request_streams, so we can't iterate on it directly
for rid, stream in list(self._request_streams.items()):
stream.put(exc)
self.abort_request(rid)