biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
14,985 Commits 2 Branches 140 Tags
738d0a281fab2e151a67b370c26b4e4360362f8f
Commit Graph

335 Commits

Author SHA1 Message Date
Ricky Xu
584f0ae40d [V1] Make AsyncLLMEngine v1-v0 opaque (#11383) 
Signed-off-by: Ricky Xu <xuchen727@hotmail.com>
 2024-12-21 15:14:08 +08:00
Joe Runde
2d1b9baa8f [Bugfix] Fix request cancellation without polling (#11190)
Some checks failed
Create Release / Create Release (push) Has been cancelled
Details
Create Release / Build Wheel (11.8, ubuntu-20.04, 3.10, 2.4.0) (push) Has been cancelled
Details
Create Release / Build Wheel (11.8, ubuntu-20.04, 3.11, 2.4.0) (push) Has been cancelled
Details
Create Release / Build Wheel (11.8, ubuntu-20.04, 3.12, 2.4.0) (push) Has been cancelled
Details
Create Release / Build Wheel (11.8, ubuntu-20.04, 3.9, 2.4.0) (push) Has been cancelled
Details
Create Release / Build Wheel (12.1, ubuntu-20.04, 3.10, 2.4.0) (push) Has been cancelled
Details
Create Release / Build Wheel (12.1, ubuntu-20.04, 3.11, 2.4.0) (push) Has been cancelled
Details
Create Release / Build Wheel (12.1, ubuntu-20.04, 3.12, 2.4.0) (push) Has been cancelled
Details
Create Release / Build Wheel (12.1, ubuntu-20.04, 3.9, 2.4.0) (push) Has been cancelled
Details
2024-12-17 12:26:32 -08:00
Cyrus Leung
0920ab9131 [Doc] Reorganize online pooling APIs (#11172) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-14 00:22:22 +08:00
Russell Bryant
ccede2b264 [Core] cleanup zmq ipc sockets on exit (#11115) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-12-11 19:12:24 -08:00
Cyrus Leung
8f10d5e393 [Misc] Split up pooling tasks (#10820) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-11 01:28:00 -08:00
Joe Runde
980ad394a8 [Frontend] Use request id from header (#10968) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2024-12-10 13:46:29 +08:00
Russell Bryant
69d357ba12 [Core] Cleanup startup logging a bit (#10961) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-12-07 02:30:23 +00:00
Ricky Xu
519e8e4182 [v1] EngineArgs for better config handling for v1 (#10382) 
Signed-off-by: rickyx <rickyx@anyscale.com>
 2024-11-25 21:09:43 -08:00
Chauncey
d04b13a380 [Bug]: Authorization ignored when root_path is set (#10606) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2024-11-25 16:21:41 +00:00
Maximilien de Bayser
214efc2c3c Support Cross encoder models (#10400) 
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Flavia Beo <flavia.beo@ibm.com>
Co-authored-by: Flavia Beo <flavia.beo@ibm.com>
 2024-11-24 18:56:20 -08:00
Cyrus Leung
32e46e000f [Frontend] Automatic detection of chat content format from AST (#9919) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-11-16 13:35:40 +08:00
Zijin Xiao
554af9228d [Bugfix] use AF_INET6 for OpenAI Compatible Server with ipv6 (#9583) 
Signed-off-by: xiaozijin <xiaozijin@bytedance.com>
 2024-11-14 16:38:53 -08:00
zifeitong
47db6ec831 [Frontend] Add per-request number of cached token stats (#10174) 2024-11-12 16:42:28 +00:00
Robert Shaw
6ace6fba2c [V1] AsyncLLM Implementation (#9826) 
Signed-off-by: Nick Hill <nickhill@us.ibm.com>
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2024-11-11 23:05:38 +00:00
cjackal
d88bff1b96 [Frontend] add add_request_id middleware (#9594) 
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
 2024-11-09 10:18:29 +00:00
Russell Bryant
5952d81139 [Frontend] Fix tcp port reservation for api server (#10012) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-11-05 07:50:57 -08:00
Robert Shaw
04cef2c6ab [Bugfix] Fix MQLLMEngine hanging (#9973) 
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
 2024-11-04 16:01:43 -05:00
Cyrus Leung
06386a64dd [Frontend] Chat-based Embeddings API (#9759) 2024-11-01 08:13:35 +00:00
Kevin H. Luu
890ca36072 Revert "[Bugfix] Use host argument to bind to interface (#9798)" (#9852) 2024-10-31 01:44:51 +00:00
Joe Runde
3b3f1e7436 [Bugfix][core] replace heartbeat with pid check (#9818) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2024-10-30 09:34:07 -07:00
Sven Seeberg
0f43387157 [Bugfix] Use host argument to bind to interface (#9798) 2024-10-29 10:37:59 -07:00
Daniele
9a94ca4a5d [Bugfix] fix OpenAI API server startup with --disable-frontend-multiprocessing (#8537) 2024-10-08 09:38:40 -07:00
Alex Brooks
069d3bd8d0 [Frontend] Add Early Validation For Chat Template / Tool Call Parser (#9151) 
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
 2024-10-08 14:31:26 +00:00
代君
3dbb215b38 [Frontend][Feature] support tool calling for internlm/internlm2_5-7b-chat model (#8405) 2024-10-04 10:36:39 +08:00
Jiaxin Shan
260d40b5ea [Core] Support Lora lineage and base model metadata management (#6315) 2024-09-20 06:20:56 +00:00
Nick Hill
d9cd78eb71 [BugFix] Nonzero exit code if MQLLMEngine startup fails (#8572) 2024-09-18 20:17:55 +00:00
Alexander Matveev
7c7714d856 [Core][Bugfix][Perf] Introduce MQLLMEngine to avoid asyncio OH (#8157) 
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-09-18 13:56:58 +00:00
Jiaxin Shan
e351572900 [Misc] Add argument to disable FastAPI docs (#8554) 2024-09-18 09:51:59 +00:00
Kevin Lin
47f5e03b5b [Bugfix] Bind api server port before starting engine (#8491) 2024-09-16 13:56:28 -07:00
Nick Hill
acd5511b6d [BugFix] Fix clean shutdown issues (#8492) 2024-09-16 09:33:46 -07:00
lewtun
837c1968f9 [Frontend] Expose revision arg in OpenAI server (#8501) 2024-09-16 15:55:26 +00:00
Jiaxin Shan
db3bf7c991 [Core] Support load and unload LoRA in api server (#6566) 
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
 2024-09-05 18:10:33 -07:00
Kyle Mistele
e02ce498be [Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models (#5649) 
Co-authored-by: constellate <constellate@1-ai-appserver-staging.codereach.com>
Co-authored-by: Kyle Mistele <kyle@constellate.ai>
 2024-09-04 13:18:13 -07:00
Nick Hill
d4db9f53c8 [Benchmark] Add --async-engine option to benchmark_throughput.py (#7964) 2024-09-03 20:57:41 -04:00
Joe Runde
b903e1ba7f [Frontend] error suppression cleanup (#7786) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2024-08-22 21:50:21 +00:00
Joe Runde
cde9183b40 [Bug][Frontend] Improve ZMQ client robustness (#7443) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2024-08-22 02:18:11 +00:00
William Lin
dd53c4b023 [misc] Add Torch profiler support (#7451) 
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
 2024-08-21 15:39:26 -07:00
Robert Shaw
970dfdc01d [Frontend] Improve Startup Failure UX (#7716) 2024-08-21 19:53:01 +00:00
Robert Shaw
f7e3b0c5aa [Bugfix][Frontend] Fix Issues Under High Load With zeromq Frontend (#7394) 
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-08-21 13:34:14 -04:00
Cyrus Leung
baaedfdb2d [mypy] Enable following imports for entrypoints (#7248) 
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Fei <dfdfcai4@gmail.com>
 2024-08-20 23:28:21 -07:00
Robert Shaw
e3b318216d [ Bugfix ] Fix Prometheus Metrics With zeromq Frontend (#7279) 
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-08-18 20:19:48 +00:00
Gordon Wong
0e39a33c6d [Bugfix][Hardware][AMD][Frontend] add quantization param to embedding checking method (#7513) 2024-08-16 10:05:18 -06:00
Michael Goin
9c8e2d1161 [Bugfix][Harmless] Fix float16 dtype for model_is_embedding (#7566) 2024-08-15 18:26:19 -07:00
youkaichao
33e5d7e6b6 [frontend] spawn engine process from api server process (#7484) 2024-08-13 15:40:17 -07:00
Joe Runde
21b9c49aa3 [Frontend] Kill the server on engine death (#6594) 
Signed-off-by: Joe Runde <joe@joerun.de>
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2024-08-08 09:47:48 -07:00
Maximilien de Bayser
fde47d3bc2 [BugFix] Fix frontend multiprocessing hang (#7217) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
 2024-08-07 18:09:36 +00:00
Robert Shaw
564985729a [ BugFix ] Move zmq frontend to IPC instead of TCP (#7222) 2024-08-07 16:24:56 +00:00
Aditya Paliwal
57f560aa23 [BugFix] Use args.trust_remote_code (#7121) 2024-08-05 09:26:14 -07:00
Cyrus Leung
cc08fc7225 [Frontend] Reapply "Factor out code for running uvicorn" (#7095) 2024-08-04 20:40:51 -07:00
Robert Shaw
ed812a73fa [ Frontend ] Multiprocessing for OpenAI Server with zeromq (#6883) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Joe Runde <joe@joerun.de>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-08-02 18:27:28 -07:00