biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
6,079 Commits 2 Branches 140 Tags
f8acd01ff758fad3a44302f5be6407243cbec193
Commit Graph

579 Commits

Author SHA1 Message Date
Wallas Henrique
cfd3219f58 [Hardware][Apple] Native support for macOS Apple Silicon (#11696) 
Signed-off-by: Wallas Santos <wallashss@ibm.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2025-01-08 16:35:49 +08:00
Rui Qiao
f8fcca100b [Misc] Fix typo for valid_tool_parses (#11753) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
 2025-01-06 07:12:38 +00:00
Robert Shaw
33fc1e2e86 [Frontend] Improve StreamingResponse Exception Handling (#11752) 2025-01-05 16:35:01 -05:00
Robert Shaw
ad0d567e1c [V1] Chore: cruft removal (#11724) 2025-01-03 23:25:02 +00:00
Robert Shaw
80c751e7f6 [V1] Simplify Shutdown (#11659) 2025-01-03 17:25:38 +00:00
Nathan Azrak
68d37809b9 [Misc] Minimum requirements for SageMaker compatibility (#11576) 2025-01-02 15:59:25 -08:00
Joe Runde
4db72e57f6 [Bugfix][Refactor] Unify model management in frontend (#11660) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2025-01-01 02:21:51 +00:00
Robert Shaw
5886aa496e [V1] [6/N] API Server: Better Shutdown (#11586) 2024-12-30 15:51:02 +00:00
Robert Shaw
df04dffade [V1] [4/N] API Server: ZMQ/MP Utilities (#11541) 2024-12-28 01:45:08 +00:00
Cyrus Leung
7af553ea30 [Misc] Abstract the logic for reading and writing media content (#11527) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-27 19:21:23 +08:00
Robert Shaw
55fb97f7bd [2/N] API Server: Avoid ulimit footgun (#11530) 2024-12-26 23:43:05 +00:00
Robert Shaw
720b10fdc6 [1/N] API Server (Remove Proxy) (#11529) 2024-12-26 23:03:43 +00:00
Cyrus Leung
9edca6bf8f [Frontend] Online Pooling API (#11457) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-24 17:54:30 +08:00
dpxa
4f074fbf53 [Misc]Suppress irrelevant exception stack trace information when CUDA… (#11438) 
Co-authored-by: shiquan <shiquan>
 2024-12-24 08:43:39 +00:00
Rafael Vasquez
32aa2059ad [Docs] Convert rST to MyST (Markdown) (#11145) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-12-23 22:35:38 +00:00
Ricky Xu
584f0ae40d [V1] Make AsyncLLMEngine v1-v0 opaque (#11383) 
Signed-off-by: Ricky Xu <xuchen727@hotmail.com>
 2024-12-21 15:14:08 +08:00
Michael Goin
d573aeadcc [Bugfix] Don't log OpenAI field aliases as ignored (#11378) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-20 19:03:50 +00:00
Yanyi Liu
5aef49806d [Feature] Add load generation config from model (#11164) 
Signed-off-by: liuyanyi <wolfsonliu@163.com>
Signed-off-by: Yanyi Liu <wolfsonliu@163.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2024-12-19 10:50:38 +00:00
Travis Johnson
17ca964273 [Model] IBM Granite 3.1 (#11307) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2024-12-19 11:27:24 +08:00
Joe Runde
2d1b9baa8f [Bugfix] Fix request cancellation without polling (#11190)
Some checks failed
Create Release / Create Release (push) Has been cancelled
Details
Create Release / Build Wheel (11.8, ubuntu-20.04, 3.10, 2.4.0) (push) Has been cancelled
Details
Create Release / Build Wheel (11.8, ubuntu-20.04, 3.11, 2.4.0) (push) Has been cancelled
Details
Create Release / Build Wheel (11.8, ubuntu-20.04, 3.12, 2.4.0) (push) Has been cancelled
Details
Create Release / Build Wheel (11.8, ubuntu-20.04, 3.9, 2.4.0) (push) Has been cancelled
Details
Create Release / Build Wheel (12.1, ubuntu-20.04, 3.10, 2.4.0) (push) Has been cancelled
Details
Create Release / Build Wheel (12.1, ubuntu-20.04, 3.11, 2.4.0) (push) Has been cancelled
Details
Create Release / Build Wheel (12.1, ubuntu-20.04, 3.12, 2.4.0) (push) Has been cancelled
Details
Create Release / Build Wheel (12.1, ubuntu-20.04, 3.9, 2.4.0) (push) Has been cancelled
Details
2024-12-17 12:26:32 -08:00
kYLe
66d4b16724 [Frontend] Add OpenAI API support for input_audio (#11027) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-16 22:09:58 -08:00
Michael Goin
0064f697d3 [CI] Add test case with JSON schema using references + use xgrammar by default with OpenAI parse (#10935) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-17 11:39:58 +08:00
Isotr0py
d927dbcd88 [Model] Refactor Ultravox to use merged input processor (#11198) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-12-16 10:09:53 +00:00
yansh97
17138af7c4 [Bugfix] Fix the default value for temperature in ChatCompletionRequest (#11219) 2024-12-16 00:15:40 -08:00
Brad Hilton
9c3dadd1c9 [Frontend] Add logits_processors as an extra completion argument (#11150) 
Signed-off-by: Brad Hilton <brad.hilton.nw@gmail.com>
 2024-12-14 16:46:42 +00:00
Russell Bryant
4863e5fba5 [Core] V1: Use multiprocessing by default (#11074) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-12-13 16:27:32 -08:00
Cyrus Leung
0920ab9131 [Doc] Reorganize online pooling APIs (#11172) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-14 00:22:22 +08:00
zhangjf
5b0ed8391d [Bugfix] using len(tokenizer) instead of tokenizer.vocab_size in AllowedTokenIdsLogitsProcessor (#11156) 2024-12-13 15:56:19 +00:00
Cyrus Leung
eeec9e3390 [Frontend] Separate pooling APIs in offline inference (#11129) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-13 10:40:07 +00:00
Jiaxin Shan
85362f028c [Misc][LoRA] Ensure Lora Adapter requests return adapter name (#11094) 
Signed-off-by: Jiaxin Shan <seedjeffwan@gmail.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
 2024-12-12 09:25:16 +00:00
Russell Bryant
ccede2b264 [Core] cleanup zmq ipc sockets on exit (#11115) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-12-11 19:12:24 -08:00
Clayton
7439a8b5fc [Bugfix] Multiple fixes to tool streaming with hermes and mistral (#10979) 
Signed-off-by: cedonley <clayton@donley.io>
 2024-12-12 01:10:12 +00:00
Cyrus Leung
cad5c0a6ed [Doc] Update docs to refer to pooling models (#11093) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-11 13:36:27 +00:00
Cyrus Leung
8f10d5e393 [Misc] Split up pooling tasks (#10820) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-11 01:28:00 -08:00
Rafael Vasquez
40766ca1b8 [Bugfix]: Clamp -inf logprob values in prompt_logprobs (#11073) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-12-11 01:27:39 -08:00
Maximilien de Bayser
e39400a4b6 Fix streaming for granite tool call when <|tool_call|> is present (#11069) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
 2024-12-11 04:51:40 +00:00
Travis Johnson
beb16b2c81 [Bugfix] Handle <|tool_call|> token in granite tool parser (#11039) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2024-12-10 10:27:11 +00:00
Joe Runde
980ad394a8 [Frontend] Use request id from header (#10968) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2024-12-10 13:46:29 +08:00
Cyrus Leung
391d7b2763 [Bugfix] Fix usage of deprecated decorator (#11025) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-10 13:45:47 +08:00
youkaichao
fd57d2b534 [torch.compile] allow candidate compile sizes (#10984) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-12-08 11:05:21 +00:00
Russell Bryant
69d357ba12 [Core] Cleanup startup logging a bit (#10961) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-12-07 02:30:23 +00:00
Cyrus Leung
3257d449fa [Misc] Remove deprecated names (#10817) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-03 06:52:57 +00:00
Cyrus Leung
d2f058e76c [Misc] Rename embedding classes to pooling (#10801) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-01 14:36:51 +08:00
tomeras91
395b1c7454 [Frontend] don't block event loop in tokenization (preprocess) in OpenAI compatible server (#10635) 
Signed-off-by: Tomer Asida <tomera@ai21.com>
 2024-11-27 13:21:10 -08:00
Michael Goin
9a99273b48 [Bugfix] Fix using -O[0,3] with LLM entrypoint (#10677) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-26 10:44:01 -08:00
Ricky Xu
519e8e4182 [v1] EngineArgs for better config handling for v1 (#10382) 
Signed-off-by: rickyx <rickyx@anyscale.com>
 2024-11-25 21:09:43 -08:00
zhou fan
b1d920531f [Model]: Add support for Aria model (#10514) 
Signed-off-by: xffxff <1247714429@qq.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
 2024-11-25 18:10:55 +00:00
Chauncey
d04b13a380 [Bug]: Authorization ignored when root_path is set (#10606) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2024-11-25 16:21:41 +00:00
Maximilien de Bayser
214efc2c3c Support Cross encoder models (#10400) 
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Flavia Beo <flavia.beo@ibm.com>
Co-authored-by: Flavia Beo <flavia.beo@ibm.com>
 2024-11-24 18:56:20 -08:00
Varun Vinayak Shenoy
7d8ffb344f [Bugfix] Internal Server Error when tool_choice is incorrect. (#10567) 
Signed-off-by: Varun Shenoy <varun.vinayak.shenoy@oracle.com>
 2024-11-22 21:13:29 -08:00