biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
3,443 Commits 2 Branches 140 Tags
4f168f69a3e856bda3f30e02fcee7db2a01ff32b
Commit Graph

64 Commits

Author SHA1 Message Date
Cyrus Leung
0b8bb86bf1 [1/N] Initial prototype for multi-modal processor (#10044) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-11-13 12:39:03 +00:00
Nick Hill
29862b884b [Frontend] Adjust try/except blocks in API impl (#10056) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2024-11-06 20:07:51 -08:00
Cyrus Leung
06386a64dd [Frontend] Chat-based Embeddings API (#9759) 2024-11-01 08:13:35 +00:00
Zhong Qishuai
ef7865b4f9 [Frontend] re-enable multi-modality input in the new beam search implementation (#9427) 
Signed-off-by: Qishuai Ferdinandzhong@gmail.com
 2024-10-29 11:49:47 +00:00
Nick Hill
25aeb7d4c9 [BugFix] Fix and simplify completion API usage streaming (#9475) 2024-10-18 14:10:26 +00:00
Chang Su
ba30942240 [Bugfix] Fix vLLM UsageInfo and logprobs None AssertionError with empty token_ids (#9034) 
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-10-15 15:40:43 -07:00
Brendan Wong
4d31cd424b [Frontend] merge beam search implementations (#9296) 2024-10-14 15:05:52 -07:00
Brendan Wong
8c746226c9 [Frontend] API support for beam search for MQLLMEngine (#9117) 2024-10-08 05:51:43 +00:00
Brendan Wong
168cab6bbf [Frontend] API support for beam search (#9087) 
Co-authored-by: youkaichao <youkaichao@126.com>
 2024-10-05 23:39:03 -07:00
Sebastian Schoennenbeck
35bd215168 [Core] [Frontend] Priority scheduling for embeddings and in the OpenAI-API (#8965) 2024-10-01 09:58:06 +00:00
Joe Runde
062c89e7c9 [Frontend][Core] Move guided decoding params into sampling params (#8252) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-10-01 09:34:25 +08:00
Adam Tilghman
1ac3de09cd [Frontend] OpenAI server: propagate usage accounting to FastAPI middleware layer (#8672) 2024-09-25 07:49:26 +00:00
Jiaxin Shan
260d40b5ea [Core] Support Lora lineage and base model metadata management (#6315) 2024-09-20 06:20:56 +00:00
Alexander Matveev
7c7714d856 [Core][Bugfix][Perf] Introduce MQLLMEngine to avoid asyncio OH (#8157) 
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-09-18 13:56:58 +00:00
Nick Hill
551ce01078 [Core] Add engine option to return only deltas or final output (#7381) 2024-09-12 12:02:00 -07:00
Cyrus Leung
baaedfdb2d [mypy] Enable following imports for entrypoints (#7248) 
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Fei <dfdfcai4@gmail.com>
 2024-08-20 23:28:21 -07:00
Grant Pinkert
f878c8feb0 [Feature]: Add OpenAI server prompt_logprobs support #6508 (#7453) 2024-08-16 02:38:08 +00:00
Nick Hill
9a3f49ae07 [BugFix] Overhaul async request cancellation (#7111) 2024-08-07 13:21:41 +08:00
Robert Shaw
ed812a73fa [ Frontend ] Multiprocessing for OpenAI Server with zeromq (#6883) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Joe Runde <joe@joerun.de>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-08-02 18:27:28 -07:00
zifeitong
3c10591ef2 [Bugfix] Set SamplingParams.max_tokens for OpenAI requests if not provided by user (#6954) 2024-07-31 21:13:34 -07:00
Nick Hill
9f69d8245a [Frontend] New allowed_token_ids decoding request parameter (#6753) 2024-07-29 23:37:27 +00:00
Evan Z. Liu
5689e256ba [Frontend] Represent tokens with identifiable strings (#6626) 2024-07-25 09:51:00 +08:00
Cyrus Leung
739b61a348 [Frontend] Refactor prompt processing (#4028) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-07-22 10:13:53 -07:00
Nick Hill
e2fbaee725 [BugFix][Frontend] Use LoRA tokenizer in OpenAI APIs (#6227) 
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-07-18 15:13:30 +08:00
sasha0552
7a3d2a5b95 [Frontend] Support for chat completions input in the tokenize endpoint (#5923) 2024-07-16 20:18:09 +08:00
Swapnil Parekh
4d6ada947c [CORE] Adding support for insertion of soft-tuned prompts (#4645) 
Co-authored-by: Swapnil Parekh <swapnilp@ibm.com>
Co-authored-by: Joe G <joseph.granados@h2o.ai>
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
 2024-07-09 13:26:36 -07:00
kczimm
16620f439d do not exclude object field in CompletionStreamResponse (#6196) 2024-07-08 10:32:57 +08:00
jvlunteren
f1e15da6fe [Frontend] Continuous usage stats in OpenAI completion API (#5742) 2024-07-05 10:37:09 -07:00
Alexander Matveev
3476ed0809 [Core] Optimize block_manager_v2 vs block_manager_v1 (to make V2 default) (#5602) 2024-07-01 20:10:37 -07:00
sasha0552
c54269d967 [Frontend] Add tokenize/detokenize endpoints (#5054) 2024-06-26 16:54:22 +00:00
Ronen Schaffer
7879f24dcc [Misc] Add OpenTelemetry support (#4687) 
This PR adds basic support for OpenTelemetry distributed tracing.
It includes changes to enable tracing functionality and improve monitoring capabilities.

I've also added a markdown with print-screens to guide users how to use this feature. You can find it here
 2024-06-19 01:17:03 +09:00
Cyrus Leung
640052b069 [Bugfix][Frontend] Cleanup "fix chat logprobs" (#5026) 2024-06-10 22:36:46 -07:00
Itay Etelis
774d1035e4 [Feature][Frontend]: Continued stream_options implementation also in CompletionRequest (#5319) 2024-06-10 14:22:09 +00:00
Toshiki Kataoka
06b2550cbb [Bugfix] Support prompt_logprobs==0 (#5217) 2024-06-03 17:59:30 -07:00
Breno Faria
87d41c849d [BUGFIX] [FRONTEND] Correct chat logprobs (#5029) 
Co-authored-by: Breno Faria <breno.faria@intrafind.com>
 2024-05-30 02:52:14 -07:00
Cyrus Leung
5ae5ed1e60 [Core] Consolidate prompt arguments to LLM engines (#4328) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-05-28 13:29:31 -07:00
Cyrus Leung
f12b20decc [Frontend] Move async logic outside of constructor (#4674) 2024-05-08 22:48:33 -07:00
Roy
7134303cbb [Bugfix][Core] Fix get decoding config from ray (#4335) 2024-04-27 11:30:08 +00:00
Cyrus Leung
8947bc3c15 [Frontend][Bugfix] Disallow extra fields in OpenAI API (#4355) 2024-04-27 05:08:24 +00:00
SangBin Cho
0ae11f78ab [Mypy] Part 3 fix typing for nested directories for most of directory (#4161) 2024-04-22 21:32:44 -07:00
Harry Mellor
66ded03067 Allow model to be served under multiple names (#2894) 
Co-authored-by: Alexandre Payot <alexandrep@graphcore.ai>
 2024-04-18 00:16:26 -07:00
Noam Gat
05434764cd LM Format Enforcer Guided Decoding Support (#3868) 
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-04-16 05:54:57 +00:00
Cyrus Leung
7fd3949a0b [Frontend][Core] Move merge_async_iterators to utils (#4026) 2024-04-12 05:30:54 +00:00
Dylan Hawk
95e7d4a97c Fix echo/logprob OpenAI completion bug (#3441) 
Co-authored-by: Dylan Hawk <dylanwawk@gmail.com>
 2024-04-11 22:15:50 +00:00
Thomas Parnell
1d7c940d74 Add option to completion API to truncate prompt tokens (#3144) 2024-04-05 10:15:42 -07:00
Roy
f510395bbf [BugFix][Frontend] Fix completion logprobs=0 error (#3731) 2024-03-29 09:38:21 -07:00
Nick Hill
dfeb2ecc3a [Misc] Include matched stop string/token in responses (#2976) 
Co-authored-by: Sahil Suneja <sahilsuneja@gmail.com>
 2024-03-25 17:31:32 -07:00
Dylan Hawk
0b4997e05c [Bugfix] API stream returning two stops (#3450) 
Co-authored-by: Dylan Hawk <dylanwawk@gmail.com>
 2024-03-25 10:14:34 -07:00
SangBin Cho
01bfb22b41 [CI] Try introducing isort. (#3495) 2024-03-25 07:59:47 -07:00
Robert Shaw
10585e035e Removed Extraneous Print Message From OAI Server (#3440) 2024-03-16 00:35:36 +00:00