biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
9,263 Commits 2 Branches 140 Tags
b9a1c4c8a2430555f1959938b87b10640c751e57
Commit Graph

92 Commits

Author SHA1 Message Date
Simon Mo
ed59a7ed23 Update test_ignore_eos (#4898) 2024-06-02 02:21:53 +00:00
Cyrus Leung
5ae5ed1e60 [Core] Consolidate prompt arguments to LLM engines (#4328) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-05-28 13:29:31 -07:00
Cyrus Leung
350f9e107f [CI/Build] Move test_utils.py to tests/utils.py (#4425) 
Since #4335 was merged, I've noticed that the definition of ServerRunner in the tests is the same as in the test for OpenAI API. I have moved the class to the test utilities to avoid code duplication. (Although it only has been repeated twice so far, I will add another similar test suite in #4200 which would duplicate the code a third time)

Also, I have moved the test utilities file (test_utils.py) to under the test directory (tests/utils.py), since none of its code is actually used in the main package. Note that I have added __init__.py to each test subpackage and updated the ray.init() call in the test utilities file in order to relative import tests/utils.py.
 2024-05-13 23:50:09 +09:00
Chang Su
e254497b66 [Model][Misc] Add e5-mistral-7b-instruct and Embedding API (#3734) 2024-05-11 11:30:37 -07:00
Woosuk Kwon
190bc838e1 [Misc] Remove unnecessary ModelRunner imports (#4703) 2024-05-09 00:17:17 -07:00
Cody Yu
f942efb5a3 [Dynamic Spec Decoding] Auto-disable by the running queue size (#4592) 
Co-authored-by: Cade Daniel <edacih@gmail.com>
 2024-05-08 21:44:00 +00:00
SangBin Cho
3521ba4f25 [Core][Model runner refactoring 1/N] Refactor attn metadata term (#4518) 2024-05-03 10:20:12 -07:00
SangBin Cho
6f1df80436 [Test] Add ignore_eos test (#4519) 2024-05-01 08:45:42 -04:00
Nick Hill
81661da7b2 [BugFix] Fix min_tokens when eos_token_id is None (#4389) 
Co-authored-by: DefTruth <31974251+deftruth@users.noreply.github.com>
 2024-04-27 09:52:46 -07:00
SangBin Cho
603ad84815 [Core] Refactoring sampler and support prompt logprob for chunked prefill (#4309) 2024-04-26 13:02:02 +00:00
Cade Daniel
62b8aebc6f [Speculative decoding 7/9] Speculative decoding end-to-end correctness tests. (#3951) 2024-04-23 08:02:36 +00:00
Antoni Baum
69e1d2fb69 [Core] Refactor model loading code (#4097) 2024-04-16 11:34:39 -07:00
Nick Hill
e46a60aa4c [BugFix] Fix handling of stop strings and stop token ids (#3672) 2024-04-11 15:34:12 -07:00
Travis Johnson
0258b7a94b [Bugfix] handle prompt_logprobs in _apply_min_tokens_penalty (#3876) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2024-04-10 01:39:56 -07:00
胡译文
b3104b2a10 [Bugfix] Fix logits processor when prompt_logprobs is not None (#3899) 2024-04-10 00:09:36 -07:00
SangBin Cho
26422e477b [Test] Make model tests run again and remove --forked from pytest (#3631) 
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-03-28 21:06:40 -07:00
Nick Hill
dfeb2ecc3a [Misc] Include matched stop string/token in responses (#2976) 
Co-authored-by: Sahil Suneja <sahilsuneja@gmail.com>
 2024-03-25 17:31:32 -07:00
Simon Mo
f408d05c52 hotfix isort on logprobs ranks pr (#3622) 2024-03-25 11:55:46 -07:00
Travis Johnson
c13ad1b7bd feat: implement the min_tokens sampling parameter (#3124) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-03-25 10:14:26 -07:00
Swapnil Parekh
819924e749 [Core] Adding token ranks along with logprobs (#3516) 
Co-authored-by: Swapnil Parekh <swapnilp@ibm.com>
 2024-03-25 10:13:10 -07:00
SangBin Cho
01bfb22b41 [CI] Try introducing isort. (#3495) 2024-03-25 07:59:47 -07:00
Woosuk Kwon
925f3332ca [Core] Refactor Attention Take 2 (#3462) 2024-03-25 04:39:33 +00:00
Roy
f1c0fc3919 Migrate logits computation and gather to model_runner (#3233) 2024-03-20 23:25:01 +00:00
Antoni Baum
426ec4ec67 [1/n] Triton sampling kernel (#3186) 
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
 2024-03-20 14:45:08 -07:00
Zhuohan Li
2f8844ba08 Re-enable the 80 char line width limit (#3305) 2024-03-10 19:49:14 -07:00
Antoni Baum
22de45235c Push logprob generation to LLMEngine (#3065) 
Co-authored-by: Avnish Narayan <avnish@anyscale.com>
 2024-03-04 19:54:06 +00:00
Nick Hill
7d2dcce175 Support per-request seed (#2514) 2024-02-21 11:47:00 -08:00
Zhuohan Li
63e2a6419d [FIX] Fix beam search test (#2930) 2024-02-20 14:37:39 -08:00
Kunshang Ji
96b6f475dd Remove hardcoded device="cuda" to support more devices (#2503) 
Co-authored-by: Jiang Li <jiang1.li@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2024-02-01 15:46:39 -08:00
Antoni Baum
9b945daaf1 [Experimental] Add multi-LoRA support (#1804) 
Co-authored-by: Chen Shen <scv119@gmail.com>
Co-authored-by: Shreyas Krishnaswamy <shrekris@anyscale.com>
Co-authored-by: Avnish Narayan <avnish@anyscale.com>
 2024-01-23 15:26:37 -08:00
shiyi.c_98
d10f8e1d43 [Experimental] Prefix Caching Support (#1669) 
Co-authored-by: DouHappy <2278958187@qq.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
 2024-01-17 16:32:10 -08:00
Simon Mo
6e01e8c1c8 [CI] Add Buildkite (#2355) 2024-01-14 12:37:58 -08:00
陈序
218dc2ccda Aligning top_p and top_k Sampling (#1885) 
* Align top_p and top_k with huggingface

* remove _get_prompt_and_output_tokens

* rename _apply_top_p_top_k

* compare top_p top_k with hf

* fix test errors
 2024-01-12 22:51:03 +01:00
Cade Daniel
79d64c4954 [Speculative decoding 1/9] Optimized rejection sampler (#2336) 2024-01-09 15:38:41 -08:00
Woosuk Kwon
5f09cbdb63 Fix broken sampler tests (#1896) 
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
 2023-12-02 16:06:17 -08:00
Simon Mo
5ffc0d13a2 Migrate linter from pylint to ruff (#1665) 2023-11-20 11:58:01 -08:00
Noam Gat
555bdcc5a3 Added logits processor API to sampling params (#1469) 2023-11-03 14:12:15 -07:00
Woosuk Kwon
d3a5bd9fb7 Fix sampler test (#1379) 2023-10-16 12:57:26 -07:00
Zhuohan Li
9d9072a069 Implement prompt logprobs & Batched topk for computing logprobs (#1328) 
Co-authored-by: Yunmo Chen <16273544+wanmok@users.noreply.github.com>
 2023-10-16 10:56:50 -07:00
Zhuohan Li
ba0bfd40e2 TP/quantization/weight loading refactor part 1 - Simplify parallel linear logic (#1181) 2023-10-02 15:36:09 -07:00
Zhuohan Li
947b794146 [Sampler] Vectorized sampling (simplified) (#1048) 
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
 2023-09-22 17:48:04 -07:00
Zhuohan Li
002800f081 Align vLLM's beam search implementation with HF generate (#857) 2023-09-04 17:29:42 -07:00