Jie Fu (傅杰)
|
6840a71610
|
[Misc] Remove unused cuda_utils.h in CPU backend (#5345)
|
2024-06-07 14:09:13 -07:00 |
|
Roger Wang
|
7a9cb294ae
|
[Frontend] Add OpenAI Vision API Support (#5237)
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
|
2024-06-07 11:23:32 -07:00 |
|
Dipika Sikka
|
ca3ea51bde
|
[Kernel] Dynamic Per-Token Activation Quantization (#5037)
Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
|
2024-06-07 09:36:26 -07:00 |
|
limingshu
|
dc49fb892c
|
Addition of lacked ignored_seq_groups in _schedule_chunked_prefill (#5296)
|
2024-06-07 13:35:42 +00:00 |
|
Antoni Baum
|
18a277b52d
|
Remove Ray health check (#4693)
|
2024-06-07 10:01:56 +00:00 |
|
Tyler Michael Smith
|
8d75fe48ca
|
[Kernel] Switch fp8 layers to use the CUTLASS kernels (#5183)
Switching from torch._scaled_mm to vLLM's cutlass fp8 kernels when supported as we are seeing 5-15% improvement in e2e performance on neuralmagic/Meta-Llama-3-8B-Instruct-FP8
see https://docs.google.com/spreadsheets/d/1GiAnmzyGHgZ6zL_LDSTm35Bdrt4A8AaFEurDlISYYA4/ for some quick e2e benchmarks and #5144 for comparisons across different GEMM sizes.
|
2024-06-07 08:42:35 +00:00 |
|
youkaichao
|
388596c914
|
[Misc][Utils] allow get_open_port to be called for multiple times (#5333)
|
2024-06-06 22:15:11 -07:00 |
|
Itay Etelis
|
baa15a9ec3
|
[Feature][Frontend]: Add support for stream_options in ChatCompletionRequest (#5135)
|
2024-06-07 03:29:24 +00:00 |
|
Jie Fu (傅杰)
|
15063741e3
|
[Misc] Missing error message for custom ops import (#5282)
|
2024-06-06 20:17:21 -07:00 |
|
Antoni Baum
|
ccdc490dda
|
[Core] Change LoRA embedding sharding to support loading methods (#5038)
|
2024-06-06 19:07:57 -07:00 |
|
Antoni Baum
|
a31cab7556
|
[Core] Avoid copying prompt/output tokens if no penalties are used (#5289)
|
2024-06-06 18:12:00 -07:00 |
|
Matthew Goldey
|
828da0d44e
|
[Frontend] enable passing multiple LoRA adapters at once to generate() (#5300)
|
2024-06-06 15:48:13 -05:00 |
|
Philipp Moritz
|
abe855d637
|
[Kernel] Retune Mixtral 8x22b configs for FP8 on H100 (#5294)
|
2024-06-06 09:29:29 -07:00 |
|
liuyhwangyh
|
4efff036f0
|
Bugfix: fix broken of download models from modelscope (#5233)
Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>
|
2024-06-06 09:28:10 -07:00 |
|
Cyrus Leung
|
89c920785f
|
[CI/Build] Update vision tests (#5307)
|
2024-06-06 05:17:18 -05:00 |
|
Breno Faria
|
7b0a0dfb22
|
[Frontend][Core] Update Outlines Integration from FSM to Guide (#4109)
Co-authored-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: Breno Faria <breno.faria@intrafind.com>
|
2024-06-05 16:49:12 -07:00 |
|
Simon Mo
|
3a6ae1d33c
|
[CI] Disable flash_attn backend for spec decode (#5286)
|
2024-06-05 15:49:27 -07:00 |
|
Simon Mo
|
8f1729b829
|
[Docs] Add Ray Summit CFP (#5295)
|
2024-06-05 15:25:18 -07:00 |
|
Woosuk Kwon
|
6a7c7711a2
|
[Misc] Skip for logits_scale == 1.0 (#5291)
|
2024-06-05 15:19:02 -07:00 |
|
Alex Wu
|
0f83ddd4d7
|
[Bugfix][Frontend/Core] Don't log exception when AsyncLLMEngine gracefully shuts down. (#5290)
|
2024-06-05 15:18:12 -07:00 |
|
Michael Goin
|
065aff6c16
|
[Bugfix] Make EngineArgs use named arguments for config construction (#5285)
|
2024-06-05 15:16:56 -07:00 |
|
Nick Hill
|
3d33e372a1
|
[BugFix] Fix log message about default max model length (#5284)
|
2024-06-05 14:53:16 -07:00 |
|
Nick Hill
|
faf71bcd4b
|
[Speculative Decoding] Add ProposerWorkerBase abstract class (#5252)
|
2024-06-05 14:53:05 -07:00 |
|
Simon Mo
|
f270a39537
|
[Docs] Add Sequoia as sponsors (#5287)
|
2024-06-05 18:02:56 +00:00 |
|
Philipp Moritz
|
51a08e7d8f
|
[Kernel] Re-tune Mixtral MoE configurations for FP8 on H100 (#5238)
|
2024-06-05 10:59:14 -07:00 |
|
DriverSong
|
eb8fcd2666
|
[BugFix] Apply get_cached_tokenizer to the tokenizer setter of LLM (#5207)
Co-authored-by: qiujiawei9 <qiujiawei9@jd.com>
|
2024-06-05 10:59:02 -07:00 |
|
Cody Yu
|
5563a4dea8
|
[Model] Correct Mixtral FP8 checkpoint loading (#5231)
|
2024-06-05 10:58:50 -07:00 |
|
Tyler Michael Smith
|
ccd4f129e8
|
[Kernel] Add GPU architecture guards to the CUTLASS w8a8 kernels to reduce binary size (#5157)
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
|
2024-06-05 10:44:15 -07:00 |
|
Tyler Michael Smith
|
02cc3b51a7
|
[misc] benchmark_serving.py -- add ITL results and tweak TPOT results (#5263)
|
2024-06-05 10:17:51 -07:00 |
|
Simon Mo
|
d5b1eb081e
|
[CI] Add nightly benchmarks (#5260)
|
2024-06-05 09:42:08 -07:00 |
|
tomeras91
|
f0a500545f
|
[Frontend] OpenAI API server: Add add_special_tokens to ChatCompletionRequest (default False) (#5278)
|
2024-06-05 09:32:58 -07:00 |
|
Woosuk Kwon
|
c65146e75e
|
[Misc] Fix docstring of get_attn_backend (#5271)
|
2024-06-05 09:18:59 -07:00 |
|
Woosuk Kwon
|
41ca62cf03
|
[Misc] Add CustomOp interface for device portability (#5255)
|
2024-06-05 09:18:19 -07:00 |
|
zifeitong
|
974fc9b845
|
[Bugfix] Fix prompt_logprobs when SamplingParams.detokenize is set to True (#5226)
|
2024-06-04 19:37:28 -07:00 |
|
youkaichao
|
fee4dcc33a
|
[Misc] update collect env (#5261)
|
2024-06-04 17:29:09 -05:00 |
|
Michael Goin
|
650a4cc55e
|
[Misc] Add transformers version to collect_env.py (#5259)
|
2024-06-04 12:52:28 -07:00 |
|
Simon Mo
|
9ca62d8668
|
[CI] mark AMD test as softfail to prevent blockage (#5256)
|
2024-06-04 11:34:53 -07:00 |
|
Li, Jiang
|
45c35f0d58
|
[CI/Build] Reducing CPU CI execution time (#5241)
|
2024-06-04 10:26:40 -07:00 |
|
Cyrus Leung
|
9ba093b4f4
|
[CI/Build] Simplify model loading for HfRunner (#5251)
|
2024-06-04 10:09:19 -07:00 |
|
Woosuk Kwon
|
27208be66e
|
[Kernel] Add back batch size 1536 and 3072 to MoE tuning (#5242)
|
2024-06-04 09:58:47 -07:00 |
|
Jie Fu (傅杰)
|
87d5abef75
|
[Bugfix] Fix a bug caused by pip install setuptools>=49.4.0 for CPU backend (#5249)
|
2024-06-04 09:57:51 -07:00 |
|
Cyrus Leung
|
ec784b2526
|
[CI/Build] Add inputs tests (#5215)
|
2024-06-03 21:01:46 -07:00 |
|
zifeitong
|
a58f24e590
|
[Bugfix] Fix torch.compile() error when using MultiprocessingGPUExecutor (#5229)
|
2024-06-03 20:55:50 -07:00 |
|
afeldman-nm
|
f42a006b15
|
[Bugfix]: During testing, use pytest monkeypatch for safely overriding the env var that indicates the vLLM backend (#5210)
|
2024-06-03 20:32:57 -07:00 |
|
Woosuk Kwon
|
3a434b07ed
|
[Kernel] Enhance MoE benchmarking & tuning script (#4921)
|
2024-06-03 20:06:59 -07:00 |
|
Zhuohan Li
|
bd0e7802e0
|
[Bugfix] Add warmup for prefix caching example (#5235)
|
2024-06-03 19:36:41 -07:00 |
|
Toshiki Kataoka
|
06b2550cbb
|
[Bugfix] Support prompt_logprobs==0 (#5217)
|
2024-06-03 17:59:30 -07:00 |
|
Breno Faria
|
f775a07e30
|
[FRONTEND] OpenAI tools support named functions (#5032)
|
2024-06-03 18:25:29 -05:00 |
|
Kevin H. Luu
|
4f0d17c05c
|
New CI template on AWS stack (#5110)
Signed-off-by: kevin <kevin@anyscale.com>
|
2024-06-03 16:16:43 -07:00 |
|
Kaiyang Chen
|
10c38e3e46
|
[Misc]: Implement CPU/GPU swapping in BlockManagerV2 (#3834)
|
2024-06-03 13:37:11 -07:00 |
|