biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Commit Graph

  • aa50b17ca7 Change plotting script submission Woosuk Kwon 2023-04-17 04:49:14 +00:00
  • 0f4b32199e Support various block sizes & Change default block size to 16 (#38) Woosuk Kwon 2023-04-15 09:03:24 -07:00
  • 84eee24e20 Collect system stats in scheduler & Add scripts for experiments (#30) Woosuk Kwon 2023-04-12 15:03:49 -07:00
  • e3cec88aa5 Memcpy kernel for flash attention (#29) Siyuan (Ryans) Zhuang 2023-04-10 18:22:49 -07:00
  • b9926f7f66 Support block size 32 (#35) Woosuk Kwon 2023-04-09 23:07:18 -07:00
  • ee88a7e5f3 Add an option to use dummy model weights (#33) Woosuk Kwon 2023-04-08 23:36:12 -07:00
  • c267b1a02c Add query stride to multi_query_cached_kv_attention & Add kernel benchmark script (#27) Woosuk Kwon 2023-04-08 13:36:09 -07:00
  • 0f40557af6 Implement block copy kernel to optimize beam search (#32) Woosuk Kwon 2023-04-07 17:45:07 -07:00
  • a490aafa36 Fix potential bugs in FastAPI frontend and add comments (#28) Zhuohan Li 2023-04-06 13:44:24 +08:00
  • 12659a0bd7 Add CUDA graph-based all reduce launcher (#26) Woosuk Kwon 2023-04-05 11:16:57 -07:00
  • 21b3671bbc Basic attention kernel that supports cached KV + (multi-)prompts (#24) Siyuan (Ryans) Zhuang 2023-04-04 20:34:46 -07:00
  • 897cb2ae28 Optimize data movement (#20) Woosuk Kwon 2023-04-02 00:30:17 -07:00
  • 1f01a18d39 Merge QKV into one linear layer (#15) Zhuohan Li 2023-04-02 15:23:29 +08:00
  • 2c5cd0defe Add ninja to dependency (#21) Woosuk Kwon 2023-04-01 19:00:20 -07:00
  • a90c97d727 Use FP32 for log probabilities (#19) Woosuk Kwon 2023-03-31 23:33:43 -07:00
  • e3f00d191e Modify README to include info on loading LLaMA (#18) Zhuohan Li 2023-04-01 01:07:57 +08:00
  • 09e9245478 Add custom kernel for RMS normalization (#16) Woosuk Kwon 2023-03-31 09:51:22 -07:00
  • c45f3c3ab6 Optimize tensor parallel execution speed (#17) Zhuohan Li 2023-04-01 00:51:08 +08:00
  • 7a7929abe8 Implement preemption via recomputation & Refactor scheduling logic (#12) Woosuk Kwon 2023-03-30 14:51:46 -07:00
  • 88c0268a18 Implement custom kernel for LLaMA rotary embedding (#14) Woosuk Kwon 2023-03-30 11:04:21 -07:00
  • 80a2f812f1 Implement LLaMA (#9) Woosuk Kwon 2023-03-29 21:25:32 -07:00
  • a1b3de86cd Refactor the test code for attention kernels (#13) Woosuk Kwon 2023-03-29 18:59:27 -07:00
  • 64e0e38314 Add cache watermark to avoid frequent cache eviction (#11) Woosuk Kwon 2023-03-29 16:38:48 -07:00
  • 721fa3df15 FastAPI-based working frontend (#10) Zhuohan Li 2023-03-29 14:48:56 +08:00
  • d359cda5fa Minor Woosuk Kwon 2023-03-26 08:00:39 +00:00
  • 2f49f15585 Support tensor parallel (#2) Zhuohan Li 2023-03-22 04:45:42 +08:00
  • cfae35b861 Add miscellaneous updates (#8) Woosuk Kwon 2023-03-13 13:48:38 -07:00
  • e9d3f2ff77 Add memory analyzer & utomatically configure KV cache size (#6) Woosuk Kwon 2023-03-11 23:23:14 -08:00
  • 1a7eb7da61 Support beam search & parallel generation (#7) Woosuk Kwon 2023-03-10 09:58:21 -08:00
  • 04e5acc08e Fix a bug in 1D input shape (#5) Woosuk Kwon 2023-03-06 10:05:27 -08:00
  • 3e9f991d6a Use FlashAttention for multi_query_kv_attention (#4) Woosuk Kwon 2023-03-01 21:13:08 -08:00
  • 0deacbce6e Implement single_query_cached_kv_attention kernel (#3) Woosuk Kwon 2023-03-01 15:02:19 -08:00
  • cbf8779afa Fix a bug in tying OPT embeddings (#1) Woosuk Kwon 2023-02-24 16:29:36 -08:00
  • c84c708a1d Add README Woosuk Kwon 2023-02-24 12:04:49 +00:00
  • fa16389a2e Clean up the server script Woosuk Kwon 2023-02-24 11:56:21 +00:00
  • 6aef2278f4 [Minor] Fix printing format Woosuk Kwon 2023-02-24 11:56:06 +00:00
  • 1132fae0ca Add Frontend Woosuk Kwon 2023-02-24 11:46:43 +00:00
  • 46ce1356f7 Add max_num_steps to SamplingParams Woosuk Kwon 2023-02-24 11:44:40 +00:00
  • b39f149a08 Add is_finished Woosuk Kwon 2023-02-24 11:44:21 +00:00
  • ef6098ec51 Merge pre_step and step Woosuk Kwon 2023-02-24 10:36:08 +00:00
  • 53f70e7334 Reduce the number of states in scheduler Woosuk Kwon 2023-02-24 10:22:39 +00:00
  • 762fd1c3fa Refactor and annotate types for attention Woosuk Kwon 2023-02-24 08:58:46 +00:00
  • 7f22f90e8c Remove xformers Woosuk Kwon 2023-02-24 08:36:16 +00:00
  • afdbe5d373 [WIP] Add server script Woosuk Kwon 2023-02-24 01:33:37 +00:00
  • 932844f1cd Fix attention Woosuk Kwon 2023-02-23 23:02:25 +00:00
  • ba84b8728a Fix attention Woosuk Kwon 2023-02-23 22:29:46 +00:00
  • 87e0bcd426 Fix attention Woosuk Kwon 2023-02-23 21:32:02 +00:00
  • 1ce1333573 Set default dtype to half Woosuk Kwon 2023-02-23 21:31:39 +00:00
  • de0fabbc5c Fix sampler Woosuk Kwon 2023-02-23 20:30:12 +00:00
  • fdd0f2f472 Minor Woosuk Kwon 2023-02-23 20:23:47 +00:00
  • 7f985166f7 Consider pempty tensor Woosuk Kwon 2023-02-23 20:20:33 +00:00
  • 86f9eb6d39 Fix typo Woosuk Kwon 2023-02-23 20:19:24 +00:00
  • 1f6c7ef437 Add controller Woosuk Kwon 2023-02-23 09:32:19 +00:00
  • d4bc1a4d24 Add unoptimized OPT Attention Woosuk Kwon 2023-02-23 09:31:55 +00:00
  • b56b6ca0d6 Add greedy sampler Woosuk Kwon 2023-02-23 09:26:09 +00:00
  • 343cea3dbc Add seq_ids to input metadata Woosuk Kwon 2023-02-23 09:25:01 +00:00
  • 4f6f4967f6 Add get_block_table Woosuk Kwon 2023-02-23 07:55:14 +00:00
  • 331fa0b042 Implement scheduler.step & Add a threshold for batch size Woosuk Kwon 2023-02-23 07:54:20 +00:00
  • 501c4bd0cd decoding.py -> sampling_params.py Woosuk Kwon 2023-02-23 07:39:20 +00:00
  • 86c682cd32 DecodingParams -> SamplingParams Woosuk Kwon 2023-02-23 07:38:43 +00:00
  • af16c05074 Add get_len Woosuk Kwon 2023-02-23 05:58:04 +00:00
  • d094512296 Move max_context_len Woosuk Kwon 2023-02-23 04:57:46 +00:00
  • 4b1ac23f53 Fix slot mapping Woosuk Kwon 2023-02-23 00:10:07 +00:00
  • 8290fce47d Add Worker class Woosuk Kwon 2023-02-22 19:01:38 +00:00
  • 7b6844e590 Add input metadata Woosuk Kwon 2023-02-22 19:01:20 +00:00
  • 608f74ffe5 Minor Woosuk Kwon 2023-02-22 18:08:25 +00:00
  • 709a69176e Move worker/models -> models Woosuk Kwon 2023-02-22 18:03:48 +00:00
  • af68ec1c5c Add tests for kernels Woosuk Kwon 2023-02-18 19:23:07 +00:00
  • c413c41cda Add reshape_and_cache op Woosuk Kwon 2023-02-18 19:22:57 +00:00
  • ffad4e1e03 cache_kernel -> cache_kernels Woosuk Kwon 2023-02-16 20:05:45 +00:00
  • 6d2f74efb3 Remove redundant fn Woosuk Kwon 2023-02-16 09:24:42 +00:00
  • 3b41f16596 Add gitignore Woosuk Kwon 2023-02-16 07:47:21 +00:00
  • 6f058c7ba8 Implement cache ops Woosuk Kwon 2023-02-16 07:47:03 +00:00
  • a1c67e6db8 Minor Woosuk Kwon 2023-02-16 01:42:53 +00:00
  • 9e68a6827e Fix return type error Woosuk Kwon 2023-02-16 01:33:03 +00:00
  • 8edcabc737 Add warning Woosuk Kwon 2023-02-16 01:28:17 +00:00
  • 2f4887de77 Fix KVCache shape Woosuk Kwon 2023-02-16 01:24:45 +00:00
  • 3363c27d19 Add __repr__ Woosuk Kwon 2023-02-14 09:34:07 +00:00
  • 2729087efe Fix a ref count bug in swapping Woosuk Kwon 2023-02-14 09:20:12 +00:00
  • c128c2ed03 Fix double free Woosuk Kwon 2023-02-14 09:14:05 +00:00
  • c80ed212a1 Minor Woosuk Kwon 2023-02-14 09:06:27 +00:00
  • 263e91b386 Minor Woosuk Kwon 2023-02-14 09:05:39 +00:00
  • be1e2163c9 Fix memory leak in swapping Woosuk Kwon 2023-02-14 08:45:59 +00:00
  • 7e5f604e68 Fix bugs in scheduler Woosuk Kwon 2023-02-14 02:25:32 +00:00
  • 1f739f9b03 Fix a bug in swap_in and swap_out Woosuk Kwon 2023-02-14 02:12:58 +00:00
  • b1644f764f Fix can_swap_in Woosuk Kwon 2023-02-14 01:57:40 +00:00
  • c574f19506 Fix typo Woosuk Kwon 2023-02-14 01:19:27 +00:00
  • e40fa136aa Fix typo Woosuk Kwon 2023-02-14 01:19:05 +00:00
  • ee9442518d Fix get_model Woosuk Kwon 2023-02-13 22:51:03 +00:00
  • 531e1c74e8 Fix typo Woosuk Kwon 2023-02-13 18:51:33 +00:00
  • d3d317665b Fix scheduler Woosuk Kwon 2023-02-13 09:37:00 +00:00
  • fffa2e1f4b Add model_utils Woosuk Kwon 2023-02-13 09:36:12 +00:00
  • bb59a3e730 Fix cache engine Woosuk Kwon 2023-02-13 09:35:48 +00:00
  • 5a309bb588 Add scheduler Woosuk Kwon 2023-02-13 02:39:53 +00:00
  • 0961f5a49a Add find method to sequence group Woosuk Kwon 2023-02-13 02:39:12 +00:00
  • eb52db1bea Fix can_swap_in Woosuk Kwon 2023-02-13 02:38:50 +00:00
  • a2a9869cb7 SERVING -> RUNNING Woosuk Kwon 2023-02-12 08:25:05 +00:00
  • 46958cf941 BlockAllocator -> BlockManager Woosuk Kwon 2023-02-12 08:15:18 +00:00
  • 3be29a1104 Add blank setup file Woosuk Kwon 2023-02-09 11:37:06 +00:00
  • 38ed06a846 Minor Woosuk Kwon 2023-02-09 11:30:55 +00:00