biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

[Kernel] Tuned FP8 Kernels for Ada Lovelace (#6677)

Browse Source 
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
This commit is contained in:
Varun Sundar Rabindranath
2024-07-29 11:42:35 -04:00
committed by GitHub
parent 7cbd9ec7a9
commit 766435e660
6 changed files with 877 additions and 490 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
benchmarks/cutlass_benchmarks/w8a8_benchmarks.py
View File

@@ -13,7 +13,7 @@ from weight_shapes import WEIGHT_SHAPES
from vllm import _custom_ops as ops
from vllm.utils import FlexibleArgumentParser
DEFAULT_MODELS = list(WEIGHT_SHAPES.keys())[1:]
DEFAULT_MODELS = list(WEIGHT_SHAPES.keys())
DEFAULT_BATCH_SIZES = [1, 16, 32, 64, 128, 256, 512]
DEFAULT_TP_SIZES = [1]