biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Fix INT8 quantization error on Blackwell GPUs (SM100+) (#25935)

Browse Source 
Signed-off-by: padg9912 <phone.and.desktop@gmail.com>
This commit is contained in:
Param
2025-09-30 22:19:53 -04:00
committed by GitHub
parent 1244948885
commit 99028fda44
2 changed files with 9 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
docs/features/quantization/int8.md
View File

@@ -6,7 +6,11 @@ This quantization method is particularly useful for reducing model size while ma
Please visit the HF collection of [quantized INT8 checkpoints of popular LLMs ready to use with vLLM](https://huggingface.co/collections/neuralmagic/int8-llms-for-vllm-668ec32c049dca0369816415).
!!! note
INT8 computation is supported on NVIDIA GPUs with compute capability > 7.5 (Turing, Ampere, Ada Lovelace, Hopper, Blackwell).
INT8 computation is supported on NVIDIA GPUs with compute capability > 7.5 (Turing, Ampere, Ada Lovelace, Hopper).
!!! warning
**Blackwell GPU Limitation**: INT8 is not supported on compute capability >= 100 (e.g., RTX 6000 Blackwell).
Use [FP8 quantization](fp8.md) instead, or run on Hopper/Ada/Ampere architectures.
## Prerequisites