biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

[BugFix] Fix DeepSeek-V3.1 + DeepGEMM incompatible scale shapes (#32361)

Browse Source 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Eldar Kurtić <8884008+eldarkurtic@users.noreply.github.com>
This commit is contained in:
Lucas Wilkinson
2026-01-14 23:32:22 -07:00
committed by GitHub
parent 9d7ae3fcdb
commit 2c9b4cf5bf
1 changed files with 3 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
vllm/model_executor/layers/quantization/utils/quant_utils.py
View File

@@ -300,6 +300,9 @@ def get_and_maybe_dequant_weights(
if ( if (
isinstance(layer.quant_method, Fp8LinearMethod) isinstance(layer.quant_method, Fp8LinearMethod)
and not layer.quant_method.use_marlin and not layer.quant_method.use_marlin
# DeepGEMM transforms the scales using `transform_sf_into_required_layout` into
# a layout that is not compatible with `scaled_dequantize`.
and not layer.quant_method.use_deep_gemm
): ):
weight_scales = get_attribute_fallback( weight_scales = get_attribute_fallback(
layer, ["weight_scale", "weight_scale_inv"] layer, ["weight_scale", "weight_scale_inv"]