CRITICAL FIX: don't fold input_scale into NVFP4 weight dequant

input_scale is the activation quantization scale (for FP8 inputs).
Since we use BF16 activations, the weight dequant is simply:
  lut[weight] * weight_scale * weight_scale_2

Folding input_scale in produced weights ~4000x too small,
causing all attention and FFN outputs to be effectively zero.
This commit is contained in:
2026-05-31 22:03:55 +00:00
parent 4e64acbb64
commit acc20dffd7

View File

@@ -84,7 +84,9 @@ def dequant_nvfp4(weight, weight_scale, weight_scale_2=None, input_scale=None):
w = torch.stack([lo_f, hi_f], -1).reshape(O, I)
s = weight_scale.float().repeat_interleave(16, 1)
if weight_scale_2 is not None: s = s * weight_scale_2.float()
if input_scale is not None: s = s * input_scale.float()
# NOTE: input_scale is intentionally NOT used. It's the activation
# quantization scale (for FP8 inputs). Since we use BF16 activations,
# the weight dequant is: lut[weight] * weight_scale * weight_scale_2.
return (w * s).bfloat16()
def nvfp4_linear(x, weight, weight_scale, weight_scale_2=None, input_scale=None):