fix: cutlass.Float32 not cutlass.float32_t in fused epilogue
Step 1 SiLU validation: PASS - cute.exp works on register tensors - SiLU (x / (1+exp(-x))) in registers matches PyTorch reference - Relative error: 0.034%, Max abs error: 0.0625 (BF16 precision limit)
This commit is contained in:
@@ -2156,9 +2156,9 @@ class FusedSwiGLUScaledGroupedGemmKernel:
|
||||
# With interleaved weights (granularity 8 BF16 = 4 FP4),
|
||||
# the accumulator N dimension has gate/up pairs adjacent.
|
||||
# SiLU(x) = x * sigmoid(x) = x / (1 + exp(-x))
|
||||
neg_acc = acc_vec * cutlass.float32_t(-1.0)
|
||||
neg_acc = acc_vec * cutlass.Float32(-1.0)
|
||||
exp_neg = cute.exp(neg_acc)
|
||||
sigmoid = cutlass.float32_t(1.0) / (cutlass.float32_t(1.0) + exp_neg)
|
||||
sigmoid = cutlass.Float32(1.0) / (cutlass.Float32(1.0) + exp_neg)
|
||||
swiglu_result = acc_vec * sigmoid
|
||||
|
||||
# Write SwiGLU result as BF16 to C tensor
|
||||
|
||||
Reference in New Issue
Block a user