biondizzle/nvfp4-megamoe-kernel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: L1 gate/up split — intermediate_size is per-projection, not fused

Browse Source 
intermediate_size=3072 is the size of gate OR up, not gate+up.
Split L1 output at intermediate_size, not intermediate_size//2.
gate = l1_out[:, :3072], up = l1_out[:, 3072:]
This commit is contained in:
biondizzle
2026-05-16 04:04:40 +00:00
parent 37aa0cbeab
commit 6d17988b51
1 changed files with 2 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
vllm/nvfp4_cutedsl.py
View File

@@ -165,9 +165,8 @@ class CuTeDSLMoERunner:
# ════════════════════════════════════════════════════════════
# SiLU(gate) * up (BF16)
# ════════════════════════════════════════════════════════════
half = self.intermediate_size // 2
gate = l1_out[:, :half]
up = l1_out[:, half:]
gate = l1_out[:, :self.intermediate_size]
up = l1_out[:, self.intermediate_size:]
activated = torch.nn.functional.silu(gate) * up
# ════════════════════════════════════════════════════════════