fix: shared_experts gate_up_proj stacking was skipped by .experts. check
The stacking logic skipped any key containing '.experts.' to avoid MoE routed expert weights. But 'shared_experts' also matches that substring, so gate_proj and up_proj were never stacked into gate_up_proj. Changed to '.ffn.experts.' which only matches the routed experts path. Also includes POST-LOAD all-zero param scan.
This commit is contained in:
@@ -1311,8 +1311,10 @@ class DeepseekV4Model(nn.Module):
|
||||
break # first match wins (order matters)
|
||||
|
||||
for param_name, weight_name, shard_id in stacked_params_mapping:
|
||||
# Skip non-stacked layers and experts (experts handled below).
|
||||
if ".experts." in name:
|
||||
# Skip MoE routed experts (handled separately below).
|
||||
# Use .ffn.experts. (not .experts.) to avoid skipping
|
||||
# shared_experts which also contains ".experts.".
|
||||
if ".ffn.experts." in name:
|
||||
continue
|
||||
if weight_name not in name:
|
||||
continue
|
||||
@@ -1370,7 +1372,7 @@ class DeepseekV4Model(nn.Module):
|
||||
loaded_params.add(name)
|
||||
break
|
||||
else:
|
||||
if ".experts." in name:
|
||||
if ".ffn.experts." in name:
|
||||
# E8M0 scales are stored as float8_e8m0fnu in
|
||||
# MXFP4 checkpoints but NVFP4 uses float8_e4m3fn.
|
||||
# The uint8 view+copy path is only valid for MXFP4;
|
||||
|
||||
Reference in New Issue
Block a user