K comes from hidden_states.size(-1) which is the full BF16 dimension (7168), not the packed weight dimension. K*2=14336 is wrong. The MoE output is always hidden_dim (7168).
K comes from hidden_states.size(-1) which is the full BF16 dimension (7168), not the packed weight dimension. K*2=14336 is wrong. The MoE output is always hidden_dim (7168).