biondizzle a10c582cf4 Add swiglu_limit=10.0 activation clamping (was missing) ...

DeepSeek-V4 uses SiluAndMulWithClamp(10.0) which clamps:
- silu(gate) to max 10.0
- up to [-10.0, 10.0]

Our runner was doing plain F.silu(gate) * up without clamping.
Large gate values could produce unbounded SiLU output, causing
numerical issues in the L2 GEMM. This is likely contributing to
garbage model output.

2026-05-17 17:52:16 +00:00

..

patches

Add swiglu_limit=10.0 activation clamping (was missing)

2026-05-17 17:52:16 +00:00

nvfp4_cutedsl.py

Add swiglu_limit=10.0 activation clamping (was missing)

2026-05-17 17:52:16 +00:00