biondizzle
7594968482
WIP: cudagraph-compatible CuTeDSL MoE runner
- Cache compiled CuTeDSL kernel (compile once, reuse every forward)
- Remove torch.cuda.synchronize() from forward path
- Add quantize_activation_nvfp4() (no .max() CPU-GPU sync)
- Pre-allocate buffers (token_indices, expert_id_range, output_bufs)
- GPU-only expert offset computation (bincount + cumsum)
- Replace Python for-loop scale assembly with GPU-vectorized version
Still TODO:
- Test with FULL_AND_PIECEWISE cudagraph mode
- Add vllm::deepseek_v4_mega_moe_experts to splitting_ops
- Verify CuTeDSL kernel launch is cudagraph-safe