Commit Graph

5 Commits

Author SHA1 Message Date
5a79065b2b fix: GEMM output should be 2x packed N (float4_e2m1fn_x2 packs 2 per element) 2026-05-16 18:27:44 +00:00
533089c9d2 fix: token_indices slice bug + torch.zeros for float4/float8 dtypes 2026-05-16 18:21:27 +00:00
7594968482 WIP: cudagraph-compatible CuTeDSL MoE runner
- Cache compiled CuTeDSL kernel (compile once, reuse every forward)
- Remove torch.cuda.synchronize() from forward path
- Add quantize_activation_nvfp4() (no .max() CPU-GPU sync)
- Pre-allocate buffers (token_indices, expert_id_range, output_bufs)
- GPU-only expert offset computation (bincount + cumsum)
- Replace Python for-loop scale assembly with GPU-vectorized version

Still TODO:
- Test with FULL_AND_PIECEWISE cudagraph mode
- Add vllm::deepseek_v4_mega_moe_experts to splitting_ops
- Verify CuTeDSL kernel launch is cudagraph-safe
2026-05-16 16:36:19 +00:00
baf44c92f8 fix: memory-efficient E2M1 quantization — no 32x distance tensor
quantize_to_nvfp4 was allocating a (..., n_blocks, block_size, 8)
float32 tensor for nearest-neighbor distances to all 8 E2M1 values.
That's 32x the input size — 10.5GB for a typical batch, causing OOM
with only 3GB free.

New approach: clamp to [0, 6], scale to half-integer steps, round,
then map through a 13-byte lookup table to E2M1 indices.
Peak memory is now ~2x input (x_f32 + x_scaled) instead of 32x.

This makes activation quantization CUDA-graph-safe for the
memory-constrained DeepSeek-V4 on B200 (175GB model / 178GB GPU).
2026-05-16 07:49:38 +00:00
0cdcc4144a refactor: add cutedsl/bridge.py, rewrite layertest to use it
bridge.py: clean API for CuTeDSL kernel
- quantize_to_nvfp4 / quantize_weight_to_nvfp4
- assemble_scales_2d_side / assemble_scales_3d_side
- make_b_k_major (stride conversion)
- compute_expert_offsets
- run_nvfp4_grouped_gemm (full kernel launch)

layertest.py: now uses bridge layer, tests with real
DeepSeek-V4 layer 0 weights (7168 hidden, 6144 intermediate).

The bridge code will be reused by the vLLM integration layer.
2026-05-16 03:13:54 +00:00