- Add dense_router_dispatch_nvfp4_fused() in dense_router_decode.py:
single-kernel NVFP4 blockscaled GEMM + fused router epilogue
- Router.load_nvfp4_fused_gate(): stores raw NVFP4 tensors for fused path
- Router._run_dense_impl() dispatch priority: fused > 2-kernel > BF16
- single_shot_inference.py: loads raw NVFP4 gate weights for fused kernel
instead of building Nvfp4Linear (which was the 2-kernel path)
- Fix selection sort bug in nvfp4_fused_router_kernel.py: pass 0 was
missing t_s/t_i/t_a temp save before swap, causing undefined vars
- Export dense_router_dispatch_nvfp4_fused from __init__.py
The dense router now uses NVFP4 GEMM via Nvfp4Linear for the gate
projection when NVFP4 scales are available in the checkpoint. This
replaces the BF16 cuBLAS GEMM with Blackwell SM100 tensor-core
NVFP4 acceleration.
Changes:
- dsv4/layers/router.py: add gate_lin (Nvfp4Linear) alongside W_gate
fallback. New load_nvfp4_gate() method.
- dsv4/kernels/router/dense_router_decode.py: add
dense_router_dispatch_nvfp4() using Nvfp4Linear + activation_topk
- dsv4/kernels/router/__init__.py: export new function
- single_shot_inference.py: load NVFP4 gate weights when available,
fall back to BF16 when not
The first draft had a fake CuTeDSL kernel body with pass statements and
Python lists as register heaps. That is not the right way. This commit
replaces it with honest documentation of what the kernel does and what
needs to happen.
Current working path:
- All N routes through torch.nn.functional.linear + activation_topk.cu
- activation_topk is a single-pass fused CUDA kernel (all 6 steps)
- This is correct and performant for all N
CuTeDSL fused decode kernel (DenseRouterDecodeKernel):
- Class structure and warp specialization defined
- Full documentation of the TMA/MMA/epilogue pipeline
- The novel part is the row-level top-k epilogue (cross-subtile heap)
- EFC framework does not apply — our epilogue is not per-element
- Implementation deferred until profiling shows the GMEM round-trip
on logits matters for decode latency
No fake code. No pass statements. No Python lists as GPU registers.
The working path is the activation_topk kernel. The CuTeDSL kernel
will be built on top of it when the optimization is needed.
