biondizzle/nvfp4-megamoe-kernel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,307 Commits 1 Branch 17 Tags
019a3a34b7eeccbccaa80200e4fcfdbcb2a16e66
Commit Graph

4 Commits

Author SHA1 Message Date
biondizzle
5493a8727e P7: compressor early return + decode buffering (skip GEMMs when n_complete=0); sampler SMEM fix (LK=24 fits 48KB default); topk on float not bf16 2026-06-01 22:29:56 +00:00
biondizzle
60715f89bc Fix CUDA kernel compilation: use c10::cuda::getCurrentCUDAStream 
- amax_gsa.cu: fix at::cuda::getCurrentCUDAStream → c10::
- amax_gsa.cu: fix torch::TensorOptions().device() → x.options()
- sampler.cu: same fixes for compilation on B200
- Both kernels now compile cleanly with torch.utils.cpp_extension.load
 2026-06-01 20:49:55 +00:00
biondizzle
2dc5b4ec19 Fix sampler kernel stack overflow: reduce MAX_K from 256 to 128 
128 * (sizeof(float) + sizeof(int)) = 1KB — within CUDA default stack limit.
256 * 8 = 2KB would overflow.
 2026-06-01 20:42:53 +00:00
biondizzle
4f698baa5d Production fused CUDA sampler + decode loop optimizations 
- Add dsv4/kernels/cuda/sampler.cu: fused temperature + repetition penalty
  + top-k + top-p (nucleus) sampling, single kernel launch, zero CPU syncs
- Add dsv4/model/sampler.py: CUDASampler wrapper + PyTorch reference
- Update single_shot_inference.py:
  - Use CUDASampler for non-greedy decoding (temperature=0.6, top_k=50, top_p=0.95)
  - Pre-allocate decode buffers (no per-step torch.tensor allocation)
  - Track thinking tokens (128821/128822) — not garbage for reasoning model
  - Reduce diagnostic CPU syncs (top-5 every 5 steps, NaN check every 20)
  - Add --top-k and --top-p CLI args
  - Default: temperature=0.6 (was 0.0 greedy), rep_penalty=1.1 (was 1.2)
 2026-06-01 20:29:57 +00:00