biondizzle/nvfp4-megamoe-kernel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Commit Graph

  • e614d0894c Clean up SMEM acc epilogue: flat indexing sO_acc->sC, TMA store from sC_s0 biondizzle 2026-05-27 04:57:40 +00:00
  • 1724eeb8ec Fix TMA store: use epi_s view of sC for proper layout compatibility biondizzle 2026-05-27 04:55:18 +00:00
  • 3a7d87adba Fix test_smem_acc: use keyword args for lse/row_sums biondizzle 2026-05-27 04:54:23 +00:00
  • 6a621bdf64 D1.5: SMEM accumulator FMHA kernel — one-way TMEM→REGS→SMEM, no round-trip biondizzle 2026-05-27 04:53:40 +00:00
  • 81acf1593c Revert "D1.5: WIP SMEM accumulator — framework in place, accumulation logic TODO" biondizzle 2026-05-27 02:17:26 +00:00
  • 72d88af400 D1.5: WIP SMEM accumulator — framework in place, accumulation logic TODO biondizzle 2026-05-27 02:15:23 +00:00
  • a6da93ddfb Revert "D1.5: Try O rescale with tCtO_base layout (epilogue-proven TMEM addressing)" biondizzle 2026-05-27 02:12:20 +00:00
  • 79e2eb3b42 D1.5: Try O rescale with tCtO_base layout (epilogue-proven TMEM addressing) biondizzle 2026-05-27 02:10:39 +00:00
  • f94978ffa7 D1.5: Prepare for SMEM accumulator implementation biondizzle 2026-05-26 21:00:41 +00:00
  • afb93eae22 D1.5: Revert broken TMEM round-trip O rescale, document as fundamentally broken biondizzle 2026-05-26 20:55:16 +00:00
  • 42c5793add D1.5: Add isolated round-trip test comparing s_k=128 vs s_k=256 with NOOP rescale biondizzle 2026-05-26 20:45:58 +00:00
  • e35b30dae6 D1.5 debug: try corr_tile_size=32 for O rescale round-trip biondizzle 2026-05-26 20:43:29 +00:00
  • 20ed6d5114 D1.5: Add TMEM load fence before PV with ACCUMULATE, revert debug rescale factor biondizzle 2026-05-26 20:31:28 +00:00
  • 34d64137ec D1.5 debug: force rescale_factor=0.5 to test if round-trip code executes biondizzle 2026-05-26 20:29:34 +00:00
  • 3be708d923 D1.5 debug: add NOOP rescale test (acc_scale=1.0) to isolate TMEM round-trip corruption biondizzle 2026-05-26 20:28:55 +00:00
  • c3648e4ebf D1.5 debug: add targeted s_k=256 rescale diagnostic test biondizzle 2026-05-26 20:27:37 +00:00
  • bf2c7c8bb8 D1.5: Implement in-kernel O rescale via CUTLASS correction_rescale pattern biondizzle 2026-05-26 20:26:06 +00:00
  • 064ececc9a Update docs: D1.5 TMEM round-trip fundamentally broken, Python KV merge is production path biondizzle 2026-05-26 19:53:10 +00:00
  • 2b4f4ce538 Remove broken D1.5 paired-atom test (TMEM round-trick is fundamentally broken) biondizzle 2026-05-26 19:50:31 +00:00
  • ffb3e736bb D1.5: Revert broken paired-atom O rescale — TMEM round-trip fundamentally broken biondizzle 2026-05-26 19:50:11 +00:00
  • 40cbf0c223 Add D1.5 paired-atom O rescale test (s_k=256/384, hd=64/128) biondizzle 2026-05-26 19:46:19 +00:00
  • 43f0b5d1e8 D1.5: Fix O rescale with paired atoms (incremental approach) biondizzle 2026-05-26 19:34:26 +00:00
  • 4bb0e063cc D1.5: Replace broken TMEM round-trip with correction epilogue (paired atoms) biondizzle 2026-05-26 19:11:19 +00:00
  • f97aee6eed plan update biondizzle 2026-05-26 19:00:22 +00:00
  • 487d960a6a D5c multi-tile: VERIFIED cos 0.999996 with Python KV merge + sink bias biondizzle 2026-05-26 15:40:45 +00:00
  • c9eab3c7e0 diag: rewrite multi-tile test with explicit per-segment compile and reference biondizzle 2026-05-26 15:39:39 +00:00
  • 7f983fb855 diag: add direct segment 0 test to compare with run_segment biondizzle 2026-05-26 15:37:06 +00:00
  • 2a5f9dc6e3 fix: simplify run_segment to use full hd V tensor (was incorrectly splitting by pv_n_tile) biondizzle 2026-05-26 15:34:57 +00:00
  • aa2df1a202 diag: test n_comp=96 with sink bias directly biondizzle 2026-05-26 15:33:38 +00:00
  • 25b236fe00 diag: test D5c multi-tile with no sink bias to isolate issue biondizzle 2026-05-26 15:31:38 +00:00
  • a3989929de diag: per-segment reference comparison for D5c multi-tile biondizzle 2026-05-26 15:29:52 +00:00
  • bbc29945e8 diag: add per-segment debug output for D5c multi-tile biondizzle 2026-05-26 15:28:19 +00:00
  • e64392f1ac D5c: add apply_sink_bias flag (independent of n_comp) biondizzle 2026-05-26 15:26:52 +00:00
  • 60b6f8ee79 fix: use separate kernels for segments with/without compressed KV (n_comp is compile-time) biondizzle 2026-05-26 15:23:21 +00:00
  • 2efd15c852 fix: correct swa_len_local calculation per segment for D5c multi-tile biondizzle 2026-05-26 15:22:03 +00:00
  • 3abcc7ff09 D5c: multi-tile test using Python KV merge with sink bias biondizzle 2026-05-26 15:20:45 +00:00
  • 57a8316bc1 update README: D5c sink bias DONE (cos 0.999996, single KV tile) biondizzle 2026-05-26 15:17:10 +00:00
  • 8f7df4d8b5 fix: mRowSums dummy tensor must match mLSE layout (3D, not 1D) biondizzle 2026-05-26 15:14:35 +00:00
  • ffc4b542bc D5c: use single KV tile (s_k=128) to avoid broken O rescale biondizzle 2026-05-26 15:11:50 +00:00
  • fc0f4bcf23 diag: test D5c with single KV tile (s_k=128) to isolate O rescale issue biondizzle 2026-05-26 15:09:49 +00:00
  • e5381b7312 diag: add baseline test (s_k=256 D3 mask, no sink bias) to isolate D5c issue biondizzle 2026-05-26 15:08:40 +00:00
  • 016edbcc97 D5c: add row_sum output for proper external normalization biondizzle 2026-05-26 15:07:22 +00:00
  • 31e6426049 fix: normalize kernel output using per-row LSE for D5c test biondizzle 2026-05-26 15:04:47 +00:00
  • 014d647ba3 fix: sink bias domain correction — add attn_sink/scale to raw logits biondizzle 2026-05-26 15:03:49 +00:00
  • dbdbcecadc fix: sink_bias must be pre-converted to CuTe tensor before passing to compile biondizzle 2026-05-26 15:02:43 +00:00
  • 04b66e0f9c fix: test_d5c use float for attn_sink in reference functions biondizzle 2026-05-26 15:01:31 +00:00
  • 9d64434954 D5c: add sink bias (attn_sink) logit modification to FMHA kernel biondizzle 2026-05-26 14:59:52 +00:00
  • 60a6f2d296 update README: D5b per-row LSE, D3/D4 DONE biondizzle 2026-05-26 11:03:57 +00:00
  • 865eed0d33 cleanup: remove debug test file biondizzle 2026-05-26 11:03:45 +00:00
  • 5b55cf0767 fix: k_seg is already 3D from slicing, don't add extra unsqueeze(-1) biondizzle 2026-05-26 11:02:44 +00:00
  • 375a682206 debug: isolated KV merge test biondizzle 2026-05-26 11:01:22 +00:00
  • 2252d7c865 fix: make K/V segments contiguous before passing to kernel (TMA needs contiguous tensors) biondizzle 2026-05-26 11:00:36 +00:00
  • 5407dc768a test: minor comment fix in D5b test biondizzle 2026-05-26 10:59:51 +00:00
  • bb7ec341cb fix: D5b test uses reference attn_sum for normalization, correct D5 merge formula biondizzle 2026-05-26 10:59:04 +00:00
  • 6c73069cb9 D5b: Per-row LSE output + Python KV merge test biondizzle 2026-05-26 10:57:54 +00:00
  • 4656fa81f9 update README: D3 and D4 status DONE biondizzle 2026-05-26 10:56:57 +00:00
  • 24993428a2 fix: D4 test reference computation only applies causal mask when is_causal=True biondizzle 2026-05-26 10:56:04 +00:00
  • e3e01071f4 fix: swa_len as Int32 scalar instead of CuTe tensor biondizzle 2026-05-26 10:54:41 +00:00
  • df84420414 fix: add is_causal to FmhaKernel __init__ signature biondizzle 2026-05-26 10:53:14 +00:00
  • 841a3e87b2 D4: Causal mask on SWA branch biondizzle 2026-05-26 10:52:30 +00:00
  • b6b581777a D3: In-kernel SWA sequence length masking biondizzle 2026-05-26 10:51:23 +00:00
  • d6a56342cc D3: add swa_lens parameter to FmhaKernel (in-kernel masking TBD) biondizzle 2026-05-25 17:31:01 +00:00
  • e9f476b6dc fix typo: from_dlset → from_dlpack biondizzle 2026-05-25 17:28:43 +00:00
  • f278348f44 D3: SWA mask with BF16 min pre-masking approach (K[invalid]=BF16_MIN → scores≈-inf) biondizzle 2026-05-25 17:27:35 +00:00
  • cfbeb9c454 D3: SWA mask test with zero-masking approach (pre-mask K/V in Python) biondizzle 2026-05-25 17:23:03 +00:00
  • 68cb0236b5 D3: add SWA sequence length mask test (reference oracle + full-window regression) biondizzle 2026-05-25 17:20:53 +00:00
  • 7f69979c5f D1.5: add multi-KV-tile attention test with Python KV merge biondizzle 2026-05-25 17:18:50 +00:00
  • 8f35b75164 D2: comprehensive head-packed test (n_h=1, 64, 128, hd=64, 128) biondizzle 2026-05-25 17:16:05 +00:00
  • dbe2ecbd41 D2: add num_query_heads/batch_size params + batch grid dimension biondizzle 2026-05-25 17:15:08 +00:00
  • 7c6fdd151d fix: use reference attn_sum for normalization (kernel LSE per-row may be wrong) biondizzle 2026-05-25 17:13:34 +00:00
  • 673825c242 rewrite D2 regression test: match existing Stage D1 test pattern with cute.compile + PV tiles biondizzle 2026-05-25 17:11:59 +00:00
  • 06cb800242 fix regression test: use normalize=False + external LSE normalization biondizzle 2026-05-25 17:06:21 +00:00
  • 13b5afc471 fully revert FmhaKernel changes to debug regression biondizzle 2026-05-25 17:04:31 +00:00
  • 0b9f9da2f7 revert grid change to debug regression biondizzle 2026-05-25 17:03:19 +00:00
  • aa66f44ff9 add n_h=1 regression test biondizzle 2026-05-25 17:00:56 +00:00
  • efdedab399 fix tests: use 3D tensors (M, hd, 1) matching kernel local_tile expectations biondizzle 2026-05-25 16:54:56 +00:00
  • a4499f5aa8 fix tests: pad Q to 128 rows (M tile size) for all configs biondizzle 2026-05-25 16:53:17 +00:00
  • af136eee27 fix: use CUstream instead of cuStream(0) biondizzle 2026-05-25 16:51:52 +00:00
  • 4826fa6afb D2: add num_query_heads/batch_size params + head-packed test biondizzle 2026-05-25 16:50:49 +00:00
  • d53e0a33a9 NVFP4-3: add use_2cta_instrs conditional to gemm_runner biondizzle 2026-05-25 16:42:02 +00:00
  • 22a2fc563e cleanup: remove diagnostic test file biondizzle 2026-05-25 16:25:05 +00:00
  • a064b99d3d fix test 4: use silu(gate)+swiglu interleaved (matching fused kernel output) biondizzle 2026-05-25 16:24:04 +00:00
  • e76ea36337 fix test: use proper global_scale from quantize_to_nvfp4 for larger shape test biondizzle 2026-05-25 16:23:00 +00:00
  • 5290c91c35 fix quantize_nvfp4 kernel: use proven single-thread-per-CTA pattern from deinterleave_quantize.cu biondizzle 2026-05-25 16:21:44 +00:00
  • 5508f29625 add GPU quantize diagnostic test biondizzle 2026-05-25 16:20:29 +00:00
  • c2e3d15633 NVFP4-1.1 integration: GPU-only quantize kernel + MoE pipeline wiring biondizzle 2026-05-25 16:19:04 +00:00
  • 6504f091ca NVFP4-1.1 Step 3: post-SWiGLU quantization test suite (all PASS) biondizzle 2026-05-25 09:08:01 +00:00
  • 5e8347836f NVFP4-1.1: working BF16→FP4 quantize kernel (cos 0.979) biondizzle 2026-05-25 08:58:19 +00:00
  • 52d11d7f92 NVFP4-1.1: standalone BF16→FP4 quantize kernel (WIP) + dequantize verification biondizzle 2026-05-25 03:23:44 +00:00
  • 1f310defa0 fix: quantize_activation_nvfp4 returns 2 values, not 3 biondizzle 2026-05-25 03:17:13 +00:00
  • 6dac3bcaf0 NVFP4-1.1: add FP4 quantize round-trip test (step 1 of kernel fusion) biondizzle 2026-05-25 03:15:40 +00:00
  • eb46e4d15e NVFP4-0.2-0.4: add FP4 primitives diagnostic test biondizzle 2026-05-25 03:07:53 +00:00
  • 29ad36934d cleanup: remove D2 diagnostic/experimental files, keep working codebase clean biondizzle 2026-05-25 02:40:12 +00:00
  • d5b69ac122 D2: simpler shape diagnostic using CuTe from Python (no kernel needed) biondizzle 2026-05-25 02:36:41 +00:00
  • 684e9a85fe fix: use utils.sm100 instead of sm100 in diagnostic biondizzle 2026-05-25 02:34:25 +00:00
  • 7599801f57 D2: add flat_divide shape diagnostic kernel for multi-CTA grid biondizzle 2026-05-25 02:33:15 +00:00
  • 32850f6974 Update README, STAGE_D, STAGE_D2 with D1 rescale findings and D2 status biondizzle 2026-05-25 01:18:48 +00:00
  • 6cc151097e Revert D2 multi-CTA attempts - keeping per-head launch approach (works correctly) biondizzle 2026-05-25 01:08:38 +00:00
  • 34f5beb767 D2: fix gC coordinate to match 5-mode flat_divide result biondizzle 2026-05-24 23:44:39 +00:00
  • a3559538cf D2: try 6-mode coordinate for flat_divide result biondizzle 2026-05-24 23:43:23 +00:00