1ba304db3e
stuff
2026-05-28 21:08:13 +00:00
adc88613fa
Milestone 5 COMPLETE: multi-head FMHA grid launch verified on B200
...
All HD=16/64/128/256 pass across MHA (4+8 heads), MQA, batched modes.
cos 0.999997+, LSE matches reference. Updated CURRENT_ISSUE.md.
2026-05-28 19:35:06 +00:00
aa41cfa2e5
Multi-head FMHA kernel (Milestone 5): grid launch with MHA/MQA/batch support
...
- fmha_6warp_multihead.cuh: grid=(1, n_h, batch) kernel with FmhaParams
- MQA support via k_head_stride=0 / v_head_stride=0
- LSE output for multi-segment KV merge composition
- test_fmha_6warp_multihead.cu: MHA (4+8 heads), MQA, batched tests
- HD-specific wrappers for hd=16/64/128/256
- Marked E2M1 dequant bug as FIXED in consultant issue file
2026-05-28 19:32:35 +00:00
090f2866ae
Update CURRENT_ISSUE: 6-warp Milestone 1 complete
2026-05-28 16:35:02 +00:00
a18d9c1584
Update CURRENT_ISSUE: ALL HD=16/64/128/256 PASS cos 0.999997+
...
Documented Layout D N=64 bug and N=16 sub-tile workaround.
2026-05-28 16:03:05 +00:00
2b32b51882
Update CURRENT_ISSUE with final session status
2026-05-28 15:22:32 +00:00
6896d1aebb
Update CURRENT_ISSUE: HD=16 done, HD=64 in progress
2026-05-28 15:16:19 +00:00
5b2e690936
Milestone: Full FMHA HD=16 with PV SS MMA (SMEM-P) — cosine 0.9997
2026-05-28 14:50:43 +00:00
ed8f48dddf
Add systematic SS+TS sequence test to debug MMA coexistence crash
2026-05-28 14:10:07 +00:00
efa03f53d4
docs: update CURRENT_ISSUE and MEMORY — full FMHA HD=64 pipeline working
2026-05-28 13:11:32 +00:00
71c774027c
test: fix HD=64 QK — zero TMEM, fence after MMA, single-thread MMA call
2026-05-28 12:47:51 +00:00
df34cae9c6
UMMA QK GEMM WORKING! Update docs — 4x was scale factor, not bug
...
Major milestone: UMMA QK GEMM produces correct attention scores at HD=16!
- MMA computes raw dot product; apply 1/sqrt(HD) scaling manually
- tcgen05.fence::after_thread_sync for MMA→TMEM fence
- 32x32b.x8 TMEM reads for Layout D output
- 4 warps (128 threads) required for M=128
- Next: HD=64 multi-K-tile, PV GEMM, full FMHA pipeline
2026-05-28 11:41:19 +00:00
32f7fa7bce
Update CURRENT_ISSUE.md and MEMORY.md with UMMA 4× bug details
...
- MMA produces exactly 4× scalar reference for all output values
- SMEM data verified correct, descriptor values correct
- 4× persists across different N, warp counts, padding
- TMEM multi-store bug documented (16x256b.x1 crashes on 2nd store)
- Layout D read with 32x32b.x8 works
- Next: study CUTLASS FMHA TMEM output layout to fix 4× factor
2026-05-28 10:15:14 +00:00
427410d94a
UMMA: Rewrite fmha_umma_desc.cuh with correct K-major core-matrix layout + minimal QK GEMM test
...
- Core-matrix layout: each 8x8 BF16 tile (128B) contiguous in SMEM
- K-major NONE descriptor: LBO=1 (16B), SBO=block_k/8, lbo_mode=0
- MMA K-tiling: tcgen05.mma uses K=16 per call, tile for hd>16
- write_smem_kmajor: converts row-major to core-matrix layout
- write_smem_ktile: extracts single K-tile in core-matrix layout
- test_umma_qk.cu: minimal hd=16, sk=128 test (single MMA call)
- Previous UMMA descriptors were wrong (row-major SMEM, wrong LBO/SBO)
2026-05-28 09:15:40 +00:00
