ef398006a7
fix: correct scale factor dimensions in warmup (K_sf = ceil_div(K_packed,8) not ceil_div(K_packed,16))
...
K_packed = original_K // 2. The scale factor dimension is
K_sf = ceil_div(original_K, 16) = ceil_div(K_packed * 2, 16) = ceil_div(K_packed, 8).
The previous code used ceil_div(K_packed, 16) which was wrong.
2026-05-20 02:08:26 +00:00
8f1a20562f
fix: root-cause JIT memory corruption myth, add eager warmup, remove _needs_token_refill
...
Bug #1 fix: The _needs_token_refill workaround was a band-aid over a
misdiagnosis. cute.compile does NOT corrupt GPU memory (verified on B200).
The original corruption was from a different bug (likely OOB write or
weight loading issue).
Changes:
- bridge.py: Add warmup_compilation() for eager JIT before runtime buffers
exist. Pre-allocate workspace per cache entry (no torch.full in hot path).
Cache stores {compiled, workspace, workspace_size} instead of just compiled.
CuTe tensor wrappers re-created per call (cheap metadata, avoids stale refs).
- runner.py: Remove _needs_token_refill hack. Add eager warmup call in
_ensure_stacked() for both L1 and L2 GEMM shapes.
- nvfp4_linear.py: Add eager warmup in finalize_weights() for single GEMM.
The warmup approach ensures cute.compile runs exactly once per shape during
model init, before any forward pass. This is deterministic and eliminates
any possible interaction between JIT and runtime GPU memory.
2026-05-20 02:08:01 +00:00
6ec0afc318
fix: handle 3D swa_indices and correct kv_bf16 expand dims
2026-05-20 01:36:27 +00:00
aa593361e7
feat: add native CuTeDSL SWA decode attention kernel stub + batched SDPA fallback
2026-05-20 01:28:05 +00:00
3599b44c0f
fix: replace _allocate_buffers with _ensure_buffer_size for dynamic sizing
2026-05-20 00:02:10 +00:00
1d5e70adfb
fix: dynamic buffer sizing in nvfp4_linear for varying token counts
2026-05-19 23:59:55 +00:00
1901bf585e
nuke vllm because this keep confusing people
2026-05-19 23:04:36 +00:00
5fb70b4cd2
Update README.md and CURRENT_BUG.md: eliminate stale issues, document NaN investigation, clarify our kernels are clean
2026-05-19 20:22:10 +00:00
2e6559402c
Add full layer NaN test (attention + MoE, multi-layer chain)
2026-05-19 18:36:49 +00:00
cca145e35c
Use 16 experts for MoE runner test (fits in memory)
2026-05-19 18:35:40 +00:00
7893e7514d
Add MoE runner NaN test (grouped GEMM with real weights)
2026-05-19 18:34:56 +00:00
7b432da754
Fix intermediate size: 3072 not 18432
2026-05-19 18:34:12 +00:00
293f14a179
Rewrite MoE NaN test: per-expert format, activation quantization, grouped GEMM
2026-05-19 18:33:57 +00:00
62f2395e30
Fix MoE weight key names, add fallback
2026-05-19 18:32:49 +00:00
9455466648
Add MoE NaN reproduction test, update CURRENT_BUG.md with NaN tracing and test plan
2026-05-19 18:32:14 +00:00
0316cec6fb
Add input NaN debug to trace where NaN starts
2026-05-19 18:15:53 +00:00
4c45d73b82
Add prefill inputs NaN debug
2026-05-19 18:04:18 +00:00
0773c9608c
Add prefill attention value debug check
2026-05-19 17:55:35 +00:00
4f02113aa0
Use module-level Blackwell flag in compressor (works during torch.compile)
2026-05-19 17:37:26 +00:00
8cf6ac3e8c
CRITICAL FIX: Remove double Q normalization and fix RoPE sin slice
2026-05-19 17:27:33 +00:00
a94ad73c64
Fix imports in vLLM codepaths test
2026-05-19 17:26:50 +00:00
f3f9674810
Fix f-string syntax
2026-05-19 17:26:40 +00:00
6cc2312e61
Add test for exact vLLM codepaths (fused_qnorm, kv_write, decode)
2026-05-19 17:26:10 +00:00
aade8593f7
CRITICAL FIX: Properly dequantize fp8 KV in decode using per-token inv_scale
2026-05-19 17:08:58 +00:00
2f811bc8bd
FIX: Use vLLM's decode_swa_indices for correct paged KV cache access during decode
2026-05-19 16:55:44 +00:00
da6fa2f1d6
Fix UnboundLocalError: move num_decode_tokens before debug print
2026-05-19 16:43:28 +00:00
76fff5fc8b
CRITICAL FIX: Skip compressor fused attention kernel on Blackwell — it bypasses our attention path
2026-05-19 16:35:07 +00:00
0554332352
Add debug logging to Blackwell attention path
2026-05-19 16:31:55 +00:00
f9a09df81a
Fix wrapper attribute access: kv_cache, attn_sink, max_model_len via mla_attn
2026-05-19 16:19:28 +00:00
b95e934703
Add CSA/HCA decode + prefill attention to Blackwell path
2026-05-19 16:06:24 +00:00
abff942edd
Fix N for C128A (need 128 tokens)
2026-05-19 16:04:53 +00:00
49c2e088d4
Fix compressor key name
2026-05-19 16:04:38 +00:00
7d89ede9f9
Add CSA sparse attention test (compressed KV gather + SWA merge)
2026-05-19 16:04:19 +00:00
51a7a89c5c
Update CURRENT_BUG: KV cache pipeline verified, all tests passing
2026-05-19 16:01:10 +00:00
696a890df7
Add decode vs prefill consistency test
2026-05-19 16:00:33 +00:00
359654f08e
Test with all 61 layers (shared experts only)
2026-05-19 15:55:41 +00:00
3e6041d752
Fix view→reshape for non-contiguous tensor
2026-05-19 15:54:40 +00:00
ff9f373633
Add e2e decode test (3 layers: C128A, C4A, SWA)
2026-05-19 15:53:29 +00:00
a5870fa05c
Vectorize paged KV cache read/write, kill container
2026-05-19 15:48:16 +00:00
9e428b83c7
Fix KV cache: write to paged cache, handle uint8→fp8 conversion, fix RoPE bug
2026-05-19 15:34:09 +00:00
0023fee706
Add blackwell_attention module and comprehensive test
2026-05-19 15:30:29 +00:00
142a4a1ad4
Fix attention for decode (1 query vs N cached KVs)
2026-05-19 15:28:52 +00:00
4b85605edf
Fix fp8 amax in decode test
2026-05-19 15:28:17 +00:00
4f23055450
Add decode attention pipeline test — reproduces KV cache bug
2026-05-19 15:27:55 +00:00
31b9cfbdbd
Update README and CURRENT_BUG: BUILD YOUR OWN KERNELS. Stop patching vLLM.
2026-05-19 15:19:55 +00:00
dca8bfc3a8
Fix _apply_rope_kv: use inline RoPE instead of 3D apply_gptj_rope
2026-05-19 10:36:21 +00:00
8e6721917e
Fix syntax in RoPE KV test
2026-05-19 10:31:07 +00:00
cbf440f75a
Add RoPE KV test
2026-05-19 10:28:15 +00:00
a5fabbdf66
Apply RoPE to KV in Blackwell attention path - fix NaN output
2026-05-19 10:27:15 +00:00
7e97551fd3
Fix: use self.scale instead of self.softmax_scale in Blackwell attention path
2026-05-19 10:04:46 +00:00
