docs: update DEBUG_LOG with cosine≈0 finding + new hypotheses

DEBUG_LOG.md

@@ -57,8 +57,30 @@ The vLLM weight loader does:
 
 This may need to stay in native BF16 and route through a BF16 matmul path instead.
 
-### 7. 🔍 BF16 reference comparison
-**Status: In progress.** Adding a diagnostic that dequantizes FP4 activation + FP4 weights back to BF16, runs a reference matmul, then compares to the NVFP4 GEMM output via cosine similarity. This will isolate whether the CUTLASS kernel is producing correct output given the same quantized inputs.
+### 7. ✅ BF16 reference comparison — COSINE ≈ 0
+**Status: CONFIRMED.** The BF16 reference comparison ran (after fixing several bugs in the diagnostic code). Result: **cosine similarity ≈ 0** between NVFP4 GEMM output and BF16 dequantized reference. This means the CUTLASS kernel is producing output that is essentially uncorrelated with the correct result.
+
+Results from all 8 TP ranks:
+```
+[TP0] cosine=-0.001789  mse=1.0201e+01  nvfp4_amax=8.5625  ref_amax=8.0000
+[TP1] cosine= 0.030470  mse=1.0157e+01  nvfp4_amax=8.0625  ref_amax=8.3125
+[TP2] cosine=-0.009217  mse=9.5978e+00  nvfp4_amax=9.1875  ref_amax=7.5312
+[TP3] cosine= 0.001786  mse=9.4161e+00  nvfp4_amax=8.6875  ref_amax=8.8750
+[TP4] cosine= 0.007108  mse=7.5709e+00  nvfp4_amax=7.3125  ref_amax=8.8750
+[TP5] cosine=-0.000572  mse=7.8290e+00  nvfp4_amax=7.5938  ref_amax=10.562
+[TP6] cosine= 0.012143  mse=9.2720e+00  nvfp4_amax=8.0000  ref_amax=8.1250
+[TP7] cosine=-0.010009  mse=9.0296e+00  nvfp4_amax=6.6250  ref_amax=36.500
+```
+
+**Key insight:** The magnitudes are in the same ballpark (amax 7-10 vs 8-10), but the *direction* is completely wrong. This is NOT a scaling error — it's a systematic misalignment. The output vectors are essentially random relative to the correct answer.
+
+**This proves the problem is in the CUTLASS GEMM itself** (or the data layout going into it), NOT in the attention, weight loading, or scaling math. The standalone test with random data showed cosine 1.0, but real data gives cosine ≈ 0. The difference must be in data layout/stride/alignment that the random test didn't exercise.
+
+### 8. 🔍 CUTLASS GEMM layout mismatch
+**Status: Active investigation.** The standalone test used random data with simple row-major layout and got cosine 1.0. Real data also uses row-major layout, but cosine ≈ 0. Possible causes:
+- **SF remap incorrect for specific M/N/K dimensions** — the remap was verified with coordinate probes for the standalone test dimensions, but real MoE dimensions (M=1, N=6144, K=7168) may expose a different code path
+- **Activation layout** — `stage_activation` produces flat row-major packed E2M1, but CUTLASS may expect a different micro-tiling for the A matrix
+- **Weight transpose convention** — after `transform_nvfp4_weights_for_mega_moe` transpose, the weight may not be in the layout CUTLASS expects for B (column-major vs row-major interpretation)
 
 ## Key Commits
 
@@ -69,6 +91,11 @@ This may need to stay in native BF16 and route through a BF16 matmul path instea
 | `995589a` | Add FP4 quantization round-trip diagnostic |
 | `c421a66` | Add BF16 reference GEMM + cosine comparison for L1 |
 | `2fd55a9` | Fix weight reshape bug (K_half,N×2 → K,N) + igs double-count |
+| `9159cb6` | Add DEBUG_LOG.md documentation |
+| `de8acc7` | Dump raw GEMM inputs + first 8 output values |
+| `755f9ad` | Fix per_expert_alpha ref + clean up BF16 reference scaling |
+| `df916b8` | Fix gs.item() for multi-element tensor |
+| `7739674` | Fix gs scalar conversion with .cpu().tolist() + add traceback |
 
 ## Bugs Fixed During This Debug Session
 
@@ -88,6 +115,17 @@ This may need to stay in native BF16 and route through a BF16 matmul path instea
 
 **Impact:** Only affected the BF16 reference diagnostic, not the kernel.
 
+### BF16 reference diagnostic: multiple bugs (commits `c421a66`→`7739674`)
+
+The BF16 reference comparison had a cascade of bugs that took 4 iterations to fix:
+
+1. **Weight reshape bug (commit `2fd55a9`):** `reshape(K_half, -1)` on 2D weight flattened N dimension. Fixed: `reshape(K_half*2, N)`.
+2. **per_expert_alpha not defined (commit `755f9ad`):** The reference code ran before `per_expert_alpha` was computed. Fixed: use `l1_alpha * l1_global_sf[e0]` directly.
+3. **gs.item() on multi-element tensor (commits `df916b8`, `7739674`):** `gs` is shape (2,) — `gs[0].item()` should work but didn't in context. Fixed: `gs.detach().cpu().tolist()`.
+4. **igs double-count (commit `2fd55a9`):** Multiplying by igs in both x_bf16 and the final output. Fixed: apply igs once in x, apply gs per-half separately.
+
+**Impact:** All bugs only in diagnostic code. The actual NVFP4 kernel was never affected.
+
 ## Architecture Notes
 
 ### DeepSeek-V4 MoE Layer Forward Pass