Update CURRENT_BUG: warmup gs is irrelevant, bug is in vLLM pipeline

CURRENT_BUG.md

@@ -1,44 +1,22 @@
 # CURRENT_BUG.md
 
-## Status: Fix committed, needs vLLM container rebuild + test
+## Status: CuTeDSL kernels confirmed correct. Bug is in vLLM's attention/FFN pipeline.
 
-### Root Cause Found: Wrong Activation Global Scale
+### Key Findings from test_model_forward_b200.py
 
-**All CuTeDSL NVFP4 kernels are correct** (verified with standalone test, cosine 0.989-0.995 vs BF16 reference). The bug was in the vLLM integration, NOT our kernels.
+1. **Warmup gs is IRRELEVANT** — CuTeDSL `runner.run()` recomputes gs internally per-call. Changing gs by 10x has no effect on output (cosine 0.9993).
+2. **CuTeDSL kernels are correct** — cosine 0.999 vs BF16 for q_a_proj with both warmup and dynamic gs.
+3. **BF16 reference produces reasonable logits** — logit std 3.05, top5 valid token IDs.
+4. **The bug is NOT in our NVFP4 kernels** — it's in vLLM's pipeline.
 
-The `CuTeDSLNvFp4LinearKernel.process_weights_after_loading` (in `vllm/kernels/linear/nvfp4/cutedsl.py`) was using the checkpoint's `input_global_scale_inv` as the activation global scale. This is a calibration-time value that doesn't match what `quantize_activation_nvfp4` expects at runtime, producing garbage output.
+### Most Likely Causes
 
-### Fix
-
-Changed `cutedsl.py` to use warmup-based activation global scale computation (same as standalone test):
-```python
-# BEFORE (broken):
-runner._activation_global_scale = input_global_scale_inv  # wrong!
-
-# AFTER (fixed):
-runner.compute_activation_global_scale(sample)  # warmup-based, correct
-```
+1. **FlashMLA kernel on Blackwell (SM100)** — `fused_deepseek_v4_qnorm_rope_kv_rope_quant_insert` is a C++ CUDA kernel. If it doesn't work on B200, attention output is garbage.
+2. **Weight sharding with TP=8** — The model is loaded with TP=8. If weight sharding is wrong, all projections produce garbage. But our standalone test uses the full (non-sharded) weights, which works.
+3. **MoE produces garbage** — The MoE path (384 experts, top-6) is complex. If expert routing or grouped GEMM is wrong, the output is dominated by MoE noise.
 
 ### Next Steps
 
-1. Rebuild vLLM container on B200 with this fix
-2. Run `build_and_run.sh` 
-3. Test with curl chat completions
-4. If still broken, also fix MoE runner warmup (currently using checkpoint input_scale mean)
-
-### Standalone Test Results (test_full_layer_b200.py)
-
-```
-q_a_proj:     cosine=0.994599 ✅
-kv_proj:      cosine=0.994777 ✅
-q_b_proj:     cosine=0.994834 ✅
-wo_b_proj:    cosine=0.994768 ✅
-comp.kv_proj: cosine=0.994152 ✅
-comp.gate:    cosine=0.994766 ✅
-shared_expert: cosine=0.989745 ✅
-```
-
-### Remaining Issues
-
-1. **MoE warmup**: `compute_activation_global_scales` is never called on the MoE runner. Currently uses checkpoint input_scale mean. Needs warmup too.
-2. **Shared expert in vLLM**: Check if the vLLM shared expert path uses CuTeDSL or falls through to broken vLLM kernels.
+- Write a test that runs the FULL model (all 61 layers) in BF16 and checks the final output
+- Add hook/logging to the vLLM container to capture layer-by-layer output
+- Test if the FlashMLA C++ kernel works on B200