diff --git a/CURRENT_BUG.md b/CURRENT_BUG.md index 07f2f586..20df2585 100644 --- a/CURRENT_BUG.md +++ b/CURRENT_BUG.md @@ -1,13 +1,13 @@ # Current Bug: CuTeDSLMoERunner — Status & Debug History -## Current Status (May 17, 2026 17:54 UTC) +## Current Status (May 17, 2026 21:30 UTC) -**Build #11 in progress (includes swiglu_limit fix). Previous builds produced empty/invisible token output.** +**Bug 26 FIXED. All tests pass.** - ✅ `layertest.py` — 0.988 cosine - ✅ `cudagraph_test.py` — capture + replay works -- ✅ vLLM container starts, loads weights, warmup gs computed, cudagraph capture succeeds -- ❌ Model output was empty content (30 invisible tokens) — **swiglu_limit fix not yet tested in container** +- ✅ `test_pipeline_real_weights.py` — 0.988 cosine (8 tokens, 48 experts) +- ⏳ vLLM container — needs rebuild + test with Bug 26 fix **Latest fix: Missing swiglu_limit=10.0 activation clamping (Bug 25).** DeepSeek-V4 uses `SiluAndMulWithClamp(10.0)` which clamps `silu(gate)` to max 10.0 and `up` to [-10, 10]. Our runner was doing plain `F.silu(gate) * up` without clamping. Large gate values → unbounded SiLU output → corrupted L2 GEMM input → garbage logits → model outputs BOS/thinking tokens. @@ -121,6 +121,11 @@ activated = gate_silu * up ``` Called from `deepseek_v4.py` after warmup: `self._cutedsl_runner.set_swiglu_limit(float(self.swiglu_limit))`. +### Bug 26: Padded Buffer x_sf Mismatch — Experts 1+ Get Zero Output — FIXED +**Symptom:** Runner produces cosine 0.285 vs BF16. Some tokens get exactly zero output. Expert 0 L1 cosine 0.996, experts 1+ get cosine 0.0. +**Root cause:** Runner quantized `padded_hidden` (4096 rows with zero padding) → `quantize_activation_nvfp4` returns x_sf with 4096 rows. Then `x_sf[:num_slots]` (first 48 rows) only covers expert 0's tokens (padded rows 0-127). Expert 1's tokens are at padded row 128, but x_sf[4] corresponds to padded row 64 (still expert 0's padding), not expert 1's data. The scale assembly scattered wrong scales for experts 1+, producing zero GEMM output. +**Fix:** Quantize `slot_hidden` (sorted tokens, num_slots rows) instead of `padded_hidden`. This gives x_sf with num_slots rows (one per token), which the scale assembly correctly scatters into padded layout. Scatter x_fp4 into a new `hidden_fp4` padded buffer (uint8→view float4). Same fix for L2 with `activated_fp4` buffer. + --- ## Current Architecture: Variable Padded Expert Offsets @@ -190,6 +195,7 @@ DeepseekV4MoE.forward(): |------|---------| | `tests/layertest.py` | Reference vs runner, 3 experts. Must pass ≥0.98 cosine. | | `tests/cudagraph_test.py` | Cudagraph capture + replay. Must pass. | +| `tests/test_pipeline_real_weights.py` | Full runner vs BF16 reference, 48 experts, 8 tokens. Must pass ≥0.98 cosine. | **Run order after any code change:** 1. `python3 tests/layertest.py` — must pass