From 6692166d0f1c61bd723e01cb66e08ab1009c6ec9 Mon Sep 17 00:00:00 2001 From: biondizzle Date: Sun, 17 May 2026 17:56:04 +0000 Subject: [PATCH] Update CURRENT_BUG.md: Bug 25 (swiglu_limit), shared expert path verification, variable padded offsets --- CURRENT_BUG.md | 158 +++++++++++++++++++++++++++++-------------------- 1 file changed, 93 insertions(+), 65 deletions(-) diff --git a/CURRENT_BUG.md b/CURRENT_BUG.md index b69be6ae..07f2f586 100644 --- a/CURRENT_BUG.md +++ b/CURRENT_BUG.md @@ -1,21 +1,17 @@ # Current Bug: CuTeDSLMoERunner — Status & Debug History -## Current Status (May 17, 2026 16:52 UTC) +## Current Status (May 17, 2026 17:54 UTC) -**vLLM runs, cudagraph capture succeeds, but model output is empty/invisible tokens (garbage logits). Going back to layer tests to debug GEMM output quality.** +**Build #11 in progress (includes swiglu_limit fix). Previous builds produced empty/invisible token output.** -- ✅ `layertest.py` — 0.988 cosine (with dynamic gs reference) +- ✅ `layertest.py` — 0.988 cosine - ✅ `cudagraph_test.py` — capture + replay works -- ✅ Container builds, loads weights, warmup gs computed (no L2 gs=0) -- ✅ With `--gpu_memory_utilization=0.9` and `max_cudagraph_capture_size=8`, container starts and serves -- ❌ Model output is empty content (30 tokens of invisible/BS token) — MoE GEMM output is wrong +- ✅ vLLM container starts, loads weights, warmup gs computed, cudagraph capture succeeds +- ❌ Model output was empty content (30 invisible tokens) — **swiglu_limit fix not yet tested in container** -**Next step:** Debug WHY the runner produces 0.988 cosine in layertest but garbage in vLLM. Likely issues: -1. The warmup gs values (computed from random data) don't match real runtime activation magnitudes -2. The scale assembly layout is subtly wrong for 48 experts vs 3 experts in the test -3. The padded buffer scatter (clamped_local) is dropping real token data +**Latest fix: Missing swiglu_limit=10.0 activation clamping (Bug 25).** DeepSeek-V4 uses `SiluAndMulWithClamp(10.0)` which clamps `silu(gate)` to max 10.0 and `up` to [-10, 10]. Our runner was doing plain `F.silu(gate) * up` without clamping. Large gate values → unbounded SiLU output → corrupted L2 GEMM input → garbage logits → model outputs BOS/thinking tokens. -**Current vLLM launch config:** +**vLLM launch config:** ``` --gpu_memory_utilization=0.9 --compilation-config='{"cudagraph_mode": "FULL_DECODE_ONLY", "custom_ops": ["all"], "cudagraph_capture_sizes": [1, 2, 4, 8], "max_cudagraph_capture_size": 8}' @@ -23,7 +19,7 @@ --- -## Bugs Found & Fixed (1–21) +## Bugs Found & Fixed ### Bug 1: Scale Assembly — Global vs Per-Expert Swizzle **Fix:** Two-phase scatter + per-expert swizzle. @@ -50,106 +46,141 @@ **Fix:** Separate `_padded_x_sf_buf_l1`/`_l2`, separate per-expert bufs. ### Bug 8: Global→Local Expert ID Mismatch — CUDA_ERROR_ASSERT +**Symptom:** `IndexKernel.cu:111` OOB, cascading CUDA_ERROR_ASSERT (710). **Root cause:** `topk_ids` contains global IDs (0-255), runner treated as local. **Fix:** `experts_start_idx`, remap global→local, mask non-local tokens. ### Bug 8b: `.cpu()` Sync Breaking Cudagraph **Fix:** `_token_indices` on GPU, `_fill_token_indices()` CPU→GPU copy. -### Bug 9: `padded_x_sf` Buffer Too Small -**Fix:** Iterative — see Bugs 14, 16. - -### Bug 10: Wrong `top_k`/`max_num_tokens` Defaults -**Fix:** Pass from `deepseek_v4.py`. - -### Bug 11: Full-Buffer Swizzle Wrong for GEMM -**Fix:** Per-expert swizzle. +### Bug 9–11: Buffer sizing and swizzle layout +See previous versions for details. ### Bug 12: `torch.full()` During Cudagraph Capture **Symptom:** `cudaErrorStreamCaptureUnsupported`. **Fix:** Pre-allocated buffers, `.fill_()` instead of `torch.full()`. ### Bug 13: Warmup Passed Global Expert IDs -**Symptom:** L2 gs=0.0 on EP5/EP7. -**Fix:** Pass local IDs. +**Fix:** Pass local IDs (0..num_experts-1). -### Bug 14: GEMM Scale Layout Mismatch — 128-Row Fixed vs Variable +### Bug 14: GEMM Scale Layout Mismatch — Fixed 128-Row vs Variable **Symptom:** BOS token repeat (garbage logits). -**Root cause:** Scale assembly at `e*128` offsets, GEMM reads by real `expert_offsets`. Expert with 500 tokens → GEMM reads 500 scale rows but only 128 have data. -**Fix:** Fixed-layout padding: each expert gets `max_chunks * 128` rows. Pad `slot_hidden`. Pass `padded_expert_offsets` to GEMM. Extract via `l1_out[padded_dst]`. +**Root cause:** Scale assembly at `e*128`, GEMM reads by real expert_offsets. Expert with 500 tokens → GEMM reads 500 scale rows but only 128 have data. +**Fix:** Variable padded expert offsets, scatter into real padded positions. ### Bug 15: OOM — Per-Layer Padded Buffers (4.3 GB) -**Root cause:** `padded_hidden_buf` + `padded_activated_buf` at 72 MB × 60 layers. +**Root cause:** 72 MB × 60 layers = 4.3 GB. Not enough room for KV cache. **Fix:** Shared buffers (Bug 21). ### Bug 16: `padded_max_slots` Mismatch -**Root cause:** Sized for `max_tokens*top_k` but needed `num_experts*max_chunks*128`. -**Fix:** Size correctly. +**Fix:** Size for `num_experts * max_chunks * 128`. ### Bug 17: Shape Mismatch (49152 vs 3072) **Root cause:** Cap `max_num_tokens` to 512 made buffers too small for 8192-token warmup. -**Fix:** Reverted cap, use shared buffers (Bug 21). +**Fix:** Reverted cap, use shared buffers. -### Bug 18–20: Cudagraph Capture Failures -**Root cause:** Dynamic tensor allocation (`torch.zeros`), variable-trip loops, GPU scalars in Python control flow. +### Bug 18–20: Cudagraph Capture Failures (dynamic allocs, variable loops, GPU scalars) **Fix:** Pre-allocate everything, fixed loop counts, Python constants for offsets. -### Bug 21: OOM (correct fix) — Shared Padded Buffers -**Root cause:** Per-layer allocation of padded buffers. -**Fix:** Class-level shared buffers dict keyed by device. Layers execute sequentially → safe to share. Also shared `padded_x_sf_buf` and `output_buf`. Total ~150 MB instead of ~4.3 GB. +### Bug 21: OOM — Shared Padded Buffers +**Fix:** Class-level shared buffers dict keyed by device. `padded_hidden`, `padded_activated`, `padded_xsf_l1`/`l2`, `output` all shared. ~150 MB total instead of ~4.3 GB. + +### Bug 22: Token Dropping via `clamped_local` +**Symptom:** Garbage model output (empty/invisible tokens). +**Root cause:** `local_row.clamp(max=max_rows_per_expert-1)` silently dropped tokens when an expert got more than `max_chunks*128` tokens. `max_chunks` was computed as average (ceil(total_slots / (num_experts*128))), not worst-case. MoE routing is uneven — some experts get 200+ tokens while others get 10. +**Fix:** Use real padded expert offsets (variable per expert, padded to 128). No clamping needed — each expert gets exactly the space it needs. + +### Bug 23: cudaErrorStreamCaptureUnsupported from Dynamic GPU Slicing +**Symptom:** All 8 workers fail during cudagraph capture. +**Root cause:** `buf[:total_padded_slots]` where `total_padded_slots` is a GPU scalar — dynamic tensor slicing with a GPU index is a CUDA operation not permitted during stream capture. +**Fix:** Use full pre-allocated buffers, no dynamic GPU slicing. Pass `x_sf[:num_slots]` (Python int) to scale assembly. + +### Bug 24: Scale Assembly `.cpu().tolist()` Breaks Cudagraph +**Symptom:** `cudaErrorStreamCaptureInvalidated` during capture. +**Root cause:** Per-expert Python loops with GPU-derived offsets required `.cpu().tolist()` for slicing — CPU-GPU sync invalidates stream capture. +**Fix:** Full-buffer Blackwell 32_4_4 swizzle. Apply `to_blocked` transform to entire `padded_x_sf` buffer at once. No CPU syncs, no Python loops. The buffer is already 128-row aligned per expert and 4-col aligned, so the full-buffer swizzle produces the correct layout. GEMM reads `scale_a` using `padded_expert_offsets`, matching the scatter layout. + +### Bug 25: Missing `swiglu_limit=10.0` Activation Clamping — LIKELY CAUSE OF GARBAGE OUTPUT +**Symptom:** Model generates 30 tokens of empty/invisible content (BOS or thinking token). Not meaningful text. +**Root cause:** DeepSeek-V4 uses `SiluAndMulWithClamp(10.0)` which: +- Clamps `silu(gate)` to max 10.0 +- Clamps `up` to [-10.0, 10.0] + +Our runner did plain `F.silu(gate) * up` without clamping. Large gate values produce unbounded SiLU output (silu(20) ≈ 20, silu(50) ≈ 50). These large values get multiplied by the up projection, producing activations with amax >> 10. This: +1. Corrupts the L2 GEMM input (quantized with wrong gs) +2. Produces garbage L2 output +3. Final logits are wrong → model collapses to most frequent token (BOS) + +**Fix:** Added `set_swiglu_limit(limit)` to runner. In `run()`, apply clamping: +```python +gate_silu = F.silu(gate) +if self._swiglu_limit is not None: + gate_silu = gate_silu.clamp(max=self._swiglu_limit) + up = up.clamp(min=-self._swiglu_limit, max=self._swiglu_limit) +activated = gate_silu * up +``` +Called from `deepseek_v4.py` after warmup: `self._cutedsl_runner.set_swiglu_limit(float(self.swiglu_limit))`. --- -## Current Architecture: Fixed-Layout Padding +## Current Architecture: Variable Padded Expert Offsets ``` -Each expert gets max_chunks * 128 rows at offset (e * max_chunks * 128). +Each expert padded to next multiple of 128 tokens. +padded_expert_offsets computed from real tokens_per_expert (GPU). -Scatter: padded_dst = expert_assign * max_rows_per_expert + clamped_local_row -GEMM input: padded_hidden (total = num_experts * max_chunks * 128 rows) -GEMM offsets: [0, max_rows, 2*max_rows, ...] (fixed, pre-computed) -GEMM output: same total rows -Extract: l1_out[padded_dst] → only real token rows +Scatter: padded_dst = padded_expert_offsets[expert_assign] + local_row +GEMM input: padded_hidden (full pre-allocated buffer, not sliced) +GEMM offsets: padded_expert_offsets[1:] (GPU tensor) +GEMM output: full buffer size; extract via l1_out[padded_dst] Scale assembly: - Phase 1: Scatter x_sf into padded_x_sf at same fixed offsets - Phase 2: Per-expert, per-chunk swizzle (fixed loop: max_chunks iterations) - No dynamic tensor allocation, no GPU→CPU syncs + Phase 1: Scatter x_sf into padded_x_sf at padded_expert_offsets + Phase 2: Full-buffer Blackwell 32_4_4 swizzle (no CPU syncs) + Zero CPU syncs, zero Python loops -Shared buffers (class-level): +Activation: + SiLU(gate) clamped to swiglu_limit (10.0) + up clamped to [-swiglu_limit, swiglu_limit] + activated = clamped_silu * clamped_up + +Shared buffers (class-level, ~150 MB total): padded_hidden, padded_activated, padded_xsf_l1, padded_xsf_l2, output - ~150 MB total (not per-layer) ``` ### Cudagraph Constraints (All Resolved) - No `.item()`, `.cpu()`, `.tolist()` -- No `torch.zeros/ones/full/empty/arange()` during capture -- No dynamic Python control flow from GPU values -- Per-expert Python loops OK (fixed `num_experts`) +- No `torch.zeros/ones/full/empty/arange()` during capture — pre-allocate everything +- No dynamic GPU slicing (`buf[:gpu_scalar]`) — use full buffers +- No Python loops with GPU-derived values — full-buffer ops instead +- No `torch.full()` — pre-allocated `.fill_()` - Shared buffers OK (layers sequential during capture and replay) +- `F.silu().clamp()` and `.clamp()` are GPU ops — cudagraph-safe ✅ ### EP Configuration (DeepSeek-V4-Pro on 8×B200) -- 256 total experts, top_k=6 -- EP=8 → 48 local experts per rank +- 256 total experts, top_k=6, swiglu_limit=10.0 +- EP=8 → 48 local experts per rank (n_routed_experts / ep_size = 256/8 = 32, but logs show 48) - `experts_start_idx` = rank × 32 - `max_num_tokens` = 8192 - `max_chunks_per_expert` = ceil(8192 × 6 / (48 × 128)) = 8 --- -## Outstanding Issue: Garbage Model Output +## Shared Expert Path (verified correct) -**Symptom:** Model generates 30 tokens of empty/invisible content (BOS or thinking token). Not meaningful text. +``` +DeepseekV4MoE.forward(): + 1. gate → fused_topk_bias → topk_weights, topk_ids + 2. self.experts(hidden_states, topk_weights, topk_ids) → routed_output + 3. EP all-reduce across ranks + 4. self.shared_experts(hidden_states) → shared_output + 5. final = routed_output + shared_output +``` -**What works:** layertest gives 0.988 cosine with 3 experts, 8 tokens, top_k=8. - -**What doesn't:** vLLM with 48 experts, variable tokens, top_k=6 produces garbage. - -**Hypotheses to investigate:** -1. **Warmup gs from random data ≠ real activation magnitudes.** The warmup uses `torch.randn` (amax ~3) but real activations have amax ~8-10. The gs values would be wrong, causing quantization errors. -2. **Scale assembly with 48 experts × 8 chunks.** With max_chunks=8 and 48 experts, there are 384 swizzle blocks. The fixed-layout scatter with `clamped_local` may be dropping tokens that overflow the expert's max_rows section. -3. **`clamped_local = local_row.clamp(max=max_rows_per_expert - 1)`.** If an expert has more than `max_chunks*128` real tokens, overflow tokens all map to the same row, overwriting each other. This silently drops data. -4. **The `_needs_token_refill` path.** After GEMM JIT, `_token_indices` may get corrupted. The refill happens AFTER the first run, but the first run already used corrupted indices. +- Shared experts: `DeepseekV4MLP` (not NVFP4, uses standard quantization) +- `routed_scaling_factor`: Applied in `fused_topk_bias` to topk_weights ✅ +- `renormalize`: Top-k weights normalized to sum to 1 ✅ +- `scoring_func=sqrtsoftplus`: Applied in routing ✅ --- @@ -159,9 +190,6 @@ Shared buffers (class-level): |------|---------| | `tests/layertest.py` | Reference vs runner, 3 experts. Must pass ≥0.98 cosine. | | `tests/cudagraph_test.py` | Cudagraph capture + replay. Must pass. | -| `tests/test_runner_vs_pipeline.py` | Runner vs pipeline comparison. | -| `tests/test_scale_assembly.py` | Scale assembly comparison. | -| `tests/test_warmup_gs.py` | Warmup gs computation. | **Run order after any code change:** 1. `python3 tests/layertest.py` — must pass