From 87a223f1ac1e51fc3e272ac9b89269bb561f6649 Mon Sep 17 00:00:00 2001 From: biondizzle Date: Sun, 17 May 2026 16:52:40 +0000 Subject: [PATCH] Update CURRENT_BUG.md: current status, outstanding garbage output issue, hypotheses --- CURRENT_BUG.md | 196 +++++++++++++++++++----------------------- docker-compose.yml | 3 +- tests/debug_output.py | 70 +++++++++++++++ 3 files changed, 162 insertions(+), 107 deletions(-) create mode 100644 tests/debug_output.py diff --git a/CURRENT_BUG.md b/CURRENT_BUG.md index f9177f1f..b69be6ae 100644 --- a/CURRENT_BUG.md +++ b/CURRENT_BUG.md @@ -1,145 +1,113 @@ # Current Bug: CuTeDSLMoERunner — Status & Debug History -## Current Status (May 17, 2026 15:51 UTC) +## Current Status (May 17, 2026 16:52 UTC) -**vLLM container build in progress. Previous crash was from OOM + shape mismatch. Both now fixed.** +**vLLM runs, cudagraph capture succeeds, but model output is empty/invisible tokens (garbage logits). Going back to layer tests to debug GEMM output quality.** -- ✅ `layertest.py` — 0.988 cosine +- ✅ `layertest.py` — 0.988 cosine (with dynamic gs reference) - ✅ `cudagraph_test.py` — capture + replay works - ✅ Container builds, loads weights, warmup gs computed (no L2 gs=0) -- 🔧 Build #7 in progress on B200 (shared buffer fix) -- ❌ Haven't gotten to serving yet (crashes were during init/capture) +- ✅ With `--gpu_memory_utilization=0.9` and `max_cudagraph_capture_size=8`, container starts and serves +- ❌ Model output is empty content (30 tokens of invisible/BS token) — MoE GEMM output is wrong -**Latest fixes (Bugs 17→21):** -- Bug 17 (shape mismatch 49152 vs 3072): Root cause was capping `max_num_tokens` to 512 for buffer sizing, but the actual warmup runs with 8192 tokens. Reverted the cap. -- Bug 21 (OOM): Instead of per-layer padded buffers (4.3 GB for 60 layers), use SHARED buffers across all runners. Only 72 MB total since layers run sequentially. +**Next step:** Debug WHY the runner produces 0.988 cosine in layertest but garbage in vLLM. Likely issues: +1. The warmup gs values (computed from random data) don't match real runtime activation magnitudes +2. The scale assembly layout is subtly wrong for 48 experts vs 3 experts in the test +3. The padded buffer scatter (clamped_local) is dropping real token data + +**Current vLLM launch config:** +``` +--gpu_memory_utilization=0.9 +--compilation-config='{"cudagraph_mode": "FULL_DECODE_ONLY", "custom_ops": ["all"], "cudagraph_capture_sizes": [1, 2, 4, 8], "max_cudagraph_capture_size": 8}' +``` --- -## Bugs Found & Fixed +## Bugs Found & Fixed (1–21) -### Bug 1: Scale Assembly — Global Swizzle vs Per-Expert Swizzle +### Bug 1: Scale Assembly — Global vs Per-Expert Swizzle **Fix:** Two-phase scatter + per-expert swizzle. -### Bug 2: `searchsorted(right=False)` — Wrong Expert Assignment +### Bug 2: `searchsorted(right=False)` **Fix:** Changed to `right=True`. ### Bug 3: CuTeDSL `cute.compile` GPU Memory Corruption — CRITICAL **Symptom:** `_token_indices` all zeros after JIT. -**Root cause:** `cute.compile` corrupts GPU memory. Tensors allocated before/during JIT get zeroed. -**Fix:** `_fill_token_indices()` builds on CPU, copies to GPU. `_needs_token_refill` for GEMM JIT. +**Root cause:** `cute.compile` corrupts GPU memory. +**Fix:** `_fill_token_indices()` builds on CPU, copies to GPU. `_needs_token_refill` flag. ### Bug 4: `expert_offsets` With Leading 0 -**Fix:** Pass `expert_offsets[1:num_experts + 1]` to the GEMM. +**Fix:** Pass `expert_offsets[1:]` to GEMM. -### Bug 5: Checkpoint `input_scale` Is Wrong for Activation Global Scale -**Root cause:** Checkpoint `input_scale` (~0.000286) is a calibration value. Too-small gs → block scale overflow → garbage. +### Bug 5: Checkpoint `input_scale` Wrong for Runtime gs +**Root cause:** Calibration value, too-small gs → block scale overflow. **Fix:** `compute_activation_global_scales()` warmup method. -### Bug 6: L1 and L2 Need Separate Activation Global Scales -**Fix:** Compute L2 gs from actual L1 output after SiLU*up. +### Bug 6: L1/L2 Need Separate gs +**Fix:** Compute L2 gs from L1 output after SiLU*up. -### Bug 7: L1 and L2 Need Separate Padded Scale Buffers -**Fix:** Separate `_padded_x_sf_buf_l1`/`_l2`, separate per-expert scale bufs. +### Bug 7: L1/L2 Need Separate Scale Buffers +**Fix:** Separate `_padded_x_sf_buf_l1`/`_l2`, separate per-expert bufs. ### Bug 8: Global→Local Expert ID Mismatch — CUDA_ERROR_ASSERT -**Symptom:** `IndexKernel.cu:111` OOB, cascading CUDA_ERROR_ASSERT (710) on all workers. -**Root cause:** `topk_ids` contains global IDs (0-255), runner treated as local (0-31/48). -**Fix:** Added `experts_start_idx`, remap global→local, mask non-local tokens. +**Root cause:** `topk_ids` contains global IDs (0-255), runner treated as local. +**Fix:** `experts_start_idx`, remap global→local, mask non-local tokens. -### Bug 8b: `.cpu()` Sync Breaking Cudagraph Compatibility -**Fix:** Moved `_token_indices` to GPU, `_fill_token_indices()` (CPU→GPU copy). +### Bug 8b: `.cpu()` Sync Breaking Cudagraph +**Fix:** `_token_indices` on GPU, `_fill_token_indices()` CPU→GPU copy. -### Bug 9: `padded_x_sf` Buffer Too Small — Index Out of Bounds -**Root cause:** Buffer sized for `num_experts * 128` rows, but scatter positions exceeded this with real token distributions. -**Fix:** Iterative — see Bugs 11, 14, 16 for the final solution. +### Bug 9: `padded_x_sf` Buffer Too Small +**Fix:** Iterative — see Bugs 14, 16. -### Bug 10: Wrong `top_k` and `max_num_tokens` Defaults -**Root cause:** Runner defaulted to `top_k=8`, vLLM uses top_k=6. -**Fix:** Pass values from `deepseek_v4.py`. +### Bug 10: Wrong `top_k`/`max_num_tokens` Defaults +**Fix:** Pass from `deepseek_v4.py`. -### Bug 11: Full-Buffer Swizzle Produced Wrong GEMM Input -**Symptom:** L2 gs=0.0 on EP5/EP7. -**Root cause:** Swizzled entire buffer at once; GEMM expects per-expert swizzled blocks. -**Fix:** Reverted to per-expert swizzle. +### Bug 11: Full-Buffer Swizzle Wrong for GEMM +**Fix:** Per-expert swizzle. ### Bug 12: `torch.full()` During Cudagraph Capture -**Symptom:** `cudaErrorStreamCaptureUnsupported` on all 8 workers. -**Root cause:** `torch.full()` allocates new tensor during stream capture. -**Fix:** Pre-allocated `_l1_gsa_buf`, `_l2_gsa_buf`, `_output_buf`, `_row_indices_buf`. Use `.fill_()`. +**Symptom:** `cudaErrorStreamCaptureUnsupported`. +**Fix:** Pre-allocated buffers, `.fill_()` instead of `torch.full()`. -### Bug 13: Warmup Passed Global Expert IDs Instead of Local +### Bug 13: Warmup Passed Global Expert IDs **Symptom:** L2 gs=0.0 on EP5/EP7. -**Root cause:** Warmup passed global IDs (336+) against local range (0..47). -**Fix:** Pass local IDs (0..num_experts-1). +**Fix:** Pass local IDs. -### Bug 14: GEMM Scale Layout Mismatch — Fixed 128-Row vs Variable -**Symptom:** Model generates BOS token repeatedly (garbage logits). -**Root cause:** Scale assembly placed data at fixed `e*128` offsets, but GEMM reads `scale_a` according to real `expert_offsets`. When expert 0 has 500 tokens, GEMM reads `scale_a[0:500]` but only rows 0-127 have valid data. -**Fix:** Fixed-layout padding: each expert gets `max_chunks * 128` rows at offset `e * max_chunks * 128`. Pad `slot_hidden` into this layout. Pass fixed `padded_expert_offsets` to GEMM. Extract real outputs via `l1_out[padded_dst]`. +### Bug 14: GEMM Scale Layout Mismatch — 128-Row Fixed vs Variable +**Symptom:** BOS token repeat (garbage logits). +**Root cause:** Scale assembly at `e*128` offsets, GEMM reads by real `expert_offsets`. Expert with 500 tokens → GEMM reads 500 scale rows but only 128 have data. +**Fix:** Fixed-layout padding: each expert gets `max_chunks * 128` rows. Pad `slot_hidden`. Pass `padded_expert_offsets` to GEMM. Extract via `l1_out[padded_dst]`. -### Bug 15: OOM — Padded Buffers Sized for 8192 Tokens (per-layer) -**Symptom:** `torch.OutOfMemoryError` trying to allocate 1008 MiB. -**Root cause:** `padded_hidden_buf` + `padded_activated_buf` at 72 MB per layer × 60 layers = 4.3 GB. Model+KV already at 175 GB on 178 GB GPUs. -**Fix (attempt 1 — wrong):** Cap `max_num_tokens` at 512. Caused Bug 17. -**Fix (attempt 2 — correct):** Shared buffers. See Bug 21. +### Bug 15: OOM — Per-Layer Padded Buffers (4.3 GB) +**Root cause:** `padded_hidden_buf` + `padded_activated_buf` at 72 MB × 60 layers. +**Fix:** Shared buffers (Bug 21). ### Bug 16: `padded_max_slots` Mismatch -**Root cause:** Computed from `max_tokens*top_k` (3072) but `total_padded_slots` is `num_experts*max_chunks*128` (6144). -**Fix:** Size for `num_experts * max_chunks * 128`. +**Root cause:** Sized for `max_tokens*top_k` but needed `num_experts*max_chunks*128`. +**Fix:** Size correctly. -### Bug 17: Shape Mismatch — slot_hidden 49152 vs padded_dst 3072 -**Symptom:** `RuntimeError: shape mismatch: [49152, 7168] cannot be broadcast to [3072, 7168]` -**Root cause:** Bug 15 fix capped `max_num_tokens` to 512, making `_token_indices` and buffers sized for 3072 slots. But the actual warmup/cudagraph forward pass uses 8192 tokens → `sorted_token_ids` has 49152 elements → `slot_hidden` has 49152 rows → doesn't fit in 3072-slot buffer. -**Fix:** Reverted the 512 cap. Use shared buffers (Bug 21) instead. +### Bug 17: Shape Mismatch (49152 vs 3072) +**Root cause:** Cap `max_num_tokens` to 512 made buffers too small for 8192-token warmup. +**Fix:** Reverted cap, use shared buffers (Bug 21). -### Bug 18: Dynamic Tensor Allocation in Scale Assembly -**Symptom:** `cudaErrorStreamCaptureInvalidated`. -**Root cause:** `torch.zeros()` for `padded_expert_offsets` inside `_assemble_scales_cudagraph_safe`. -**Fix:** Use fixed offsets from Python constants. - -### Bug 19: Variable-Trip `while` Loop in Scale Assembly -**Symptom:** `cudaErrorStreamCaptureInvalidated`. -**Root cause:** `while remaining > 0` loop with GPU scalar in condition → CPU sync. -**Fix:** Fixed `for c in range(max_chunks)` loop. - -### Bug 20: Another `torch.zeros()` in Scale Assembly -**Fix:** Removed. Use fixed `e * max_chunks * 128 + c * 128` offsets. +### Bug 18–20: Cudagraph Capture Failures +**Root cause:** Dynamic tensor allocation (`torch.zeros`), variable-trip loops, GPU scalars in Python control flow. +**Fix:** Pre-allocate everything, fixed loop counts, Python constants for offsets. ### Bug 21: OOM (correct fix) — Shared Padded Buffers -**Symptom:** Same as Bug 15 (4.3 GB for per-layer padded buffers). -**Root cause:** Per-layer allocation of `padded_hidden_buf` and `padded_activated_buf` at 72 MB × 60 layers. -**Fix:** Single shared set of padded buffers across all runners. Layers execute sequentially during both capture and replay, so the same buffer is reused. Total: 72 MB (not 4.3 GB). Stored as class-level dict keyed by device. +**Root cause:** Per-layer allocation of padded buffers. +**Fix:** Class-level shared buffers dict keyed by device. Layers execute sequentially → safe to share. Also shared `padded_x_sf_buf` and `output_buf`. Total ~150 MB instead of ~4.3 GB. --- -## vLLM Integration Status +## Current Architecture: Fixed-Layout Padding -| Component | Status | Notes | -|-----------|--------|-------| -| Weight loading | ✅ | Direct NVFP4 path, no BF16 round-trip | -| Weight stacking | ✅ | `make_b_k_major` + `assemble_scales_3d_side` | -| Global→local ID remap | ✅ | `experts_start_idx`, mask non-local tokens | -| Warmup gs computation | ✅ | Per-layer, local expert IDs, L1+L2 gs | -| Scale assembly | ✅ | Fixed max_chunks layout, no dynamic allocs | -| Cudagraph compatibility | ✅ | No dynamic allocs, no CPU syncs, fixed loops | -| Buffer sizing | ✅ | Shared buffers avoid OOM | -| Model output | ❓ | Build #7 in progress — never reached serving without crash | - ---- - -## Key Architecture: Fixed-Layout Padding - -### Current Design ``` -Each expert gets max_chunks * 128 rows at fixed offset (e * max_chunks * 128). - -padded_hidden: [exp0_chunk0][exp0_chunk1]...[exp1_chunk0]... - 128 rows 128 rows 128 rows +Each expert gets max_chunks * 128 rows at offset (e * max_chunks * 128). Scatter: padded_dst = expert_assign * max_rows_per_expert + clamped_local_row GEMM input: padded_hidden (total = num_experts * max_chunks * 128 rows) -GEMM offsets: [0, max_rows, 2*max_rows, ...] (fixed, pre-computed in _allocate_buffers) +GEMM offsets: [0, max_rows, 2*max_rows, ...] (fixed, pre-computed) GEMM output: same total rows Extract: l1_out[padded_dst] → only real token rows @@ -148,36 +116,52 @@ Scale assembly: Phase 2: Per-expert, per-chunk swizzle (fixed loop: max_chunks iterations) No dynamic tensor allocation, no GPU→CPU syncs -Shared buffers: - padded_hidden and padded_activated are class-level (not per-layer). - 72 MB total instead of 4.3 GB. Layers run sequentially → safe to share. +Shared buffers (class-level): + padded_hidden, padded_activated, padded_xsf_l1, padded_xsf_l2, output + ~150 MB total (not per-layer) ``` ### Cudagraph Constraints (All Resolved) -- No `.item()`, `.cpu()`, `.tolist()` — zero CPU-GPU syncs -- No `torch.zeros/ones/full/empty/arange()` during capture — pre-allocate everything -- No dynamic Python control flow from GPU values — fixed loop counts -- Per-expert Python loops OK (fixed `num_experts`, unrolled at capture time) -- Shared buffers OK (layers execute sequentially during capture and replay) +- No `.item()`, `.cpu()`, `.tolist()` +- No `torch.zeros/ones/full/empty/arange()` during capture +- No dynamic Python control flow from GPU values +- Per-expert Python loops OK (fixed `num_experts`) +- Shared buffers OK (layers sequential during capture and replay) ### EP Configuration (DeepSeek-V4-Pro on 8×B200) - 256 total experts, top_k=6 - EP=8 → 48 local experts per rank - `experts_start_idx` = rank × 32 -- `max_num_tokens` = 8192 (from `scheduler_config.max_num_batched_tokens`) +- `max_num_tokens` = 8192 - `max_chunks_per_expert` = ceil(8192 × 6 / (48 × 128)) = 8 --- +## Outstanding Issue: Garbage Model Output + +**Symptom:** Model generates 30 tokens of empty/invisible content (BOS or thinking token). Not meaningful text. + +**What works:** layertest gives 0.988 cosine with 3 experts, 8 tokens, top_k=8. + +**What doesn't:** vLLM with 48 experts, variable tokens, top_k=6 produces garbage. + +**Hypotheses to investigate:** +1. **Warmup gs from random data ≠ real activation magnitudes.** The warmup uses `torch.randn` (amax ~3) but real activations have amax ~8-10. The gs values would be wrong, causing quantization errors. +2. **Scale assembly with 48 experts × 8 chunks.** With max_chunks=8 and 48 experts, there are 384 swizzle blocks. The fixed-layout scatter with `clamped_local` may be dropping tokens that overflow the expert's max_rows section. +3. **`clamped_local = local_row.clamp(max=max_rows_per_expert - 1)`.** If an expert has more than `max_chunks*128` real tokens, overflow tokens all map to the same row, overwriting each other. This silently drops data. +4. **The `_needs_token_refill` path.** After GEMM JIT, `_token_indices` may get corrupted. The refill happens AFTER the first run, but the first run already used corrupted indices. + +--- + ## Test Files | File | Purpose | |------|---------| -| `tests/layertest.py` | Reference: moe_pipeline with dynamic gs, 3 experts, layer 0. Must pass (≥0.98 cosine). | -| `tests/cudagraph_test.py` | CuTeDSLMoERunner cudagraph capture + replay. Must pass. | +| `tests/layertest.py` | Reference vs runner, 3 experts. Must pass ≥0.98 cosine. | +| `tests/cudagraph_test.py` | Cudagraph capture + replay. Must pass. | +| `tests/test_runner_vs_pipeline.py` | Runner vs pipeline comparison. | +| `tests/test_scale_assembly.py` | Scale assembly comparison. | | `tests/test_warmup_gs.py` | Warmup gs computation. | -| `tests/test_runner_vs_pipeline.py` | Compare runner.run() vs moe_pipeline. | -| `tests/test_scale_assembly.py` | Compare cudagraph-safe vs reference scale assembly. | **Run order after any code change:** 1. `python3 tests/layertest.py` — must pass diff --git a/docker-compose.yml b/docker-compose.yml index b93990f7..947481c5 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -17,7 +17,8 @@ services: - --tensor-parallel-size=8 #- --enforce-eager - --compilation-config - - '{"cudagraph_mode": "FULL_DECODE_ONLY", "custom_ops": ["all"], "cudagraph_capture_sizes": [1, 2, 4, 8], "max_cudagraph_capture_size": 8}' + #- '{"cudagraph_mode": "NONE", "custom_ops": ["all"]}' + - '{"cudagraph_mode": "FULL_DECODE_ONLY", "custom_ops": ["all"], "cudagraph_capture_sizes": [1, 2, 4, 8], "max_cudagraph_capture_size": 8}' # This is what is runing right now #- '{"cudagraph_mode":"FULL_AND_PIECEWISE", "custom_ops":["all"]}' #- --moe-backend=deep_gemm_mega_moe - --tokenizer-mode=deepseek_v4 diff --git a/tests/debug_output.py b/tests/debug_output.py new file mode 100644 index 00000000..17cc0453 --- /dev/null +++ b/tests/debug_output.py @@ -0,0 +1,70 @@ +"""Debug: Compare runner output vs reference pipeline output. +Focus on whether the scale assembly + GEMM produces correct values.""" +import torch +import sys +sys.path.insert(0, '/root/nvfp4-megamoe-kernel/cutedsl') +sys.path.insert(0, '/root/nvfp4-megamoe-kernel/vllm') + +from cutedsl.reference.moe_pipeline import moe_pipeline +from vllm.nvfp4_cutedsl import CuTeDSLMoERunner + +torch.cuda.set_device(0) + +# Load real model weights for layer 0 +from cutedsl.weight_loader import load_layer_weights +weights = load_layer_weights(layer_idx=0, num_experts=3) + +# Run reference pipeline with dynamic gs +ref_out = moe_pipeline( + hidden_states=torch.randn(4, 256, dtype=torch.bfloat16, device='cuda'), + topk_weights=torch.ones(4, 2, dtype=torch.float32, device='cuda') / 2, + topk_ids=torch.tensor([[0,1],[0,1],[0,1],[0,1]], dtype=torch.int64, device='cuda'), + l1_fp4=weights['l1_fp4'], + l1_sf=weights['l1_sf'], + l1_gs=weights['l1_gs'], + l2_fp4=weights['l2_fp4'], + l2_sf=weights['l2_sf'], + l2_gs=weights['l2_gs'], + num_experts=3, + hidden_size=256, + intermediate_size=512, +) + +print(f"Reference output: amax={ref_out.amax().item():.4f} mean={ref_out.mean().item():.4f}") + +# Run runner with warmup gs +runner = CuTeDSLMoERunner( + num_experts=3, hidden_size=256, intermediate_size=512, + max_num_tokens=4, top_k=2, device='cuda' +) +# Set weights directly +runner.l1_fp4 = weights['l1_fp4'] +runner.l1_sf = weights['l1_sf'] +runner.l1_gs = weights['l1_gs'] +runner.l2_fp4 = weights['l2_fp4'] +runner.l2_sf = weights['l2_sf'] +runner.l2_gs = weights['l2_gs'] + +# Compute warmup gs +hs = torch.randn(4, 256, dtype=torch.bfloat16, device='cuda') +tw = torch.ones(4, 2, dtype=torch.float32, device='cuda') / 2 +ti = torch.tensor([[0,1],[0,1],[0,1],[0,1]], dtype=torch.int64, device='cuda') +runner.compute_activation_global_scales(hs, tw, ti) +print(f"Warmup gs: L1={runner._l1_activation_global_scale} L2={runner._l2_activation_global_scale}") + +# Run with same input as reference +runner_out = runner.run(hs, tw, ti) +print(f"Runner output: amax={runner_out.amax().item():.4f} mean={runner_out.mean().item():.4f}") + +# Cosine similarity +cos = torch.nn.functional.cosine_similarity(ref_out.flatten().unsqueeze(0), runner_out.flatten().unsqueeze(0)).item() +print(f"Cosine similarity: {cos:.6f}") + +# Check for NaN/Inf +print(f"Runner NaN: {torch.isnan(runner_out).any().item()} Inf: {torch.isinf(runner_out).any().item()}") +print(f"Ref NaN: {torch.isnan(ref_out).any().item()} Inf: {torch.isinf(ref_out).any().item()}") + +# Per-token comparison +for i in range(4): + cos_i = torch.nn.functional.cosine_similarity(ref_out[i].unsqueeze(0), runner_out[i].unsqueeze(0)).item() + print(f" Token {i}: cosine={cos_i:.6f} ref_max={ref_out[i].amax().item():.4f} run_max={runner_out[i].amax().item():.4f}")