From 5122cadc945849671a6c80e970e6c48b435ad6da Mon Sep 17 00:00:00 2001 From: biondizzle Date: Tue, 19 May 2026 07:21:30 +0000 Subject: [PATCH] Update CURRENT_BUG.md: root cause found + fix committed --- CURRENT_BUG.md | 114 ++++++++++++++----------------------------------- 1 file changed, 33 insertions(+), 81 deletions(-) diff --git a/CURRENT_BUG.md b/CURRENT_BUG.md index b4acaaf9..90b5bbe4 100644 --- a/CURRENT_BUG.md +++ b/CURRENT_BUG.md @@ -1,92 +1,44 @@ -# CURRENT_BUG.md — Empty Output from vLLM +# CURRENT_BUG.md -**Status:** BLOCKED — model loads, serves requests, returns empty content -**Date:** 2026-05-19 +## Status: Fix committed, needs vLLM container rebuild + test -## Symptom +### Root Cause Found: Wrong Activation Global Scale -``` -curl -X POST http://127.0.0.1:8000/v1/chat/completions \ - -d '{"model":"/model","stream":true,"max_tokens":10,"temperature":0, - "messages":[{"role":"user","content":"The capital of France is"}]}' +**All CuTeDSL NVFP4 kernels are correct** (verified with standalone test, cosine 0.989-0.995 vs BF16 reference). The bug was in the vLLM integration, NOT our kernels. -→ Returns: content="" finish_reason="length" (generates 10 tokens of nothing → EOS) +The `CuTeDSLNvFp4LinearKernel.process_weights_after_loading` (in `vllm/kernels/linear/nvfp4/cutedsl.py`) was using the checkpoint's `input_global_scale_inv` as the activation global scale. This is a calibration-time value that doesn't match what `quantize_activation_nvfp4` expects at runtime, producing garbage output. + +### Fix + +Changed `cutedsl.py` to use warmup-based activation global scale computation (same as standalone test): +```python +# BEFORE (broken): +runner._activation_global_scale = input_global_scale_inv # wrong! + +# AFTER (fixed): +runner.compute_activation_global_scale(sample) # warmup-based, correct ``` -Model loads fine. CUDAGraph captures. Inference runs. But every token is EOS. +### Next Steps -## What's Been Tried +1. Rebuild vLLM container on B200 with this fix +2. Run `build_and_run.sh` +3. Test with curl chat completions +4. If still broken, also fix MoE runner warmup (currently using checkpoint input_scale mean) -| Attempt | Result | Why it failed | -|---------|--------|---------------| -| BF16 dequant all NVFP4 weights | Still garbage | Something else in vLLM pipeline is broken | -| Patch `process_weights_after_loading` | Overwritten by vLLM quant method | Timing issue — our fix runs, then vLLM's quant overwrites it | -| Forward pre-hooks | Never fire | torch.compile + model wrappers bypass them | -| Patch `utils.py` with `_post_quant_fix()` | 305 projections dequantized, still garbage | vLLM's pipeline has deeper issues beyond the linear layers | -| Replace `deepseek_v4_attention.py` with v0.21.0 version | Import errors (`breakable_cudagraph` missing) | Docker image uses v0.20.2rc1, not v0.21.0 stable | -| Replace with Docker image's original | Nuked all our existing patches | The file already had our BF16 wo_a work — dumb mistake | +### Standalone Test Results (test_full_layer_b200.py) -## Confirmed Working (standalone tests on B200) +``` +q_a_proj: cosine=0.994599 ✅ +kv_proj: cosine=0.994777 ✅ +q_b_proj: cosine=0.994834 ✅ +wo_b_proj: cosine=0.994768 ✅ +comp.kv_proj: cosine=0.994152 ✅ +comp.gate: cosine=0.994766 ✅ +shared_expert: cosine=0.989745 ✅ +``` -| Component | Test | Result | -|-----------|------|--------| -| MoE CuTeDSL kernel | `layertest.py` | ✅ cosine 0.988 | -| CUDAGraph | `cudagraph_test.py` | ✅ capture + replay | -| wo_a BMM (BF16) | `test_o_projection_b200.py` | ✅ z amax=0.8, no NaN | -| wo_b NVFP4 GEMM | `test_o_projection_b200.py` | ✅ cosine 0.996 | -| inv RoPE roundtrip | `test_o_projection.py` | ✅ BF16 precision | -| wo_a BMM vs einsum | `test_o_projection.py` | ✅ exact match | -| All NVFP4 weight dequant | standalone | ✅ Valid BF16 output | +### Remaining Issues -## The Core Problem - -**We don't control the execution path.** vLLM intercepts our weights and routes them through kernels we can't debug: - -1. **`FlashInferCutlassNvFp4LinearKernel`** — vLLM's built-in NVFP4 linear kernel. Same class of C++ CUTLASS FP4 bugs we hit with MoE. Used for ALL attention projections (fused_wqa_wkv, wq_b, wo_b) and shared experts (gate_up_proj, down_proj). Our `CuTeDSLNvFp4LinearKernel` is registered and selected by `init_nvfp4_linear_kernel`, but it's still going through vLLM's quantization layer which may interfere. - -2. **FP8 einsum for wo_a** — The attention forward (`deepseek_v4_attention.py`) does `fused_inv_rope_fp8_quant` + `deepseek_v4_fp8_einsum`. This expects `wo_a.weight_scale_inv` (FP8). Our checkpoint has wo_a as BF16. The `has_fp8_weights` check in our patched forward should handle this, but we need to verify it's actually running our path. - -3. **MHC (Hyper-connections)** — Pure PyTorch replacement. Should work. Not verified end-to-end. - -4. **CSA/HCA/SWA** — vLLM's Triton kernels for sparse attention. JIT compiles on first request (the warnings Mike saw). These are vLLM internals, not our code. Should work. Not verified. - -## What We Don't Know - -- **Is `CuTeDSLNvFp4LinearKernel` actually being used?** Logs say yes, but we haven't verified the output of a single attention projection inside the container. -- **Is the BF16 wo_a path actually running?** The `has_fp8_weights` check should detect our BF16 wo_a, but we haven't confirmed inside the container. -- **Is the activation global scale correct?** We fixed the double-inversion bug (`1.0 / input_global_scale_inv` → just `input_global_scale_inv`), but haven't tested in the container. -- **Are the MHC hyper-connections producing valid output?** Not tested. -- **Is the attention itself (FlashMLA sparse) working on B200?** Not tested outside vLLM. - -## The Decision: Our Own Path - -**Stop fighting vLLM's kernels.** Every time we patch around one, another breaks. The CuTeDSL kernel works (0.988 cosine). The CuTeDSL linear kernel works (0.996 cosine). But vLLM's pipeline keeps routing things through broken paths we can't control. - -**The plan:** Bypass vLLM's kernel selection entirely. Replace the model's forward methods to call our runners directly, not through vLLM's quantization layer. This means: - -1. **Attention projections** — Call `CuTeDSLNvfp4Linear` runners directly from the model forward, not through `ColumnParallelLinear`/`RowParallelLinear` which go through vLLM's quant dispatch. -2. **Shared experts** — Same. Call `CuTeDSLNvfp4Linear` directly from `DeepseekV4MoE.forward()`. -3. **MoE** — Already doing this (cutedsl_moe.py calls our runner directly). -4. **wo_a** — BF16 BMM, already in our patched attention forward. -5. **MHC** — Already our pure PyTorch replacement. - -The key insight: vLLM's `ModelOptNvFp4LinearMethod.process_weights_after_loading` is the bottleneck. It's the one that converts weights and routes to the kernel. We need to intercept BEFORE it, grab the raw weights, and store them in our runner format. Then the model forward calls our runner, not vLLM's quant method. - -## Test Infrastructure - -- `tests/` — CPU-only tests (run locally) -- `tests/venv/` — B200 venv with torch + CuTeDSL + safetensors -- `tests/test_o_projection_b200.py` — Loads real weights, tests O-projection (✅ PASS) -- `tests/test_o_projection.py` — CPU tests for inv RoPE + BMM (✅ PASS) - -**Rule: Test everything standalone on B200 BEFORE touching the container.** - -## Files That Have Our Patches (DO NOT NUKE) - -| File | What it has | DON'T | -|------|-------------|-------| -| `vllm/patches/deepseek_v4.py` | Model patch: BF16 wo_a, weight mapper, NVFP4 config | Don't replace with stock vLLM | -| `vllm/patches/deepseek_v4_attention.py` | BF16 wo_a path with `has_fp8_weights` check, `_apply_inv_rope_bf16`, BMM + all-gather, FP8 fallback | Don't replace with stock vLLM | -| `vllm/patches/layers/mhc.py` | Pure PyTorch MHC (replaces TileLang) | Don't replace | -| `vllm/kernels/linear/nvfp4/cutedsl.py` | `CuTeDSLNvFp4LinearKernel` — our NVFP4 linear kernel | Don't replace | -| `vllm/patches/fused_moe/experts/cutedsl_moe.py` | CuTeDSL MoE backend | Don't replace | +1. **MoE warmup**: `compute_activation_global_scales` is never called on the MoE runner. Currently uses checkpoint input_scale mean. Needs warmup too. +2. **Shared expert in vLLM**: Check if the vLLM shared expert path uses CuTeDSL or falls through to broken vLLM kernels.