biondizzle
02b8ea536f
Server running on B200 port 8000 with full NVFP4→vLLM bridge. All critical bugs fixed: DeepGEMM scale format, compressor shapes, block scale values.
59 lines
3.4 KiB
Markdown
59 lines
3.4 KiB
Markdown
# 2026-05-11 — DeepSeek V4 NVFP4 vLLM Serving: Full End-to-End
|
||
|
||
## 🎉 SERVER RUNNING ON PORT 8000
|
||
|
||
The vLLM server successfully loads the NVFP4 model and serves API requests on 8× B200.
|
||
|
||
### What We Fixed (Session Summary)
|
||
|
||
#### 1. DeepGEMM `sf.dim()` Assertion (CRITICAL)
|
||
- **Error**: `Assertion error layout.hpp:94: sf.dim() == num_groups + 2`
|
||
- **Cause**: `weight_scale_inv` was 1D per-tensor scale. DeepGEMM expects 2D/3D block-scale tensor from `transform_sf_into_required_layout`.
|
||
- **Fix**: Use `deepgemm_post_process_fp8_weight_block(wq, ws, quant_block_shape=(128,128), use_e8m0=True)` to produce correct block-scale format. Store result in `weight_scale_inv`.
|
||
- **Key insight**: The attention runtime reads `self.wo_a.weight_scale_inv` as `b_scale` for the einsum. It MUST be the DeepGEMM-formatted block scale.
|
||
|
||
#### 2. Block Scale dtype
|
||
- **Error**: `Expected float32 or float8_e8m0fnu, got float8_e4m3fn`
|
||
- **Fix**: Create block scale as `dtype=torch.float32`
|
||
|
||
#### 3. Missing `deepgemm_post_process` args
|
||
- **Error**: `missing 2 required positional arguments: 'quant_block_shape' and 'use_e8m0'`
|
||
- **Fix**: Pass `quant_block_shape=(128, 128)` and `use_e8m0=True`
|
||
|
||
#### 4. Compressor Indexer Shape Mismatch (CRITICAL)
|
||
- **Error**: `split_with_sizes expects 2048, got split_sizes=[256, 256]`
|
||
- **Cause**: `_reconstruct_compressor_weight` used wrong checkpoint prefix for indexer. Main compressor keys: `compressor.kv_proj.*`. Indexer keys: `compressor.indexer.kv_proj.*`. Loading main compressor weight into indexer's fused_wkv_wgate = 4× size mismatch.
|
||
- **Fix**: Added `sub_path` parameter, pass `".indexer"` for indexer compressors.
|
||
|
||
#### 5. All-Ones Block Scale → Garbage Output (CRITICAL)
|
||
- **Symptom**: Server runs, outputs tokens, but text is incoherent gibberish (repeating "Palm", "sulfuric", "东海")
|
||
- **Cause**: Block scale was `torch.ones(...)` = 1.0. DeepGEMM divides by block scale at runtime, so output was divided by 1.0 instead of actual fp8_scale.
|
||
- **Fix**: `torch.full(..., fp8_scale.item())` — fill each block with the per-tensor FP8 scale value.
|
||
|
||
### Conversion Summary
|
||
- 61 NVFP4→FP8 (wo_a attention, DeepGEMM block-scale BMM einsum)
|
||
- 0 BF16→FP8
|
||
- 305 attn/shared→BF16 (UnquantizedLinearMethod)
|
||
- 91 compressor→BF16 (reconstructed from separate NVFP4 kv_proj+gate_proj)
|
||
- MoE experts: stay NVFP4 (FLASHINFER_TRTLLM FusedMoE backend)
|
||
|
||
### Architecture Map
|
||
```
|
||
wo_a → FP8 + DeepGEMM block scale (weight_scale_inv = dg_ws)
|
||
fused_wqa_wkv, wo_b → BF16 (UnquantizedLinearMethod)
|
||
compressor.fused_wkv_wgate → BF16 (read from checkpoint, unpack, dequant, cat)
|
||
shared_expert → FP8 (Fp8LinearMethod, DeepGEMM)
|
||
MoE w13/w2 → NVFP4 (FusedMoE, FLASHINFER_TRTLLM)
|
||
```
|
||
|
||
### Key Code Locations
|
||
- Patch: `/root/nvidia-meeting/deepseek-v4-quant/patches/deepseek_v4.py`
|
||
- Runtime attention: `deepseek_v4_attention.py:319` — reads `wo_a.weight_scale_inv`
|
||
- Runtime einsum: `deepseek_v4_fp8_einsum` → DeepGEMM `fp8_einsum`
|
||
- DeepGEMM scale format: `deepgemm_post_process_fp8_weight_block` in `fp8_utils.py`
|
||
- Compressor forward: `deepseek_compressor.py:281` — `kv, score = kv_score.split(...)`
|
||
|
||
### Outstanding Issues
|
||
- **Output quality**: Still producing garbled text after block-scale fix. Need to verify the latest fix (fp8_scale in block scale) produces coherent output.
|
||
- Possible causes if still garbled: subtle dequant bug, sign handling in E2M1 unpack, wrong scale ordering
|