deepseek-v4-quant/memory/2026-05-11.md

2026-05-11 — DeepSeek V4 NVFP4 vLLM Serving: Full End-to-End

🎉 SERVER RUNNING ON PORT 8000

The vLLM server successfully loads the NVFP4 model and serves API requests on 8× B200.

What We Fixed (Session Summary)

1. DeepGEMM sf.dim() Assertion (CRITICAL)

• Error: Assertion error layout.hpp:94: sf.dim() == num_groups + 2
• Cause: weight_scale_inv was 1D per-tensor scale. DeepGEMM expects 2D/3D block-scale tensor from transform_sf_into_required_layout.
• Fix: Use deepgemm_post_process_fp8_weight_block(wq, ws, quant_block_shape=(128,128), use_e8m0=True) to produce correct block-scale format. Store result in weight_scale_inv.
• Key insight: The attention runtime reads self.wo_a.weight_scale_inv as b_scale for the einsum. It MUST be the DeepGEMM-formatted block scale.

2. Block Scale dtype

• Error: Expected float32 or float8_e8m0fnu, got float8_e4m3fn
• Fix: Create block scale as dtype=torch.float32

3. Missing deepgemm_post_process args

• Error: missing 2 required positional arguments: 'quant_block_shape' and 'use_e8m0'
• Fix: Pass quant_block_shape=(128, 128) and use_e8m0=True

4. Compressor Indexer Shape Mismatch (CRITICAL)

• Error: split_with_sizes expects 2048, got split_sizes=[256, 256]
• Cause: _reconstruct_compressor_weight used wrong checkpoint prefix for indexer. Main compressor keys: compressor.kv_proj.*. Indexer keys: compressor.indexer.kv_proj.*. Loading main compressor weight into indexer's fused_wkv_wgate = 4× size mismatch.
• Fix: Added sub_path parameter, pass ".indexer" for indexer compressors.

5. All-Ones Block Scale → Garbage Output (CRITICAL)

• Symptom: Server runs, outputs tokens, but text is incoherent gibberish (repeating "Palm", "sulfuric", "东海")
• Cause: Block scale was torch.ones(...) = 1.0. DeepGEMM divides by block scale at runtime, so output was divided by 1.0 instead of actual fp8_scale.
• Fix: torch.full(..., fp8_scale.item()) — fill each block with the per-tensor FP8 scale value.

Conversion Summary

• 61 NVFP4→FP8 (wo_a attention, DeepGEMM block-scale BMM einsum)
• 0 BF16→FP8
• 305 attn/shared→BF16 (UnquantizedLinearMethod)
• 91 compressor→BF16 (reconstructed from separate NVFP4 kv_proj+gate_proj)
• MoE experts: stay NVFP4 (FLASHINFER_TRTLLM FusedMoE backend)

Architecture Map

wo_a → FP8 + DeepGEMM block scale (weight_scale_inv = dg_ws)
fused_wqa_wkv, wo_b → BF16 (UnquantizedLinearMethod)
compressor.fused_wkv_wgate → BF16 (read from checkpoint, unpack, dequant, cat)
shared_expert → FP8 (Fp8LinearMethod, DeepGEMM)
MoE w13/w2 → NVFP4 (FusedMoE, FLASHINFER_TRTLLM)

Key Code Locations

• Patch: /root/nvidia-meeting/deepseek-v4-quant/patches/deepseek_v4.py
• Runtime attention: deepseek_v4_attention.py:319 — reads wo_a.weight_scale_inv
• Runtime einsum: deepseek_v4_fp8_einsum → DeepGEMM fp8_einsum
• DeepGEMM scale format: deepgemm_post_process_fp8_weight_block in fp8_utils.py
• Compressor forward: deepseek_compressor.py:281 — kv, score = kv_score.split(...)

Outstanding Issues

• Output quality: Still producing garbled text after block-scale fix. Need to verify the latest fix (fp8_scale in block scale) produces coherent output.
• Possible causes if still garbled: subtle dequant bug, sign handling in E2M1 unpack, wrong scale ordering