deepseek-v4-quant/memory/2026-05-10.md

2026-05-10

DeepSeek V4 Pro NVFP4 — vLLM Serving Debug Session

• Quantization completed successfully (Run 11, 881GB NVFP4)
• Spent the day debugging vLLM serving of the modelopt NVFP4 checkpoint
• Key finding: modelopt and vllm were never integrated for NVFP4 on DeepSeek V4
• NVIDIA themselves haven't gotten this far — we're in uncharted territory

What we fixed:

• Expert weight name mapping (gate_proj→w1, up_proj→w3, down_proj→w2)
• mlp→ffn module naming
• Attention: self_attn→attn.mla_attn, kv_proj→wkv, etc.
• Compressor: kv_proj→wkv, gate_proj→wgate
• kv_norm moved from compressor to attention level
• Class attribute patching (hf_to_vllm_mapper)
• Source file patching (workers are separate processes)
• E2M1 FP4→BF16 unpacking for stacked attention params
• Skip patterns for NVFP4 scale tensors on MergedColumnParallelLinear

What we abandoned:

• mega_moe: No NVFP4 kernel exists, format mismatch (16-col vs 32-col blocks)
• Runtime monkey-patching: Workers don't inherit patches

Open issues (stop point):

1. MergedColumnParallelLinear + NVFP4 incompatibility — ModelOptNvFp4Config only handles Linear, not MergedColumn. Weight param is bf16 (should be uint8), no weight_scale registered for stacked params
2. Unknown params from modelopt (compressor.position_bias) crash loading
3. Current approach (unpack uint8→bf16, skip scales) loses calibration-optimized scales for attention weights

Repo state:

• All code/patches/docker-compose synced and committed on modelopt-nvfp4 branch
• README fully updated with vLLM serving run history, open issues, bug list
• B200 node at 45.76.247.107, weights at /root/nvidia-meeting/DeepSeek-V4-Pro-NVFP4

2026-05-10

DeepSeek V4 Pro NVFP4 — vLLM Serving Debug Session

• Quantization completed successfully (Run 11, 881GB NVFP4)
• Spent the day debugging vLLM serving of the modelopt NVFP4 checkpoint
• Key finding: modelopt and vllm were never integrated for NVFP4 on DeepSeek V4
• NVIDIA themselves haven't gotten this far — we're in uncharted territory

What we fixed:

• Expert weight name mapping (gate_proj→w1, up_proj→w3, down_proj→w2)
• mlp→ffn module naming
• Attention: self_attn→attn.mla_attn, kv_proj→wkv, etc.
• Compressor: kv_proj→wkv, gate_proj→wgate
• kv_norm moved from compressor to attention level
• Class attribute patching (hf_to_vllm_mapper)
• Source file patching (workers are separate processes)
• E2M1 FP4→BF16 unpacking for stacked attention params
• Skip patterns for NVFP4 scale tensors on MergedColumnParallelLinear

What we abandoned:

• mega_moe: No NVFP4 kernel exists, format mismatch (16-col vs 32-col blocks)
• Runtime monkey-patching: Workers don't inherit patches

Open issues (stop point):

1. MergedColumnParallelLinear + NVFP4 incompatibility — ModelOptNvFp4Config only handles Linear, not MergedColumn. Weight param is bf16 (should be uint8), no weight_scale registered for stacked params
2. Unknown params from modelopt (compressor.position_bias) crash loading
3. Current approach (unpack uint8→bf16, skip scales) loses calibration-optimized scales for attention weights

Repo state:

• All code/patches/docker-compose synced and committed on modelopt-nvfp4 branch
• README fully updated with vLLM serving run history, open issues, bug list
• B200 node at 45.76.247.107, weights at /root/nvidia-meeting/DeepSeek-V4-Pro-NVFP4

---

vLLM NVFP4 Serving — Second Session (16:28–19:35 UTC)

Mike gave autonomous work instructions. Key directive: use weights AS-IS (NVFP4), do NOT convert to MXFP4. Try FusedMoE first, then if stuck, build a mega_moe NVFP4 kernel from scratch.

Major breakthroughs (S11→S14 progress):

Key insight: vLLM attention forward bypasses quant_method, uses deepseek_v4_fp8_einsum directly

• The attention code reads self.wo_a.weight (expects fp8) and self.wo_a.weight_scale_inv directly
• NVFP4 uint8 weights are incompatible with this FP8 kernel
• Solution: NVFP4→bf16→FP8 dequantize/requant at load time for attention layers

S12 fixes applied (weight loading now succeeds to 94%):

1. Substr mapping fix: Removed .mla_attn. prefix from attention projections. The model has fused_wqa_wkv, wq_b, wo_a, wo_b at attn.* level, not attn.mla_attn.*. The stacking code then correctly maps attn.wq_a → attn.fused_wqa_wkv.
2. Skip patterns fix: Only skip compressor scale tensors (compressor uses UnquantizedLinearMethod with quant_config=None). Attention and shared expert scales now correctly load via stacking logic.
3. Suffix mapping fix: Removed "head.weight": "lm_head.weight" which caused lm_head.weight → lm_lm_head.weight doubling.
4. Resilient loading: Unknown params (e.g., compressor.position_bias) silently skipped.

S13 — Weight loading SUCCESS (32 seconds!)

• All 95 safetensors loaded without KeyError
• New error: MergedColumnParallelLinear has no weight_scale_inv (FP8 attribute)

S13.5 — o_a_proj discovery:

• modelopt did NOT quantize o_a_proj — it's bf16 in the checkpoint (no scales)
• But vLLM creates wo_a with NVFP4 quant (uint8 weight + scales)
• Fix: convert bf16→FP8 directly at load time, set weight_scale_inv

S14 — NVFP4→FP8 post-load conversion approach:

• Added _convert_nvfp4_attention_to_fp8() and _convert_nvfp4_module_to_fp8() methods to DeepseekV4Model
• Converts all uint8 NVFP4 attention weights (fused_wqa_wkv, wq_b, wo_a, wo_b, gate_up_proj) to FP8 at load time
• Steps: unpack E2M1 FP4→bf16, dequantize with block/global scales, requantize to FP8 e4m3, set weight_scale_inv
• For o_a_proj (bf16, no scales): convert directly bf16→FP8
• For compressor fused_wkv_wgate: stays bf16 (UnquantizedLinearMethod)
• For MoE experts: handled natively by ModelOptNvFp4FusedMoE

Bug found: E2M1 LUT indexing off-by-one

• FP4 4-bit values are 0-15 (bit 3 = sign, bits 0-2 = magnitude)
• LUT has 8 entries (magnitudes 0-7), but code was indexing with full 4-bit value (0-15) → CUDA assert
• Fix: mask with & 0x07 for magnitude index, apply sign from bit 3 separately

Bug found: method placement inside Python class

• _convert_nvfp4_attention_to_fp8 was being placed at top level (0 indent) instead of inside DeepseekV4Model
• The class actually ends at finalize_mega_moe_weights() (line ~1600), followed by top-level hc_head function
• Had to insert methods BEFORE the @torch.compile decorator that marks the class boundary

Bug found: logger not available in method

• logger.info_once() isn't accessible inside the conversion methods
• Replaced with print(f"...") for now

Current status (as of 19:35 UTC):

• Weight loading + NVFP4→FP8 conversion code is in place
• Last test was running (loading 880GB checkpoint)
• E2M1 sign handling fix applied but NOT YET TESTED
• Need to fix logger → print issue
• After load succeeds: FusedMoE expert weight handling needs verification
• If FusedMoE fails: need to build mega_moe NVFP4 kernel

Key files on B200 node:

• Patch: /root/nvidia-meeting/deepseek-v4-quant/patches/deepseek_v4.py
• Docker: docker compose up -d (TP=8, no mega_moe, FLASHINFER_TRTLLM attn)
• Weights: /root/nvidia-meeting/DeepSeek-V4-Pro-NVFP4/

Architecture decisions:

• NVFP4→FP8 for attention/shared_experts (requant, preserves FP8 kernel compat)
• BF16 for compressor (UnquantizedLinearMethod, no quant_config)
• Native NVFP4 for MoE experts (ModelOptNvFp4FusedMoE handles it)
• UnquantizedLinearMethod as no-op quant_method (attention forward bypasses it anyway)

---

vLLM NVFP4 Serving — Third Session (23:05+ UTC)

Current state of the B200 node:

• Docker container ran 27 min ago and crashed with BFloat16 != Float8_e4m3fn
• Uncommitted changes to patches/deepseek_v4.py (the _convert_nvfp4_post_load methods)
• Repo on modelopt-nvfp4 branch, last commit db16be8

Crash analysis (S15 — BFloat16 != Float8_e4m3fn):

Weight loading succeeds (95/95, 330s). Post-load conversion reports: 122 layers → FP8, 183 → BF16. MoE setup runs. Crash during profile_run/_dummy_run.

Root cause: _convert_nvfp4_post_load converts fused_wqa_wkv to FP8 and sets quant_method = UnquantizedLinearMethod(). The attention forward calls self.fused_wqa_wkv(hidden_states) which goes through UnquantizedLinearMethod.forward() → F.linear(bf16_input, fp8_weight) → dtype mismatch.

Key insight about the attention forward paths:

• wo_a: Attention code reads self.wo_a.weight and self.wo_a.weight_scale_inv DIRECTLY, passes to deepseek_v4_fp8_einsum. This bypasses quant_method. FP8 conversion works here.
• fused_wqa_wkv: Called via self.fused_wqa_wkv(hidden_states) → MergedColumnParallelLinear.forward() → quant_method.forward(). Cannot be FP8 with UnquantizedLinearMethod.
• wq_b, wo_b: Called via normal .forward(). Need BF16 + UnquantizedLinearMethod.
• compressor.fused_wkv_wgate: Called via torch.mm(hidden_states, weight.T, out_dtype=torch.float32) DIRECTLY. Needs BF16 weight — currently uint8 (not in any conversion set!).

Critical finding from safetensors: o_a_proj.weight is BF16 (modelopt did NOT quantize it). So wo_a weight is already BF16, not NVFP4. The post-load conversion code's dtype != uint8 check skips it. This means wo_a.weight stays BF16 and wo_a.weight_scale_inv is never created. When deepseek_v4_fp8_einsum tries to read it as FP8 → crash.

Wait, but the log says 122 → FP8. 61 layers × 2 (fused_wqa_wkv + wo_a) = 122. If wo_a.weight is BF16 and gets skipped, only 61 → FP8. The 122 count means wo_a IS being converted somehow. Hypothesis: ModelOptNvFp4LinearMethod.create_weights() creates wo_a.weight as uint8. When the BF16 checkpoint data is loaded into the uint8 param, the weight_loader might be casting it, or the param might be updated to BF16. Need to verify.

Unfixed bugs from S14 (still present):

1. E2M1 sign handling fix applied but NOT TESTED
2. logger → print issue in conversion methods

Compressor fused_wkv_wgate — PENDING CRASH:

• NOT in any conversion set (fp8_proj_names, bf16_proj_names, bf16_shared_names)
• Weight is uint8 after loading (NVFP4 packed)
• Forward uses torch.mm(hidden_states, weight.T, out_dtype=torch.float32) directly
• uint8 × BF16 would crash with a different error than the current one
• Needs BF16 dequantization in post-load conversion

Checkpoint key format (verified from safetensors):

• model.layers.0.self_attn.q_a_proj.weight — uint8
• model.layers.0.self_attn.q_a_proj.weight_scale — float8_e4m3fn (block scale)
• model.layers.0.self_attn.q_a_proj.weight_scale_2 — float32 (per-tensor)
• model.layers.0.self_attn.q_a_proj.input_scale — float32
• model.layers.0.self_attn.o_a_proj.weight — BF16 (NOT quantized by modelopt)
• model.layers.0.self_attn.o_b_proj.weight — uint8
• model.layers.0.self_attn.kv_proj.weight — uint8
• model.layers.0.self_attn.compressor.kv_proj.weight — uint8
• model.layers.0.self_attn.compressor.gate_proj.weight — uint8
• model.layers.0.self_attn.compressor.position_bias — BF16 (unknown param, skipped)
• Expert scales: .weight_scale, .weight_scale_2, .input_scale (NOT .scale)

FusedMoE NVFP4 status:

• ModelOptNvFp4FusedMoE creates proper uint8 weights + float8_e4m3fn block scales + float32 per-tensor/input scales
• process_weights_after_loading calls convert_to_nvfp4_moe_kernel_format then make_nvfp4_moe_kernel
• Uses cutlass_fp4_gemm via nvfp4 backend
• Warning: w1_weight_scale_2 must match w3_weight_scale_2 — modelopt gives different global scales to w1 and w3, but FusedMoE uses a single w13_weight_scale_2 (takes w1's). Minor accuracy impact.
• expert_dtype: fp4 in config — causes weight mapper to use .scale → .weight_scale regex, but checkpoint already uses .weight_scale directly, so regex is a no-op. Correct behavior.
• scale_fmt: "ue8m0" in config — used by attention FP8 einsum. Correct for NVFP4.

Config verification:

• compress_ratios ✅ (copied from BF16 source)
• scale_fmt: "ue8m0" ✅ (added by us)
• rope_parameters ✅ (flattened)
• expert_dtype: fp4 ✅ (original, correct for weight mapper regex)