deepseek-v4-quant/memory/2026-05-08.md

2026-05-08 DeepSeek V4 Pro NVFP4 Quantization

Status: Blocked on mixed-precision model's deeply non-standard weight format

The mixed-precision DeepSeek-V4-Pro model has three levels of compression:

• Attention layers: FP8 (float8_e4m3fn + float8_e8m0fnu scales) — our dequant_fp8_to_bf16.py handles these ✅
• Routed experts: INT8 at half dimensions — e.g. w1: int8 [3072, 3584] instead of expected [3072, 7168], with separate scale factors. Our dequant script does NOT handle these ❌
• Shared experts: FP8 (float8_e4m3fn) — also not dequantized ❌

Result: from_pretrained sees shape mismatches on expert weights because the INT8 expert tensors are at half the expected dimensions. ignore_mismatched_sizes won't help — it'd reinitialize with random weights.

What needs to happen

A complete dequantization script that handles ALL compressed tensor types:

1. FP8 attention weights (weight × block_scale → BF16) — already working
2. INT8 routed expert weights (weight × scale → BF16, expand dimensions back) — need to find scale tensors
3. FP8 shared expert weights — same as attention dequant Then produce a truly pure-BF16 model with standard dimensions.

Attempt history on B200 node (45.76.247.107)

1. Custom quantizer (Path A): Output 840GB, wrong format — rejected
2. FP8 model + modelopt: DeepGEMM crashes on Blackwell (SM100 unsupported), Triton finegrained FP8 matmul has K mismatch
3. Naive upcast (upcast_to_bf16.py): Just cast FP8 scales to BF16 without dequantizing — broken, FP8Linear.forward still tries FP8 kernel path
4. Proper dequant (dequant_fp8_to_bf16.py): Handles FP8 attention ✅, but missed INT8 experts and FP8 shared experts — shape mismatch on load
5. nvfp4_experts_only on mixed-precision: Crashes in calibration forward pass (FP8Linear kernel issues)
6. nvfp4_experts_only on dequantized BF16: OOM killed during quantizer setup (94% RAM with seq_device_map)
7. nvfp4_experts_only, 128 calib, gpu_max_mem_percentage 0.9: Shape mismatch — INT8 expert weights at half dimensions

Repo

• Branch: modelopt-nvfp4 on https://sweetapi.com/biondizzle/deepseek-v4-quant.git
• Auth: user biondizzle, password Gold1234@x
• Scripts: scripts/dequant_fp8_to_bf16.py, scripts/upcast_to_bf16.py, scripts/model_opt_nvfp4_experts_only.py
• Patches: patches/patch_finegrained_fp8_blackwell.py, patches/quant_module_patched.py

Key learnings

• Mixed-precision DeepSeek-V4-Pro is NOT just "BF16 with some FP8" — it has INT8 expert weights at half dimensions too
• DeepGEMM only supports SM90 (Hopper), not SM100 (Blackwell)
• FP8Linear.forward() has an escape hatch: if self.weight.element_size() > 1: return F.linear(...) — if weight is BF16, it skips the FP8 kernel
• --use_seq_device_map causes OOM on 2.8TB RAM with 782GB model + quantizer overhead
• --gpu_max_mem_percentage 0.9 is the right approach — fits ~1.3TB on 8×B200 GPUs
• Calibration size (256 vs 128) only affects scale quality, not model format or inference

Mike's preferences

• Name output dirs with strategy: DeepSeek-V4-Pro_NVFP4-<strategy>_kv_fp8_cast
• One script per strategy, copy and tweak for each
• Track everything in git, don't lose working code
• Target output: ~600GB NVFP4

01:37 UTC - Complete dequant script updated, strategy pivot to full NVFP4

Script update: dequant_fp8_to_bf16.py now handles ALL compressed types:

1. FP8 weights (attention + shared experts): block-wise (128×128) dequant via float8_e4m3fn × float8_e8m0fnu scale → BF16
2. FP4 (E2M1) expert weights: Unpack 2 FP4 values per int8 byte (lower nibble first, upper second), E2M1 LUT lookup × E8M0 scale, per-row 32-column block scaling (MXFP4 microscaling), output dimensions 2× stored dimensions → BF16
3. Scale tensors removed after dequantization
4. Progress output per shard with ETA

Mike's directive: strategy change

• Drop nvfp4_experts_only — go full nvfp4 quantization on the properly dequantized BF16 model
• Need to create scripts/model_opt_nvfp4_full.py (copy from experts_only, change --quantization from nvfp4_experts_only to nvfp4)
• Steps: (1) Run complete dequant to get true BF16 model, (2) Run full nvfp4 quantization on it

FP4 (E2M1) expert weight format confirmed (was incorrectly assumed INT4)

• w1: int8 [3072, 3584] with scale float8_e8m0fnu [3072, 224] → output [3072, 7168] (224 * 32 = 7168)
• w2: int8 [7168, 1536] with scale float8_e8m0fnu [7168, 96] → output [7168, 3072] (96 * 32 = 3072)
• w3: int8 [3072, 3584] with scale float8_e8m0fnu [3072, 224] → output [3072, 7168]
• Proof: nibble index 0 (+0.0) vs index 8 (-0.0) ratio = 0.996. INT4 would have -8 at index 8 (rare). FP4 has -0.0 at index 8 (same as +0.0).

Previous BF16 model is WRONG

• The /root/nvidia-meeting/DeepSeek-V4-Pro-BF16 (782GB) only dequantized FP8 attention weights
• FP4 expert weights and FP8 shared experts were left compressed → shape mismatches on load
• Must re-run complete dequant, output will likely be larger (~1.1-1.2TB?)