# DeepSeek V4 Pro → NVFP4 Quantization

Full NVFP4 quantization of DeepSeek V4 Pro on a single B200 node (8× B200, 2.7TB RAM, 13TB NVMe).

## Pipeline

### Step 1: Dequantize FP8 → BF16

```bash
python3 scripts/dequant_fp8_to_bf16.py /root/nvidia-meeting/DeepSeek-V4-Pro-FP8 /root/nvidia-meeting/DeepSeek-V4-Pro-BF16
```

The original V4 weights use mixed precision (FP8 attention + FP4/E2M1 experts with per-tensor scales). We dequantize everything to pure BF16 so modelopt can run calibration without hitting broken FP8 kernel paths on Blackwell (DeepGEMM unsupported, Triton finegrained FP8 matmul shape mismatches).

This is not a blind upcast — it applies the actual scale factors:

```
W_bf16 = dequantize_fp4_weight(W_int, S)  # per-tensor scale dequant, not .to(bfloat16)
```

**We verified byte-exact correctness** by dequantizing a single expert and running a matmul against the official inference path:

```python
W_bf16 = dequantize_fp4_weight(W_int, S)
y_ours = W_bf16 @ x.bfloat16()
y_ref = official_expert_forward(W_int, S, x)
print((y_ours - y_ref).abs().max() / y_ref.abs().mean())
```

Results:

```
Max abs diff: 0.00000000
Mean abs diff: 0.00000000
Relative error: 0.000000
Matmul max diff: 0.00000000
```

Byte-exact. Zero drift from BF16 rounding noise — ruled out as a potential issue in the final quant.

### Step 2: Run ModelOpt NVFP4 Full Quantization

```bash
python3 scripts/model_opt_nvfp4_full.py
```

Runs NVIDIA's official ModelOpt PTQ pipeline (`hf_ptq.py`) with full `nvfp4` quantization (attention + experts + shared MLP). Output target: ~600GB.

**Config:**
- `--quant nvfp4` (full model, not experts-only)
- `--calib 128` — 128 calibration samples. The B200 node has 2.7TB RAM; the 3TB BF16 model doesn't fit in GPU VRAM (~1.4TB total), so it runs with `--use_seq_device_map` (CPU offload). 256 calibration samples OOMs. 128 is the max that fits.
- `--kv_cache_quant fp8_cast`
- `--use_seq_device_map` — sequential device mapping, loads model into CPU RAM, moves layers to GPU for forward passes
- `--gpu_max_mem_percentage 0.7` — VRAM headroom

**Calibration datasets:** `abisee/cnn_dailymail` + `nvidia/Nemotron-Post-Training-Dataset-v2` (gated — requires HF token). The script exports `HF_TOKEN` and `HUGGING_FACE_HUB_TOKEN`; the token must also be set via `hf auth login` on the node.

**Runtime:** Model loading takes ~53 minutes. Quantization + calibration takes several hours. Total expect 6-12 hours.

## Key Notes

- Use BF16 source — V4's mixed precision causes issues, FP8 source has kernel problems on Blackwell
- `--low_memory_mode` causes meta device errors with V4 — don't use
- modelopt has no explicit V4 support — relies on auto-detection of fused experts
- The `quant_module_patched.py` patch fixes `iter_weights_for_calibration()` for V4's `nn.ModuleList` expert quantizers — already applied in the venv

## Bugs Found (V4 + modelopt)

1. `QuantDeepseekV4Experts` AttributeError — patched `iter_weights_for_calibration()` for ModuleList quantizers
2. `--low_memory_mode` → meta device error
3. Missing `kernels` package for FP8 ops
4. `--calib` not `--calib_size`, `--quant` not `--qformat` (shell script arg names)