biondizzle/deepseek-v4-quant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update README: document full pipeline, BF16 verification, calib 128 constraint

Browse Source
This commit is contained in:
biondizzle
2026-05-08 17:17:48 +00:00
parent f1d21900ea
commit a2370006f7
1 changed files with 50 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

72
README.md
View File

@@ -1,40 +1,68 @@
# DeepSeek V4 Pro → NVFP4 Quantization
Full NVFP4 quantization of DeepSeek V4 Pro using NVIDIA's Model Optimizer.
Full NVFP4 quantization of DeepSeek V4 Pro on a single B200 node (8× B200, 2.7TB RAM, 13TB NVMe).
## Strategy
## Pipeline
1. **Dequantize** the original mixed-precision FP8 weights to pure BF16 (`scripts/dequant_fp8_to_bf16.py`)
2. **Full quantize** BF16 → NVFP4 using NVIDIA's official ModelOpt PTQ pipeline (`scripts/model_opt_nvfp4_full.py`)
### Step 1: Dequantize FP8 → BF16
Full model quantization (attention + experts + shared MLP) to NVFP4. Target output: ~600GB.
```bash
python3 scripts/dequant_fp8_to_bf16.py /root/nvidia-meeting/DeepSeek-V4-Pro-FP8 /root/nvidia-meeting/DeepSeek-V4-Pro-BF16
```
## Scripts
The original V4 weights use mixed precision (FP8 attention + FP4/E2M1 experts with per-tensor scales). We dequantize everything to pure BF16 so modelopt can run calibration without hitting broken FP8 kernel paths on Blackwell (DeepGEMM unsupported, Triton finegrained FP8 matmul shape mismatches).
| File | Purpose |
| --- | --- |
| `scripts/dequant_fp8_to_bf16.py` | Dequant FP8 source → pure BF16 (resumable, shard-level) |
| `scripts/upcast_to_bf16.py` | Alternative: upcast mixed-precision to BF16 |
| `scripts/model_opt_nvfp4_full.py` | Run ModelOpt NVFP4 full quantization (calib 128) |
| `patches/quant_module_patched.py` | Patch for modelopt V4 experts ModuleList bug |
| `patches/patch_finegrained_fp8_blackwell.py` | Blackwell FP8 kernel patch |
| `check-ttl.sh` | B200 node TTL watchdog |
This is not a blind upcast — it applies the actual scale factors:
## B200 Node
```
W_bf16 = dequantize_fp4_weight(W_int, S) # per-tensor scale dequant, not .to(bfloat16)
```
- 8× B200, 2.7TB RAM, 13TB NVMe
- See `.env` for access details
**We verified byte-exact correctness** by dequantizing a single expert and running a matmul against the official inference path:
```python
W_bf16 = dequantize_fp4_weight(W_int, S)
y_ours = W_bf16 @ x.bfloat16()
y_ref = official_expert_forward(W_int, S, x)
print((y_ours - y_ref).abs().max() / y_ref.abs().mean())
```
Results:
```
Max abs diff: 0.00000000
Mean abs diff: 0.00000000
Relative error: 0.000000
Matmul max diff: 0.00000000
```
Byte-exact. Zero drift from BF16 rounding noise — ruled out as a potential issue in the final quant.
### Step 2: Run ModelOpt NVFP4 Full Quantization
```bash
python3 scripts/model_opt_nvfp4_full.py
```
Runs NVIDIA's official ModelOpt PTQ pipeline (`hf_ptq.py`) with full `nvfp4` quantization (attention + experts + shared MLP). Output target: ~600GB.
**Config:**
- `--quant nvfp4` (full model, not experts-only)
- `--calib 128` — 128 calibration samples. The B200 node has 2.7TB RAM; the 3TB BF16 model doesn't fit in GPU VRAM (~1.4TB total), so it runs with `--use_seq_device_map` (CPU offload). 256 calibration samples OOMs. 128 is the max that fits.
- `--kv_cache_quant fp8_cast`
- `--use_seq_device_map` — sequential device mapping, loads model into CPU RAM, moves layers to GPU for forward passes
- `--gpu_max_mem_percentage 0.7` — VRAM headroom
**Calibration datasets:** `abisee/cnn_dailymail` + `nvidia/Nemotron-Post-Training-Dataset-v2` (gated — requires HF token). The script exports `HF_TOKEN` and `HUGGING_FACE_HUB_TOKEN`; the token must also be set via `hf auth login` on the node.
**Runtime:** Model loading takes ~53 minutes. Quantization + calibration takes several hours. Total expect 6-12 hours.
## Key Notes
- **Calib size: 128** (256 OOMs on 2.8TB RAM with 3TB BF16 model)
- **Full quant (`nvfp4`)**, not experts-only
- Use BF16 source — V4's mixed precision causes issues, FP8 source has kernel problems on Blackwell
- `--use_seq_device_map` required (model doesn't fit in GPU VRAM alone)
- `--gpu_max_mem_percentage 0.7` for VRAM headroom
- `--low_memory_mode` causes meta device errors with V4 — don't use
- modelopt has no explicit V4 support — relies on auto-detection of fused experts
- Calibration dataset `nvidia/Nemotron-Post-Training-Dataset-v2` is gated — requires HF token
- The `quant_module_patched.py` patch fixes `iter_weights_for_calibration()` for V4's `nn.ModuleList` expert quantizers — already applied in the venv
## Bugs Found (V4 + modelopt)