diff --git a/README.md b/README.md index 7406cfd..98a3451 100644 --- a/README.md +++ b/README.md @@ -1,40 +1,68 @@ # DeepSeek V4 Pro → NVFP4 Quantization -Full NVFP4 quantization of DeepSeek V4 Pro using NVIDIA's Model Optimizer. +Full NVFP4 quantization of DeepSeek V4 Pro on a single B200 node (8× B200, 2.7TB RAM, 13TB NVMe). -## Strategy +## Pipeline -1. **Dequantize** the original mixed-precision FP8 weights to pure BF16 (`scripts/dequant_fp8_to_bf16.py`) -2. **Full quantize** BF16 → NVFP4 using NVIDIA's official ModelOpt PTQ pipeline (`scripts/model_opt_nvfp4_full.py`) +### Step 1: Dequantize FP8 → BF16 -Full model quantization (attention + experts + shared MLP) to NVFP4. Target output: ~600GB. +```bash +python3 scripts/dequant_fp8_to_bf16.py /root/nvidia-meeting/DeepSeek-V4-Pro-FP8 /root/nvidia-meeting/DeepSeek-V4-Pro-BF16 +``` -## Scripts +The original V4 weights use mixed precision (FP8 attention + FP4/E2M1 experts with per-tensor scales). We dequantize everything to pure BF16 so modelopt can run calibration without hitting broken FP8 kernel paths on Blackwell (DeepGEMM unsupported, Triton finegrained FP8 matmul shape mismatches). -| File | Purpose | -| --- | --- | -| `scripts/dequant_fp8_to_bf16.py` | Dequant FP8 source → pure BF16 (resumable, shard-level) | -| `scripts/upcast_to_bf16.py` | Alternative: upcast mixed-precision to BF16 | -| `scripts/model_opt_nvfp4_full.py` | Run ModelOpt NVFP4 full quantization (calib 128) | -| `patches/quant_module_patched.py` | Patch for modelopt V4 experts ModuleList bug | -| `patches/patch_finegrained_fp8_blackwell.py` | Blackwell FP8 kernel patch | -| `check-ttl.sh` | B200 node TTL watchdog | +This is not a blind upcast — it applies the actual scale factors: -## B200 Node +``` +W_bf16 = dequantize_fp4_weight(W_int, S) # per-tensor scale dequant, not .to(bfloat16) +``` -- 8× B200, 2.7TB RAM, 13TB NVMe -- See `.env` for access details +**We verified byte-exact correctness** by dequantizing a single expert and running a matmul against the official inference path: + +```python +W_bf16 = dequantize_fp4_weight(W_int, S) +y_ours = W_bf16 @ x.bfloat16() +y_ref = official_expert_forward(W_int, S, x) +print((y_ours - y_ref).abs().max() / y_ref.abs().mean()) +``` + +Results: + +``` +Max abs diff: 0.00000000 +Mean abs diff: 0.00000000 +Relative error: 0.000000 +Matmul max diff: 0.00000000 +``` + +Byte-exact. Zero drift from BF16 rounding noise — ruled out as a potential issue in the final quant. + +### Step 2: Run ModelOpt NVFP4 Full Quantization + +```bash +python3 scripts/model_opt_nvfp4_full.py +``` + +Runs NVIDIA's official ModelOpt PTQ pipeline (`hf_ptq.py`) with full `nvfp4` quantization (attention + experts + shared MLP). Output target: ~600GB. + +**Config:** +- `--quant nvfp4` (full model, not experts-only) +- `--calib 128` — 128 calibration samples. The B200 node has 2.7TB RAM; the 3TB BF16 model doesn't fit in GPU VRAM (~1.4TB total), so it runs with `--use_seq_device_map` (CPU offload). 256 calibration samples OOMs. 128 is the max that fits. +- `--kv_cache_quant fp8_cast` +- `--use_seq_device_map` — sequential device mapping, loads model into CPU RAM, moves layers to GPU for forward passes +- `--gpu_max_mem_percentage 0.7` — VRAM headroom + +**Calibration datasets:** `abisee/cnn_dailymail` + `nvidia/Nemotron-Post-Training-Dataset-v2` (gated — requires HF token). The script exports `HF_TOKEN` and `HUGGING_FACE_HUB_TOKEN`; the token must also be set via `hf auth login` on the node. + +**Runtime:** Model loading takes ~53 minutes. Quantization + calibration takes several hours. Total expect 6-12 hours. ## Key Notes -- **Calib size: 128** (256 OOMs on 2.8TB RAM with 3TB BF16 model) -- **Full quant (`nvfp4`)**, not experts-only - Use BF16 source — V4's mixed precision causes issues, FP8 source has kernel problems on Blackwell -- `--use_seq_device_map` required (model doesn't fit in GPU VRAM alone) -- `--gpu_max_mem_percentage 0.7` for VRAM headroom - `--low_memory_mode` causes meta device errors with V4 — don't use - modelopt has no explicit V4 support — relies on auto-detection of fused experts -- Calibration dataset `nvidia/Nemotron-Post-Training-Dataset-v2` is gated — requires HF token +- The `quant_module_patched.py` patch fixes `iter_weights_for_calibration()` for V4's `nn.ModuleList` expert quantizers — already applied in the venv ## Bugs Found (V4 + modelopt)