biondizzle/deepseek-v4-quant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Clarify: V4 is NOT BF16, dequantize first

Browse Source
This commit is contained in:
biondizzle
2026-05-08 17:31:35 +00:00
parent 24e3b3745d
commit 50348989b2
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@@ -68,7 +68,7 @@ The `quant_module_patched.py` fix is for **modelopt 0.45.0.dev64** specifically.
## Key Notes
- Use BF16 source — V4's mixed precision causes issues, FP8 source has kernel problems on Blackwell
- V4 is NOT BF16 — it ships as mixed-precision FP8/FP4. You MUST dequantize to BF16 first (Step 1). The raw FP8 source has kernel problems on Blackwell; the mixed-precision source causes modelopt errors
- `--low_memory_mode` causes meta device errors with V4 — don't use
- modelopt has no explicit V4 support — relies on auto-detection of fused experts
- The `quant_module_patched.py` patch fixes `iter_weights_for_calibration()` for V4's `nn.ModuleList` expert quantizers — already applied in the venv