biondizzle
220649c188
Build 17-18 'scale_vec not supported' error was because we targeted sm_100 instead of sm_100a. The 'a' suffix enables FP4 block-scaled instructions. No need to fall back to mxf8f6f4 with UE8M0. Path forward: target sm_100a, use mxf4nvf4.scale_vec::4X, keep native UE4M3 scales + block16. No scale conversion needed.
291 lines
13 KiB
Markdown
291 lines
13 KiB
Markdown
# DeepSeek V4 Pro → NVFP4 Quantization + vLLM Serving
|
||
|
||
Full NVFP4 quantization of DeepSeek V4 Pro and vLLM serving on 8× NVIDIA B200 GPUs.
|
||
|
||
## Quick Status
|
||
|
||
| Component | Status |
|
||
|-----------|--------|
|
||
| NVFP4 Quantization | ✅ 881GB (Run 11), modelopt 0.45.0.dev64 |
|
||
| Weight Loading | ✅ 95 safetensors shards, all 8 TP ranks |
|
||
| NVFP4→FP8 Conversion (wo_a) | ✅ DeepGEMM block-scale format |
|
||
| NVFP4→BF16 Dequantization | ✅ 305 attn/shared, 91 compressor layers |
|
||
| Compressor Reconstruction | ✅ Separate kv_proj/gate_proj → fused_wkv_wgate |
|
||
| MoE Expert Serving (FusedMoE) | ✅ FLASHINFER_TRTLLM backend |
|
||
| MoE Expert Serving (MegaMoE) | 🔧 Kernel compiles, runs, but garbled (wrong arch flag sm_100 vs sm_100a) |
|
||
| API Server | ✅ Running on port 8000 |
|
||
| Output Quality | 🔧 Garbled — wrong MMA kind (used mxf8f6f4 instead of mxf4nvf4) |
|
||
|
||
## B200 Node
|
||
|
||
- **IP**: `45.76.247.107`
|
||
- **User**: `root`
|
||
- **Password**: see `.env`
|
||
- **GPUs**: 8× NVIDIA B200 (SM100)
|
||
- **RAM**: ~2.7 TB
|
||
- **Model weights**: `/root/nvidia-meeting/DeepSeek-V4-Pro-NVFP4/`
|
||
- **BF16 reference**: `/root/nvidia-meeting/DeepSeek-V4-Pro-BF16/`
|
||
|
||
## Repositories
|
||
|
||
| Repo | Branch | Purpose |
|
||
|------|--------|---------|
|
||
| `deepseek-v4-quant` | `modelopt-nvfp4` | Main patch (FlashInfer FusedMoE path) |
|
||
| `deepseek-v4-quant` | `mega-moe-nvfp4` | MegaMoE patch (DeepGEMM mega_moe path) |
|
||
| `DeepGEMM` | `nvfp4-mega-moe` | NVFP4 mega_moe kernel fork |
|
||
|
||
## Architecture
|
||
|
||
```
|
||
DeepSeek V4 Pro (1.2T params, 61 layers)
|
||
├── MLA Attention (61 layers)
|
||
│ ├── fused_wqa_wkv → BF16 (UnquantizedLinearMethod)
|
||
│ ├── wo_a → FP8 (DeepGEMM block-scale, BMM einsum)
|
||
│ ├── wo_b → BF16 (UnquantizedLinearMethod)
|
||
│ └── compressor.fused_wkv_wgate → BF16 (reconstructed from NVFP4)
|
||
├── MoE Experts (384 experts, 61 layers)
|
||
│ ├── [FusedMoE path] → NVFP4 (FLASHINFER_TRTLLM backend)
|
||
│ └── [MegaMoE path] → NVFP4 (DeepGEMM mxf8f6f4, UE4M3→UE8M0 adapted)
|
||
└── Shared Expert → FP8 (Fp8LinearMethod, DeepGEMM)
|
||
```
|
||
|
||
## The NVFP4 → vLLM Gap
|
||
|
||
ModelOpt quantizes to NVFP4 (4-bit FP4 with block scales). vLLM's DeepSeek V4
|
||
attention code expects FP8 with DeepGEMM block-scale einsum. These formats were
|
||
**never integrated** — we're ahead of NVIDIA on this. Key gaps we had to bridge:
|
||
|
||
### 1. wo_a: NVFP4 → FP8 + DeepGEMM Block Scale
|
||
|
||
**Problem**: `wo_a` uses `deepseek_v4_fp8_einsum` (BMM with DeepGEMM), which expects:
|
||
- Weight: `float8_e4m3fn` in 3D shape `(g, r, d)` for batched matmul
|
||
- Scale: DeepGEMM-formatted block scale tensor (not a per-tensor scalar)
|
||
|
||
Our NVFP4 weights are uint8 packed FP4 with separate block/global scales.
|
||
|
||
**Solution** (`_convert_nvfp4_to_fp8`):
|
||
1. Unpack NVFP4 uint8 → BF16 using E2M1 lookup table
|
||
2. Dequantize: `weight_bf16 * block_scale * global_scale` (NO input_scale)
|
||
3. Re-quantize BF16 → FP8 e4m3 with per-tensor scale (`w_amax / fp8_max`)
|
||
4. Create block scale tensor filled with `fp8_scale`
|
||
5. Call `deepgemm_post_process_fp8_weight_block` with `quant_block_shape=(128,128)`, `use_e8m0=True`, `is_bmm=True`
|
||
6. Store: `weight_scale_inv = dg_ws`, `weight = w_fp8` (3D BMM shape)
|
||
|
||
### 2. Attention Layers: NVFP4 → BF16
|
||
|
||
**Solution** (`_convert_nvfp4_to_bf16`): Unpack → dequantize → set `UnquantizedLinearMethod`.
|
||
|
||
### 3. Compressor: Reconstructing fused_wkv_wgate from NVFP4
|
||
|
||
**Solution** (`_reconstruct_compressor_weight`): Read `kv_proj`+`gate_proj` from
|
||
safetensors, dequantize, concatenate. **Critical**: indexer compressor at
|
||
`.compressor.indexer.{kv_proj,gate_proj}` not `.compressor.{kv_proj,gate_proj}`.
|
||
|
||
### 4. MoE Experts: NVFP4 FusedMoE
|
||
|
||
**Solution**: Keep expert weights as NVFP4, use `FLASHINFER_TRTLLM` MoE backend.
|
||
|
||
### 5. BF16 wo_a Layers: BF16 → FP8
|
||
|
||
**Solution** (`_convert_bf16_to_fp8`): Directly quantize BF16 → FP8 with block scale.
|
||
|
||
## Bugs Found and Fixed
|
||
|
||
| # | Bug | Impact | Fix |
|
||
|---|-----|--------|-----|
|
||
| 1 | DeepGEMM `sf.dim()` crash | Server crash | `deepgemm_post_process_fp8_weight_block` for block-scale format |
|
||
| 2 | Block scale dtype `float8_e4m3fn` | Crash | Use `float32` |
|
||
| 3 | Missing `deepgemm_post_process` args | Crash | Pass `quant_block_shape`, `use_e8m0` |
|
||
| 4 | Compressor indexer shape mismatch | Crash | `.indexer.` sub-path in checkpoint keys |
|
||
| 5 | All-ones block scale | Garbage output | `torch.full(..., fp8_scale)` not `torch.ones` |
|
||
| 6 | `fused_skip_regex` skipping q_b/o_a/o_b scales | Garbage output | Remove non-fused scale entries from skip list |
|
||
| 7 | UE8M0 block scale misinterpreted as E4M3 | Garbled output | `_ue8m0_to_float32()`: reinterpret raw uint8 as IEEE 754 exponent |
|
||
| 8 | wo_a BF16 weight into uint8 param (suspected) | Double-conversion loss | On-the-fly BF16→NVFP4 in weight_loader, or BF16→FP8 directly |
|
||
|
||
### Bug #7 Detail: UE8M0 → float32 Misinterpretation
|
||
|
||
**Root cause**: `weight_scale` bytes are E8M0 format (power-of-2 only, 8-bit exponent),
|
||
but `.to(torch.float32)` interprets the raw byte as E4M3 (8-bit: sign+exp+mantissa),
|
||
producing a completely wrong float value.
|
||
|
||
**Fix**: `_ue8m0_to_float32()` — reinterpret the raw uint8 bits as the upper 8 bits
|
||
of an IEEE 754 float32 exponent: `(uint8_value << 23).view(float32)`. Applied to all
|
||
dequant paths.
|
||
|
||
### Bug #8 Detail: wo_a BF16 Loading
|
||
|
||
`o_a_proj.weight` is BF16 in checkpoint, but `ModelOptNvFp4Config` creates a uint8
|
||
param (shape mismatch: BF16 `(16384,4096)` vs uint8 `(16384,2048)`). The weight_loader
|
||
does on-the-fly BF16→NVFP4 quantization, but the double conversion (BF16→NVFP4→BF16→FP8)
|
||
is lossy. Diagnostics added but fix pending.
|
||
|
||
## NVFP4 Mega MoE Kernel
|
||
|
||
### What We Built
|
||
|
||
A native NVFP4 mega_moe kernel in our DeepGEMM fork that avoids dequantizing
|
||
expert weights to BF16 before the GEMM. The kernel keeps weights in E2M1 packed
|
||
format and uses block-scaled MMA directly.
|
||
|
||
### SM100a (B200) Hardware Support (CORRECTED)
|
||
|
||
**B200 (SM100a) DOES support `kind::mxf4nvf4`** with `scale_vec::4X` (block16, UE4M3 scales).
|
||
This is documented in PTX ISA 8.7 (CUDA 12.8+) and confirmed by NVIDIA/CUTLASS/Colfax.
|
||
|
||
Build 22 produced garbled output because we incorrectly fell back to `mxf8f6f4` (UE8M0/block32),
|
||
losing 3 bits of mantissa precision per scale. The `mxf4nvf4` instruction was never actually
|
||
tried with the correct target. The real issues were:
|
||
|
||
1. **Wrong arch flag**: JIT compiled for `sm_100` instead of `sm_100a` (`a` suffix required)
|
||
2. **No NVFP4 MMA kind enum**: DeepGEMM only had BF16 + MXFP8FP4 — NVFP4 was just MXFP4 in disguise
|
||
3. **Wrong SF layout**: Block16 (scale_vec::4X) has different TMEM layout than block32 (2X)
|
||
|
||
**Fix**: Target `sm_100a`, emit `tcgen05.mma.kind::mxf4nvf4.block_scale.scale_vec::4X`,
|
||
keep E2M1 weights + UE4M3 scales + block16. No scale conversion needed. Full precision.
|
||
|
||
| Parameter | NVFP4 Checkpoint | Kernel (sm_100a, CORRECT) |
|
||
|-----------|-----------------|--------------------------|
|
||
| Weight format | E2M1 uint8 | E2M1 uint8 (unchanged) |
|
||
| Block scale format | UE4M3 (float8_e4m3fn) | UE4M3 (native, no conversion) |
|
||
| Block size | 16 | 16 (native) |
|
||
| Global scale | float32 | Folded in before packing |
|
||
| PTX instruction | `mxf4nvf4.block_scale.scale_vec::4X` | Same |
|
||
| Instruction descriptor | `float_ue4m3_t` | Same |
|
||
|
||
### Kernel Architecture (TARGET — sm_100a with mxf4nvf4)
|
||
|
||
```
|
||
sm100_fp8_nvfp4_mega_moe_impl (adapted from sm100_fp8_fp4_mega_moe_impl)
|
||
├── kGranK = 16 (NVFP4 native block size)
|
||
├── kind::mxf4nvf4.block_scale.scale_vec::4X PTX instruction
|
||
├── float_ue4m3_t instruction descriptor
|
||
├── Block16 SF layout: scale_vec::4X, 4 TMEM sub-columns per UMMA atom
|
||
├── UTCCP copy: i*8 stride (4X layout)
|
||
├── kNumSFATmemCols = SF_BLOCK_M / 32 * 4
|
||
├── kNumSFBTmemCols = SF_BLOCK_N / 32 * 4
|
||
├── kNumSFUint32 = kHidden / 64 (4 UE4M3 per int32)
|
||
├── UE4M3 L1 epilogue (float → cutlass::float_e4m3_t cast)
|
||
└── recipe = (1, 1, 16)
|
||
|
||
Python API:
|
||
├── fp8_nvfp4_mega_moe() — recipe=(1,1,16)
|
||
├── transform_nvfp4_weights_for_mega_moe()
|
||
│ ├── fold_global_scale(): UE4M3 * FP32 → UE4M3
|
||
│ ├── NO block16→block32 merge (native block16)
|
||
│ ├── NO UE4M3→UE8M0 conversion (native UE4M3)
|
||
│ └── pack_ue4m3_to_int32() + transform_sf_into_required_layout(gran_k=16)
|
||
└── get_symm_buffer_for_nvfp4_mega_moe() — 2x SF buffer
|
||
|
||
C++ Bindings:
|
||
├── csrc/apis/mega_nvfp4.hpp — kGranK=16, SF stride K/16
|
||
├── csrc/jit_kernels/impls/sm100_fp8_nvfp4_mega_moe.hpp
|
||
└── csrc/apis/layout.hpp — gran_k=16 support
|
||
```
|
||
|
||
### Current State (Build 22 — WRONG PATH)
|
||
|
||
Build 22 used `mxf8f6f4` with UE8M0/block32, which produces garbled output.
|
||
The kernel architecture below is what was deployed. It needs to be reverted to the
|
||
`mxf4nvf4` path with sm_100a targeting.
|
||
|
||
```
|
||
Build 22 (GARBLED — do not use):
|
||
├── kGranK = 32, mxf8f6f4, float_ue8m0_t
|
||
├── Same TMEM layout as MXFP4 (2X, block32)
|
||
├── UE4M3→UE8M0 scale conversion (lossy)
|
||
├── block16→block32 scale merge (lossy)
|
||
└── recipe = (1, 1, 32)
|
||
```
|
||
|
||
The earlier Build 17 code had the RIGHT instruction (`mxf4nvf4.scale_vec::4X`)
|
||
but wrong arch flag (`sm_100` instead of `sm_100a`). We need to go back to that
|
||
code and fix the arch flag.
|
||
|
||
### Container Build Pipeline
|
||
|
||
```
|
||
Dockerfile → FROM atl.vultrcr.com/vllm/vllm-with-lmcache:dream-build
|
||
├── DeepGEMM (nvfp4-mega-moe branch) — JIT-compiled at runtime
|
||
├── vLLM patch (deepseek_v4.py) — COPY over model file
|
||
└── NVRTC symlink for CUDA compilation
|
||
|
||
build_push.sh → build → login to CR → push → update docker-compose
|
||
Container registry: atl.vultrcr.com/vllm/vllm-dsv4-nvfp4:latest
|
||
Always run builds in screen: screen -S nvfp4-build
|
||
```
|
||
|
||
### Debugging Log (Builds 1–22)
|
||
|
||
| Build | Error | Fix |
|
||
|-------|-------|-----|
|
||
| 1–6 | Various Dockerfile/build issues | NVRTC symlink, CPATH, PYTHONPATH |
|
||
| 7 | `kPackedFP4` type mismatch | uint8→int8 view on weights |
|
||
| 9 | SF stride assertion | Need MN-major layout + TMA alignment |
|
||
| 10 | `transform_sf_into_required_layout` no gran_k=16 | C++ fix |
|
||
| 11 | SF dtype float8_e4m3fn rejected | Pack UE4M3→int32 first |
|
||
| 12–14 | SF stride layout | Transpose to MN-major before transform |
|
||
| 15 | SymmBuffer too small (NVFP4 has 2× SF) | NVFP4-specific SymmBuffer |
|
||
| 16 | `ImportError: deep_gemm.mega.nvfp4` | Python wrapper in mega/__init__.py |
|
||
| 17 | NVCC: `scale_vec::4X` not supported on sm_100f | Wrong arch: need `sm_100a` not `sm_100` |
|
||
| 18 | NVCC: `scale_vec::2X` ALSO not supported | Same — `sm_100a` required |
|
||
| 19 | kGranK=16 still in C++ binding | → 32 |
|
||
| 20 | UE4M3→UE8M0 `uint32 >> 23` fails | Cast to int32 first |
|
||
| 22 | Server UP, but garbled output | Fell back to mxf8f6f4 — should use mxf4nvf4 on sm_100a |
|
||
|
||
## Path Forward
|
||
|
||
The `mxf4nvf4` instruction IS supported on B200 (SM100a). Build 17-18 failed because
|
||
we targeted `sm_100` instead of `sm_100a`. Build 22 garbled because we fell back to
|
||
`mxf8f6f4` unnecessarily. The correct fix:
|
||
|
||
### Fix the NVFP4 MegaMoE Kernel (Recommended)
|
||
1. Target `sm_100a` in DeepGEMM's JIT compiler
|
||
2. Add NVFP4 MMA kind enum (not just MXFP8FP4 with an NVFP4 hat)
|
||
3. Emit `tcgen05.mma.kind::mxf4nvf4.block_scale.scale_vec::4X`
|
||
4. Use `float_ue4m3_t` instruction descriptor
|
||
5. Block16 SF layout (scale_vec::4X) — different TMEM layout from block32 (2X)
|
||
6. Keep E2M1 weights + UE4M3 scales + block16 — no scale conversion needed
|
||
|
||
This gives full NVFP4 precision with zero scale format conversion.
|
||
|
||
## Running
|
||
|
||
```bash
|
||
# On B200 node
|
||
cd /root/nvidia-meeting
|
||
docker compose up -d
|
||
|
||
# Check logs
|
||
docker logs -f nvidia-meeting-vllm-1
|
||
|
||
# Test
|
||
curl http://localhost:8000/v1/models
|
||
curl http://localhost:8000/v1/chat/completions \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"model": "/model", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 50}'
|
||
```
|
||
|
||
## Files
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `patches/deepseek_v4.py` | Main patch: NVFP4 post-load conversion, weight reconstruction |
|
||
| `patches/modelopt.py` | ModelOpt FP4 config patches for weight loading |
|
||
| `.env` | B200 node credentials |
|
||
| `Dockerfile` | Container build (extends dream-build with DeepGEMM + patch) |
|
||
| `build_push.sh` | Build, push to CR, update docker-compose |
|
||
|
||
## NVFP4 Format Specification
|
||
|
||
- **Weights**: E2M1 packed uint8 (2 values per byte)
|
||
- **Block scales**: float8_e4m3fn (UE4M3), group_size=16
|
||
- **Global scales**: float32 (weight_scale_2), per-tensor
|
||
- **Dequant formula**: `value = packed_E2M1 * block_scale * global_scale`
|
||
- **Block scale range**: [0, 448] (UE4M3 max = 448, E2M1 max = 6, so 6×448 = 2688)
|
||
- **UE8M0**: Power-of-2 only. Encoded as uint8 = float32_exponent_bits[31:23]
|
||
- **UE4M3**: 3-bit mantissa + 4-bit exponent + sign. Max = 448.
|
||
|
||
## HARD RULES
|
||
|
||
- **NEVER convert DeepSeek MoE experts to MXFP4.** Experts stay in NVFP4. Period.
|