# DeepSeek V4 Pro → NVFP4 Quantization + vLLM Serving

Full NVFP4 quantization of DeepSeek V4 Pro on a single B200 node (8× B200, 2.7TB RAM, 13TB NVMe). **Result: 881GB NVFP4 (Run 11).** Now working on vLLM serving of the quantized checkpoint.

**Cost:** ~$161/run at $23/hr (7 hours each). Don't waste runs.

## ✅ Final Quantization Result (Run 11)

- **Output:** `/root/nvidia-meeting/DeepSeek-V4-Pro-NVFP4` — 881GB, 95 safetensors
- **Config:** `nvfp4` full quantization, 128 calib samples, `kv_cache_qformat=fp8_cast`
- **Total runtime:** ~7,783s (~2h10m end-to-end)
- **Peak GPU mem:** ~163GB per B200
- **Amax snapshots:** 47,696 quantizers, 15.4MB
- **Calibrated state:** 721.4GB (insurance, can re-export with `--export-only`)
- A few experts (11, 83, 100, 112, 254) had uncalibrated amax — weight-derived fallback used (normal for sparse MoE with 256 experts)

---

## 🔧 vLLM Serving (In Progress)

### Current Status: Debugging weight loading

The modelopt NVFP4 export and vllm have a chain of incompatibilities. We're progressively fixing them. The fundamental problem: **modelopt's NVFP4 quantization format and vllm's DeepSeek V4 serving code were never integrated.** NVIDIA's own published NVFP4 exports (DeepSeek-V3.2, MiniMax-M2.7) don't have these issues because they don't use MLA attention compression or 256-expert MoE — both of which create stacked/fused weight parameters that modelopt doesn't account for.

### Approach: Patched deepseek_v4.py + disabled mega_moe

Instead of runtime monkey-patching (which doesn't propagate to worker processes), we patch the vllm source file directly. The patched `deepseek_v4.py` is mounted into the Docker container and copied over the original before vllm starts.

We also disabled `--moe-backend=deep_gemm_mega_moe` because:
1. The NVFP4 mega_moe kernel doesn't exist yet (NVIDIA hasn't built it)
2. MegaMoE uses MXFP4 block scale format (32-col blocks), but modelopt exports NVFP4 (16-col blocks) — format mismatch
3. MegaMoE doesn't register `weight_scale_2` or `input_scale` params, so those scales would be silently dropped

Instead, we use the standard FusedMoE path with `ModelOptNvFp4FusedMoE`, which natively supports NVFP4 expert weights.

### vLLM Serving Run History

| # | Date | Approach | Result | Root Cause | Fix/Next |
|---|------|----------|--------|------------|----------|
| S1 | May 10 09:34 | `patch_vllm_weights.py` runtime patch + mega_moe | ❌ `UnboundLocalError: name_mapped` | Expert weight names don't match any mapping → `name_mapped` never assigned | Add gate_proj→w1, up_proj→w3, down_proj→w2 mappings |
| S2 | May 10 ~10:30 | Same, added expert rename regexes | ❌ Same error | `DeepseekV4ForCausalLM.hf_to_vllm_mapper` is a **class attribute** set at import time — patching the function doesn't update the cached mapper | Patch the class attribute directly |
| S3 | May 10 ~11:00 | Patched class attr, but workers are separate processes | ❌ Same error in workers | Workers don't inherit in-memory patches — they fork before the patch runs | Patch the source file directly (`deepseek_v4.py`) |
| S4 | May 10 ~11:30 | Direct source file patch + mega_moe | ❌ `KeyError: 'layers.0.mlp.experts.0.w2.weight'` | modelopt uses `mlp`, vllm uses `ffn` internally — missing `.mlp.` → `.ffn.` mapping | Add substr mapping |
| S5 | May 10 ~12:00 | Added `mlp→ffn` mapping + mega_moe | ❌ `KeyError: 'fused_wkv_wgate.input_scale'` | Compressor fused params don't register `input_scale`/`weight_scale_2` | Add skip patterns for compressor/attention scale tensors |
| S6 | May 10 ~12:30 | Added skip patterns + mega_moe | ❌ Shape mismatch: `w2_weight_scale (7168, 96) vs (7168, 192)` | NVFP4 uses 16-col block scales, mega_moe expects 32-col MXFP4 — format incompatibility | **Abandon mega_moe** — no NVFP4 mega_moe kernel exists |
| S7 | May 10 ~13:00 | Disabled mega_moe, standard FusedMoE | ❌ `fused_wkv_wgate.weight` shape mismatch: param=(1024,7168) bf16, loaded=(512,3584) uint8 | `MergedColumnParallelLinear` creates weight as bf16 (not uint8), but modelopt exports NVFP4 packed uint8. `ModelOptNvFp4Config` only handles `Linear`, not `MergedColumnParallelLinear` | Unpack uint8→bf16 at load time |
| S8 | May 10 ~13:30 | Added E2M1 unpacking for fused weights | ❌ `KeyError: 'fused_wkv_wgate.weight_scale'` | No `weight_scale` param registered for `MergedColumnParallelLinear` (same `ModelOptNvFp4Config` gap) | Skip all NVFP4 scale tensors for stacked/fused attention+compressor params |
| S9 | May 10 ~14:00 | Added weight_scale skip patterns | ❌ `KeyError: 'compressor.kv_norm.weight'` | modelopt puts `kv_norm` under `compressor`, vllm has it at attention level (`attn.kv_norm`) | Add `compressor.kv_norm` → `kv_norm` mapping |
| S10 | May 10 ~14:15 | Fixed kv_norm mapping | ❌ `KeyError: 'compressor.position_bias'` | modelopt exports params that don't exist in the vllm model | Make loading resilient to unknown params |

### Open Issues (as of May 10 ~16:00 UTC)

1. **MergedColumnParallelLinear + NVFP4 incompatibility** — The core problem. `ModelOptNvFp4Config.create_weights()` only handles `Linear` layers. For `MergedColumnParallelLinear` (used for `fused_wqa_wkv`, `fused_wkv_wgate`, `gate_up_proj`):
   - Weight param is created as `ModelWeightParameter` (bf16) instead of `PackedColumnParameter` (uint8)
   - `weight_scale`, `weight_scale_2`, `input_scale` params are never registered
   - `adjust_shard_indexes_for_packing` applies `packed_factor` to rows, but NVFP4 packs along columns
   - Current workaround: unpack uint8→bf16 at load time, skip scale tensors, let `process_weights_after_loading` re-quantize. This loses the calibration-optimized scales for attention/compressor/shared_expert weights.

2. **No NVFP4 mega_moe kernel** — We disabled mega_moe to avoid the format mismatch. Standard FusedMoE with `ModelOptNvFp4FusedMoE` works for expert weights, but loses the mega_moe performance optimization. When NVIDIA builds an NVFP4 mega_moe kernel, we can re-enable it.

3. **Resilient loading needed** — modelopt exports params (e.g., `compressor.position_bias`) that don't exist in the vllm model. Need to skip unknown params gracefully instead of crashing.

4. **Expert `weight_scale_2` handling with FusedMoE** — The standard FusedMoE path registers `w13_weight_scale_2` and `w2_weight_scale_2`, so expert global scales CAN be loaded. This works for experts. The issue is only with the stacked/fused attention params.

### What Each Patch Does

**`patches/deepseek_v4.py`** — Patched vllm source file, copied over the original at container startup. Contains:
- **Regex mappings** (applied first by WeightsMapper):
  - Skip `weight_scale`, `weight_scale_2`, `input_scale` for compressor/attention fused params (no stacked param registered)
  - Skip `weight_scale`, `weight_scale_2`, `input_scale` for shared expert gate/up projections (stacked into `gate_up_proj`)
  - Expert projection rename: `gate_proj→w1`, `up_proj→w3`, `down_proj→w2` (only for `.experts.N.`, not `.shared_experts.`)
- **Substr mappings** (applied after regex):
  - Attention: `self_attn→attn.mla_attn` with proper sub-projection names
  - `kv_norm` moved from compressor to attention level
  - `compressor.kv_proj→compressor.wkv`, `compressor.gate_proj→compressor.wgate`
  - `shared_experts.gate_proj→shared_experts.w1`, `shared_experts.up_proj→shared_experts.w3`
  - `.mlp.→.ffn.` (modelopt uses `mlp`, vllm uses `ffn`)
- **E2M1 FP4→BF16 unpacking** for stacked params: When a uint8 packed NVFP4 weight is loaded into a bf16 param (MergedColumnParallelLinear), unpack using the E2M1 lookup table
- **Resilient loading**: Skip unknown params that modelopt exports but vllm doesn't have

**`patches/patch_vllm_weights.py`** — Legacy runtime monkey-patch approach. Doesn't work because vllm workers are separate processes that don't inherit in-memory patches. Kept for reference.

**`docker-compose.yml`** — Docker Compose config:
- Copies patched `deepseek_v4.py` before starting vllm
- Removed `--moe-backend=deep_gemm_mega_moe` (no NVFP4 kernel exists)
- All other vllm flags are critical for V4 (see `serve_vllm.py` for documentation)

---

## ⚠️ Model Config Patches (post-export)

modelopt 0.45.0.dev64's export produces configs that don't match what vllm expects at runtime. **NVIDIA's own published NVFP4 exports have the same gaps** — we compared against `nvidia/DeepSeek-V3.2-NVFP4` and `nvidia/MiniMax-M2.7-NVFP4` on HuggingFace. Neither includes `compress_ratios` or `scale_fmt` either. This is a modelopt ↔ vllm integration gap, not a problem with our quantization.

All patches below are to `DeepSeek-V4-Pro-NVFP4/config.json` unless noted.

| # | Field | modelopt export (original) | vllm requires | Patch applied | Why modelopt doesn't export it |
|---|-------|---------------------------|--------------|---------------|------------------------------ |
| 1 | `compress_ratios` | Missing (transformers 5.8.0 renamed to `compress_rates` dict) | List of ints indexed by layer_id | Copied from BF16 source model's `compress_ratios` (62 items) | modelopt doesn't add fields the source config lacks; transformers 5.8.0 renamed the field |
| 2 | `quantization_config.scale_fmt` | Missing | `"ue8m0"` string | Added | modelopt doesn't include vllm-specific runtime fields |
| 3 | `rope_parameters` | Nested dict `{'main': {...}, 'compress': {...}}` (transformers 5.8.0 format) | Flat dict `{'rope_theta': ..., 'rope_type': ..., ...}` | Flattened to `main` sub-dict | transformers 5.8.0 changed rope_parameters from flat → nested per-component |
| 4 | `rope_scaling` | Nested dict `{'main': {...}, 'compress': {...}}` (same as above) | Flat dict | Flattened to `main` sub-dict | Same transformers 5.8.0 schema change |

**NVIDIA's own NVFP4 exports confirmed to also lack patches 1 and 2.** We checked:
- `nvidia/DeepSeek-V3.2-NVFP4` — no `compress_ratios`, no `scale_fmt`, no `quantization_config` in config.json at all (V3.2 doesn't use MLA compression so it sidesteps the issue)
- `nvidia/MiniMax-M2.7-NVFP4` — has `quantization_config` in config.json (same schema as ours) but no `scale_fmt`

The `compress_rates` → `compress_ratios` rename and `rope_parameters` nesting are transformers 5.8.0 regressions that modelopt doesn't account for. `scale_fmt` is a vllm runtime field that modelopt has never exported.

## Architecture

We call modelopt's `hf_ptq.main()` directly — the same entry point the shell script uses. We don't rewrite the pipeline. We just:

1. **Patch** modelopt at runtime (GPU tensor safety, before anything runs)
2. **Hook** `export_quantized` to snapshot amax + save state before export
3. **Call** `hf_main(args)` with properly parsed args

This avoids the cascade of missing-arg bugs from manually constructing `argparse.Namespace` (Runs 4–8).

## Pipeline

### Step 1: Dequantize FP8 → BF16

```bash
python3 scripts/dequant_fp8_to_bf16.py /root/nvidia-meeting/DeepSeek-V4-Pro-FP8 /root/nvidia-meeting/DeepSeek-V4-Pro-BF16
```

The original V4 weights use mixed precision (FP8 attention + FP4/E2M1 experts with per-tensor scales). We dequantize everything to pure BF16 so modelopt can run calibration without hitting broken FP8 kernel paths on Blackwell (DeepGEMM unsupported, Triton finegrained FP8 matmul shape mismatches).

This is not a blind upcast — it applies the actual scale factors:

```
W_bf16 = dequantize_fp4_weight(W_int, S)  # per-tensor scale dequant, not .to(bfloat16)
```

**Byte-exact verified** — matmul diff is 0.000000 against the official inference path.

### Step 2: Run NVFP4 Quantization

```bash
cd /root/nvidia-meeting/modelopt-repo/examples/llm_ptq
python3 /root/nvidia-meeting/deepseek-v4-quant/scripts/quantize_nvfp4.py
```

Must run from the modelopt example directory (relative imports).

What happens inside:
1. **Apply patches** — 3 runtime monkey-patches for GPU tensor safety (see below)
2. **Parse args** — uses `hf_ptq.parse_args()` with our config via `sys.argv` replacement, then applies the same post-parse conversions (`dataset` split, `calib_size` int list) that `hf_ptq.__main__` normally does
3. **Hook export** — monkey-patch `export_quantized` to snapshot amax + save state before export
4. **Call `hf_main(args)`** — the exact same pipeline the shell script uses

If the export crashes:

```bash
python3 /root/nvidia-meeting/deepseek-v4-quant/scripts/quantize_nvfp4.py --export-only
```

To validate saved state without running anything:

```bash
python3 /root/nvidia-meeting/deepseek-v4-quant/scripts/quantize_nvfp4.py --validate-only
```

**Config:** `nvfp4`, 128 calib samples, `calib_seq=512`, `kv_cache_qformat=fp8_cast`, `gpu_max_mem_percentage=0.7`, `use_seq_device_map`, `inference_tensor_parallel=8`

**Calibration datasets:** `abisee/cnn_dailymail` + `nvidia/Nemotron-Post-Training-Dataset-v2` (default when no `--dataset` specified).

**Runtime:** Model loading ~50 min. Calibration ~5.5 hours. Export ~30-60 min. Total 7-8 hours.

### Step 3: Serve with vLLM

```bash
cd /root/nvidia-meeting
docker compose up -d
```

Or without Docker:

```bash
source /root/nvidia-meeting/venv/bin/activate
python3 /root/nvidia-meeting/deepseek-v4-quant/scripts/serve_vllm.py
```

**Note:** `serve_vllm.py` still references `--moe-backend=deep_gemm_mega_moe`. This needs to be removed when mega_moe support is ready. For now, use the Docker Compose setup which has it removed.

## Quantization Run History

| Run | Date | Commit | Calib | Result | Root Cause | Fix |
|-----|------|--------|-------|--------|------------|-----|
| 1 | May 7 | shell wrapper | 256 | ❌ Batch probing crash | `o_b_proj` shape mismatch — finegrained_fp8 wraps MLA projections incorrectly with FP8 source | Use BF16 source (dequantized) |
| 2 | May 8-9 | shell wrapper | 128 | ❌ Export crash (calib ✅) | `get_activation_scaling_factor` reads stale GPU amax → CUDA illegal memory access | Snapshot amax to CPU after calibration |
| 3 | May 9 06:10 | `3907838` | 128 | ❌ Model loading OOM | `AutoModelForCausalLM.from_pretrained` OOM during expert weight `torch.cat` | Use modelopt `get_model()` with `max_memory` |
| 4 | May 9 ~07:00 | `86dd8df` | 128 | ❌ Import error | `mtq.KV_QUANT_CFG_CHOICES` doesn't exist — it's `hf_ptq.KV_QUANT_CFG_CHOICES` | Import from `hf_ptq`, not `mtq` |
| 5 | May 9 ~08:05 | `f9bbef8` | 128 | ❌ Same as Run 4 | Fix wasn't synced properly | Properly synced |
| 6 | May 9 ~09:25 | `6c1bff6` | 128 | ❌ Dataloader crash | `make_calib_dataloader` AttributeError — missing args | Added args to Namespace |
| 7 | May 9 ~13:40 | `25b4d8d` | 128 | ❌ Dataloader crash | `dataset=None`, `len()` on None | Provided dataset list |
| 8 | May 9 ~14:00 | `b2849a8` | 128 | ❌ Argparse crash | Wrong flag names (shell script names vs `hf_ptq.py` names) | Use `hf_ptq.py` flag names |
| 9 | May 9 ~14:30 | `a300302` | 128 | ❌ TypeError | Skipped `__main__` post-parse conversions (`calib_size` still string, not int list) | Apply same conversions after `parse_args()` |
| 10 | May 9 ~15:30 | `5a72da7` | 128 | ❌ Export crash (calib ✅) | `get_weight_scaling_factor` reads stale GPU weight → `cudaErrorIllegalAddress` | Patch `_export_quantized_weight` to force weight to CPU at entry point |
| 11 | May 9 ~22:50 | `07cd50e` | 128 | ✅ **SUCCESS** | — | 8 patches covering full export chain |

### Key Lessons (Quantization)

**Run 2 — Stale GPU tensors:** `use_seq_device_map` shuffles layers through GPU for calibration. Quantizer amax tensors sit in VRAM for 5+ hours while CUDA's allocator churns memory. By export time, the GPU tensor metadata is valid but the underlying memory has been recycled — reading it triggers `cudaErrorIllegalAddress`. Fix: copy amax to CPU immediately after calibration.

**Run 3 — Expert weight OOM:** `AutoModelForCausalLM.from_pretrained` does `torch.cat` on GPU for expert `gate_up_proj` (31.5GB alloc, 25.9GB free). Fix: use modelopt's `get_model()` which sets `max_memory` per GPU before loading. (Note: Run 10 uses `hf_main()` which calls `get_model()` internally.)

**Runs 4–8 — Pipeline rewriting trap:** Trying to reconstruct hf_ptq's pipeline by importing individual functions and building a fake `argparse.Namespace` causes an endless stream of missing-attribute and type errors. Each fix reveals the next bug. Fix: call `hf_main(args)` directly with a properly parsed args object.

**Run 9 — `__main__` gap:** `hf_ptq.py` does critical type conversions in its `__main__` block (string → list for `dataset`, string → int list for `calib_size`). When calling `main()` directly, these are skipped. Fix: apply the same conversions after `parse_args()`.

**Run 10 — Stale GPU weight tensors in export:** The amax patches (Patch 1-3) only cover quantizer state. The model *weights* themselves are also on stale GPU. `get_weight_scaling_factor` does `weight_scaling_factor_2.to(weight.device)` which triggers `cudaErrorIllegalAddress` because `weight` is on stale GPU. Fix: patch `_export_quantized_weight` (the entry point for each module's export) to force `weight` to CPU before any downstream code reads it. This covers the entire chain: `get_weight_scaling_factor`, `get_weights_scaling_factor_from_quantizer`, `to_quantized_weight`, `weight.to(dtype)` — all resolve to CPU because `weight.device` is CPU.

### Do NOT Repeat These Mistakes

- Don't use FP8 source model — kernel issues on Blackwell (Run 1)
- Don't use `--low_memory_mode` with V4 — meta device errors
- Don't use `calib_size=256` — OOMs with 3TB BF16 on CPU offload
- Don't use `AutoModelForCausalLM.from_pretrained` directly — OOM during expert weight concat (Run 3)
- Don't assume GPU tensor integrity after 5+ hours of sequential calibration (Run 2, Run 10)
- Don't rewrite the hf_ptq pipeline — call `hf_main()` directly (Runs 4–8)
- Don't skip the `__main__` post-parse conversions — `calib_size` must be int list, `dataset` must be list (Run 9)
- Don't use shell script arg names (`--quant`, `--calib`, `--kv_cache_quant`, `--tp`) — use `hf_ptq.py` names (`--qformat`, `--calib_size`, `--kv_cache_qformat`, `--inference_tensor_parallel`)
- Don't patch individual export functions one at a time — patch the entry point (`_export_quantized_weight`) so weight is on CPU for the entire chain (Run 10)
- Don't use runtime monkey-patching for vllm serving — workers are separate processes that don't inherit patches. Patch the source file directly instead.

## Runtime Patches Applied by quantize_nvfp4.py

These are monkey-patches applied at runtime — no modelopt source files are modified.

### Calibration-time patches (applied before pipeline runs)

1. **`TensorQuantizer.load_calib_amax`** — After calibration writes `_amax` to GPU, immediately moves it to CPU. Prevents stale GPU tensors.
2. **`TensorQuantizer.export_amax`** — If `_amax` is still on GPU at export time, moves to CPU before reading. Safety net.
3. **`NVFP4QTensor.get_activation_scaling_factor`** — Moves amax to CPU, clamps bad values instead of hard assert. Prevents crash on garbage from GPU corruption.

### Export-time patches (force stale GPU tensors to CPU at entry points)

4. **`_export_quantized_weight`** (KEY PATCH) — Forces weight + all quantizer state to CPU *before* any downstream code reads them. This is the entry point for exporting each linear layer. By forcing weight to CPU here, every downstream `.to(weight.device)` resolves to CPU, covering the entire chain: `get_weight_scaling_factor`, `get_weights_scaling_factor_from_quantizer`, `to_quantized_weight`, `weight.to(dtype)`.
5. **`_export_fused_experts`** — Same treatment for MoE expert weights (DeepseekV4Experts go through this path). Forces expert weights, buffers, and quantizer state to CPU.
6. **`to_quantized_weight`** — Forces weight and scaling factors to CPU. Redundant if Patch 4 works, but catches any code path that reaches this function without going through `_export_quantized_weight`.
7. **`get_weight_scaling_factor`** — Forces weight + quantizer to CPU. Redundant if Patch 4 works.
8. **`get_weight_scaling_factor_2`** — Forces quantizer state to CPU. Redundant if Patch 4 works.

Patches 6-8 are belt-and-suspenders. Patch 4 is the one that matters — it moves weight to CPU at the earliest possible point in the export chain, making all downstream stale GPU reads impossible.

### Post-Calibration Hook

`export_quantized` is monkey-patched to run these steps before the real export:

4. **`snapshot_amax_to_cpu()`** — Walks all quantizers, copies `_amax` to CPU, saves to disk (~50MB). Insurance policy.
5. **`force_all_amax_to_cpu()`** — Moves `_pre_quant_scale`, `_global_amax` to CPU too. Nuclear option.
6. **`save_calibrated_state()`** — Saves full model state dict to disk (~1.5TB). Enables `--export-only` recovery if export crashes.

## Bugs Found (V4 + modelopt 0.45.0.dev64)

1. ~~`QuantDeepseekV4Experts` AttributeError~~ — **Already fixed** in modelopt 0.45.0.dev64 (handles `nn.ModuleList` quantizers natively).
2. `--low_memory_mode` → meta device error. Don't use with V4.
3. Missing `kernels` package for FP8 ops. `pip install -U kernels`.
4. ~~Shell script arg names~~ — Resolved by calling `hf_main()` directly.
5. **Export crash — stale GPU tensors in `export_amax()`.** After hours of calibration, quantizer `_amax` on GPU becomes unreadable. Fixed by patching `export_amax` to move `_amax` to CPU before reading.
6. **Export crash — `assert torch.all(activation_scaling_factor > 0)`.** Amax values from stale GPU reads are garbage (zeros, negatives, NaN). Fixed by clamping instead of asserting, plus snapshotting valid amax to CPU before corruption can occur.
7. **Model loading OOM during expert weight conversion.** `AutoModelForCausalLM.from_pretrained` does `torch.cat` on GPU for expert `gate_up_proj` (31.5GB alloc), but only 25.9GB free with `device_map="sequential"`. Fixed by using modelopt's `get_model()` which sets `max_memory` per GPU before loading.
8. **Export crash — stale GPU weight tensors in `get_weight_scaling_factor`.** Patches 1-3 only covered quantizer amax. The model weights themselves are also on stale GPU. `weight_scaling_factor_2.to(weight.device)` triggers `cudaErrorIllegalAddress`. Fixed by patching `_export_quantized_weight` to force weight to CPU at the entry point, covering the entire export chain.

### Bugs Found (V4 NVFP4 + vLLM serving)

1. **modelopt uses `mlp`, vllm uses `ffn`** — Module naming mismatch. Fixed with substr mapping.
2. **modelopt uses `gate_proj`/`up_proj`/`down_proj`, vllm expects `w1`/`w3`/`w2`** — Expert weight naming mismatch. Fixed with regex mapping (only for `.experts.N.`, not `.shared_experts.`).
3. **modelopt uses `self_attn` prefix, vllm uses `attn.mla_attn`** — Attention module naming. Fixed with substr mapping.
4. **`kv_proj` maps to `wkv`, not `kv_proj`** — vllm stacks `wkv` + `wq_a` into `fused_wqa_wkv`. Fixed with substr mapping.
5. **`compressor.kv_proj` → `compressor.wkv`** — Similar stacking for compressor. Fixed with substr mapping.
6. **`compressor.kv_norm` → `attn.kv_norm`** — modelopt puts `kv_norm` under compressor, vllm has it at attention level. Fixed with substr mapping (must come before general compressor mapping).
7. **`MergedColumnParallelLinear` + NVFP4 incompatibility** — `ModelOptNvFp4Config.create_weights()` only handles `Linear`, not `MergedColumnParallelLinear`. This causes:
   - Weight param created as bf16 instead of uint8 (PackedColumnParameter)
   - `weight_scale`/`weight_scale_2`/`input_scale` not registered for stacked params
   - `adjust_shard_indexes_for_packing` applies packed_factor to rows, but NVFP4 packs along columns
   - **Workaround:** Unpack uint8→bf16 at load time, skip scale tensors, rely on `process_weights_after_loading` re-quantization
8. **No NVFP4 mega_moe kernel** — `DeepseekV4MegaMoEExperts` expects MXFP4 (32-col blocks), modelopt exports NVFP4 (16-col blocks). No kernel exists. **Abandoned mega_moe**, using standard FusedMoE instead.
9. **`DeepseekV4ForCausalLM.hf_to_vllm_mapper` is a class attribute** — Runtime monkey-patching the factory function doesn't update the cached class attribute. Must patch the source file directly or update the class attribute explicitly.
10. **vllm workers are separate processes** — In-memory monkey-patches don't propagate to workers. Must patch the source file directly.
11. **modelopt exports params vllm doesn't have** — e.g., `compressor.position_bias`. Need resilient loading that skips unknown params.

## Dependencies (pinned versions)

- **nvidia-modelopt:** `0.45.0.dev64+g579fc6c31` (installed from git, not PyPI)
- **transformers:** `5.8.0.dev0` (from git, required for DeepSeekV4 support)
- **kernels:** latest (`pip install -U kernels` — needed for finegrained FP8 ops)
- **Python:** 3.10

The patches in `quantize_nvfp4.py` are for **modelopt 0.45.0.dev64** specifically. Later versions may include fixes natively — check before applying.

## Key Notes

- V4 is NOT BF16 — it ships as mixed-precision FP8/FP4. You MUST dequantize to BF16 first (Step 1).
- `--low_memory_mode` causes meta device errors with V4 — don't use.
- modelopt has no explicit V4 support — relies on auto-detection of fused experts.
- The calibration state save (`v4_nvfp4_calibrated_state.pt`) is ~1.5TB. It lives on NVMe, not in git.
- The amax snapshot (`v4_nvfp4_amax_snapshots.pt`) is ~50MB. Small, critical, cheap insurance.
- The script calls `hf_main(args)` — the exact same entry point as the shell script. No pipeline divergence.
- Must run from `/root/nvidia-meeting/modelopt-repo/examples/llm_ptq` (relative imports).
- For vllm serving, the patched `deepseek_v4.py` must be mounted into the container — workers don't inherit in-memory patches.
- We disabled `--moe-backend=deep_gemm_mega_moe` because no NVFP4 mega_moe kernel exists yet. Standard FusedMoE with `ModelOptNvFp4FusedMoE` handles expert weights correctly.

## File Layout

```
scripts/
  dequant_fp8_to_bf16.py   — Step 1: FP8/FP4 → BF16 dequantization
  quantize_nvfp4.py         — Step 2: NVFP4 quantization (patches + hf_main)
  serve_vllm.py             — Step 3: vLLM serving (legacy, still has mega_moe flag)

patches/
  deepseek_v4.py            — Patched vllm source file (copied over original at container startup)
  patch_vllm_weights.py     — Legacy runtime monkey-patch (doesn't work with workers, kept for reference)
  quant_module_patched.py   — (legacy) quant module patches
  patch_finegrained_fp8_blackwell.py  — (legacy) FP8 kernel patches for Blackwell

docker-compose.yml           — Docker Compose config for serving (uses patched deepseek_v4.py, no mega_moe)
```

The `patches/` directory contains earlier approaches that modified modelopt source files directly. The current approach (`quantize_nvfp4.py`) uses runtime monkey-patching instead — no source files are modified.