deepseek-v4-quant/MEMORY.md

MEMORY.md — Long-Term Memory

Mike

• Working on DeepSeek V4 Pro NVFP4 quantization + vLLM serving on B200 node
• B200 node: 45.76.247.107, root, password in project .env
• Repo: https://sweetapi.com/biondizzle/deepseek-v4-quant.git (modelopt-nvfp4 branch)

DeepSeek V4 NVFP4 Project

• Successfully quantized: 881GB NVFP4 (Run 11), 8× B200, $161/run
• modelopt 0.45.0.dev64 + transformers 5.8.0.dev0
• vLLM server running on B200 port 8000 as of May 11, 2026 🎉
• We built the entire NVFP4→vLLM bridge from scratch (NVIDIA hasn't done this)
• Abandoned mega_moe (no kernel, format mismatch), using standard FusedMoE instead

Key Technical Decisions

• wo_a: NVFP4→BF16→FP8 with DeepGEMM block-scale format for BMM einsum
• Attention layers: NVFP4→BF16 dequantization, UnquantizedLinearMethod
• Compressor: Reconstructed fused_wkv_wgate from separate kv_proj+gate_proj in checkpoint
• MoE experts: Stay NVFP4, use FLASHINFER_TRTLLM FusedMoE backend

Critical Bugs Fixed (May 11)

1. DeepGEMM sf.dim() crash: weight_scale_inv must be DeepGEMM-formatted block scale tensor
2. Compressor indexer shape mismatch: checkpoint keys have .indexer. sub-path
3. All-ones block scale → garbage output: must use torch.full(..., fp8_scale) not torch.ones
4. Block scale dtype: must be float32, not float8_e4m3fn

Outstanding

• Output quality under investigation — FP4 is aggressive quantization
• All code in patches/deepseek_v4.py on modelopt-nvfp4 branch