deepseek-v4-quant/memory/2026-05-10-vllm-modelopt-patch.md

Session: 2026-05-10 13:12:29 UTC

• Session Key: agent:main:main
• Session ID: 8eeafe17-997c-452c-97fe-38e25742c5eb
• Source: webchat

Conversation Summary

assistant: Now let me see what vllm's mapper expects. Let me dump the full mapper:Now I can see the mapping. vllm expects names like:

• layers.0.attn.compressor.gate_proj... → model.layers.0.atn.mla_attn.compressor.gate_proj...
• layers.0.shared_experts.w2 → model.layers.0.shared_experts.down_proj

Our modelopt export has:

• model.layers.0.self_attn.compressor.gate_proj... (has self_atn not attn)
• model.layers.0.mlp.shared_experts.down_proj (already uses down_proj)

The prefix mapper converts layers. → model.layers. and then the substr mapper converts .attn.compressor. → .attn.mla_attn.compressor.. But our weights already have model.layers. prefix AND use self_attn instead of atn.

The issue: modelopt exports with self_attn but vllm's mapper expects atn (which it then maps to attn.mla_attn). Let me also check what other name mismatches exist:Now I have the full picture. The main mapping issue is self_atn → attn.mla_attn. But there are also quantization suffixes (input_scale, weight_scale, weight_scale_2) that modelopt adds. vllm's modelopt_fp4 quantizer should handle those natively — the WeightsMapper just needs to map the base names correctly.

Let me write a mountable patch script that fixes the mapper at container startup:Now I need to figure out how to inject this patch into the container. The easiest way is to mount the patches dir and add it as an entrypoint wrapper. Let me update docker-compose:YAML escape issue. Let me fix the entrypoint:YAML's being a pain with indentation. Let me just write it properly:YAML validates. Let me start it:It's past the weight mapper crash! The patch is working. I can see:

• Detected ModelOpt NVFP4 checkpoint ✅
• Using max model len 1048576 ✅
• DeepSeekV4MTPModel architecture resolved ✅
• Now it's initializing the engine...

This is loading the 881GB model across 8 GPUs. This will take a while. Let me check back in a few minutes: user: Sender (untrusted metadata):

{
  "label": "openclaw-control-ui",
  "id": "openclaw-control-ui"
}

[Sun 2026-05-10 09:30 UTC] fuck it. do #2 and see where ya get. If ya dont get anywhere, then ya need to look deeper into model opt and we have to patch model opt, or see if there's a seeting we missed that does that assistant: Already on it — I patched the weight mapper and restarted the container. It got past the UnboundLocalError crash. Let me check where it's at now: assistant: SSH connection is timing out. The machine might be under heavy load from model loading (881GB across 8 GPUs). Let me retry: assistant: The execs keep getting aborted. Let me try a different approach: user: System: [2026-05-10 09:31:01 UTC] Exec completed (calm-ree, code 0) :: INFO 05-10 09:30:59 [nixl_utils.py:20] Setting UCX_RCACHE_MAX_UNRELEASED to '1024' to avoid a rare memory leak in UCX when using NIXL. INFO 05-10 09:30:59 [nixl_utils.py:32] NIXL … System: [2026-05-10 09:31:19 UTC] Exec completed (calm-dai, code 0) :: __', 'le', 'lt', 'match_args', 'module', 'ne', 'new', 'or', 'reduce', 'reduce_ex', 'repr', 'setattr', 'sizeof', 'str', '__subcl… System: [2026-05-10 09:31:40 UTC] Exec completed (swift-br, code 0) :: === orig_to_new_substr === .attn.compressor. -> .attn.mla_attn.compressor. .shared_experts.w2 -> .shared_experts.down_proj === orig_to_new_prefix === layers. -> model.layers. embe… System: [2026-05-10 09:34:38 UTC] Exec completed (briny-ri, code 0) :: ver pid=1)[0;0m [33mWARNING[0m [90m05-10 09:34:35[0m [90m[vllm.py:1252][0m Auto-initialization of reasoning token IDs failed. Please check whether your reasoning parser has implem…

Sender (untrusted metadata):

{
  "label": "openclaw-control-ui",
  "id": "openclaw-control-ui"
}

[Sun 2026-05-10 09:35 UTC] you did this in detached mode? If you did ill watch and tell you when we're good assistant: Yeah, it's detached — docker compose up -d. The container's already past the weight mapper crash (the patch worked 🎉), so it's in the model loading phase now. 881GB across 8 GPUs takes a hot minute.

You can tail it with:

docker compose -f /root/nvidia-meeting/docker-compose.yml logs -f

What to look for:

• Good sign: Loading safetensors checkpoint shards: X/95 progressing
• Bad sign: Another traceback, UnboundLocalError again, or KeyError
• Victory: Uvicorn running on http://0.0.0.0:8000 or Application startup complete

If it crashes again, save the last 50 lines of logs and I'll dig into whatever's next. The mapper patch covered the self_attn → attn.mla_attn renaming, but there might be other weight name mismatches I couldn't fully predict without seeing what vllm's internal model actually expects vs what modelopt gave us. user: Sender (untrusted metadata):

{
  "label": "openclaw-control-ui",
  "id": "openclaw-control-ui"
}

[Sun 2026-05-10 10:58 UTC] im in bed. check if it came up or if we neeed more patches assistant: Crashed again. Container exited. Let me get the root cause — the worker error: assistant: Same UnboundLocalError: name_mapped. The patch didn't take. Let me check if the patch script actually ran: assistant: The patch ran in the APIServer process (pid=1) but the worker processes (pid=1068-1075) are separate processes spawned later. They don't inherit the monkeypatch! The WeightsMapper is rebuilt from scratch inside each worker. I need a different approach — patch the actual file on disk inside the container, or use a volume mount to override it.

Let me go deeper. The real issue is in utils.py's _load_module → AutoWeightsLoader.load_weights → _map_name. The name_mapped is set by the mapper. If the mapper can't map a name, name_mapped is never assigned. Let me look at that function: assistant: Wait — the error is at deepseek_v4.py line 1579, which is loader.load_weights(weights, mapper=self.hf_to_vllm_mapper). But name_mapped isn't in that code. It's in the deepseek_v2.py code. Let me check if the DeepSeek V4 model is using the V2 load_weights somehow, or if the error is coming from the AutoWeightsLoader internally: assistant: There's ANOTHER load_weights in deepseek_v4.py at line 1417 — that's NOT the DeepseekV4ForCausalLM.load_weights at line 1579. This is a different class (probably DeepseekV4Model or a sub-model). Let me see: assistant: There it is. Line 1417: loaded_params.add(name_mapped). If none of the expert_mapping entries match (the continue on line 1403 skips), name_mapped is never set, and it falls through to loaded_params.add(name_mapped) which blows up.

This is the MoE expert weight loader. The expert weights from modelopt have names like model.layers.0.mlp.experts.0.gate_proj.weight but this code expects a different naming convention. The expert_mapping comes from get_expert_mapping(). Let me see what it expects: user: Sender (untrusted metadata):

{
  "label": "openclaw-control-ui",
  "id": "openclaw-control-ui"
}

[Sun 2026-05-10 13:06 UTC] what happened?