biondizzle
02b8ea536f
Server running on B200 port 8000 with full NVFP4→vLLM bridge. All critical bugs fixed: DeepGEMM scale format, compressor shapes, block scale values.
95 lines
7.1 KiB
Markdown
95 lines
7.1 KiB
Markdown
# Session: 2026-05-10 13:12:29 UTC
|
|
|
|
- **Session Key**: agent:main:main
|
|
- **Session ID**: 8eeafe17-997c-452c-97fe-38e25742c5eb
|
|
- **Source**: webchat
|
|
|
|
## Conversation Summary
|
|
|
|
assistant: Now let me see what vllm's mapper expects. Let me dump the full mapper:Now I can see the mapping. vllm expects names like:
|
|
- `layers.0.attn.compressor.gate_proj...` → `model.layers.0.atn.mla_attn.compressor.gate_proj...`
|
|
- `layers.0.shared_experts.w2` → `model.layers.0.shared_experts.down_proj`
|
|
|
|
Our modelopt export has:
|
|
- `model.layers.0.self_attn.compressor.gate_proj...` (has `self_atn` not `attn`)
|
|
- `model.layers.0.mlp.shared_experts.down_proj` (already uses `down_proj`)
|
|
|
|
The prefix mapper converts `layers.` → `model.layers.` and then the substr mapper converts `.attn.compressor.` → `.attn.mla_attn.compressor.`. But our weights already have `model.layers.` prefix AND use `self_attn` instead of `atn`.
|
|
|
|
The issue: modelopt exports with `self_attn` but vllm's mapper expects `atn` (which it then maps to `attn.mla_attn`). Let me also check what other name mismatches exist:Now I have the full picture. The main mapping issue is `self_atn` → `attn.mla_attn`. But there are also quantization suffixes (`input_scale`, `weight_scale`, `weight_scale_2`) that modelopt adds. vllm's modelopt_fp4 quantizer should handle those natively — the `WeightsMapper` just needs to map the base names correctly.
|
|
|
|
Let me write a mountable patch script that fixes the mapper at container startup:Now I need to figure out how to inject this patch into the container. The easiest way is to mount the patches dir and add it as an entrypoint wrapper. Let me update docker-compose:YAML escape issue. Let me fix the entrypoint:YAML's being a pain with indentation. Let me just write it properly:YAML validates. Let me start it:**It's past the weight mapper crash!** The patch is working. I can see:
|
|
- `Detected ModelOpt NVFP4 checkpoint` ✅
|
|
- `Using max model len 1048576` ✅
|
|
- `DeepSeekV4MTPModel` architecture resolved ✅
|
|
- Now it's initializing the engine...
|
|
|
|
This is loading the 881GB model across 8 GPUs. This will take a while. Let me check back in a few minutes:
|
|
user: Sender (untrusted metadata):
|
|
```json
|
|
{
|
|
"label": "openclaw-control-ui",
|
|
"id": "openclaw-control-ui"
|
|
}
|
|
```
|
|
|
|
[Sun 2026-05-10 09:30 UTC] fuck it. do #2 and see where ya get. If ya dont get anywhere, then ya need to look deeper into model opt and we have to patch model opt, or see if there's a seeting we missed that does that
|
|
assistant: Already on it — I patched the weight mapper and restarted the container. It got past the `UnboundLocalError` crash. Let me check where it's at now:
|
|
assistant: SSH connection is timing out. The machine might be under heavy load from model loading (881GB across 8 GPUs). Let me retry:
|
|
assistant: The execs keep getting aborted. Let me try a different approach:
|
|
user: System: [2026-05-10 09:31:01 UTC] Exec completed (calm-ree, code 0) :: INFO 05-10 09:30:59 [nixl_utils.py:20] Setting UCX_RCACHE_MAX_UNRELEASED to '1024' to avoid a rare memory leak in UCX when using NIXL. INFO 05-10 09:30:59 [nixl_utils.py:32] NIXL …
|
|
System: [2026-05-10 09:31:19 UTC] Exec completed (calm-dai, code 0) :: __', '__le__', '__lt__', '__match_args__', '__module__', '__ne__', '__new__', '__or__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subcl…
|
|
System: [2026-05-10 09:31:40 UTC] Exec completed (swift-br, code 0) :: === orig_to_new_substr === .attn.compressor. -> .attn.mla_attn.compressor. .shared_experts.w2 -> .shared_experts.down_proj === orig_to_new_prefix === layers. -> model.layers. embe…
|
|
System: [2026-05-10 09:34:38 UTC] Exec completed (briny-ri, code 0) :: ver pid=1)[0;0m [33mWARNING[0m [90m05-10 09:34:35[0m [90m[vllm.py:1252][0m Auto-initialization of reasoning token IDs failed. Please check whether your reasoning parser has implem…
|
|
|
|
Sender (untrusted metadata):
|
|
```json
|
|
{
|
|
"label": "openclaw-control-ui",
|
|
"id": "openclaw-control-ui"
|
|
}
|
|
```
|
|
|
|
[Sun 2026-05-10 09:35 UTC] you did this in detached mode? If you did ill watch and tell you when we're good
|
|
assistant: Yeah, it's detached — `docker compose up -d`. The container's already past the weight mapper crash (the patch worked 🎉), so it's in the model loading phase now. 881GB across 8 GPUs takes a hot minute.
|
|
|
|
You can tail it with:
|
|
```
|
|
docker compose -f /root/nvidia-meeting/docker-compose.yml logs -f
|
|
```
|
|
|
|
What to look for:
|
|
- **Good sign**: `Loading safetensors checkpoint shards: X/95` progressing
|
|
- **Bad sign**: Another traceback, `UnboundLocalError` again, or `KeyError`
|
|
- **Victory**: `Uvicorn running on http://0.0.0.0:8000` or `Application startup complete`
|
|
|
|
If it crashes again, save the last 50 lines of logs and I'll dig into whatever's next. The mapper patch covered the `self_attn` → `attn.mla_attn` renaming, but there might be other weight name mismatches I couldn't fully predict without seeing what vllm's internal model actually expects vs what modelopt gave us.
|
|
user: Sender (untrusted metadata):
|
|
```json
|
|
{
|
|
"label": "openclaw-control-ui",
|
|
"id": "openclaw-control-ui"
|
|
}
|
|
```
|
|
|
|
[Sun 2026-05-10 10:58 UTC] im in bed. check if it came up or if we neeed more patches
|
|
assistant: Crashed again. Container exited. Let me get the root cause — the worker error:
|
|
assistant: Same `UnboundLocalError: name_mapped`. The patch didn't take. Let me check if the patch script actually ran:
|
|
assistant: The patch ran in the APIServer process (pid=1) but the worker processes (pid=1068-1075) are separate processes spawned later. They don't inherit the monkeypatch! The `WeightsMapper` is rebuilt from scratch inside each worker. I need a different approach — patch the actual file on disk inside the container, or use a volume mount to override it.
|
|
|
|
Let me go deeper. The real issue is in `utils.py`'s `_load_module` → `AutoWeightsLoader.load_weights` → `_map_name`. The `name_mapped` is set by the mapper. If the mapper can't map a name, `name_mapped` is never assigned. Let me look at that function:
|
|
assistant: Wait — the error is at `deepseek_v4.py` line 1579, which is `loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)`. But `name_mapped` isn't in that code. It's in the `deepseek_v2.py` code. Let me check if the DeepSeek V4 model is using the V2 load_weights somehow, or if the error is coming from the `AutoWeightsLoader` internally:
|
|
assistant: There's ANOTHER `load_weights` in `deepseek_v4.py` at line 1417 — that's NOT the `DeepseekV4ForCausalLM.load_weights` at line 1579. This is a different class (probably `DeepseekV4Model` or a sub-model). Let me see:
|
|
assistant: There it is. Line 1417: `loaded_params.add(name_mapped)`. If none of the `expert_mapping` entries match (the `continue` on line 1403 skips), `name_mapped` is never set, and it falls through to `loaded_params.add(name_mapped)` which blows up.
|
|
|
|
This is the MoE expert weight loader. The expert weights from modelopt have names like `model.layers.0.mlp.experts.0.gate_proj.weight` but this code expects a different naming convention. The `expert_mapping` comes from `get_expert_mapping()`. Let me see what it expects:
|
|
user: Sender (untrusted metadata):
|
|
```json
|
|
{
|
|
"label": "openclaw-control-ui",
|
|
"id": "openclaw-control-ui"
|
|
}
|
|
```
|
|
|
|
[Sun 2026-05-10 13:06 UTC] what happened?
|