Add layer-by-layer diagnostic prints (CLAWMINE_DEBUG=1, enforce-eager)

When CLAWMINE_DEBUG=1, prints amax/mean/NaN/Inf after each layer.
Must run with --enforce-eager (data-dependent prints break Dynamo).
Gated by os.environ so dead-code-eliminated during compilation.
This commit is contained in:
2026-05-18 12:51:51 +00:00
parent 2d1e9f42b1
commit 9e7639fba4
3 changed files with 84 additions and 200 deletions

View File

@@ -10,14 +10,14 @@ services:
- CUDA_LAUNCH_BLOCKING=0
- PYTHONUNBUFFERED=1
- VLLM_RPC_TIMEOUT_MS=600000
- CLAWMINE_NAN_CHECK=1
- CLAWMINE_DEBUG=1
command:
- /model
- --trust-remote-code
- --enable-expert-parallel
- --tensor-parallel-size=8
#- --enforce-eager
- --compilation-config
- --enforce-eager
#- --compilation-config
#- '{"cudagraph_mode": "NONE", "custom_ops": ["all"]}'
- '{"cudagraph_mode": "FULL_DECODE_ONLY", "custom_ops": ["all"], "cudagraph_capture_sizes": [1, 2, 4, 8], "max_cudagraph_capture_size": 8}' # This is what is runing right now
#- '{"cudagraph_mode":"FULL_AND_PIECEWISE", "custom_ops":["all"]}'