diff --git a/README.md b/README.md index f8437c2a..c56a8dde 100644 --- a/README.md +++ b/README.md @@ -158,30 +158,35 @@ Both harnesses follow the same discipline: 4. **Run in screen** — survives SSH drops, has a timeout 5. **One test at a time** — no parallel launches, ever -### Python test (one command) +### Python test ```bash # From local machine — auto-pushes, runs, polls, dumps log +# DEFAULT timeout: 600s (10 min). Override with all 4 args: +~/.openclaw/workspace/fire_b200_test [screen_name] [log_file] [timeout_sec] + +# Examples: ~/.openclaw/workspace/fire_b200_test tests/unit/test_fmha_v3_stage_c.py +~/.openclaw/workspace/fire_b200_test tests/unit/test_degeneration_2_mhc_falsify.py kernel-test /tmp/kernel-test.log 1800 ``` -### CUDA test (one command) +### CUDA test ```bash # From local machine — compiles with nvcc, runs, polls, dumps log -# Default timeout: 60s. Pass a second arg for custom timeout. ~/.openclaw/workspace/fire_b200_cuda_test tests/unit/test_fmha_sm100_standalone.cu -~/.openclaw/workspace/fire_b200_cuda_test tests/unit/test_tmem_minimal.cu 30 +~/.openclaw/workspace/fire_b200_cuda_test tests/unit/test_tmem_minimal.cu 30 # custom timeout ``` -### Check on a running CUDA test +### Check on a running test ```bash -# Show current log + screen status +# Check CUDA test log + screen status ~/.openclaw/workspace/check_b200_cuda +~/.openclaw/workspace/check_b200_cuda kill # kill a hung test -# Kill a hung test + show the log -~/.openclaw/workspace/check_b200_cuda kill +# Check Python test — SSH to B200 and tail the log: +ssh root@ tail -f /tmp/kernel-test.log ``` ### Manual B200 cycle (emergency only) @@ -193,7 +198,44 @@ bash tests/run_test.sh tests/unit/test_<...>.py bash tests/check_log.sh ``` -`run_test.sh` kills any prior `kernel-test` screen (with SIGKILL on stuck GPU procs), deletes the old log, starts a fresh `screen -dmS kernel-test`, and logs to `/tmp/kernel-test.log`. +### ⚠️ Test harness gotchas (READ THIS — cost real time) + +1. **The timeout is the 4th argument, not the 2nd.** + - WRONG: `fire_b200_test test.py 1800` ← this makes `1800` the SCREEN NAME + - RIGHT: `fire_b200_test test.py kernel-test /tmp/kernel-test.log 1800` + - When you pass just a number as the 2nd arg, the screen gets a numeric name + and the harness can't kill the old `kernel-test` screen on the next run. + - **Always pass all 4 args** when you need a custom timeout. + +2. **After a timeout, the harness kills the screen but NOT the GPU process.** + - The `timeout` command inside screen kills the shell, but CUDA processes survive. + - Before re-running, check: `ssh root@ nvidia-smi --query-compute-apps=pid --format=csv,noheader` + - Kill stale processes: `kill -9 ` for each GPU process listed + - Or: `for pid in $(nvidia-smi --query-compute-apps=pid --format=csv,noheader); do kill -9 $pid; done` + +3. **After an OOM or crash, stale GPU processes WILL be left behind.** + - Always check `nvidia-smi` before running a new test after a failure. + - The harness kills `python.*test_` and `python.*inference` procs, but if the + process name doesn't match the pattern, it survives. + +4. **Single-shot tests MUST use the harness too.** + - `single_shot_inference.py` is NOT a unit test, but it MUST be run via the harness. + - WRONG: ssh to B200 and run `python single_shot_inference.py` directly + - RIGHT: `fire_b200_test single_shot_inference.py kernel-test /tmp/kernel-test.log 1800 -- --max-tokens 512` + - Extra args after `--` are passed to the Python script. + - If the harness can't handle your use case, FIX THE HARNESS, don't bypass it. + +5. **Weight loading + CuTeDSL compilation takes 5-10 minutes.** + - First FMHA call triggers JIT compile of CuTeDSL kernels. + - This is EXPECTED. Do NOT kill the process because it "seems stuck". + - Use 1800s (30 min) timeout for full-model tests. + +6. **The screen name must match between runs.** + - The harness kills the old screen by name. If you used a different name last time, + the old screen survives and holds GPU memory. + - Always use `kernel-test` for Python tests and `cuda-test` for CUDA tests. + - If you accidentally used a numeric screen name, clean up manually: + `ssh root@ screen -S -X quit` ### Environment