README: document test harness gotchas (timeout arg, stale procs, screen names)

README.md

@@ -158,30 +158,35 @@ Both harnesses follow the same discipline:
 4. **Run in screen** — survives SSH drops, has a timeout
 5. **One test at a time** — no parallel launches, ever
 
-### Python test (one command)
+### Python test
 
 ```bash
 # From local machine — auto-pushes, runs, polls, dumps log
+# DEFAULT timeout: 600s (10 min). Override with all 4 args:
+~/.openclaw/workspace/fire_b200_test <test_file> [screen_name] [log_file] [timeout_sec]
+
+# Examples:
 ~/.openclaw/workspace/fire_b200_test tests/unit/test_fmha_v3_stage_c.py
+~/.openclaw/workspace/fire_b200_test tests/unit/test_degeneration_2_mhc_falsify.py kernel-test /tmp/kernel-test.log 1800
 ```
 
-### CUDA test (one command)
+### CUDA test
 
 ```bash
 # From local machine — compiles with nvcc, runs, polls, dumps log
-# Default timeout: 60s. Pass a second arg for custom timeout.
 ~/.openclaw/workspace/fire_b200_cuda_test tests/unit/test_fmha_sm100_standalone.cu
-~/.openclaw/workspace/fire_b200_cuda_test tests/unit/test_tmem_minimal.cu 30
+~/.openclaw/workspace/fire_b200_cuda_test tests/unit/test_tmem_minimal.cu 30   # custom timeout
 ```
 
-### Check on a running CUDA test
+### Check on a running test
 
 ```bash
-# Show current log + screen status
+# Check CUDA test log + screen status
 ~/.openclaw/workspace/check_b200_cuda
+~/.openclaw/workspace/check_b200_cuda kill   # kill a hung test
 
-# Kill a hung test + show the log
-~/.openclaw/workspace/check_b200_cuda kill
+# Check Python test — SSH to B200 and tail the log:
+ssh root@<B200> tail -f /tmp/kernel-test.log
 ```
 
 ### Manual B200 cycle (emergency only)
@@ -193,7 +198,44 @@ bash tests/run_test.sh tests/unit/test_<...>.py
 bash tests/check_log.sh
 ```
 
-`run_test.sh` kills any prior `kernel-test` screen (with SIGKILL on stuck GPU procs), deletes the old log, starts a fresh `screen -dmS kernel-test`, and logs to `/tmp/kernel-test.log`.
+### ⚠️ Test harness gotchas (READ THIS — cost real time)
+
+1. **The timeout is the 4th argument, not the 2nd.**
+   - WRONG: `fire_b200_test test.py 1800` ← this makes `1800` the SCREEN NAME
+   - RIGHT: `fire_b200_test test.py kernel-test /tmp/kernel-test.log 1800`
+   - When you pass just a number as the 2nd arg, the screen gets a numeric name
+     and the harness can't kill the old `kernel-test` screen on the next run.
+   - **Always pass all 4 args** when you need a custom timeout.
+
+2. **After a timeout, the harness kills the screen but NOT the GPU process.**
+   - The `timeout` command inside screen kills the shell, but CUDA processes survive.
+   - Before re-running, check: `ssh root@<B200> nvidia-smi --query-compute-apps=pid --format=csv,noheader`
+   - Kill stale processes: `kill -9 <pid>` for each GPU process listed
+   - Or: `for pid in $(nvidia-smi --query-compute-apps=pid --format=csv,noheader); do kill -9 $pid; done`
+
+3. **After an OOM or crash, stale GPU processes WILL be left behind.**
+   - Always check `nvidia-smi` before running a new test after a failure.
+   - The harness kills `python.*test_` and `python.*inference` procs, but if the
+     process name doesn't match the pattern, it survives.
+
+4. **Single-shot tests MUST use the harness too.**
+   - `single_shot_inference.py` is NOT a unit test, but it MUST be run via the harness.
+   - WRONG: ssh to B200 and run `python single_shot_inference.py` directly
+   - RIGHT: `fire_b200_test single_shot_inference.py kernel-test /tmp/kernel-test.log 1800 -- --max-tokens 512`
+   - Extra args after `--` are passed to the Python script.
+   - If the harness can't handle your use case, FIX THE HARNESS, don't bypass it.
+
+5. **Weight loading + CuTeDSL compilation takes 5-10 minutes.**
+   - First FMHA call triggers JIT compile of CuTeDSL kernels.
+   - This is EXPECTED. Do NOT kill the process because it "seems stuck".
+   - Use 1800s (30 min) timeout for full-model tests.
+
+6. **The screen name must match between runs.**
+   - The harness kills the old screen by name. If you used a different name last time,
+     the old screen survives and holds GPU memory.
+   - Always use `kernel-test` for Python tests and `cuda-test` for CUDA tests.
+   - If you accidentally used a numeric screen name, clean up manually:
+     `ssh root@<B200> screen -S <wrong_name> -X quit`
 
 ### Environment