Add deployment runbook

RUNBOOK.md

@@ -0,0 +1,128 @@
+# SmolLM3-3B LoRA Training — Deployment Runbook
+
+## Prerequisites
+
+- [ ] GPU server deployed and accessible via SSH
+- [ ] SSH creds from Mike (host, user, key/password)
+- [ ] Docker + Docker Compose + NVIDIA Container Toolkit installed on GPU server
+- [ ] This repo at `/home/openclaw/dev/smollora`
+
+## Step 1: SSH In & Prep the Host
+
+```bash
+ssh <user>@<host>
+```
+
+Verify GPU is visible:
+```bash
+nvidia-smi
+docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
+```
+
+If Docker/nvidia toolkit missing, install before continuing.
+
+## Step 2: Create Persistent Directories
+
+```bash
+sudo mkdir -p /srv/smollora/{data,output,hf-cache}
+sudo chown -R $(whoami):$(whoami) /srv/smollora
+```
+
+## Step 3: Copy Project Files to Server
+
+From the OpenClaw box:
+```bash
+scp -r /home/openclaw/dev/smollora <user>@<host>:/tmp/smollora
+```
+
+On the GPU server:
+```bash
+cp -r /tmp/smollora ~/smollora
+cd ~/smollora
+```
+
+## Step 4: Build & Start Container
+
+```bash
+cd ~/smollora
+docker compose build
+docker compose up -d
+```
+
+Verify it's running:
+```bash
+docker compose ps
+docker compose logs --tail=20
+```
+
+## Step 5: Exec In & Kick Off Training
+
+```bash
+docker compose exec smollora bash
+```
+
+Inside the container, run the pipeline:
+```bash
+/app/run.sh
+```
+
+Watch for:
+- ✅ Datasets downloading successfully
+- ✅ Samples counted (should be thousands)
+- ✅ Model loading without OOM
+- ✅ First few training steps completing (check loss is decreasing)
+- ✅ No CUDA OOM errors in first 50 steps
+
+If data prep already ran and you just want to re-train:
+```bash
+SKIP_PREP=1 /app/run.sh
+```
+
+## Step 6: Monitor
+
+From the host (no need to stay in the container):
+```bash
+# Follow logs
+docker compose logs -f
+
+# GPU utilization
+watch -n5 nvidia-smi
+```
+
+Expected timeline on a single A100:
+- Data prep: ~10-20 min
+- Training (3 epochs, ~20-40k samples): ~2-4 hours
+- Could be longer on smaller GPUs
+
+## Step 7: Verify Output
+
+```bash
+# Check the LoRA adapter was saved
+ls -la /srv/smollora/output/final/
+# Should see: adapter_config.json, adapter_model.safetensors, tokenizer files
+```
+
+## Step 8: Notify Mike
+
+Message Mike:
+> 🎭 LoRA training is running on the GPU box. Data prep done, [N] samples, training started at [time]. Estimated completion: [est]. I'll check back periodically — will ping you if anything blows up or when it finishes.
+
+## Troubleshooting
+
+| Problem | Fix |
+|---------|-----|
+| CUDA OOM | Reduce `BATCH_SIZE` to 2, increase `GRAD_ACCUM` to 8, or reduce `MAX_LENGTH` to 2048 |
+| Dataset download fails | Check internet; can pre-download and mount into `/data` |
+| Docker can't see GPU | Install nvidia-container-toolkit: `sudo apt-get install -y nvidia-container-toolkit && sudo systemctl restart docker` |
+| Training loss not decreasing | Check LR — try `1e-4` or `5e-5`; verify labels aren't all -100 |
+| Disk full | Clean up `/srv/smollora/hf-cache` after model loads; processed data is small |
+
+## Rollback
+
+If everything goes sideways:
+```bash
+docker compose down
+rm -rf /srv/smollora/output/*
+# Fix whatever broke, then:
+docker compose up -d
+```