Update README: haproxy proxy layer, /health probe fix, current state

haproxy 2.4 (Ubuntu 22.04) doesn't support http-request return with
payload/content-type syntax (that's 2.8+). Switch to errorfile-based
stub responses: http-request deny deny_status N + errorfile N path.

Dockerfile

@@ -1,5 +1,11 @@
 FROM lmsysorg/sglang-rocm:v0.5.10rc0-rocm700-mi30x-20260411
 
+# ---------------------------------------------------------------
+# haproxy: routes /metrics stub, proxies everything else to SGLang
+# ---------------------------------------------------------------
+RUN apt-get update && apt-get install -y --no-install-recommends haproxy \
+    && rm -rf /var/lib/apt/lists/*
+
 # ---------------------------------------------------------------
 # Replace the vllm binary with our shim
 # ---------------------------------------------------------------

README.md

@@ -8,22 +8,51 @@ The vLLM production stack handles model lifecycle, scaling, and routing — but
 
 ## How It Works
 
-Two interception paths:
+### Invocation interception
+
+Two interception paths catch however the vLLM stack tries to start the server:
 
 | What the stack calls | What happens |
 |---|---|
-| `vllm serve <model> [flags]` | Shell shim (`vllm-shim.sh`) parses args, execs `python -m sglang.launch_server` |
-| `python -m vllm.entrypoints.openai.api_server` | Python shim (shadow module on `PYTHONPATH`) does the same |
+| `vllm serve <model> [flags]` | Shell shim (`vllm-shim.sh`) replaces the `vllm` binary |
+| `python -m vllm.entrypoints.openai.api_server` | Python shim (shadow module on `PYTHONPATH`) intercepts the import |
 
-Both extract `--host` and `--port` from whatever the stack sends and forward them to SGLang. Everything else is currently hardcoded for the target model.
+Both extract `--host` and `--port` from whatever the stack sends.
+
+### haproxy proxy layer
+
+Rather than launching SGLang directly on the vLLM port, the shim runs **haproxy** on the original port and **SGLang on port+1**. This solves two critical problems:
+
+1. **`/metrics` stub** — The vLLM stack expects a Prometheus metrics endpoint at `/metrics`. SGLang doesn't serve one. haproxy intercepts `/metrics` and returns an empty 200 response instantly.
+
+2. **`/health` probe timing** — SGLang's `/health` endpoint takes ~1.001s to respond, which races the 1s k8s probe timeout and causes repeated `Startup probe failed: context deadline exceeded`. haproxy health-checks SGLang in the background (every 5s, with a 3s timeout) and responds to `/health` probes **instantly** — 200 if the backend is up, 503 if it's not. No more timeout roulette.
+
+```
+┌─────────────────────────────────────────────┐
+│  k8s probes / vLLM stack                    │
+│         │                                   │
+│         ▼                                   │
+│  haproxy (port 8000)                        │
+│    /metrics ──► 200 empty (stub)            │
+│    /health  ──► 200/503 instant (backend    │
+│                 health-checked in bg)        │
+│    /*       ──► proxy to SGLang             │
+│                       │                     │
+│                       ▼                     │
+│              SGLang (port 8001)             │
+└─────────────────────────────────────────────┘
+```
+
+haproxy 2.4 compat: uses `errorfile` + `http-request deny deny_status` for stub responses (the `http-request return` payload syntax requires haproxy 2.8+).
 
 ## Current State
 
-**PoC — hardcoded for `mistralai/Devstral-2-123B-Instruct-2512` on 8× MI300X.**
+**Running in production — `mistralai/Devstral-2-123B-Instruct-2512` on 8× MI300X.**
 
 - Model path, `--tp 8`, and `--tool-call-parser mistral` are baked into both shims
 - The Dockerfile builds on `lmsysorg/sglang-rocm` and patches a broken `aiter` build from the base image
 - MI300X tuning env vars are set (`HIP_FORCE_DEV_KERNARG`, `NCCL_MIN_NCHANNELS`, etc.)
+- All received args are logged to `/tmp/vllm-shim.log` (configurable via `VLLM_SHIM_LOG` env var)
 
 ## Building
 
@@ -31,14 +60,23 @@ Both extract `--host` and `--port` from whatever the stack sends and forward the
 docker build -t vllm-to-sglang .
 ```
 
+Or use the Jenkins pipeline:
+
+```bash
+curl -X POST "https://jenkins.sweetapi.com/job/vllm-to-sglang/buildWithParameters" \
+  -u "${JENKINS_USER}:${JENKINS_PASS}" \
+  -d "BRANCH=metrics" \
+  -d "TAG=nightly3"
+```
+
 Then use this image anywhere the vLLM stack expects its server image.
 
 ## Making It Work For Other Models
 
 Right now the model config is hardcoded in three places:
 
-- `vllm-shim.sh` — the `exec python -m sglang.launch_server` line
-- `vllm_shim_module.py` — the `os.execvp()` call
+- `vllm-shim.sh` — the `python -m sglang.launch_server` line
+- `vllm_shim_module.py` — the `subprocess.Popen()` call
 - `Dockerfile` — base image and ROCm-specific patches
 
 To adapt for a different model, change `--model-path`, `--tp`, and `--tool-call-parser` in both shim files. A future pass will make this configurable via env vars or args so you don't have to edit source.
@@ -47,6 +85,6 @@ To adapt for a different model, change `--model-path`, `--tp`, and `--tool-call-
 
 | File | Purpose |
 |---|---|
-| `Dockerfile` | Builds the image: ROCm SGLang base + aiter fix + shims + MI300X env |
-| `vllm-shim.sh` | Shell shim — replaces the `vllm` binary |
-| `vllm_shim_module.py` | Python shim — shadows `vllm.*` module imports |
+| `Dockerfile` | Builds the image: ROCm SGLang base + haproxy + shims + MI300X env |
+| `vllm-shim.sh` | Shell shim — replaces the `vllm` binary, launches SGLang + haproxy |
+| `vllm_shim_module.py` | Python shim — shadows `vllm.*` module imports, launches SGLang + haproxy |

vllm-shim.sh

@@ -5,6 +5,16 @@ set -euo pipefail
 # vLLM -> SGLang shim
 # This script replaces the vllm binary. The k8s production stack
 # calls `vllm serve <model> [flags]`, and we intercept everything.
+#
+# Architecture:
+#   haproxy on the vLLM port (front door)
+#     /metrics → 200 empty (stub)
+#     /health  → 200 if SGLang backend is up, 503 if not (instant)
+#     /*       → proxy to SGLang on port+1
+#   SGLang on port+1 (internal)
+#
+# haproxy 2.4 compat: uses errorfile for stub responses instead
+# of http-request return (which needs 2.8+ for payload syntax).
 # ============================================================
 
 echo ""
@@ -22,6 +32,20 @@ done
 echo "=========================================="
 echo ""
 
+# Log to file
+LOG_PATH="${VLLM_SHIM_LOG:-/tmp/vllm-shim.log}"
+{
+  echo "$(date -Iseconds) vLLM -> SGLang Shim (shell)"
+  echo "  Invoked as: vllm $*"
+  echo "  All arguments received:"
+  i=1
+  for arg in "$@"; do
+    echo "    [$i] $arg"
+    i=$((i + 1))
+  done
+  echo ""
+} >> "$LOG_PATH"
+
 # Defaults
 HOST="0.0.0.0"
 PORT="8000"
@@ -38,12 +62,81 @@ while [[ $# -gt 0 ]]; do
   esac
 done
 
-echo "Launching SGLang on ${HOST}:${PORT}"
+# SGLang runs one port higher; haproxy binds the original port
+SGLANG_PORT=$((PORT + 1))
+
+echo "Launching SGLang on ${HOST}:${SGLANG_PORT} (internal)"
+echo "Launching haproxy on ${HOST}:${PORT} (front door, /metrics + /health stub)"
 echo ""
 
-exec python -m sglang.launch_server \
+# Prepare error files for haproxy stub responses
+# haproxy errorfile format: HTTP/1.x status_code reason\r\nheaders\r\n\r\nbody
+mkdir -p /tmp/haproxy-errors
+printf "HTTP/1.0 200 OK\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" > /tmp/haproxy-errors/200-empty.http
+printf "HTTP/1.0 503 Service Unavailable\r\nContent-Length: 16\r\nConnection: close\r\nContent-Type: text/plain\r\n\r\nSGLang not ready" > /tmp/haproxy-errors/503-sglang.http
+
+# Write haproxy config (compatible with haproxy 2.4)
+HAPROXY_CFG="/tmp/haproxy-shim.cfg"
+cat > "$HAPROXY_CFG" <<EOF
+global
+  maxconn 4096
+
+defaults
+  mode http
+  timeout connect 5s
+  timeout client 300s
+  timeout server 300s
+
+frontend proxy
+  bind ${HOST}:${PORT}
+
+  # /metrics stub — instant 200 empty (vLLM stack expects this)
+  acl is_metrics path /metrics
+  http-request deny deny_status 200 if is_metrics
+  errorfile 200 /tmp/haproxy-errors/200-empty.http
+
+  # /health — instant response based on SGLang backend state
+  # haproxy health-checks SGLang in the background; this avoids
+  # the 1s k8s probe timeout racing SGLang's ~1.001s /health response
+  acl is_health path /health
+  acl sglang_up nbsrv(sglang) gt 0
+  http-request deny deny_status 200 if is_health sglang_up
+  http-request deny deny_status 503 if is_health
+  errorfile 503 /tmp/haproxy-errors/503-sglang.http
+
+  default_backend sglang
+
+backend sglang
+  option httpchk GET /health
+  http-check expect status 200
+  server s1 127.0.0.1:${SGLANG_PORT} check inter 5s fall 3 rise 2
+EOF
+
+echo "haproxy config written to ${HAPROXY_CFG}" >> "$LOG_PATH"
+
+# Start SGLang in the background
+python -m sglang.launch_server \
   --model-path mistralai/Devstral-2-123B-Instruct-2512 \
   --host "$HOST" \
-  --port "$PORT" \
+  --port "$SGLANG_PORT" \
   --tp 8 \
-  --tool-call-parser mistral
+  --tool-call-parser mistral &
+
+SGLANG_PID=$!
+
+# Give SGLang a moment to start before haproxy starts routing
+sleep 2
+
+# Start haproxy in the foreground (this is now PID 1 for the container)
+haproxy -f "$HAPROXY_CFG" &
+
+HAPROXY_PID=$!
+
+echo "SGLang PID: ${SGLANG_PID}, haproxy PID: ${HAPROXY_PID}" >> "$LOG_PATH"
+
+# Wait for whichever dies first — if either goes, we go
+wait -n "$SGLANG_PID" "$HAPROXY_PID"
+EXIT_CODE=$?
+echo "A process exited (code ${EXIT_CODE}), shutting down" >> "$LOG_PATH"
+kill "$SGLANG_PID" "$HAPROXY_PID" 2>/dev/null || true
+exit $EXIT_CODE

vllm_shim_module.py

@@ -1,15 +1,36 @@
 """
 vLLM -> SGLang Python shim.
 Catches `python -m vllm.entrypoints.openai.api_server` (and similar)
-and launches SGLang instead.
+and launches SGLang behind haproxy instead.
+
+Architecture:
+  haproxy on the vLLM port (front door)
+    /metrics → 200 empty (stub)
+    /health  → 200 if SGLang backend is up, 503 if not (instant)
+    /*       → proxy to SGLang on port+1
+  SGLang on port+1 (internal)
+
+haproxy 2.4 compat: uses errorfile for stub responses instead
+of http-request return (which needs 2.8+ for payload syntax).
 """
 import os
 import sys
 import subprocess
+import time
 
 def main():
     args = sys.argv[1:]
 
+    log_path = os.environ.get("VLLM_SHIM_LOG", "/tmp/vllm-shim.log")
+    import datetime
+    with open(log_path, "a") as f:
+        f.write(f"\n{datetime.datetime.now().isoformat()} vLLM -> SGLang Shim (Python module)\n")
+        f.write(f"  Invoked as: python -m {__name__} {' '.join(args)}\n")
+        f.write("  All arguments received:\n")
+        for i, arg in enumerate(args, 1):
+            f.write(f"    [{i}] {arg}\n")
+        f.write("\n")
+
     print()
     print("==========================================")
     print("  vLLM -> SGLang Shim (Python module)")
@@ -42,23 +63,99 @@ def main():
         else:
             i += 1
 
-    print(f"Launching SGLang on {host}:{port}")
+    # SGLang runs one port higher; haproxy binds the original port
+    sglang_port = str(int(port) + 1)
+
+    print(f"Launching SGLang on {host}:{sglang_port} (internal)")
+    print(f"Launching haproxy on {host}:{port} (front door, /metrics + /health stub)")
     print()
 
-    os.execvp(
-        sys.executable,
+    # Prepare error files for haproxy stub responses
+    # haproxy errorfile format: HTTP/1.x status_code reason\r\nheaders\r\n\r\nbody
+    os.makedirs("/tmp/haproxy-errors", exist_ok=True)
+    with open("/tmp/haproxy-errors/200-empty.http", "w") as f:
+        f.write("HTTP/1.0 200 OK\r\nContent-Length: 0\r\nConnection: close\r\n\r\n")
+    with open("/tmp/haproxy-errors/503-sglang.http", "w") as f:
+        f.write("HTTP/1.0 503 Service Unavailable\r\nContent-Length: 16\r\nConnection: close\r\nContent-Type: text/plain\r\n\r\nSGLang not ready")
+
+    # Write haproxy config (compatible with haproxy 2.4)
+    haproxy_cfg = "/tmp/haproxy-shim.cfg"
+    with open(haproxy_cfg, "w") as f:
+        f.write(f"""global
+  maxconn 4096
+
+defaults
+  mode http
+  timeout connect 5s
+  timeout client 300s
+  timeout server 300s
+
+frontend proxy
+  bind {host}:{port}
+
+  # /metrics stub — instant 200 empty (vLLM stack expects this)
+  acl is_metrics path /metrics
+  http-request deny deny_status 200 if is_metrics
+  errorfile 200 /tmp/haproxy-errors/200-empty.http
+
+  # /health — instant response based on SGLang backend state
+  # haproxy health-checks SGLang in the background; this avoids
+  # the 1s k8s probe timeout racing SGLang's ~1.001s /health response
+  acl is_health path /health
+  acl sglang_up nbsrv(sglang) gt 0
+  http-request deny deny_status 200 if is_health sglang_up
+  http-request deny deny_status 503 if is_health
+  errorfile 503 /tmp/haproxy-errors/503-sglang.http
+
+  default_backend sglang
+
+backend sglang
+  option httpchk GET /health
+  http-check expect status 200
+  server s1 127.0.0.1:{sglang_port} check inter 5s fall 3 rise 2
+""")
+
+    with open(log_path, "a") as f:
+        f.write(f"haproxy config written to {haproxy_cfg}\n")
+        f.write(f"SGLang port: {sglang_port}, haproxy port: {port}\n")
+
+    # Start SGLang in the background
+    sglang_proc = subprocess.Popen(
         [
             sys.executable, "-m", "sglang.launch_server",
             "--model-path", "mistralai/Devstral-2-123B-Instruct-2512",
             "--host", host,
-            "--port", port,
+            "--port", sglang_port,
             "--tp", "8",
             "--tool-call-parser", "mistral",
         ],
     )
 
+    # Give SGLang a moment before haproxy starts routing
+    time.sleep(2)
+
+    # Start haproxy in the background
+    haproxy_proc = subprocess.Popen(["haproxy", "-f", haproxy_cfg])
+
+    with open(log_path, "a") as f:
+        f.write(f"SGLang PID: {sglang_proc.pid}, haproxy PID: {haproxy_proc.pid}\n")
+
+    # Wait for whichever dies first
+    while True:
+        sglang_ret = sglang_proc.poll()
+        haproxy_ret = haproxy_proc.poll()
+        if sglang_ret is not None:
+            print(f"SGLang exited (code {sglang_ret}), shutting down")
+            haproxy_proc.terminate()
+            os._exit(sglang_ret)
+        if haproxy_ret is not None:
+            print(f"haproxy exited (code {haproxy_ret}), shutting down")
+            sglang_proc.terminate()
+            os._exit(haproxy_ret)
+        time.sleep(1)
+
 if __name__ == "__main__":
     main()
 
 # Also run if imported as a module (some invocation paths just import the file)
-main()
+main()