Commit Graph

7 Commits

Author SHA1 Message Date
bbe40ac8c0 Add middleware to strip vLLM-only params (logprobs/top_logprobs) before forwarding to SGLang
SGLang's Mistral tool-call parser rejects logprobs/top_logprobs with 422,
while vLLM accepts them. Clients like OpenClaw send these by default.

New architecture: haproxy (port N) → middleware (port N+2) → SGLang (port N+1)
The middleware is a thin FastAPI app that strips incompatible params from
chat completion request bodies and passes everything else through unchanged.
2026-04-12 18:58:37 +00:00
7fb373fdfc Add haproxy proxy: /metrics returns 200 empty, everything else proxies to SGLang
SGLang now runs on port+1, haproxy binds the original vLLM port.
haproxy serves a stub /metrics endpoint (200, empty body) and
passes all other traffic through to SGLang via raw TCP proxy.
2026-04-12 17:09:58 +00:00
513f8bb5dd we dont need to compile aiter 2026-04-12 04:16:50 +00:00
71f7fe0881 fix aiter 2026-04-12 02:56:27 +00:00
b6151ba5db fix aiter 2026-04-12 02:47:33 +00:00
4d444bebbb use a shim 2026-04-12 02:19:55 +00:00
d71248d0f6 init commit 2026-04-11 23:39:36 +00:00