biondizzle 2e4ff6b8d4 fix: increase vLLM RPC timeout to 10 min for first-request JIT ...

First inference triggers Triton/TileLang kernel JIT compilation (2-3 min).
The default 5-min RPC timeout kills the engine. Bumped to 10 min via
VLLM_RPC_TIMEOUT_MS so the first request survives compilation.

Not ideal — would prefer to warm up the kernels during startup.
But CUDA graphs don't work well with grouped GEMMs and variable
expert counts. Will investigate vLLM warmup shape config later.

2026-05-16 06:02:11 +00:00

702 B

Raw Blame History

View Raw