From 72bf750a0b4ee1da976bcb166566b275e32479b9 Mon Sep 17 00:00:00 2001
From: biondizzle <biondizzle@gmail.com>
Date: Sat, 16 May 2026 08:07:44 +0000
Subject: [PATCH] =?UTF-8?q?fix:=20revert=20to=20eager=20mode=20=E2=80=94?=
 =?UTF-8?q?=20CUDA=20graphs=20OOM=20with=20175GB=20model?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

CUDA graph capture needs extra memory on top of the model weights.
With 175GB model on 178GB GPUs, there's no room.

Going back to --enforce-eager with 10-min RPC timeout. The first
inference request will be slow (2-3 min JIT compilation) but won't
crash. Subsequent requests are fast.

CUDA graph mode requires either more GPU memory or a smaller model.
---
 docker-compose.yml | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/docker-compose.yml b/docker-compose.yml
index 146862ef..11cf955f 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -15,8 +15,7 @@ services:
       - --trust-remote-code
       - --enable-expert-parallel
       - --tensor-parallel-size=8
-      - --compilation-config
-      - '{"cudagraph_mode":"FULL_AND_PIECEWISE", "custom_ops":["all"]}'
+      - --enforce-eager
       - --tokenizer-mode=deepseek_v4
       - --host=0.0.0.0
       - --port=8000