biondizzle/deepseek-v4-quant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
36e134227009b4c82353f760d392a7c3ee4677aa
deepseek-v4-quant/memory/2026-05-05-starlette-error-debug.md
biondizzle f63eed5cfd Purge INT4 references — expert weights are FP4 (E2M1), not INT4 
All docs and scripts updated. Historical memory entries annotated.
2026-05-08 02:33:46 +00:00

26 KiB
Raw Blame History

Session: 2026-05-05 07:20:02 UTC

  • Session Key: agent:main:main
  • Session ID: 87ec791b-001a-4e3e-974e-e2bace2fad56
  • Source: webchat

Conversation Summary

user: A new session was started via /new or /reset. Run your Session Startup sequence - read the required files before responding to the user. Then greet the user in your configured persona, if one is provided. Be yourself - use your defined voice, mannerisms, and mood. Keep it to 1-3 sentences and ask what they want to do. If the runtime model differs from default_model in the system prompt, mention the default model. Do not mention internal steps, files, tools, or reasoning. Current time: Tuesday, May 5th, 2026 - 6:29 AM (UTC) / 2026-05-05 06:29 UTC assistant: Yo Mike. Vinny here — 6:29 AM and already a bunch of today's notes piling up. What are we working on? user: Sender (untrusted metadata):

{
  "label": "openclaw-control-ui",
  "id": "openclaw-control-ui"
}

[Tue 2026-05-05 06:30 UTC] take a gander at /home/openclaw/dev/api.vultrinference.com

particularly the file /home/openclaw/dev/api.vultrinference.com/normalizer_proxy.py

Im seeing this error spam the logs


Traceback (most recent call last):
 File "/app/venv/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 415, in run_asgi
 result = await app( # type: ignore[func-returns-value]
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 self.scope, self.receive, self.send
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 56, in __call__
 return await self.app(scope, receive, send)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/fastapi/applications.py", line 1159, in __call__
 await super().__call__(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/applications.py", line 90, in __call__
 await self.middleware_stack(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 186, in __call__
 raise exc
 File "/app/venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 164, in __call__
 await self.app(scope, receive, _send)
 File "/app/venv/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
 await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
 raise exc
 File "/app/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
 await app(scope, receive, sender)
 File "/app/venv/lib/python3.13/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
 await self.app(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/routing.py", line 660, in __call__
 await self.middleware_stack(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/routing.py", line 680, in app
 await route.handle(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/routing.py", line 276, in handle
 await self.app(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/fastapi/routing.py", line 134, in app
 await wrap_app_handling_exceptions(app, request)(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
 raise exc
 File "/app/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
 await app(scope, receive, sender)
 File "/app/venv/lib/python3.13/site-packages/fastapi/routing.py", line 120, in app
 response = await f(request)
 ^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/fastapi/routing.py", line 674, in app
 raw_response = await run_endpoint_function(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 ...<3 lines>...
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
 return await dependant.call(**values)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/normalizer_proxy.py", line 163, in chat_completions
 r = await c.post(f"{upstream_base}/v1/chat/completions", json=body, headers=headers)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1859, in post
 return await self.request(
 ^^^^^^^^^^^^^^^^^^^
 ...<13 lines>...
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1540, in request
 return await self.send(request, auth=auth, follow_redirects=follow_redirects)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1629, in send
 response = await self._send_handling_auth(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 ...<4 lines>...
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
 response = await self._send_handling_redirects(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 ...<3 lines>...
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
 response = await self._send_single_request(request)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1730, in _send_single_request
 response = await transport.handle_async_request(request)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
 with map_httpcore_exceptions():
 ~~~~~~~~~~~~~~~~~~~~~~~^^
 File "/usr/lib/python3.13/contextlib.py", line 162, in __exit__
 self.gen.throw(value)
 ~~~~~~~~~~~~~~^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
 raise mapped_exc(message) from exc
httpx.ReadTimeout
ERROR: Exception in ASGI application
Traceback (most recent call last):
 File "/app/venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
 yield
 File "/app/venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
 resp = await self._pool.handle_async_request(req)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
 raise exc from None
 File "/app/venv/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
 response = await connection.handle_async_request(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 pool_request.request
 ^^^^^^^^^^^^^^^^^^^^
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
 return await self._connection.handle_async_request(request)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpcore/_async/http11.py", line 136, in handle_async_request
 raise exc
 File "/app/venv/lib/python3.13/site-packages/httpcore/_async/http11.py", line 106, in handle_async_request
 ) = await self._receive_response_headers(**kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpcore/_async/http11.py", line 177, in _receive_response_headers
 event = await self._receive_event(timeout=timeout)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpcore/_async/http11.py", line 217, in _receive_event
 data = await self._network_stream.read(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 self.READ_NUM_BYTES, timeout=timeout
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/httpcore/_backends/anyio.py", line 32, in read
 with map_exceptions(exc_map):
 ~~~~~~~~~~~~~~^^^^^^^^^
 File "/usr/lib/python3.13/contextlib.py", line 162, in __exit__
 self.gen.throw(value)
 ~~~~~~~~~~~~~~^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
 raise to_exc(exc) from exc
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/app/venv/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 415, in run_asgi
 result = await app( # type: ignore[func-returns-value]
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 self.scope, self.receive, self.send
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 56, in __call__
 return await self.app(scope, receive, send)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/fastapi/applications.py", line 1159, in __call__
 await super().__call__(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/applications.py", line 90, in __call__
 await self.middleware_stack(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 186, in __call__
 raise exc
 File "/app/venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 164, in __call__
 await self.app(scope, receive, _send)
 File "/app/venv/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
 await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
 raise exc
 File "/app/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
 await app(scope, receive, sender)
 File "/app/venv/lib/python3.13/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
 await self.app(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/routing.py", line 660, in __call__
 await self.middleware_stack(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/routing.py", line 680, in app
 await route.handle(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/routing.py", line 276, in handle
 await self.app(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/fastapi/routing.py", line 134, in app
 await wrap_app_handling_exceptions(app, request)(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
 raise exc
 File "/app/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
 await app(scope, receive, sender)
 File "/app/venv/lib/python3.13/site-packages/fastapi/routing.py", line 120, in app
 response = await f(request)
 ^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/fastapi/routing.py", line 674, in app
 raw_response = await run_endpoint_function(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 ...<3 lines>...
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
 return await dependant.call(**values)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/normalizer_proxy.py", line 163, in chat_completions
 r = await c.post(f"{upstream_base}/v1/chat/completions", json=body, headers=headers)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1859, in post
 return await self.request(
 ^^^^^^^^^^^^^^^^^^^
 ...<13 lines>...
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1540, in request
 return await self.send(request, auth=auth, follow_redirects=follow_redirects)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1629, in send
 response = await self._send_handling_auth(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 ...<4 lines>...
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
 response = await self._send_handling_redirects(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 ...<3 lines>...
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
 response = await self._send_single_request(request)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1730, in _send_single_request
 response = await transport.handle_async_request(request)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
 with map_httpcore_exceptions():
 ~~~~~~~~~~~~~~~~~~~~~~~^^
 File "/usr/lib/python3.13/contextlib.py", line 162, in __exit__
 self.gen.throw(value)
 ~~~~~~~~~~~~~~^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
 raise mapped_exc(message) from exc
httpx.ReadTimeout
ERROR: Exception in ASGI application
Traceback (most recent call last):
 File "/app/venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
 yield
 File "/app/venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
 resp = await self._pool.handle_async_request(req)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
 raise exc from None
 File "/app/venv/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
 response = await connection.handle_async_request(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 pool_request.request
 ^^^^^^^^^^^^^^^^^^^^
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
 return await self._connection.handle_async_request(request)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpcore/_async/http11.py", line 136, in handle_async_request
 raise exc
 File "/app/venv/lib/python3.13/site-packages/httpcore/_async/http11.py", line 106, in handle_async_request
 ) = await self._receive_response_headers(**kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpcore/_async/http11.py", line 177, in _receive_response_headers
 event = await self._receive_event(timeout=timeout)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpcore/_async/http11.py", line 217, in _receive_event
 data = await self._network_stream.read(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 self.READ_NUM_BYTES, timeout=timeout
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/httpcore/_backends/anyio.py", line 32, in read
 with map_exceptions(exc_map):
 ~~~~~~~~~~~~~~^^^^^^^^^
 File "/usr/lib/python3.13/contextlib.py", line 162, in __exit__
 self.gen.throw(value)
 ~~~~~~~~~~~~~~^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
 raise to_exc(exc) from exc
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/app/venv/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 415, in run_asgi
 result = await app( # type: ignore[func-returns-value]
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 self.scope, self.receive, self.send
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 56, in __call__
 return await self.app(scope, receive, send)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/fastapi/applications.py", line 1159, in __call__
 await super().__call__(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/applications.py", line 90, in __call__
 await self.middleware_stack(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 186, in __call__
 raise exc
 File "/app/venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 164, in __call__
 await self.app(scope, receive, _send)
 File "/app/venv/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
 await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
 raise exc
 File "/app/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
 await app(scope, receive, sender)
 File "/app/venv/lib/python3.13/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
 await self.app(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/routing.py", line 660, in __call__
 await self.middleware_stack(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/routing.py", line 680, in app
 await route.handle(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/routing.py", line 276, in handle
 await self.app(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/fastapi/routing.py", line 134, in app
 await wrap_app_handling_exceptions(app, request)(scope, receive, send)
 File "/app/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
 raise exc
 File "/app/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
 await app(scope, receive, sender)
 File "/app/venv/lib/python3.13/site-packages/fastapi/routing.py", line 120, in app
 response = await f(request)
 ^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/fastapi/routing.py", line 674, in app
 raw_response = await run_endpoint_function(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 ...<3 lines>...
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
 return await dependant.call(**values)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/normalizer_proxy.py", line 163, in chat_completions
 r = await c.post(f"{upstream_base}/v1/chat/completions", json=body, headers=headers)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1859, in post
 return await self.request(
 ^^^^^^^^^^^^^^^^^^^
 ...<13 lines>...
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1540, in request
 return await self.send(request, auth=auth, follow_redirects=follow_redirects)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1629, in send
 response = await self._send_handling_auth(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 ...<4 lines>...
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
 response = await self._send_handling_redirects(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 ...<3 lines>...
 )
 ^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
 response = await self._send_single_request(request)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_client.py", line 1730, in _send_single_request
 response = await transport.handle_async_request(request)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
 with map_httpcore_exceptions():
 ~~~~~~~~~~~~~~~~~~~~~~~^^
 File "/usr/lib/python3.13/contextlib.py", line 162, in __exit__
 self.gen.throw(value)
 ~~~~~~~~~~~~~~^^^^^^^
 File "/app/venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
 raise mapped_exc(message) from exc
httpx.ReadTimeout
root@inference-api-57cffd9545-54k2t:/app#

assistant: Classic — httpx.ReadTimeout on line 163 (non-streaming path) with zero error handling, so it just barfs a full traceback into the logs instead of returning a clean 504. Same issue would hit the streaming path's initial connection too.

The fix: wrap both httpx calls in try/except for timeout/connection errors and return proper JSON error responses instead of letting FastAPI puke stack traces. I'll also bump the timeout a bit for good measure. assistant: Also need to handle timeouts in the passthrough route: assistant: Quick sanity check: assistant: Here's what I changed:

  1. Caught httpx.ReadTimeout and httpx.ConnectTimeout on the non-streaming path (line 163 area) — returns a clean 504 with a JSON error body instead of a full traceback vomit in the logs
  2. Caught httpx.ConnectError too — returns 502
  3. Streaming path already had a try/except but now uses the same timeout constant
  4. Passthrough route got the same treatment
  5. Bumped timeout from 300s → 600s — some models are just slow, 5 minutes wasn't cutting it

The root cause is that the upstream vLLM/sglang backend is taking longer than 300s to respond on some requests. The timeout itself might still happen (slow model, queue backlog, whatever), but now instead of 50 lines of traceback spam you get a single-line ⚠ Upstream read timeout log entry and the client gets a proper 504 response. user: Sender (untrusted metadata):

{
  "label": "openclaw-control-ui",
  "id": "openclaw-control-ui"
}

[Tue 2026-05-05 06:32 UTC] while we're in here. How many simultaneous requests can fastapi handle? DOn't know if there's a limit per worker assistant: By default, uvicorn.run() with no workers arg starts 1 process, 1 event loop — but since it's async, it can handle many concurrent requests on that single loop. The bottleneck isn't FastAPI itself, it's the httpx.AsyncClient usage pattern.

Looking at your code — you're creating a new AsyncClient per request:

async with httpx.AsyncClient(timeout=...) as c:
    r = await c.post(...)

Each AsyncClient creates its own connection pool. So there's no connection reuse between requests — every request opens a fresh TCP connection to upstream. That's wasteful but doesn't cap concurrency.

The real limits:

  • No worker limit — you're running single-worker. One process, one uvicorn event loop. That's fine for async I/O-bound work; the loop can juggle thousands of coroutines.
  • No connection pool limit — since you spin up a new client each time, each one defaults to max_connections=100, max_keepalive_connections=20. But since they're ephemeral, it's effectively unlimited (and messy).
  • uvicorn --workers — not set, so 1 worker. If you wanted multi-process parallelism, you'd pass workers=N to uvicorn.run() or the CLI.

The move: Create a shared AsyncClient at app startup with tuned pool limits. Way more efficient, and lets you control concurrency explicitly. Want me to refactor to use an app-lifespan client? Something like:

@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.client = httpx.AsyncClient(
        timeout=600.0,
        limits=httpx.Limits(max_connections=200, max_keepalive_connections=50),
    )
    yield
    await app.state.client.aclose()

Then all routes use request.app.state.client — connection reuse, proper pooling, no per-request overhead. And you can tune max_connections if you need a ceiling.

Want me to make that refactor? user: Sender (untrusted metadata):

{
  "label": "openclaw-control-ui",
  "id": "openclaw-control-ui"
}

[Tue 2026-05-05 06:33 UTC] Well I spin it up via supervisor. the config is normalizer_proxy.conf assistant: There it is — --workers 1. So you've got a single uvicorn process handling everything.

Couple thoughts:

  • For your workload (pure async I/O proxy), 1 worker is probably fine. The event loop can juggle hundreds of concurrent upstream requests without breaking a sweat. You're not doing CPU-bound work — just shuttling JSON around.
  • Bumping to 2-4 workers would give you some resilience if one worker gets stuck or GC-paused, and lets you roll restarts without downtime.
  • The bigger win is the shared AsyncClient refactor. Right now every request creates and tears down a connection pool. With a shared client you get connection keepalive reuse, which means less TCP handshake overhead and faster responses under load.

Want me to do both — bump workers to something reasonable (like 4) and refactor to a shared AsyncClient? With multiple workers you'd just instantiate the client per-worker via the lifespan, which works cleanly. user: Sender (untrusted metadata):

{
  "label": "openclaw-control-ui",
  "id": "openclaw-control-ui"
}

[Tue 2026-05-05 06:35 UTC] I dont care about overhead or speed. I only care about not being limited. This container has alot of replicas on big machines, My issue is I see Idle timeout reached for "http://127.0.0.1:8100/v1/chat/completions". in my sentry logs and wasnt sure if thats the error we just fixed or something else assistant: That's the same root cause. Here's the chain:

  1. Normalizer's httpx client (was 300s timeout) waits for vLLM to respond
  2. vLLM is slow, takes longer than the fronting proxy's idle timeout (nginx/envoy/whatever)
  3. Fronting proxy gives up → Idle timeout reached in Sentry
  4. Meanwhile the normalizer's httpx call eventually times out too → unhandled ReadTimeout → traceback spam (the error you showed me)

So your fronting proxy is timing out before the normalizer even gets a chance to respond. The fix I just made stops the traceback spam and returns a clean 504, but the idle timeout is still gonna happen if the proxy's timeout is shorter than however long vLLM takes.

Two things to check:

  1. What's your fronting proxy's idle/keepalive timeout? If it's something like 60s or 120s, that's your ceiling — no matter what httpx timeout you set, the proxy will cut the connection first.
  2. For streaming requests this shouldn't be an issue since SSE chunks keep the connection alive. The idle timeout is only hitting you on non-streaming requests where the normalizer holds the connection open waiting for the full response.

What's sitting in front of this — nginx, envoy, an ingress controller? And do you know the idle timeout on it?

Reference in New Issue View Git Blame Copy Permalink