PUBLIC/gpu-turnstile
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity

Add LLM busy modes: wait (hang) or reject with Retry-After

Browse Source 
LLM_BUSY_MODE=reject answers blocked LLM requests immediately with
LLM_BUSY_STATUS (default 503, 429 works) and Retry-After, so routers
like LiteLLM can cool down and retry instead of holding a hung
connection. The default wait mode now also sends Retry-After when
LLM_WAIT_TIMEOUT expires. Document the service account (LocalSystem
default, NT SERVICE virtual-account hardening) and the Program
Files / ProgramData install layout.
This commit is contained in:
mram
2026-09-20 21:43:03 +02:00
parent e363c9e4e4
commit bacb26772a
10 changed files with 291 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
compose.yaml.example
+3
View File
@@ -18,6 +18,9 @@ services:
# UNLOAD_TIMEOUT: 60s
# JOB_TIMEOUT: 15m
# LLM_WAIT_TIMEOUT: 10m
# LLM_BUSY_MODE: reject # wait (default) hangs; reject fails fast
# LLM_BUSY_STATUS: 429 # status in reject mode (default 503)
# BUSY_RETRY_AFTER: 30 # Retry-After seconds on busy responses
# WARM_MODEL: qwen3:14b
# LOG_LEVEL: info
# LOG_FORMAT: json