PUBLIC/gpu-turnstile
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Add LLM busy modes: wait (hang) or reject with Retry-After

Browse Source 
LLM_BUSY_MODE=reject answers blocked LLM requests immediately with
LLM_BUSY_STATUS (default 503, 429 works) and Retry-After, so routers
like LiteLLM can cool down and retry instead of holding a hung
connection. The default wait mode now also sends Retry-After when
LLM_WAIT_TIMEOUT expires. Document the service account (LocalSystem
default, NT SERVICE virtual-account hardening) and the Program
Files / ProgramData install layout.
This commit is contained in:
mram
2026-09-20 21:43:03 +02:00
parent e363c9e4e4
commit bacb26772a
10 changed files with 291 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+15 -2
View File
@@ -18,7 +18,10 @@ Open WebUI / n8n ────► :8188 ───┘
- LLM endpoints (`/api/generate`, `/api/chat`, `/api/embed`, `/v1/*`) take the
LLM lock: concurrent requests allowed, but blocked while an image job is
active or waiting (image priority).
active or waiting (image priority). Blocked requests either hang until the
lock is free (`LLM_BUSY_MODE=wait`, default) or fail immediately with 503
(or 429) + `Retry-After` (`LLM_BUSY_MODE=reject`) — the latter lets routers
like LiteLLM cool down and retry instead of holding a hung connection.
- `POST /prompt` on the ComfyUI listener takes the image lock: new LLM
requests block, in-flight LLMs drain, Ollama models are unloaded, the prompt
is forwarded, and the lock is held until the job finishes and ComfyUI frees
@@ -42,7 +45,10 @@ override file values. Invalid values fail at startup.
| `COMFY_URL` | `http://127.0.0.1:8189` | ComfyUI upstream |
| `UNLOAD_TIMEOUT` | `60s` | Wait for Ollama to unload before an image job |
| `JOB_TIMEOUT` | `15m` | Wait for a ComfyUI job to finish |
| `LLM_WAIT_TIMEOUT` | `10m` | Max lock wait for an LLM request before 503 |
| `LLM_WAIT_TIMEOUT` | `10m` | Max lock wait for an LLM request before 503 (wait mode) |
| `LLM_BUSY_MODE` | `wait` | `wait` = hold blocked LLM requests; `reject` = fail them immediately |
| `LLM_BUSY_STATUS` | `503` | HTTP status for rejected LLM requests in reject mode (400–599, e.g. 429) |
| `BUSY_RETRY_AFTER` | `30` | Seconds sent as `Retry-After` on busy responses (both modes) |
| `WARM_MODEL` | _(empty)_ | Model to reload after an image job (off by default) |
| `LOGLEVEL` | `warn` | `info` logs every request (colored arrows in text mode), `debug` adds lock transitions. `LOG_LEVEL` works as an alias |
| `LOG_FORMAT` | `text` | `json` for structured JSON logs |
@@ -96,6 +102,13 @@ gpu-turnstile.exe service remove
The service uses the config file (services have no convenient
environment); set `LOG_FILE` in it since there is no console.
Suggested layout: `C:\Program Files\gpu-turnstile\` for the exe and
`gpu-turnstile.env`, logs under `C:\ProgramData\gpu-turnstile\` via
`LOG_FILE`. The service runs as `LocalSystem` by default, which can write
the install directory for self-updates. For least privilege, run it as the
virtual account `NT SERVICE\gpu-turnstile` and grant write access to just
those two directories.
**Auto-update is on by default**: the binary checks the repo's latest
release on startup and every `UPDATE_INTERVAL`, verifies the Ed25519
signature of the download against the public key embedded at build time,