PUBLIC/gpu-turnstile
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Add LLM busy modes: wait (hang) or reject with Retry-After

Browse Source 
LLM_BUSY_MODE=reject answers blocked LLM requests immediately with
LLM_BUSY_STATUS (default 503, 429 works) and Retry-After, so routers
like LiteLLM can cool down and retry instead of holding a hung
connection. The default wait mode now also sends Retry-After when
LLM_WAIT_TIMEOUT expires. Document the service account (LocalSystem
default, NT SERVICE virtual-account hardening) and the Program
Files / ProgramData install layout.
This commit is contained in:
mram
2026-09-20 21:43:03 +02:00
parent e363c9e4e4
commit bacb26772a
10 changed files with 291 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
SPEC.md
+18 -1
View File
@@ -51,6 +51,11 @@ are LLM requests and the single "writer" is an image job):
`image` **or while an image job is waiting**. Then state := `llm`, n++.
On completion (response fully written, including streamed bodies, or client
disconnect) n--; if n == 0 state := `idle`.
`LLM_BUSY_MODE` selects what a blocked LLM request sees: `wait` (default)
hangs until the lock is free or `LLM_WAIT_TIMEOUT` expires (then 503 +
`Retry-After`); `reject` answers immediately with `LLM_BUSY_STATUS`
(default 503; 429 works too) + `Retry-After: BUSY_RETRY_AFTER`, which
routers like LiteLLM honor for cooldowns/retries.
- **Image job**: `AcquireImage()` marks "image pending" (so no new LLM
requests start), waits until n == 0, sets state := `image`. Released after
the ComfyUI job finished and models were freed.
@@ -116,7 +121,10 @@ override file values. A missing file is fine; a malformed one is fatal.
| `COMFY_URL` | `http://127.0.0.1:8189` | upstream |
| `UNLOAD_TIMEOUT` | `60s` | wait for Ollama to unload |
| `JOB_TIMEOUT` | `15m` | wait for ComfyUI job |
| `LLM_WAIT_TIMEOUT` | `10m` | max time an LLM request waits for the lock before 503 |
| `LLM_WAIT_TIMEOUT` | `10m` | max time an LLM request waits for the lock before 503 (wait mode) |
| `LLM_BUSY_MODE` | `wait` | `wait` = hold blocked LLM requests; `reject` = fail them immediately |
| `LLM_BUSY_STATUS` | `503` | HTTP status for rejected LLM requests in reject mode (400–599, e.g. 429) |
| `BUSY_RETRY_AFTER` | `30` | seconds sent as `Retry-After` on busy responses (both modes) |
| `WARM_MODEL` | `` | optional model to reload after an image job |
| `LOGLEVEL` | `warn` | `info` logs every request (colored arrows in text mode), `debug` adds lock transitions. `LOG_LEVEL` is accepted as an alias |
| `LOG_FORMAT` | `text` | `json` for structured JSON logs |
@@ -146,6 +154,15 @@ Docker.
- `gpu-turnstile.exe service install [-config path]` registers an
auto-start Windows service (needs an elevated shell). Recovery actions
restart it 5 s after any failure. `service remove` uninstalls.
- **Layout**: install to `C:\Program Files\gpu-turnstile\` (exe plus
`gpu-turnstile.env`); logs belong in `C:\ProgramData\gpu-turnstile\` via
`LOG_FILE`. The service must be able to write its install directory for
self-updates — Program Files is writable by LocalSystem and admins, which
is why running as the default `LocalSystem` account is the simple choice.
- **Account**: the default `LocalSystem` works out of the box. For least
privilege, create the service with the virtual account
`NT SERVICE\gpu-turnstile` and grant it write access to the install and
log directories only (no network logon, no user profile).
- Use a config file (above) for the service — Windows services have no
convenient environment. Logs go to `LOG_FILE` since there is no console.
- **Auto-update**: on startup and every `UPDATE_INTERVAL`, the binary