Make consumers optional: a URL enables its mode, empty disables it
OLLAMA_URL and COMFY_URL no longer have defaults; each consumer (listener, client, startup probe, lock participation) is enabled by setting its URL and disabled by leaving it empty. At least one must be set. Ollama-only mode is a pure pass-through; ComfyUI-only mode skips the unload and warm-reload steps. /metrics is now served on both listeners. This is the extension pattern for future consumers such as local game detection.
This commit is contained in:
@@ -42,6 +42,21 @@ listener is an `httputil.ReverseProxy` to its upstream. Websocket upgrades
|
||||
(ComfyUI `/ws`) and streaming bodies (Ollama NDJSON / SSE) must pass through
|
||||
unbuffered (`FlushInterval = -1`).
|
||||
|
||||
### Modes of operation
|
||||
|
||||
Each GPU consumer is enabled by setting its URL and disabled by leaving it
|
||||
empty — no separate flags. At least one URL must be set; a disabled
|
||||
consumer gets no listener, no startup probe, and no lock participation:
|
||||
|
||||
- **Both set** (default deployment): full arbitration as described below.
|
||||
- **Only `OLLAMA_URL`**: pure pass-through for Ollama; the LLM lock never
|
||||
blocks since no image jobs can arrive.
|
||||
- **Only `COMFY_URL`**: image jobs are tracked and ComfyUI's VRAM is freed
|
||||
afterwards, but the Ollama unload and warm-reload steps are skipped.
|
||||
- Future consumers (e.g. detecting a local game holding VRAM) plug into the
|
||||
same lock the same way: enabled by their config knob, excluded when
|
||||
absent.
|
||||
|
||||
### Lock semantics
|
||||
|
||||
Two-mode lock with image priority (writer-preferring RW lock, where "readers"
|
||||
@@ -83,7 +98,8 @@ ComfyUI listener (`:8188` → `COMFY_URL`):
|
||||
### Image job flow (`POST /prompt`)
|
||||
|
||||
1. `AcquireImage()`.
|
||||
2. Unload Ollama: `GET /api/ps`; for each model `POST /api/generate
|
||||
2. Unload Ollama (skipped when `OLLAMA_URL` is unset): `GET /api/ps`; for
|
||||
each model `POST /api/generate
|
||||
{"model":M,"keep_alive":0}`; if that returns non-2xx (embedding-only
|
||||
models), `POST /api/embed {"model":M,"input":"x","keep_alive":0}`. Poll
|
||||
`/api/ps` every `UNLOAD_POLL_INTERVAL` (default 500 ms) until empty or
|
||||
@@ -117,8 +133,8 @@ override file values. A missing file is fine; a malformed one is fatal.
|
||||
|---|---|---|
|
||||
| `LISTEN_OLLAMA` | `:11434` | Ollama-facing listener |
|
||||
| `LISTEN_COMFY` | `:8188` | ComfyUI-facing listener |
|
||||
| `OLLAMA_URL` | `http://127.0.0.1:11435` | upstream |
|
||||
| `COMFY_URL` | `http://127.0.0.1:8189` | upstream |
|
||||
| `OLLAMA_URL` | _(empty = disabled)_ | Ollama upstream; set to enable the Ollama consumer |
|
||||
| `COMFY_URL` | _(empty = disabled)_ | ComfyUI upstream; set to enable the ComfyUI consumer |
|
||||
| `UNLOAD_TIMEOUT` | `60s` | wait for Ollama to unload |
|
||||
| `JOB_TIMEOUT` | `15m` | wait for ComfyUI job |
|
||||
| `LLM_WAIT_TIMEOUT` | `10m` | max time an LLM request waits for the lock before 503 (wait mode) |
|
||||
@@ -143,8 +159,9 @@ override file values. A missing file is fine; a malformed one is fatal.
|
||||
| `UPDATE_REPO` | `https://git.rambossek.at/PUBLIC/gpu-turnstile` | repository to check for releases |
|
||||
| `UPDATE_ASSET` | `gpu-turnstile.exe` | release asset to download |
|
||||
|
||||
Startup fails fast on unparsable values. Both upstreams are probed once at
|
||||
start (`/api/version`, `/system_stats`); failure is logged, not fatal.
|
||||
Startup fails fast on unparsable values and when neither consumer URL is
|
||||
set. Enabled upstreams are probed once at start (`/api/version`,
|
||||
`/system_stats`); failure is logged, not fatal.
|
||||
|
||||
## Native Windows deployment
|
||||
|
||||
@@ -184,7 +201,7 @@ Docker.
|
||||
|
||||
- `GET /healthz` on both listeners: 200 with JSON
|
||||
`{"state":"idle|llm|image","llm_inflight":N,"image_pending":B}`.
|
||||
- `GET /metrics` on the Ollama listener: Prometheus text format, no external
|
||||
- `GET /metrics` on both listeners: Prometheus text format, no external
|
||||
dependency needed:
|
||||
`gpu_turnstile_state{state="…"} 1`, `gpu_turnstile_llm_inflight`,
|
||||
`gpu_turnstile_image_jobs_total`, `gpu_turnstile_lock_wait_seconds`
|
||||
|
||||
Reference in New Issue
Block a user