25 Commits
Author SHA1 Message Date
mram 21e40a1774 Pin compose example to v0.2.0
ci / test (push) Successful in 14s
ci / docker (push) Successful in 1m7s
ci / release (push) Successful in 15s
2026-09-21 21:47:48 +02:00
mram bdc844872d Install narrates itself: stop/copy/start steps and the potentially long icacls tree grant 2026-09-21 21:36:45 +02:00
mram 299b6dc0bb Distinguish EACCES from ENOENT in the COMFY_DIR startup check 2026-09-21 21:32:45 +02:00
mram 9589e58ce6 Install opens up COMFY_DIR for the sandboxed service: ACL grant on Windows, BindPaths on Linux 2026-09-21 21:30:35 +02:00
mram be0bb36317 Pin compose example to v0.1.10
ci / test (push) Successful in 28s
ci / docker (push) Successful in 1m15s
ci / release (push) Successful in 17s
2026-09-21 21:20:49 +02:00
mram 9b267533c9 COMFY_DIR alone manages ComfyUI: derive the launch command from the standard venv layout 2026-09-21 21:18:33 +02:00
mram 6f092ddc12 Default GAME_POLL_INTERVAL to 15s: nvidia-smi polls keep the GPU awake 2026-09-21 21:10:21 +02:00
mram 0228ccc296 Pin compose example to v0.1.9
ci / test (push) Successful in 17s
ci / docker (push) Successful in 1m17s
ci / release (push) Successful in 19s
2026-09-21 20:57:34 +02:00
mram 9997913929 Sync the env file at startup, not just at install: updates append new settings 2026-09-21 20:56:32 +02:00
mram 482731ed9e Pin compose example to v0.1.8
ci / test (push) Successful in 14s
ci / docker (push) Successful in 1m11s
ci / release (push) Successful in 15s
2026-09-21 17:19:16 +02:00
mram e4bdc92ece Game detection: foreign GPU holders take an external lock hold (GAME_PROCS, GPU_FOREIGN_VRAM_MB) 2026-09-21 17:16:56 +02:00
mram 14120bf4a4 Supervisor probes before spawning: an external server on the port is used, never fought or killed 2026-09-21 16:06:28 +02:00
mram d7566329ae Managed ComfyUI: COMFY_CMD starts it on demand, idle stop frees VRAM (internal/supervise) 2026-09-21 13:48:07 +02:00
mram e43ad02fc4 Ignore staged-update leftovers next to the dev exe 2026-09-21 13:14:33 +02:00
mram 232f5b61f2 Periodic upstream health checks (HEALTH_INTERVAL, default 30s); log down/recovered transitions 2026-09-21 13:14:18 +02:00
mram 5e7a042cad CFG_VER is always a concrete version: dev builds stamp v0.0.0, never "dev" 2026-09-21 12:42:32 +02:00
mram 14c2e30478 Clarify listener wording: Ollama-compatible clients, not Ollama-facing 2026-09-21 12:38:03 +02:00
mram dcdb2cc72e Pin compose example to v0.1.7
ci / test (push) Successful in 13s
ci / docker (push) Successful in 1m7s
ci / release (push) Successful in 16s
2026-09-21 10:04:29 +02:00
mram 44a8e9fbdc CFG_VER/APP_VER in the env file: invalid configs replaced, updates follow APP_VER (dev/stable/pin) 2026-09-21 09:28:29 +02:00
mram 272a3a467d force-update: single-shot with a 30s fail-fast timeout 2026-09-21 09:09:27 +02:00
mram cedf28a809 force-update does not require consumer URLs (ErrNoConsumer sentinel) 2026-09-21 09:05:46 +02:00
mram e46e92bee8 Sample env carries a version marker; install appends settings missing since the writing version 2026-09-21 08:54:47 +02:00
mram cf48796183 Install writes a fully commented sample env file when no config exists 2026-09-21 08:48:51 +02:00
mram dd3187f951 Install sets a default LOG_FILE in the env file on Windows (no console as a service) 2026-09-21 08:37:25 +02:00
mram 1394a76eea Make service install converge: stop running service, refresh binary/config only on change, restart only if it was running 2026-09-21 08:30:55 +02:00
28 changed files with 2615 additions and 180 deletions
+3
View File
@@ -4,3 +4,6 @@
/compose.yml
/signing/
/gpu-turnstile.exe.old
/gpu-turnstile.exe.new
/gpu-turnstile.exe~
+99 -12
View File
@@ -2,9 +2,10 @@
GPU arbitration proxy for Ollama + ComfyUI. One consumer GPU is shared by an
LLM server (Ollama) and an image generator (ComfyUI); gpu-turnstile sits in
front of both and guarantees the GPU is always in exactly one of three states:
`idle`, `llm` (N ≥ 1 Ollama requests in flight), or `image` (exactly one
ComfyUI job, Ollama models unloaded). See [SPEC.md](SPEC.md) for the full
front of both and guarantees the GPU is always in exactly one of four states:
`idle`, `llm` (N ≥ 1 Ollama requests in flight), `image` (exactly one
ComfyUI job, Ollama models unloaded), or `external` (a foreign process such
as a game holds the GPU). See [SPEC.md](SPEC.md) for the full
design.
gpu-turnstile listens on the ports the services normally use; the actual
@@ -32,8 +33,9 @@ Open WebUI / n8n ────► :8188 ───┘
Each consumer is enabled by setting its URL (`OLLAMA_URL`, `COMFY_URL`) and
disabled by leaving it empty — at least one is required. With only Ollama
the proxy is a pass-through (no image jobs can arrive); with only ComfyUI
the Ollama unload/warm steps are skipped. Future consumers (e.g. local game
detection) plug into the same lock the same way.
the Ollama unload/warm steps are skipped. A third, URL-less consumer —
detection of foreign GPU holders such as games — is enabled by `GAME_PROCS`
and/or `GPU_FOREIGN_VRAM_MB` (see below).
## Configuration
@@ -45,8 +47,8 @@ override file values. Invalid values fail at startup.
| Var | Default | Meaning |
|---|---|---|
| `LISTEN_OLLAMA` | `:11434` | Ollama-facing listener |
| `LISTEN_COMFY` | `:8188` | ComfyUI-facing listener |
| `LISTEN_OLLAMA` | `:11434` | Listener for Ollama-compatible clients |
| `LISTEN_COMFY` | `:8188` | Listener for ComfyUI clients |
| `OLLAMA_URL` | _(empty = disabled)_ | Ollama upstream; set to enable the Ollama consumer |
| `COMFY_URL` | _(empty = disabled)_ | ComfyUI upstream; set to enable the ComfyUI consumer |
| `UNLOAD_TIMEOUT` | `60s` | Wait for Ollama to unload before an image job |
@@ -56,12 +58,21 @@ override file values. Invalid values fail at startup.
| `LLM_BUSY_STATUS` | `503` | HTTP status for rejected LLM requests in reject mode (400599, e.g. 429) |
| `BUSY_RETRY_AFTER` | `30` | Seconds sent as `Retry-After` on busy responses (both modes) |
| `WARM_MODEL` | _(empty)_ | Model to reload after an image job (off by default) |
| `COMFY_CMD` | _(derived from `COMFY_DIR`; both empty = unmanaged)_ | Supervise ComfyUI: start on demand, stop when idle to free VRAM. Requires `COMFY_URL` |
| `COMFY_DIR` | _(empty)_ | Standard venv install root: set alone to supervise ComfyUI with the derived command (`.venv` + `main.py`); also the working directory for `COMFY_CMD` |
| `COMFY_IDLE_TIMEOUT` | `5m` | Stop the managed ComfyUI after this long idle |
| `COMFY_START_TIMEOUT` | `2m` | Max wait for the managed ComfyUI to come up |
| `GAME_PROCS` | _(empty = disabled)_ | Process names (comma-separated); while any runs, the GPU counts as held: requests wait, Ollama unloads, managed ComfyUI stops |
| `GPU_FOREIGN_VRAM_MB` | `0` (disabled) | Also treat the GPU as held when a non-ignored process uses more VRAM than this (needs nvidia-smi) |
| `GPU_IGNORE_PROCS` | `ollama,ollama app,ollama_llama_server,python,pythonw` | Process names never counted as foreign GPU users |
| `GAME_POLL_INTERVAL` | `15s` | How often game/VRAM detection runs (don't go below ~10s — nvidia-smi polls keep the GPU awake) |
| `LOGLEVEL` | `warn` | `info` logs every request (colored arrows in text mode), `debug` adds lock transitions. `LOG_LEVEL` works as an alias |
| `LOG_FORMAT` | `text` | `json` for structured JSON logs |
| `LOG_FILE` | _(empty)_ | Append logs to this file instead of stderr |
| `UNLOAD_POLL_INTERVAL` | `500ms` | `/api/ps` poll interval while unloading |
| `HISTORY_POLL_INTERVAL` | `1s` | `/history/<id>` poll interval while a job runs |
| `PROBE_TIMEOUT` | `5s` | Startup probe of both upstreams |
| `PROBE_TIMEOUT` | `5s` | Startup probe of both upstreams (also per-probe health check timeout) |
| `HEALTH_INTERVAL` | `30s` | Periodic upstream probe; down/recovered changes are logged |
| `FREE_TIMEOUT` | `30s` | `POST /free` call after an image job |
| `WARM_TIMEOUT` | `2m` | Warm-model reload after an image job |
| `SHUTDOWN_TIMEOUT` | `10s` | Graceful shutdown on SIGINT/SIGTERM |
@@ -72,10 +83,11 @@ override file values. Invalid values fail at startup.
| `UPDATE_INTERVAL` | `6h` | Auto-update check interval |
| `UPDATE_REPO` | `https://git.rambossek.at/PUBLIC/gpu-turnstile` | Repository checked for releases |
| `UPDATE_ASSET` | `gpu-turnstile.exe` | Release asset to download |
| `APP_VER` | `stable` | `dev` disables updates, `stable` tracks latest, or pin an exact `vX.Y.Z` |
## Observability
- `GET /healthz` (both listeners): `{"state":"idle|llm|image","llm_inflight":N,"image_pending":B}`
- `GET /healthz` (both listeners): `{"state":"idle|llm|image|external","llm_inflight":N,"image_pending":B}`
- `GET /metrics` (both listeners): Prometheus text format — `gpu_turnstile_state`,
`gpu_turnstile_llm_inflight`, `gpu_turnstile_image_pending`,
`gpu_turnstile_image_jobs_total`, `gpu_turnstile_lock_wait_seconds`
@@ -87,6 +99,71 @@ override file values. Invalid values fail at startup.
class) in text mode, which renders in `docker compose logs` on Windows
Terminal. Set `NO_COLOR` to disable colors.
## Managed ComfyUI (`COMFY_CMD` / `COMFY_DIR`)
Don't want ComfyUI running 24/7 (it holds VRAM even when idle — and the
Desktop app kills its server when you close it)? gpu-turnstile can supervise
it: the first request starts it, it stops again after `COMFY_IDLE_TIMEOUT`
(default 5m) without work, freeing the GPU for games or the LLM.
The easy way — point `COMFY_DIR` at a standard venv install (a folder with
`.venv` and `main.py`, or `.venv` and `ComfyUI\main.py`) and the launch
command is derived from it, including `--port` from `COMFY_URL`:
```
COMFY_URL=http://127.0.0.1:8189
COMFY_DIR=C:\ComfyUI
```
For other layouts, spell the command out yourself (run it once manually to
confirm it works):
```
COMFY_URL=http://127.0.0.1:8189
COMFY_CMD="C:\ComfyUI\.venv\Scripts\python.exe" main.py --port 8189
COMFY_DIR=C:\ComfyUI
```
`POST /prompt` waits for the server to answer before taking the GPU lock
(LLM traffic keeps flowing while torch loads); crashes are logged and the
next request respawns. Shutting gpu-turnstile down stops the child too.
Running the ComfyUI Desktop app alongside is safe: if something already
answers on the port, gpu-turnstile just uses it instead of spawning
(and never kills it — it only ever stops its own child). If the managed
instance already holds the port when you open the desktop app, the
desktop's server is the one that fails to bind.
One catch when gpu-turnstile runs as a service: the sandboxed service
account may not enter your user profile, so a ComfyUI install under
`C:\Users\...` (or `/home/...`) fails with "Access is denied".
`--install-service` fixes that automatically — it grants
`NT SERVICE\gpu-turnstile` recursive access to `COMFY_DIR` on Windows and
adds a `BindPaths=` to the systemd unit on Linux. Re-run it after changing
`COMFY_DIR`; or grant by hand from an admin shell:
`icacls "<COMFY_DIR>" /grant "NT SERVICE\gpu-turnstile:(OI)(CI)M" /T`.
## Game detection
Want to game on the same GPU without Ollama/ComfyUI squatting on the VRAM?
gpu-turnstile can watch for foreign GPU holders and, while one is active,
make LLM/image requests wait (or 503, per `LLM_BUSY_MODE`), unload Ollama's
models and stop the managed ComfyUI so the game gets the memory. Two
detection paths, each optional, polled every `GAME_POLL_INTERVAL` (15s):
```
GAME_PROCS=cyberpunk2077.exe,bg3.exe # the reliable way on Windows
GPU_FOREIGN_VRAM_MB=1024 # catch-all via nvidia-smi
```
`GAME_PROCS` matches running process names (case-insensitive, `.exe`
optional). `GPU_FOREIGN_VRAM_MB` asks nvidia-smi which processes hold GPU
memory and treats anything not in `GPU_IGNORE_PROCS` above the threshold as
foreign — handy as a catch-all, but note that under Windows' WDDM driver
graphics-only games may not show up in nvidia-smi's per-process list, so
name your games in `GAME_PROCS` there; on Linux both paths work. When the
game exits, requests resume automatically.
## Build and run
```sh
@@ -112,12 +189,19 @@ gpu-turnstile.exe --remove-service
```
Layout: `C:\Program Files\gpu-turnstile\` holds the exe and
`gpu-turnstile.env`, logs go to `C:\ProgramData\gpu-turnstile\` (set
`LOG_FILE` in the env file — there is no console). The service always runs
`gpu-turnstile.env`, logs go to `C:\ProgramData\gpu-turnstile\`. If you
install without a config, the installer writes a sample env file with every
setting commented and explained — only `LOG_FILE` is active (a service has
no console). Your own `LOG_FILE` setting is always kept. The service always
runs
as the virtual account `NT SERVICE\gpu-turnstile` (low-privilege,
per-service, no password); the installer automatically grants it write
access to the install and data directories — nothing else to do.
Re-running `--install-service` is safe: it stops a running service,
replaces the installed binary only if it changed, fixes the registration
only where it drifted, and restarts the service only if it was running.
### Run natively on Linux (systemd)
The same binary works on Linux. Install it as a systemd service as root:
@@ -144,7 +228,10 @@ containers and interactive shells.
release on startup and every `UPDATE_INTERVAL`, verifies the Ed25519
signature of the download against the public key embedded at build time,
and — once the GPU lock is idle — restarts the service onto the new
version. Disable with `AUTO_UPDATE=false`. Releases are signed by CI with
version. `APP_VER` controls the target: `dev` disables updates, `stable`
(the default) tracks the latest release, and an exact `vX.Y.Z` pins that
release (even as a downgrade or to replace a dev build). Disable entirely
with `AUTO_UPDATE=false`. Releases are signed by CI with
OpenSSL; the matching public key lives in `internal/update/pubkey.go`
(one-time setup: `openssl genpkey -algorithm ed25519 -out private.pem`,
`openssl pkey -in private.pem -pubout -out public.pem`; private key goes
+138 -22
View File
@@ -10,11 +10,12 @@ becomes very slow; on Linux it would OOM instead.
## Goal
A single Go binary that sits in front of **both** services and guarantees that
at any moment the GPU is in exactly one of three states:
at any moment the GPU is in exactly one of four states:
- `idle` — nothing in flight
- `llm` — N ≥ 1 Ollama inference requests in flight (concurrency allowed)
- `image` — exactly one ComfyUI job in flight, Ollama models unloaded
- `external` — a foreign process (e.g. a game) holds the GPU; new work waits
Clients (LiteLLM, Open WebUI, n8n) point at gpu-turnstile instead of at the
services. gpu-turnstile is transparent for everything that does not touch the
@@ -53,9 +54,10 @@ consumer gets no listener, no startup probe, and no lock participation:
blocks since no image jobs can arrive.
- **Only `COMFY_URL`**: image jobs are tracked and ComfyUI's VRAM is freed
afterwards, but the Ollama unload and warm-reload steps are skipped.
- Future consumers (e.g. detecting a local game holding VRAM) plug into the
same lock the same way: enabled by their config knob, excluded when
absent.
- **Game detection** is a third, optional consumer without a URL: enabled by
`GAME_PROCS` and/or `GPU_FOREIGN_VRAM_MB` it watches for foreign processes
holding the GPU (see below) and plugs into the same lock the same way —
excluded when both knobs are unset.
### Lock semantics
@@ -75,6 +77,11 @@ are LLM requests and the single "writer" is an image job):
requests start), waits until n == 0, sets state := `image`. Released after
the ComfyUI job finished and models were freed.
- Concurrent image jobs queue FIFO behind each other.
- **External hold**: game detection calls `SetExternal(holder)` while a
foreign process holds the GPU. New LLM and image grants block (same busy
handling as above) until `ClearExternal()`; in-flight work is not
preempted, it drains. Snapshot reports state `external` once nothing else
is in flight.
- All waits are context-aware: a client that disconnects while waiting is
removed from the queue.
@@ -121,6 +128,81 @@ state is `idle`, send `POST /api/generate {"model":WARM_MODEL,"keep_alive":-1}`
with empty prompt to reload the chat model so the next chat doesn't pay the
load time. Off by default.
### Managed ComfyUI (`COMFY_CMD` / `COMFY_DIR`)
When `COMFY_CMD` is set, gpu-turnstile runs ComfyUI as a supervised child
process instead of expecting an always-on server. Setting only `COMFY_DIR`
enables the same management with the launch command derived from the
standard venv layout under it (`.venv\Scripts\python.exe` on Windows,
`.venv/bin/python` on Linux; `ComfyUI\main.py`, or a flat `main.py` when
that is what exists; `--port` from the `COMFY_URL` port). Missing layout
files are flagged in the startup log.
- **Start on demand**: any ComfyUI request spawns it (double quotes in the
command line group arguments with spaces; `COMFY_DIR` sets the working
directory). `POST /prompt` additionally waits for readiness
(`/system_stats`) for up to `COMFY_START_TIMEOUT` *before* taking the GPU
lock, so LLM traffic flows while torch loads. Other requests are bridged
by the normal retry backoff. Spawn failure or a readiness timeout
answers 502.
- **Idle stop**: after `COMFY_IDLE_TIMEOUT` without requests or finished
jobs — and only while the GPU lock is idle — the process tree is killed,
freeing the VRAM ComfyUI holds. The next request restarts it.
- **Crash**: an unexpected exit is logged; the next request respawns.
gpu-turnstile's own shutdown stops the child too.
- **Coexistence**: before spawning, the URL is probed — if another server
already answers (e.g. the ComfyUI desktop app), it is used as-is and no
child is spawned; the idle watcher and shutdown only ever stop the
supervisor's own process, never the external one. The other direction —
starting the desktop app while the managed instance holds the port —
makes the *desktop* server fail to bind; gpu-turnstile is unaffected.
- Its stdout/stderr is forwarded to the log at INFO. The health check
skips the intentionally-stopped/starting states; a failed probe while
the process is alive and was previously ready is logged as DOWN.
- **Permissions**: the service account is sandboxed (Windows virtual
account, systemd `DynamicUser`), so a ComfyUI install inside a user
profile is off-limits by default. `--install-service` opens it up —
a recursive ACL grant for `NT SERVICE\gpu-turnstile` on Windows, a
`BindPaths=` in the unit on Linux — reading `COMFY_DIR` from the config
it installs. Re-run `--install-service` after changing `COMFY_DIR`, or
grant by hand (admin shell):
`icacls "<COMFY_DIR>" /grant "NT SERVICE\gpu-turnstile:(OI)(CI)M" /T`.
## Game detection (foreign GPU holders)
Games and other foreign GPU users sit outside the URL-based consumer model —
nothing proxies through gpu-turnstile for them. Two independent detection
paths, polled every `GAME_POLL_INTERVAL` (default 15 s); either one being
configured enables the feature:
- **Process watch list** (`GAME_PROCS`, comma-separated, case-insensitive,
`.exe` optional): while any listed process runs, the GPU counts as held.
This is the reliable path on Windows.
- **Foreign VRAM threshold** (`GPU_FOREIGN_VRAM_MB`): `nvidia-smi
--query-compute-apps` lists per-process GPU memory; any process not in
`GPU_IGNORE_PROCS` (default: Ollama and python — ComfyUI runs under python)
holding more than the threshold counts as a foreign holder. Needs
nvidia-smi on the PATH (absent: logged once, path disabled) and works best
on Linux — under Windows' WDDM driver, graphics-only games may not appear
in the per-process list.
While a holder is detected, gpu-turnstile:
1. takes the external hold on the lock (`SetExternal`), so new LLM/image
requests wait or are rejected per `LLM_BUSY_MODE`; busy responses name the
holder,
2. once in-flight work has drained, **frees VRAM for the foreign process**:
the managed ComfyUI is stopped (an external server on its port is never
touched) and Ollama's resident models are unloaded,
3. refuses to spawn the managed ComfyUI; ComfyUI requests that would need a
spawn are answered 503 + `Retry-After` (a running desktop instance keeps
being proxied),
4. logs the transitions at WARN ("GPU held by an external process …" /
"released the GPU; resuming") and reports state `external` in `/healthz`
and `/metrics`.
When the holder disappears, the hold is lifted and queued requests proceed.
## Configuration (env)
Configuration comes from environment variables and/or an `.env`-style
@@ -131,8 +213,8 @@ override file values. A missing file is fine; a malformed one is fatal.
| Var | Default | Meaning |
|---|---|---|
| `LISTEN_OLLAMA` | `:11434` | Ollama-facing listener |
| `LISTEN_COMFY` | `:8188` | ComfyUI-facing listener |
| `LISTEN_OLLAMA` | `:11434` | listener for Ollama-compatible clients |
| `LISTEN_COMFY` | `:8188` | listener for ComfyUI clients |
| `OLLAMA_URL` | _(empty = disabled)_ | Ollama upstream; set to enable the Ollama consumer |
| `COMFY_URL` | _(empty = disabled)_ | ComfyUI upstream; set to enable the ComfyUI consumer |
| `UNLOAD_TIMEOUT` | `60s` | wait for Ollama to unload |
@@ -142,12 +224,21 @@ override file values. A missing file is fine; a malformed one is fatal.
| `LLM_BUSY_STATUS` | `503` | HTTP status for rejected LLM requests in reject mode (400599, e.g. 429) |
| `BUSY_RETRY_AFTER` | `30` | seconds sent as `Retry-After` on busy responses (both modes) |
| `WARM_MODEL` | `` | optional model to reload after an image job |
| `COMFY_CMD` | _(derived from `COMFY_DIR`; both empty = unmanaged)_ | spawn and supervise ComfyUI on demand: first request starts it, idle stop after `COMFY_IDLE_TIMEOUT` frees its VRAM. Requires `COMFY_URL` |
| `COMFY_DIR` | `` | standard venv install root: set alone to supervise ComfyUI with the derived launch command (`.venv` + `main.py`, `--port` from `COMFY_URL`); also the working directory for `COMFY_CMD` |
| `COMFY_IDLE_TIMEOUT` | `5m` | stop the managed ComfyUI after this long without requests or jobs |
| `COMFY_START_TIMEOUT` | `2m` | how long a request waits for the managed ComfyUI to come up |
| `GAME_PROCS` | _(empty = disabled)_ | comma-separated process names (case-insensitive, `.exe` optional); while any runs, the GPU counts as held by it: requests wait, Ollama unloads, the managed ComfyUI stops |
| `GPU_FOREIGN_VRAM_MB` | `0` (disabled) | also treat the GPU as held when a process not in `GPU_IGNORE_PROCS` uses more VRAM than this; needs nvidia-smi |
| `GPU_IGNORE_PROCS` | `ollama,ollama app,ollama_llama_server,python,pythonw` | process names never counted as foreign GPU users |
| `GAME_POLL_INTERVAL` | `15s` | how often game/VRAM detection runs (nvidia-smi polls keep the GPU awake; don't go below ~10s) |
| `LOGLEVEL` | `warn` | `info` logs every request (colored arrows in text mode), `debug` adds lock transitions. `LOG_LEVEL` is accepted as an alias |
| `LOG_FORMAT` | `text` | `json` for structured JSON logs |
| `LOG_FILE` | `` | append logs to this file instead of stderr (useful as a service) |
| `UNLOAD_POLL_INTERVAL` | `500ms` | `/api/ps` poll interval while unloading |
| `HISTORY_POLL_INTERVAL` | `1s` | `/history/<id>` poll interval while a job runs |
| `PROBE_TIMEOUT` | `5s` | startup probe of both upstreams |
| `PROBE_TIMEOUT` | `5s` | startup probe of both upstreams (also the per-probe health check timeout) |
| `HEALTH_INTERVAL` | `30s` | periodic probe of enabled upstreams; status changes (down/recovered) are logged |
| `FREE_TIMEOUT` | `30s` | `POST /free` call after an image job |
| `WARM_TIMEOUT` | `2m` | warm-model reload after an image job |
| `SHUTDOWN_TIMEOUT` | `10s` | graceful shutdown on SIGINT/SIGTERM |
@@ -158,6 +249,8 @@ override file values. A missing file is fine; a malformed one is fatal.
| `UPDATE_INTERVAL` | `6h` | auto-update check interval |
| `UPDATE_REPO` | `https://git.rambossek.at/PUBLIC/gpu-turnstile` | repository to check for releases |
| `UPDATE_ASSET` | `gpu-turnstile.exe` | release asset to download |
| `APP_VER` | `stable` | version to run: `dev` disables updates, `stable` tracks the latest release, or an exact `vX.Y.Z` pin (up- or downgraded to) |
| `CFG_VER` | _(installer-managed)_ | config format reference written by `--install-service` (always a concrete `vX.Y.Z`; a dev build stamps `v0.0.0`); missing = the file is replaced with a fresh sample (backup `.bak`). New settings are appended (commented out) at install and at every startup after an update changed the version |
Startup fails fast on unparsable values and when neither consumer URL is
set. Enabled upstreams are probed once at start (`/api/version`,
@@ -176,12 +269,27 @@ prompt instead of failing — the command relaunches itself elevated, waits
for the child, and mirrors its exit code. The legacy form
`gpu-turnstile service install|remove` does the same thing.
Re-running install on an already-registered service converges instead of
failing: a running service is stopped first, the installed binary copy is
refreshed only when the content differs, the registration (Windows service
config / systemd unit) is updated only where it drifted, and the service is
started again only if it was running before.
By default install creates the canonical layout and copies the binary into
it (Windows: `%ProgramFiles%\gpu-turnstile\`, plus
`%ProgramData%\gpu-turnstile\` for logs; Linux: `/var/lib/gpu-turnstile/`
with the config at `/etc/gpu-turnstile.env`). An existing config in the
target location is never overwritten. `--no-copy` registers the current
executable location as-is instead.
with the config at `/etc/gpu-turnstile.env`). If there is no config at all,
install writes a sample env file covering every setting — each with a
comment line, everything commented out — except the `CFG_VER`/`APP_VER`
header and `LOG_FILE`, which is active on Windows
(`%ProgramData%\gpu-turnstile\gpu-turnstile.log`) since a service has no
console; on Linux it stays commented because stderr goes to the journal.
The first line is `CFG_VER=vX.Y.Z`, recording the installer version. When a
later version's install finds an older `CFG_VER`, it appends every setting
the file does not mention (commented or not) at the end and updates
`CFG_VER`; a file without `CFG_VER` is invalid and gets replaced by a fresh
sample, with the old content kept as `<file>.bak`. `--no-copy` registers
the current executable location as-is and leaves the config untouched.
### Windows
@@ -189,8 +297,9 @@ executable location as-is instead.
`%ProgramData%\gpu-turnstile\`, copies the exe and (if none exists there
yet) the `gpu-turnstile.env` into the Program Files directory, and
registers that copy as a Windows service; recovery actions restart it
5 s after any failure. Logs go to the ProgramData directory via
`LOG_FILE` since there is no console.
5 s after any failure. The install ensures the env file sets `LOG_FILE`
to `%ProgramData%\gpu-turnstile\gpu-turnstile.log` since there is no
console — an existing `LOG_FILE` setting is kept.
- **Account**: the service always runs as the virtual account
`NT SERVICE\gpu-turnstile` — a per-service low-privilege identity the
SCM manages (no password, automatic logon-as-a-service right, no admin
@@ -238,19 +347,25 @@ executable location as-is instead.
`RestartSec=5s` brings up the staged binary after the updater exits with
code 3.
- **Auto-update**: on startup and every `UPDATE_INTERVAL`, the binary
checks `UPDATE_REPO`'s latest release; if its tag is a newer `vX.Y.Z`,
it downloads `UPDATE_ASSET` plus its `.sig` (and `.sha256` when present)
and verifies an Ed25519 signature against the public key embedded in
consults `APP_VER`: `dev` disables updates; `stable` (the default)
fetches `UPDATE_REPO`'s latest release and applies it when its tag is a
newer `vX.Y.Z` (a `dev` binary cannot be compared and is replaced by the
latest release); a `vX.Y.Z` pin fetches that exact tag and stages it on
any difference, including downgrades. Applying means downloading
`UPDATE_ASSET` plus its `.sig` (and `.sha256` when present) and verifying
an Ed25519 signature against the public key embedded in
`internal/update/pubkey.go`. A verified binary is swapped in next to the
running exe (rename-aside, allowed on Windows), and once the GPU lock is
idle the process exits with code 3 so the service recovery restarts it
on the new version. Interactive runs only log "restart to apply".
`dev` builds and builds without an embedded public key never update.
- **`--force-update`** runs the same check immediately: it downloads,
verifies and stages a newer release, and if the service is running it
restarts it right away (otherwise the new version applies on next
start). On Windows it elevates via UAC only when the stage or restart
needs permissions the caller does not have.
Builds without an embedded public key never update.
- **`--force-update`** runs the same check immediately, single-shot: one
attempt with a 30 s timeout, then exit — "up to date" (exit 0) or the
error (exit 1), no retries. When a newer release is found it downloads,
verifies and stages it, and if the service is running it restarts it
right away (otherwise the new version applies on next start). On Windows
it elevates via UAC only when the stage or restart needs permissions the
caller does not have.
- **Signing setup (one time)**: `openssl genpkey -algorithm ed25519 -out
private.pem`; `openssl pkey -in private.pem -pubout -out public.pem`.
Private key → repo secret `RELEASE_SIGNING_KEY`; public key → committed into
@@ -334,7 +449,8 @@ are new.
receives the first chunk before the last is sent (no buffering).
- `internal/config`: env-file parsing, precedence, fail-fast values.
- `internal/update`: fake Gitea releases API; staged update happy path,
tampered signature rejected, older versions and dev builds skipped.
tampered signature rejected, older versions skipped, APP_VER=dev and
pinned releases honored.
## Build and CI
+261 -11
View File
@@ -15,17 +15,20 @@ import (
"os"
"os/signal"
"path/filepath"
"runtime"
"strings"
"syscall"
"time"
"gpu-turnstile/internal/comfy"
"gpu-turnstile/internal/config"
"gpu-turnstile/internal/game"
"gpu-turnstile/internal/lock"
"gpu-turnstile/internal/metrics"
"gpu-turnstile/internal/ollama"
"gpu-turnstile/internal/proxy"
"gpu-turnstile/internal/service"
"gpu-turnstile/internal/supervise"
"gpu-turnstile/internal/update"
)
@@ -170,6 +173,7 @@ func main() {
}
log, logOut, logCloser := newLogger(cfg)
defer logCloser.Close()
syncEnvFile(resolveConfigPath(configPath), cfg.LogFile, log)
if service.IsService() {
if err := service.Run(func(ctx context.Context) error { return run(ctx, cfg, log, logOut, true) }); err != nil {
@@ -186,6 +190,27 @@ func main() {
}
}
// syncEnvFile upgrades an installer-written config file after an update:
// settings added since its CFG_VER are appended (commented out) and CFG_VER
// is bumped. Files not written by the installer (no CFG_VER), up-to-date
// files and dev builds are left untouched; a write failure is logged, not
// fatal.
func syncEnvFile(path, logFile string, log *slog.Logger) {
data, err := os.ReadFile(path)
if err != nil {
return // no config file; nothing to upgrade
}
synced, changed := config.SyncSample(string(data), version, logFile)
if !changed {
return
}
if err := os.WriteFile(path, []byte(synced), 0o644); err != nil {
log.Warn("could not append new settings to the config file", "path", path, "err", err)
return
}
log.Warn("config file updated: new settings appended", "path", path, "version", version)
}
// defaultConfigPath returns gpu-turnstile.env next to the executable.
func defaultConfigPath() string {
exe, err := os.Executable()
@@ -299,6 +324,9 @@ func forceUpdateCommand(configPath string, elevatedChild bool) int {
defer waitForEnter()
}
cfg, err := loadMergedConfig(configPath)
if errors.Is(err, config.ErrNoConsumer) {
err = nil // update settings do not depend on a consumer URL
}
if err != nil {
fmt.Fprintf(os.Stderr, "%s\n\ngpu-turnstile: %v\n", versionLine(), err)
return 1
@@ -310,9 +338,15 @@ func forceUpdateCommand(configPath string, elevatedChild bool) int {
}
log, _, logCloser := newLogger(cfg)
defer logCloser.Close()
u := &update.Updater{Repo: cfg.UpdateRepo, Asset: cfg.UpdateAsset, Version: version, Log: log}
if cfg.AppVersion == "dev" {
fmt.Printf("%s: APP_VER=dev, updates disabled\n", versionLine())
return 0
}
u := &update.Updater{Repo: cfg.UpdateRepo, Asset: cfg.UpdateAsset, Version: version, Desired: cfg.AppVersion, Log: log}
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
// Single-shot: one attempt, fail fast when the server is unreachable
// instead of hanging in a TCP connect for minutes.
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
staged, err := u.Check(ctx, exePath)
if err != nil && isPermission(err) && !service.Elevated() {
@@ -381,7 +415,7 @@ func serviceCommand(configPath string, install, noCopy, elevatedChild bool) int
if abs, absErr := filepath.Abs(path); absErr == nil {
path = abs
}
err = service.Install(path, !noCopy)
err = service.Install(path, !noCopy, version)
} else {
err = service.Remove()
}
@@ -401,7 +435,26 @@ func orDisabled(url string) string {
return url
}
// managedComfyCommand resolves how ComfyUI is launched when it is managed:
// COMFY_CMD verbatim, or the standard venv layout under COMFY_DIR. Empty
// when neither is set (unmanaged).
func managedComfyCommand(cfg config.Config) string {
if cfg.ComfyCmd != "" {
return cfg.ComfyCmd
}
if cfg.ComfyDir != "" {
return supervise.DefaultComfyCommand(runtime.GOOS, cfg.ComfyDir, cfg.ComfyURL)
}
return ""
}
func run(ctx context.Context, cfg config.Config, log *slog.Logger, logOut io.Writer, isService bool) error {
// ComfyUI can run as a managed child — COMFY_CMD verbatim, or the
// standard venv layout derived from COMFY_DIR alone: started on demand
// by the proxy, stopped after COMFY_IDLE_TIMEOUT idle (and on shutdown)
// so its VRAM is freed.
comfyCmdLine := managedComfyCommand(cfg)
// The startup line carries the version and every setting and is emitted
// at WARN so it is visible even with the default (quiet) log level.
log.Log(ctx, slog.LevelWarn, "starting gpu-turnstile",
@@ -419,6 +472,7 @@ func run(ctx context.Context, cfg config.Config, log *slog.Logger, logOut io.Wri
"unload_poll_interval", cfg.UnloadPollInterval,
"history_poll_interval", cfg.HistoryPollInterval,
"probe_timeout", cfg.ProbeTimeout,
"health_interval", cfg.HealthInterval,
"free_timeout", cfg.FreeTimeout,
"warm_timeout", cfg.WarmTimeout,
"shutdown_timeout", cfg.ShutdownTimeout,
@@ -426,6 +480,14 @@ func run(ctx context.Context, cfg config.Config, log *slog.Logger, logOut io.Wri
"backoff_max", cfg.BackoffMax,
"prompt_capture_limit", cfg.PromptCaptureLimit,
"warm_model", cfg.WarmModel,
"comfy_cmd", orDisabled(comfyCmdLine),
"comfy_dir", cfg.ComfyDir,
"comfy_idle_timeout", cfg.ComfyIdleTimeout,
"comfy_start_timeout", cfg.ComfyStartTimeout,
"game_procs", cfg.GameProcs,
"gpu_foreign_vram_mb", cfg.GPUForeignVRAMMB,
"gpu_ignore_procs", cfg.GPUIgnoreProcs,
"game_poll_interval", cfg.GamePollInterval,
"auto_update", cfg.AutoUpdate,
"update_interval", cfg.UpdateInterval,
"update_repo", cfg.UpdateRepo,
@@ -452,12 +514,45 @@ func run(ctx context.Context, cfg config.Config, log *slog.Logger, logOut io.Wri
}
}
var comfySup *supervise.Process
if comfyCmdLine != "" {
if cfg.ComfyCmd == "" {
// Derived from COMFY_DIR: flag a wrong-looking layout early,
// while the operator is still watching the startup log. Inside
// a profile the service account may not enter, os.Stat fails
// with EACCES — that reads as "not found" but means "grant
// access", so say so.
python, script := supervise.ComfyLayout(runtime.GOOS, cfg.ComfyDir)
for _, p := range []string{python, script} {
if _, err := os.Stat(p); err != nil {
if isPermission(err) {
log.Warn("COMFY_DIR: not accessible to the service account; grant access or re-run --install-service", "path", p)
} else {
log.Warn("COMFY_DIR: file not found; ComfyUI requests will fail until it exists", "path", p)
}
}
}
}
var err error
comfySup, err = supervise.New("comfy", comfyCmdLine, cfg.ComfyDir, comfyClient.Probe, cfg.ComfyStartTimeout, log)
if err != nil {
return err
}
defer comfySup.Stop()
gpuIdle := func() bool {
state, _, pending := lk.Snapshot()
return state == lock.StateIdle && !pending
}
go comfySup.WatchIdle(ctx, cfg.ComfyIdleTimeout, gpuIdle)
}
srv, err := proxy.New(proxy.Config{
OllamaURL: cfg.OllamaURL,
ComfyURL: cfg.ComfyURL,
Lock: lk,
Ollama: ollamaClient,
Comfy: comfyClient,
ComfySup: comfySup,
Metrics: metrics.New(),
Log: log,
LogColor: !cfg.LogJSON && cfg.LogFile == "" && os.Getenv("NO_COLOR") == "",
@@ -481,19 +576,52 @@ func run(ctx context.Context, cfg config.Config, log *slog.Logger, logOut io.Wri
return err
}
// Probe the enabled upstreams once; failure is logged, not fatal.
probeCtx, probeCancel := context.WithTimeout(ctx, cfg.ProbeTimeout)
// Probe the enabled upstreams once; failure is logged, not fatal. A
// managed ComfyUI is intentionally down at startup — the first request
// starts it — so neither the startup probe nor the health check treats
// that as an outage.
probes := map[string]func(context.Context) error{}
if ollamaClient != nil {
if err := ollamaClient.Probe(probeCtx); err != nil {
log.Warn("ollama probe failed", "url", cfg.OllamaURL, "err", err)
}
probes["ollama"] = ollamaClient.Probe
}
if comfyClient != nil {
if err := comfyClient.Probe(probeCtx); err != nil {
log.Warn("comfy probe failed", "url", cfg.ComfyURL, "err", err)
if comfySup == nil {
probes["comfy"] = comfyClient.Probe
} else {
// Managed upstream: an idle-stopped or still-starting server is
// not an outage, so it is skipped until it has answered once
// (Ready resets on every spawn/stop). After that, a failed
// probe while the process lives is a real "DOWN".
probes["comfy"] = func(ctx context.Context) error {
if !comfySup.Ready() {
if comfySup.Running() {
if err := comfyClient.Probe(ctx); err == nil {
comfySup.MarkReady()
}
}
return errManagedDown
}
return comfyClient.Probe(ctx)
}
}
}
probeCtx, probeCancel := context.WithTimeout(ctx, cfg.ProbeTimeout)
for name, probe := range probes {
if err := probe(probeCtx); err != nil && !errors.Is(err, errManagedDown) {
log.Warn(name+" probe failed", "err", err)
}
}
probeCancel()
if cfg.HealthInterval > 0 {
go healthLoop(ctx, cfg.HealthInterval, cfg.ProbeTimeout, log, probes)
}
// Foreign GPU holders (games, other ML jobs) — enabled by GAME_PROCS
// and/or GPU_FOREIGN_VRAM_MB — hold the lock externally while they run.
if len(cfg.GameProcs) > 0 || cfg.GPUForeignVRAMMB > 0 {
det := game.New(cfg.GameProcs, cfg.GPUForeignVRAMMB, cfg.GPUIgnoreProcs, log)
go gameLoop(ctx, cfg, log, det, lk, ollamaClient, comfySup)
}
// Bind the listeners up front so a port conflict fails fast and the
// readiness notification below really means "accepting connections".
@@ -551,6 +679,128 @@ func run(ctx context.Context, cfg config.Config, log *slog.Logger, logOut io.Wri
return nil
}
// errManagedDown marks a managed upstream that is intentionally stopped
// (idle); health checks skip it instead of logging an outage.
var errManagedDown = errors.New("managed upstream intentionally stopped")
// gameLoop polls for foreign GPU holders (a game, another ML job). While one
// is detected it holds the lock externally so new LLM and image requests
// wait (or are rejected per LLM_BUSY_MODE), and — once in-flight work has
// drained — frees VRAM for it: the managed ComfyUI is stopped and Ollama's
// resident models are unloaded.
func gameLoop(ctx context.Context, cfg config.Config, log *slog.Logger, det *game.Detector, lk *lock.Lock, ollamaClient *ollama.Client, comfySup *supervise.Process) {
ticker := time.NewTicker(cfg.GamePollInterval)
defer ticker.Stop()
held, freed := false, false
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
}
holders, err := det.Check(ctx)
if err != nil && ctx.Err() == nil {
log.Warn("game detection failed", "err", err)
}
switch {
case len(holders) > 0 && !held:
held = true
lk.SetExternal(summarizeHolders(holders))
log.Warn("GPU held by an external process; new LLM/image requests wait",
"holders", summarizeHolders(holders))
case len(holders) == 0 && held:
held, freed = false, false
lk.ClearExternal()
log.Warn("external process released the GPU; resuming")
}
if held && !freed {
if state, _, _ := lk.Snapshot(); state != lock.StateLLM && state != lock.StateImage {
freed = true
freeVRAM(ctx, cfg.UnloadTimeout, log, ollamaClient, comfySup)
}
}
}
}
// summarizeHolders joins holder descriptions for logs and busy responses,
// capping the list so a process name matching dozens of PIDs (system
// services) does not flood the log.
func summarizeHolders(holders []string) string {
const max = 5
if len(holders) > max {
return strings.Join(holders[:max], "; ") + fmt.Sprintf("; +%d more", len(holders)-max)
}
return strings.Join(holders, "; ")
}
// freeVRAM stops the managed ComfyUI (never an external server on its port)
// and unloads Ollama's resident models so the foreign process gets the GPU
// memory.
func freeVRAM(ctx context.Context, unloadTimeout time.Duration, log *slog.Logger, ollamaClient *ollama.Client, comfySup *supervise.Process) {
if comfySup != nil && comfySup.Running() {
log.Warn("stopping the managed ComfyUI to free VRAM")
comfySup.Stop()
}
if ollamaClient == nil {
return
}
uctx, cancel := context.WithTimeout(ctx, unloadTimeout)
defer cancel()
if models, err := ollamaClient.LoadedModels(uctx); err != nil || len(models) == 0 {
return // nothing resident (or ollama unreachable); nothing to free
}
elapsed, err := ollamaClient.UnloadAll(uctx)
if err != nil {
log.Warn("ollama unload incomplete; continuing", "err", err)
return
}
log.Warn("ollama models unloaded to free VRAM", "seconds", elapsed.Seconds())
}
// healthLoop probes the enabled upstreams every interval and logs status
// transitions — "is DOWN" when a previously healthy upstream stops
// answering, "recovered" when it comes back. The first round only
// establishes the baseline; the startup probe already reported that state.
func healthLoop(ctx context.Context, interval, probeTimeout time.Duration, log *slog.Logger, probes map[string]func(context.Context) error) {
ticker := time.NewTicker(interval)
defer ticker.Stop()
up := map[string]bool{}
managed := map[string]bool{}
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
}
for name, probe := range probes {
pctx, cancel := context.WithTimeout(ctx, probeTimeout)
err := probe(pctx)
cancel()
if errors.Is(err, errManagedDown) {
managed[name] = true // intentionally stopped; not an outage
continue
}
now := err == nil
if managed[name] {
// First real probe after an idle stop only re-baselines —
// an on-demand start is not a "recovery".
managed[name] = false
up[name] = now
continue
}
was, seen := up[name]
if seen && now != was {
if now {
log.Warn(name + " upstream recovered")
} else {
log.Warn(name+" upstream is DOWN", "err", err)
}
}
up[name] = now
}
}
}
// updateLoop checks for signed updates on startup and every UPDATE_INTERVAL.
// In service mode a staged update is applied by exiting with exitCodeUpdate
// once the GPU lock is idle; the service recovery configuration restarts the
@@ -561,7 +811,7 @@ func updateLoop(ctx context.Context, cfg config.Config, log *slog.Logger, lk *lo
log.Warn("auto-update disabled: cannot locate executable", "err", err)
return
}
u := &update.Updater{Repo: cfg.UpdateRepo, Asset: cfg.UpdateAsset, Version: version, Log: log}
u := &update.Updater{Repo: cfg.UpdateRepo, Asset: cfg.UpdateAsset, Version: version, Desired: cfg.AppVersion, Log: log}
for {
staged, err := u.Check(ctx, exePath)
if err != nil && ctx.Err() == nil {
+1 -1
View File
@@ -6,7 +6,7 @@
# ComfyUI --listen 0.0.0.0 --port 8189).
services:
gpu-turnstile:
image: git.rambossek.at/public/gpu-turnstile:v0.1.6
image: git.rambossek.at/public/gpu-turnstile:v0.2.0
restart: unless-stopped
environment:
# Each consumer is enabled by setting its URL; leave one unset to
+94 -1
View File
@@ -4,6 +4,7 @@ package config
import (
"bufio"
"errors"
"fmt"
"io"
"log/slog"
@@ -25,6 +26,7 @@ type Config struct {
UnloadPollInterval time.Duration
HistoryPollInterval time.Duration
ProbeTimeout time.Duration
HealthInterval time.Duration
FreeTimeout time.Duration
WarmTimeout time.Duration
ShutdownTimeout time.Duration
@@ -37,6 +39,11 @@ type Config struct {
UpdateRepo string
UpdateAsset string
// AppVersion is the version the user wants to run: "dev" disables
// updates, "stable" tracks the latest release, anything else is an
// exact vX.Y.Z release to pin. From APP_VER; defaults to "stable".
AppVersion string
// LLMBusyMode is "wait" (hold requests until the lock is free or
// LLMWaitTimeout expires) or "reject" (immediately answer with
// LLMBusyStatus + Retry-After when an image job is active or pending).
@@ -44,6 +51,30 @@ type Config struct {
LLMBusyStatus int
BusyRetryAfter int
// ComfyCmd spawns and supervises a ComfyUI server on demand. When
// ComfyCmd is empty but ComfyDir is set, management is enabled with the
// standard venv layout under ComfyDir (.venv + main.py or
// ComfyUI/main.py; --port from the COMFY_URL port) — ComfyCmd is the
// override for other layouts and doubles as the working directory when
// set explicitly. The managed server is stopped after ComfyIdleTimeout
// without requests, freeing its VRAM; ComfyStartTimeout bounds how long
// a request waits for it to come up.
ComfyCmd string
ComfyDir string
ComfyIdleTimeout time.Duration
ComfyStartTimeout time.Duration
// GameProcs (GAME_PROCS) is a watch list of process names; while any of
// them runs, the GPU is treated as held by a foreign process. The
// nvidia-smi path (GPUForeignVRAMMB, GPU_FOREIGN_VRAM_MB) does the same
// when a process not in GPUIgnoreProcs (GPU_IGNORE_PROCS) holds more than
// that many MiB of VRAM. GamePollInterval (GAME_POLL_INTERVAL) is how
// often both checks run.
GameProcs []string
GPUForeignVRAMMB int
GPUIgnoreProcs []string
GamePollInterval time.Duration
WarmModel string
LogLevel slog.Level
LogJSON bool
@@ -64,6 +95,7 @@ func Defaults() Config {
UnloadPollInterval: 500 * time.Millisecond,
HistoryPollInterval: time.Second,
ProbeTimeout: 5 * time.Second,
HealthInterval: 30 * time.Second,
FreeTimeout: 30 * time.Second,
WarmTimeout: 2 * time.Minute,
ShutdownTimeout: 10 * time.Second,
@@ -75,15 +107,29 @@ func Defaults() Config {
UpdateInterval: 6 * time.Hour,
UpdateRepo: "https://git.rambossek.at/PUBLIC/gpu-turnstile",
UpdateAsset: "gpu-turnstile.exe",
AppVersion: "stable",
LLMBusyMode: "wait",
LLMBusyStatus: 503,
BusyRetryAfter: 30,
ComfyIdleTimeout: 5 * time.Minute,
ComfyStartTimeout: 2 * time.Minute,
// ComfyUI runs under python; excluding it (and Ollama) by name keeps
// our own consumers from tripping the foreign-VRAM check.
GPUIgnoreProcs: []string{"ollama", "ollama app", "ollama_llama_server", "python", "pythonw"},
GamePollInterval: 15 * time.Second,
LogLevel: slog.LevelWarn,
}
}
// ErrNoConsumer is returned by Load when neither OLLAMA_URL nor COMFY_URL
// is set. Commands that never talk to an upstream (--force-update) may
// ignore it and proceed with the rest of the configuration.
var ErrNoConsumer = errors.New("at least one of OLLAMA_URL or COMFY_URL must be set (each URL enables its consumer)")
// ParseEnvFile parses a .env-style file: KEY=VALUE lines, blank lines and
// #-comments are ignored, no quoting. A line without '=' is an error.
func ParseEnvFile(r io.Reader) (map[string]string, error) {
@@ -110,6 +156,17 @@ func ParseEnvFile(r io.Reader) (map[string]string, error) {
return values, scanner.Err()
}
// splitList parses a comma-separated setting into trimmed, non-empty items.
func splitList(v string) []string {
var out []string
for _, item := range strings.Split(v, ",") {
if item = strings.TrimSpace(item); item != "" {
out = append(out, item)
}
}
return out
}
func envDuration(getenv func(string) string, name string, dst *time.Duration) error {
v := getenv(name)
if v == "" {
@@ -136,6 +193,8 @@ func Load(getenv func(string) string) (Config, error) {
{"OLLAMA_URL", &cfg.OllamaURL},
{"COMFY_URL", &cfg.ComfyURL},
{"WARM_MODEL", &cfg.WarmModel},
{"COMFY_CMD", &cfg.ComfyCmd},
{"COMFY_DIR", &cfg.ComfyDir},
{"UPDATE_REPO", &cfg.UpdateRepo},
{"UPDATE_ASSET", &cfg.UpdateAsset},
{"LOG_FILE", &cfg.LogFile},
@@ -154,17 +213,34 @@ func Load(getenv func(string) string) (Config, error) {
{"UNLOAD_POLL_INTERVAL", &cfg.UnloadPollInterval},
{"HISTORY_POLL_INTERVAL", &cfg.HistoryPollInterval},
{"PROBE_TIMEOUT", &cfg.ProbeTimeout},
{"HEALTH_INTERVAL", &cfg.HealthInterval},
{"FREE_TIMEOUT", &cfg.FreeTimeout},
{"WARM_TIMEOUT", &cfg.WarmTimeout},
{"SHUTDOWN_TIMEOUT", &cfg.ShutdownTimeout},
{"BACKOFF_INITIAL", &cfg.BackoffInitial},
{"BACKOFF_MAX", &cfg.BackoffMax},
{"COMFY_IDLE_TIMEOUT", &cfg.ComfyIdleTimeout},
{"COMFY_START_TIMEOUT", &cfg.ComfyStartTimeout},
{"UPDATE_INTERVAL", &cfg.UpdateInterval},
{"GAME_POLL_INTERVAL", &cfg.GamePollInterval},
} {
if err := envDuration(getenv, e.name, e.dst); err != nil {
return cfg, err
}
}
if v := getenv("GAME_PROCS"); v != "" {
cfg.GameProcs = splitList(v)
}
if v := getenv("GPU_IGNORE_PROCS"); v != "" {
cfg.GPUIgnoreProcs = splitList(v)
}
if v := getenv("GPU_FOREIGN_VRAM_MB"); v != "" {
n, err := strconv.Atoi(v)
if err != nil || n < 0 {
return cfg, fmt.Errorf("GPU_FOREIGN_VRAM_MB: must be a non-negative integer (MiB, 0 = disabled)")
}
cfg.GPUForeignVRAMMB = n
}
if v := getenv("PROMPT_CAPTURE_LIMIT"); v != "" {
n, err := strconv.ParseInt(v, 10, 64)
if err != nil || n < 0 {
@@ -199,6 +275,17 @@ func Load(getenv func(string) string) (Config, error) {
}
cfg.BusyRetryAfter = n
}
if v := getenv("APP_VER"); v != "" {
switch {
case v == "dev" || v == "stable":
cfg.AppVersion = v
default:
if _, ok := parseVersion(v); !ok {
return cfg, fmt.Errorf("APP_VER: must be \"dev\", \"stable\" or a vX.Y.Z version")
}
cfg.AppVersion = "v" + strings.TrimPrefix(v, "v")
}
}
// LOGLEVEL is the canonical spelling; LOG_LEVEL is kept as an alias.
logLevelValue := getenv("LOGLEVEL")
if logLevelValue == "" {
@@ -218,8 +305,14 @@ func Load(getenv func(string) string) (Config, error) {
default:
return cfg, fmt.Errorf("LOG_FORMAT: must be \"text\" or \"json\"")
}
if cfg.ComfyCmd != "" && cfg.ComfyURL == "" {
return cfg, fmt.Errorf("COMFY_CMD requires COMFY_URL to be set (the proxy needs somewhere to forward)")
}
if cfg.ComfyCmd == "" && cfg.ComfyDir != "" && cfg.ComfyURL == "" {
return cfg, fmt.Errorf("COMFY_DIR without COMFY_CMD requires COMFY_URL to be set (it enables the managed ComfyUI)")
}
if cfg.OllamaURL == "" && cfg.ComfyURL == "" {
return cfg, fmt.Errorf("at least one of OLLAMA_URL or COMFY_URL must be set (each URL enables its consumer)")
return cfg, ErrNoConsumer
}
return cfg, nil
}
+131
View File
@@ -1,6 +1,7 @@
package config
import (
"errors"
"log/slog"
"strings"
"testing"
@@ -39,6 +40,61 @@ func TestLoadRequiresConsumer(t *testing.T) {
if err == nil || !strings.Contains(err.Error(), "OLLAMA_URL") {
t.Fatalf("err = %v, want missing-consumer error", err)
}
if !errors.Is(err, ErrNoConsumer) {
t.Fatalf("err = %v, want errors.Is(err, ErrNoConsumer)", err)
}
}
func TestAppVersion(t *testing.T) {
load := func(appVer string) (Config, error) {
return Load(func(k string) string {
switch k {
case "OLLAMA_URL":
return "http://127.0.0.1:11435"
case "APP_VER":
return appVer
}
return ""
})
}
cfg, err := load("")
if err != nil || cfg.AppVersion != "stable" {
t.Fatalf("default AppVersion = %q, err %v; want stable", cfg.AppVersion, err)
}
for _, v := range []string{"dev", "stable"} {
if cfg, err := load(v); err != nil || cfg.AppVersion != v {
t.Fatalf("APP_VER=%s: got %q, err %v", v, cfg.AppVersion, err)
}
}
if cfg, err := load("1.2.3"); err != nil || cfg.AppVersion != "v1.2.3" {
t.Fatalf("APP_VER=1.2.3: got %q, err %v; want normalized v1.2.3", cfg.AppVersion, err)
}
if _, err := load("nightly"); err == nil {
t.Fatal("APP_VER=nightly: want validation error")
}
}
func TestComfyCmdRequiresURL(t *testing.T) {
_, err := Load(func(k string) string {
if k == "COMFY_CMD" {
return "python main.py"
}
return ""
})
if err == nil || !strings.Contains(err.Error(), "COMFY_CMD requires COMFY_URL") {
t.Fatalf("err = %v, want COMFY_CMD/COMFY_URL validation error", err)
}
if _, err := Load(func(k string) string {
switch k {
case "COMFY_CMD":
return "python main.py"
case "COMFY_URL":
return "http://127.0.0.1:8188"
}
return ""
}); err != nil {
t.Fatalf("COMFY_CMD with COMFY_URL must load: %v", err)
}
}
func TestParseEnvFile(t *testing.T) {
@@ -101,6 +157,8 @@ func TestLoadErrors(t *testing.T) {
{"LLM_BUSY_MODE", "bogus"},
{"LLM_BUSY_STATUS", "200"},
{"BUSY_RETRY_AFTER", "0"},
{"GPU_FOREIGN_VRAM_MB", "-1"},
{"GAME_POLL_INTERVAL", "bogus"},
} {
_, err := Load(func(k string) string {
if k == tc.key {
@@ -116,3 +174,76 @@ func TestLoadErrors(t *testing.T) {
}
}
}
func TestGameDetectionSettings(t *testing.T) {
cfg, err := Load(func(k string) string {
switch k {
case "OLLAMA_URL":
return "http://127.0.0.1:11435"
case "GAME_PROCS":
return " cyberpunk2077.exe, hl2.exe ,, "
case "GPU_FOREIGN_VRAM_MB":
return "1024"
case "GPU_IGNORE_PROCS":
return "ollama, my-trainer"
}
return ""
})
if err != nil {
t.Fatal(err)
}
if len(cfg.GameProcs) != 2 || cfg.GameProcs[0] != "cyberpunk2077.exe" || cfg.GameProcs[1] != "hl2.exe" {
t.Fatalf("GameProcs = %v", cfg.GameProcs)
}
if cfg.GPUForeignVRAMMB != 1024 {
t.Fatalf("GPUForeignVRAMMB = %d", cfg.GPUForeignVRAMMB)
}
if len(cfg.GPUIgnoreProcs) != 2 || cfg.GPUIgnoreProcs[1] != "my-trainer" {
t.Fatalf("GPUIgnoreProcs = %v", cfg.GPUIgnoreProcs)
}
if cfg.GamePollInterval != 15*time.Second {
t.Fatalf("GamePollInterval = %v, want 15s default", cfg.GamePollInterval)
}
// Defaults: both detection paths off, ignore list covers our consumers.
cfg, err = Load(func(k string) string {
if k == "OLLAMA_URL" {
return "http://127.0.0.1:11435"
}
return ""
})
if err != nil {
t.Fatal(err)
}
if len(cfg.GameProcs) != 0 || cfg.GPUForeignVRAMMB != 0 {
t.Fatalf("detection must be off by default: %v %d", cfg.GameProcs, cfg.GPUForeignVRAMMB)
}
if len(cfg.GPUIgnoreProcs) == 0 {
t.Fatal("GPUIgnoreProcs default must not be empty")
}
}
func TestComfyDirOnlyEnablesManaged(t *testing.T) {
// COMFY_DIR without COMFY_CMD and without COMFY_URL is a mistake.
_, err := Load(func(k string) string {
if k == "COMFY_DIR" {
return `C:\ComfyUI`
}
return ""
})
if err == nil || !strings.Contains(err.Error(), "COMFY_DIR") {
t.Fatalf("err = %v, want COMFY_DIR/COMFY_URL validation error", err)
}
// With COMFY_URL it loads — the launch command is derived from the dir.
if _, err := Load(func(k string) string {
switch k {
case "COMFY_DIR":
return `C:\ComfyUI`
case "COMFY_URL":
return "http://127.0.0.1:8189"
}
return ""
}); err != nil {
t.Fatalf("COMFY_DIR with COMFY_URL must load: %v", err)
}
}
+193
View File
@@ -0,0 +1,193 @@
package config
import (
"fmt"
"strconv"
"strings"
)
// sampleEntry is one line pair in the sample env file: a comment and the
// (usually commented-out) KEY=VALUE line.
type sampleEntry struct {
name string
value string
comment string
active bool // rendered uncommented
}
// appVerComment documents APP_VER wherever it is rendered.
const appVerComment = `Version to run: "dev" disables updates, "stable" tracks the latest release, or pin an exact release like v0.1.7`
// sampleEntries lists every setting in sample order. logFile activates the
// LOG_FILE line (Windows install); empty keeps it commented like the rest.
// CFG_VER and APP_VER are not entries — they head the file, always active.
func sampleEntries(logFile string) []sampleEntry {
return []sampleEntry{
{"LISTEN_OLLAMA", ":11434", "Listen address for Ollama-compatible clients (gpu-turnstile poses as Ollama here)", false},
{"LISTEN_COMFY", ":8188", "Listen address for ComfyUI clients (gpu-turnstile poses as ComfyUI here)", false},
{"OLLAMA_URL", "http://127.0.0.1:11434", "Ollama upstream URL; setting it enables the Ollama consumer (default: empty = disabled)", false},
{"COMFY_URL", "http://127.0.0.1:8188", "ComfyUI upstream URL; setting it enables the ComfyUI consumer (default: empty = disabled)", false},
{"WARM_MODEL", "", "Optional model to reload after an image job (default: empty = none)", false},
{"COMFY_CMD", `"C:\ComfyUI\.venv\Scripts\python.exe" ComfyUI\main.py --port 8188`, "Spawn and supervise ComfyUI on demand: the first request starts it, it stops after COMFY_IDLE_TIMEOUT to free VRAM (default: derived from COMFY_DIR; both empty = unmanaged)", false},
{"COMFY_DIR", `C:\ComfyUI`, "Root of a standard ComfyUI venv install (.venv + main.py): setting it alone supervises ComfyUI with the derived launch command; also the working directory for COMFY_CMD", false},
{"COMFY_IDLE_TIMEOUT", "5m", "Stop the managed ComfyUI after this long without requests or jobs (frees VRAM)", false},
{"COMFY_START_TIMEOUT", "2m", "How long a request waits for the managed ComfyUI to come up", false},
{"GAME_PROCS", "cyberpunk2077.exe,hl2.exe", "While a listed process runs, the GPU counts as held by it: requests wait, Ollama unloads, managed ComfyUI stops (default: empty = disabled)", false},
{"GPU_FOREIGN_VRAM_MB", "1024", "Also treat the GPU as held when a process not in GPU_IGNORE_PROCS uses more VRAM than this (needs nvidia-smi; 0/empty = disabled)", false},
{"GPU_IGNORE_PROCS", "ollama,ollama app,ollama_llama_server,python,pythonw", "Process names never counted as foreign GPU users (ComfyUI runs under python)", false},
{"GAME_POLL_INTERVAL", "15s", "How often game/VRAM detection runs (nvidia-smi polls keep the GPU awake; don't go below ~10s)", false},
{"UNLOAD_TIMEOUT", "60s", "How long to wait for Ollama to unload a model", false},
{"JOB_TIMEOUT", "15m", "Maximum time to wait for a ComfyUI job", false},
{"LLM_WAIT_TIMEOUT", "10m", "Max time an LLM request waits for the GPU before being answered 503 (wait mode)", false},
{"LLM_BUSY_MODE", "wait", `How blocked LLM requests are handled: "wait" holds them, "reject" fails them immediately`, false},
{"LLM_BUSY_STATUS", "503", "HTTP status for rejected LLM requests in reject mode (400-599, e.g. 429)", false},
{"BUSY_RETRY_AFTER", "30", "Seconds sent as Retry-After on busy responses (both modes)", false},
{"LOGLEVEL", "warn", `Log verbosity: debug, info, warn, error; "info" logs every request (LOG_LEVEL works too)`, false},
{"LOG_FORMAT", "text", `Log format: "text" or "json"`, false},
{"LOG_FILE", logFile, "Append logs to this file instead of stderr (a Windows service has no console)", logFile != ""},
{"UNLOAD_POLL_INTERVAL", "500ms", "/api/ps poll interval while unloading", false},
{"HISTORY_POLL_INTERVAL", "1s", "/history/<id> poll interval while a job runs", false},
{"PROBE_TIMEOUT", "5s", "Probe timeout for the startup probe and the periodic health check", false},
{"HEALTH_INTERVAL", "30s", "How often enabled upstreams are probed; status changes are logged", false},
{"FREE_TIMEOUT", "30s", "Timeout for the POST /free call after an image job", false},
{"WARM_TIMEOUT", "2m", "Timeout for the warm-model reload after an image job", false},
{"SHUTDOWN_TIMEOUT", "10s", "Graceful shutdown timeout on SIGINT/SIGTERM", false},
{"BACKOFF_INITIAL", "1s", "First retry wait when an upstream connection fails", false},
{"BACKOFF_MAX", "60s", "Cap for the exponential retry backoff", false},
{"PROMPT_CAPTURE_LIMIT", "65536", "Bytes of the /prompt response buffered to find prompt_id (pass-through unaffected)", false},
{"AUTO_UPDATE", "true", "Poll the releases API for signed updates", false},
{"UPDATE_INTERVAL", "6h", "Auto-update check interval", false},
{"UPDATE_REPO", "https://git.rambossek.at/PUBLIC/gpu-turnstile", "Repository to check for releases", false},
{"UPDATE_ASSET", "gpu-turnstile.exe", "Release asset to download", false},
}
}
// SampleEnv renders a sample .env file covering every setting, each with a
// comment line. Everything is commented out — so all defaults apply —
// except the CFG_VER/APP_VER header and LOG_FILE when logFile is non-empty
// (a Windows service has no console). CFG_VER records the version that
// wrote the file so installs — and startups after an update — can upgrade
// it.
func SampleEnv(version, logFile string) string {
// CFG_VER is always a concrete vX.Y.Z — never "dev". A dev build
// stamps v0.0.0, which sorts older than any release, so the next
// release install upgrades the file and stamps a proper version.
cfgVer := version
if cfgVer == "" || cfgVer == "dev" {
cfgVer = "v0.0.0"
}
var b strings.Builder
fmt.Fprintf(&b, "CFG_VER=%s\n", cfgVer)
b.WriteString("# Config format reference, written by the installer — do not edit.\n")
b.WriteString("# The installer uses it to append newly added settings on updates.\n\n")
b.WriteString("# gpu-turnstile configuration\n")
b.WriteString("# KEY=VALUE lines; \"#\" starts a comment. Every setting below is at its\n")
b.WriteString("# default and commented out — remove the \"#\" to change it.\n")
b.WriteString("# At least one of OLLAMA_URL / COMFY_URL must be set for the proxy to start.\n\n")
writeEntry(&b, sampleEntry{"APP_VER", "stable", appVerComment, true})
for _, e := range sampleEntries(logFile) {
writeEntry(&b, e)
}
return b.String()
}
func writeEntry(b *strings.Builder, e sampleEntry) {
fmt.Fprintf(b, "# %s\n", e.comment)
if e.active {
fmt.Fprintf(b, "%s=%s\n\n", e.name, e.value)
} else {
fmt.Fprintf(b, "#%s=%s\n\n", e.name, e.value)
}
}
// SyncSample upgrades an installer-written env file: when its CFG_VER says
// it was written by an older gpu-turnstile, every setting the file does not
// mention — commented or not — is appended at the end, and CFG_VER is
// updated to version. Files without CFG_VER (hand-written or foreign),
// up-to-date files and "dev" builds are returned unchanged; changed reports
// whether the returned content differs.
func SyncSample(data, version, logFile string) (string, bool) {
if version == "" || version == "dev" {
return data, false
}
values, err := ParseEnvFile(strings.NewReader(data))
if err != nil || values["CFG_VER"] == "" {
return data, false // not installer-written; the caller decides
}
if compareVersions(values["CFG_VER"], version) >= 0 {
return data, false // same or newer
}
present := make(map[string]bool)
for _, line := range strings.Split(data, "\n") {
line = strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(line), "#"))
if name, _, ok := strings.Cut(line, "="); ok {
present[strings.TrimSpace(name)] = true
}
}
lines := strings.Split(data, "\n")
for i, l := range lines {
if strings.HasPrefix(strings.TrimSpace(l), "CFG_VER=") {
lines[i] = "CFG_VER=" + version
break
}
}
out := strings.Join(lines, "\n")
if !strings.HasSuffix(out, "\n") {
out += "\n"
}
var b strings.Builder
if !present["APP_VER"] {
fmt.Fprintf(&b, "\n# Added by gpu-turnstile %s:\n", version)
writeEntry(&b, sampleEntry{"APP_VER", "stable", appVerComment, true})
}
for _, e := range sampleEntries(logFile) {
if !present[e.name] {
fmt.Fprintf(&b, "\n# Added by gpu-turnstile %s:\n", version)
writeEntry(&b, e)
}
}
out += b.String()
return out, out != data
}
// compareVersions orders two vX.Y.Z version strings. Unparseable versions
// (including "dev") sort before any release.
func compareVersions(a, b string) int {
pa, oka := parseVersion(a)
pb, okb := parseVersion(b)
if oka != okb {
if oka {
return 1
}
return -1
}
for i := range pa {
if pa[i] != pb[i] {
if pa[i] > pb[i] {
return 1
}
return -1
}
}
return 0
}
func parseVersion(v string) ([3]int, bool) {
var out [3]int
v = strings.TrimPrefix(strings.TrimSpace(v), "v")
parts := strings.Split(v, ".")
if len(parts) != 3 {
return out, false
}
for i, p := range parts {
n, err := strconv.Atoi(p)
if err != nil || n < 0 {
return out, false
}
out[i] = n
}
return out, true
}
+156
View File
@@ -0,0 +1,156 @@
package config
import (
"strings"
"testing"
)
const sampleLogPath = `C:\ProgramData\gpu-turnstile\gpu-turnstile.log`
var allSettingNames = []string{
"LISTEN_OLLAMA", "LISTEN_COMFY", "OLLAMA_URL", "COMFY_URL",
"WARM_MODEL", "UNLOAD_TIMEOUT", "JOB_TIMEOUT", "LLM_WAIT_TIMEOUT",
"LLM_BUSY_MODE", "LLM_BUSY_STATUS", "BUSY_RETRY_AFTER",
"LOGLEVEL", "LOG_FORMAT", "LOG_FILE",
"UNLOAD_POLL_INTERVAL", "HISTORY_POLL_INTERVAL", "PROBE_TIMEOUT",
"FREE_TIMEOUT", "WARM_TIMEOUT", "SHUTDOWN_TIMEOUT",
"BACKOFF_INITIAL", "BACKOFF_MAX", "PROMPT_CAPTURE_LIMIT",
"AUTO_UPDATE", "UPDATE_INTERVAL", "UPDATE_REPO", "UPDATE_ASSET",
}
func TestSampleEnv(t *testing.T) {
sample := SampleEnv("v0.1.7", sampleLogPath)
if !strings.HasPrefix(sample, "CFG_VER=v0.1.7\n") {
t.Errorf("first line does not carry CFG_VER: %q", strings.SplitN(sample, "\n", 2)[0])
}
// Every setting known to Load must appear.
for _, name := range allSettingNames {
if !strings.Contains(sample, name+"=") {
t.Errorf("sample is missing %s", name)
}
}
// The sample must parse cleanly; active values are CFG_VER, APP_VER
// and LOG_FILE.
values, err := ParseEnvFile(strings.NewReader(sample))
if err != nil {
t.Fatalf("sample does not parse: %v", err)
}
want := map[string]string{"CFG_VER": "v0.1.7", "APP_VER": "stable", "LOG_FILE": sampleLogPath}
if len(values) != len(want) {
t.Fatalf("active values = %v, want %v", values, want)
}
for k, v := range want {
if values[k] != v {
t.Errorf("%s = %q, want %q", k, values[k], v)
}
}
// Without a log path LOG_FILE stays commented out.
values, err = ParseEnvFile(strings.NewReader(SampleEnv("v0.1.7", "")))
if err != nil {
t.Fatalf("sample without log path does not parse: %v", err)
}
if len(values) != 2 || values["LOG_FILE"] != "" {
t.Fatalf("active values = %v, want only CFG_VER and APP_VER", values)
}
}
func TestSyncSample(t *testing.T) {
old := SampleEnv("v0.1.6", sampleLogPath)
// Simulate a setting that did not exist yet when the file was written.
old = strings.Replace(old, "#UPDATE_ASSET=gpu-turnstile.exe\n", "", 1)
out, changed := SyncSample(old, "v0.1.7", sampleLogPath)
if !changed {
t.Fatal("older installer file was not upgraded")
}
if !strings.Contains(out, "\nCFG_VER=v0.1.7\n") && !strings.HasPrefix(out, "CFG_VER=v0.1.7\n") {
t.Error("CFG_VER was not updated to the new version")
}
if !strings.Contains(out, "#UPDATE_ASSET=gpu-turnstile.exe") {
t.Error("missing setting was not appended")
}
if !strings.Contains(out, "LOG_FILE="+sampleLogPath) {
t.Error("existing active LOG_FILE was lost")
}
if !strings.Contains(out, "Added by gpu-turnstile v0.1.7") {
t.Error("appended section is not attributed")
}
if _, err := ParseEnvFile(strings.NewReader(out)); err != nil {
t.Fatalf("upgraded file does not parse: %v", err)
}
// A file written by the same or a newer version is left alone.
if _, changed := SyncSample(SampleEnv("v0.1.7", sampleLogPath), "v0.1.7", sampleLogPath); changed {
t.Error("same-version file was modified")
}
if _, changed := SyncSample(SampleEnv("v0.2.0", sampleLogPath), "v0.1.7", sampleLogPath); changed {
t.Error("newer-version file was modified")
}
// Files without CFG_VER are not installer-written; the installer
// replaces them, SyncSample leaves them alone.
user := "OLLAMA_URL=http://host:11434\n"
if out, changed := SyncSample(user, "v0.1.7", sampleLogPath); changed || out != user {
t.Error("file without CFG_VER was modified")
}
// A dev build never upgrades.
if _, changed := SyncSample(old, "dev", sampleLogPath); changed {
t.Error("dev build modified the file")
}
}
func TestSyncSampleAppendsMissingAppVer(t *testing.T) {
old := SampleEnv("v0.1.6", sampleLogPath)
old = strings.Replace(old, "# "+appVerComment+"\n", "", 1)
old = strings.Replace(old, "APP_VER=stable\n", "", 1)
out, changed := SyncSample(old, "v0.1.7", sampleLogPath)
if !changed {
t.Fatal("file without APP_VER was not upgraded")
}
values, err := ParseEnvFile(strings.NewReader(out))
if err != nil {
t.Fatalf("upgraded file does not parse: %v", err)
}
if values["APP_VER"] != "stable" {
t.Fatalf("APP_VER = %q, want appended default \"stable\"", values["APP_VER"])
}
}
func TestSampleEnvDevStampsConcreteVersion(t *testing.T) {
// A dev build must never write CFG_VER=dev: it stamps v0.0.0, and the
// next release install upgrades the file to a proper version.
dev := SampleEnv("dev", sampleLogPath)
if !strings.HasPrefix(dev, "CFG_VER=v0.0.0\n") {
t.Errorf("dev sample first line: %q", strings.SplitN(dev, "\n", 2)[0])
}
out, changed := SyncSample(dev, "v0.1.7", sampleLogPath)
if !changed || !strings.HasPrefix(out, "CFG_VER=v0.1.7\n") {
t.Errorf("dev-stamped file was not upgraded to v0.1.7 (changed=%v)", changed)
}
}
func TestCompareVersions(t *testing.T) {
cases := []struct {
a, b string
want int
}{
{"v0.1.6", "v0.1.7", -1},
{"v0.1.7", "v0.1.7", 0},
{"v1.0.0", "v0.9.9", 1},
{"0.1.7", "v0.1.6", 1},
{"dev", "v0.1.7", -1},
{"v0.1.7", "dev", 1},
{"dev", "dev", 0},
}
for _, c := range cases {
if got := compareVersions(c.a, c.b); got != c.want {
t.Errorf("compareVersions(%q, %q) = %d, want %d", c.a, c.b, got, c.want)
}
}
}
+159
View File
@@ -0,0 +1,159 @@
// Package game detects processes outside gpu-turnstile's control that hold
// the GPU — typically a game — so the proxy can block new GPU work and free
// VRAM while they run. Two detection paths: an explicit process watch list
// (GAME_PROCS) and a foreign-VRAM threshold via nvidia-smi
// (GPU_FOREIGN_VRAM_MB) that catches anything not on the ignore list.
package game
import (
"context"
"errors"
"fmt"
"log/slog"
"os/exec"
"strconv"
"strings"
)
// Process is one running OS process.
type Process struct {
PID int
Name string
}
// computeApp is one process holding GPU memory, as reported by nvidia-smi.
type computeApp struct {
PID int
UsedMB int
}
// Detector checks whether a foreign process holds the GPU. The zero value
// (no watch list, no threshold) never detects anything; main only starts the
// poll loop when at least one path is configured.
type Detector struct {
procs map[string]bool // normalized names from GAME_PROCS
vramMB int // foreign VRAM threshold; 0 = disabled
ignore map[string]bool // normalized names never counted as foreign
log *slog.Logger
noNvidia bool // nvidia-smi was not found; VRAM path disabled for good
}
// New builds a Detector from the configured watch list, VRAM threshold in
// MiB (0 disables the nvidia-smi path) and ignore list. Names are matched
// case-insensitively, with or without a trailing ".exe".
func New(procs []string, vramMB int, ignore []string, log *slog.Logger) *Detector {
if log == nil {
log = slog.Default()
}
return &Detector{
procs: nameSet(procs),
vramMB: vramMB,
ignore: nameSet(ignore),
log: log,
}
}
// normName lowercases a process name and strips a trailing ".exe" so the
// watch and ignore lists match on Windows and Linux spellings alike.
func normName(s string) string {
return strings.TrimSuffix(strings.ToLower(strings.TrimSpace(s)), ".exe")
}
func nameSet(names []string) map[string]bool {
set := make(map[string]bool, len(names))
for _, n := range names {
if n = normName(n); n != "" {
set[n] = true
}
}
return set
}
// Check looks once for foreign GPU holders and returns a human-readable
// description of each (empty when the GPU is free for gpu-turnstile's
// consumers). A failing nvidia-smi call is returned as an error only when
// the process list found nothing; a missing nvidia-smi binary disables the
// VRAM path permanently (logged once).
func (d *Detector) Check(ctx context.Context) ([]string, error) {
ps, psErr := processes()
if d.vramMB <= 0 || d.noNvidia {
return d.detect(ps, nil), psErr
}
apps, err := queryComputeApps(ctx)
if errors.Is(err, exec.ErrNotFound) {
d.noNvidia = true
d.log.Warn("GPU_FOREIGN_VRAM_MB is set but nvidia-smi was not found; VRAM detection disabled")
return d.detect(ps, nil), nil
}
if err != nil {
return d.detect(ps, nil), err
}
return d.detect(ps, apps), nil
}
// detect is the pure core of Check: given the process table and (optionally)
// the nvidia-smi compute-apps list, it returns the foreign holders.
func (d *Detector) detect(ps []Process, apps []computeApp) []string {
var holders []string
for _, p := range ps {
if d.procs[normName(p.Name)] {
holders = append(holders, fmt.Sprintf("%s (pid %d)", p.Name, p.PID))
}
}
if d.vramMB > 0 && apps != nil {
names := make(map[int]string, len(ps))
for _, p := range ps {
names[p.PID] = p.Name
}
for _, a := range apps {
name := names[a.PID]
if d.ignore[normName(name)] || a.UsedMB < d.vramMB {
continue
}
if name == "" {
name = "unknown process"
}
holders = append(holders, fmt.Sprintf("%s (pid %d) using %d MiB VRAM", name, a.PID, a.UsedMB))
}
}
return holders
}
// queryComputeApps runs nvidia-smi and parses the per-process VRAM list.
// Note: under Windows' WDDM driver, nvidia-smi only sees compute
// allocations, so graphics-only games may not appear there — GAME_PROCS is
// the reliable path on Windows; on Linux both work.
func queryComputeApps(ctx context.Context) ([]computeApp, error) {
out, err := exec.CommandContext(ctx, "nvidia-smi",
"--query-compute-apps=pid,used_memory", "--format=csv,noheader,nounits").Output()
if err != nil {
return nil, err
}
return parseComputeApps(string(out))
}
// parseComputeApps parses "pid, used_memory" CSV lines (no header, MiB
// units). Unsupported rows ("N/A" on WDDM) are skipped.
func parseComputeApps(out string) ([]computeApp, error) {
var apps []computeApp
for _, line := range strings.Split(out, "\n") {
line = strings.TrimSpace(line)
if line == "" {
continue
}
pidStr, memStr, ok := strings.Cut(line, ",")
if !ok {
return nil, fmt.Errorf("nvidia-smi: unexpected line %q", line)
}
pid, err := strconv.Atoi(strings.TrimSpace(pidStr))
if err != nil {
return nil, fmt.Errorf("nvidia-smi: unexpected pid in %q", line)
}
mem, err := strconv.Atoi(strings.TrimSpace(memStr))
if err != nil {
continue // "N/A" and friends: unsupported under WDDM
}
apps = append(apps, computeApp{PID: pid, UsedMB: mem})
}
return apps, nil
}
+96
View File
@@ -0,0 +1,96 @@
package game
import (
"runtime"
"slices"
"testing"
)
func TestNormName(t *testing.T) {
for in, want := range map[string]string{
"Cyberpunk2077.exe": "cyberpunk2077",
"ollama": "ollama",
"OLLAMA APP.EXE": "ollama app",
" python.exe ": "python",
"hl2": "hl2",
} {
if got := normName(in); got != want {
t.Errorf("normName(%q) = %q, want %q", in, got, want)
}
}
}
func TestParseComputeApps(t *testing.T) {
apps, err := parseComputeApps("1234, 512\n 42 , 8192 \n\n")
if err != nil {
t.Fatal(err)
}
want := []computeApp{{PID: 1234, UsedMB: 512}, {PID: 42, UsedMB: 8192}}
if !slices.Equal(apps, want) {
t.Errorf("got %+v, want %+v", apps, want)
}
// WDDM reports "N/A" for memory; those rows are skipped, not fatal.
apps, err = parseComputeApps("1234, N/A\n42, 1024\n")
if err != nil {
t.Fatal(err)
}
if !slices.Equal(apps, []computeApp{{PID: 42, UsedMB: 1024}}) {
t.Errorf("got %+v", apps)
}
if _, err := parseComputeApps("garbage\n"); err == nil {
t.Error("expected an error for a malformed line")
}
}
func TestDetect(t *testing.T) {
d := New([]string{"Cyberpunk2077.exe", "hl2"}, 1024,
[]string{"ollama", "python", "pythonw"}, nil)
ps := []Process{
{PID: 10, Name: "ollama.exe"},
{PID: 20, Name: "python.exe"},
{PID: 30, Name: "cyberpunk2077.exe"},
}
apps := []computeApp{
{PID: 10, UsedMB: 8192}, // ignored: ollama
{PID: 20, UsedMB: 4096}, // ignored: python (ComfyUI)
{PID: 40, UsedMB: 2048}, // foreign, above threshold
{PID: 50, UsedMB: 100}, // foreign but below threshold
}
holders := d.detect(ps, apps)
if len(holders) != 2 {
t.Fatalf("got %v, want 2 holders", holders)
}
if holders[0] != "cyberpunk2077.exe (pid 30)" {
t.Errorf("holders[0] = %q", holders[0])
}
if holders[1] != "unknown process (pid 40) using 2048 MiB VRAM" {
t.Errorf("holders[1] = %q", holders[1])
}
}
func TestDetectNothingConfigured(t *testing.T) {
d := New(nil, 0, nil, nil)
if got := d.detect([]Process{{PID: 1, Name: "game.exe"}}, nil); len(got) != 0 {
t.Errorf("got %v, want none", got)
}
}
func TestProcessesLive(t *testing.T) {
if runtime.GOOS != "windows" && runtime.GOOS != "linux" {
t.Skip("no process listing on this platform")
}
ps, err := processes()
if err != nil {
t.Fatal(err)
}
if len(ps) == 0 {
t.Fatal("no processes listed")
}
for _, p := range ps {
if p.Name == "" {
t.Errorf("pid %d has an empty name", p.PID)
}
}
}
+30
View File
@@ -0,0 +1,30 @@
//go:build linux
package game
import (
"os"
"strconv"
"strings"
)
// processes lists the running processes from /proc/<pid>/comm.
func processes() ([]Process, error) {
entries, err := os.ReadDir("/proc")
if err != nil {
return nil, err
}
var ps []Process
for _, e := range entries {
pid, err := strconv.Atoi(e.Name())
if err != nil {
continue
}
comm, err := os.ReadFile("/proc/" + e.Name() + "/comm")
if err != nil {
continue // process vanished mid-walk
}
ps = append(ps, Process{PID: pid, Name: strings.TrimSpace(string(comm))})
}
return ps, nil
}
+7
View File
@@ -0,0 +1,7 @@
//go:build !windows && !linux
package game
// processes is unsupported on this platform; the process watch list never
// matches and the nvidia-smi path reports PIDs without names.
func processes() ([]Process, error) { return nil, nil }
+35
View File
@@ -0,0 +1,35 @@
//go:build windows
package game
import (
"unsafe"
"golang.org/x/sys/windows"
)
// processes lists the running processes via the Toolhelp32 snapshot API.
func processes() ([]Process, error) {
h, err := windows.CreateToolhelp32Snapshot(windows.TH32CS_SNAPPROCESS, 0)
if err != nil {
return nil, err
}
defer windows.CloseHandle(h) //nolint:errcheck // best effort
var entry windows.ProcessEntry32
entry.Size = uint32(unsafe.Sizeof(entry))
if err := windows.Process32First(h, &entry); err != nil {
return nil, err
}
var ps []Process
for {
ps = append(ps, Process{
PID: int(entry.ProcessID),
Name: windows.UTF16ToString(entry.ExeFile[:]),
})
if err := windows.Process32Next(h, &entry); err != nil {
break // ERROR_NO_MORE_FILES ends the walk
}
}
return ps, nil
}
+40 -5
View File
@@ -1,6 +1,8 @@
// Package lock implements the two-mode GPU arbitration lock: any number of
// concurrent LLM requests ("readers") or exactly one image job ("writer"),
// with image jobs taking priority over newly arriving LLM requests.
// with image jobs taking priority over newly arriving LLM requests. An
// external hold (SetExternal) blocks new grants of both kinds while a
// foreign process — e.g. a game — holds the GPU; in-flight work drains.
package lock
import (
@@ -16,6 +18,7 @@ const (
StateIdle State = "idle"
StateLLM State = "llm"
StateImage State = "image"
StateExternal State = "external"
)
type imageWaiter struct{ id uint64 }
@@ -30,6 +33,7 @@ type Lock struct {
imageActive bool // an image job holds the GPU
imageQ []imageWaiter
nextID uint64
external string // non-empty: a foreign process (e.g. a game) holds the GPU
log *slog.Logger
}
@@ -52,12 +56,41 @@ func (l *Lock) logTransition(msg string, args ...any) {
}
}
// SetExternal records that a process outside gpu-turnstile's control (a
// game, another ML job) holds the GPU: new LLM and image grants block until
// ClearExternal. In-flight work is not preempted. holder describes the
// process for logs and busy responses.
func (l *Lock) SetExternal(holder string) {
l.mu.Lock()
l.external = holder
l.broadcast()
l.mu.Unlock()
l.logTransition("lock transition", "state", StateExternal, "holder", holder)
}
// ClearExternal lifts the external hold; waiting LLM and image requests
// proceed.
func (l *Lock) ClearExternal() {
l.mu.Lock()
l.external = ""
l.broadcast()
l.mu.Unlock()
l.logTransition("lock transition", "state", StateIdle)
}
// External returns the current external holder, or "" when none.
func (l *Lock) External() string {
l.mu.Lock()
defer l.mu.Unlock()
return l.external
}
// AcquireLLM blocks until no image job is active or pending, then registers
// one in-flight LLM request. Returns ctx.Err() if the context is cancelled
// while waiting; no state is changed in that case.
func (l *Lock) AcquireLLM(ctx context.Context) error {
l.mu.Lock()
for l.imageActive || len(l.imageQ) > 0 {
for l.imageActive || len(l.imageQ) > 0 || l.external != "" {
ch := l.change
l.mu.Unlock()
select {
@@ -76,10 +109,10 @@ func (l *Lock) AcquireLLM(ctx context.Context) error {
// TryAcquireLLM acquires one in-flight LLM slot without waiting and
// reports whether it succeeded. It fails when an image job is active or
// pending.
// pending or an external hold is set.
func (l *Lock) TryAcquireLLM() bool {
l.mu.Lock()
if l.imageActive || len(l.imageQ) > 0 {
if l.imageActive || len(l.imageQ) > 0 || l.external != "" {
l.mu.Unlock()
return false
}
@@ -120,7 +153,7 @@ func (l *Lock) AcquireImage(ctx context.Context) error {
for {
l.mu.Lock()
if l.imageQ[0].id == w.id && l.n == 0 && !l.imageActive {
if l.imageQ[0].id == w.id && l.n == 0 && !l.imageActive && l.external == "" {
l.imageQ = l.imageQ[1:]
l.imageActive = true
l.mu.Unlock()
@@ -166,6 +199,8 @@ func (l *Lock) Snapshot() (state State, llmInflight int, imagePending bool) {
state = StateImage
case l.n > 0:
state = StateLLM
case l.external != "":
state = StateExternal
default:
state = StateIdle
}
+67
View File
@@ -226,3 +226,70 @@ func TestRace(t *testing.T) {
t.Fatalf("leaked lock state: state=%s n=%d pending=%v", state, n, pending)
}
}
func TestExternalHoldBlocksBoth(t *testing.T) {
lk := New(nil)
ctx := context.Background()
lk.SetExternal("game.exe (pid 42)")
if lk.TryAcquireLLM() {
t.Fatal("TryAcquireLLM succeeded during external hold")
}
if got := lk.External(); got != "game.exe (pid 42)" {
t.Fatalf("External() = %q", got)
}
if state, _, _ := lk.Snapshot(); state != StateExternal {
t.Fatalf("state=%s, want external", state)
}
llmAcquired := make(chan struct{})
go func() {
if err := lk.AcquireLLM(ctx); err != nil {
t.Error(err)
}
close(llmAcquired)
}()
assertBlocked(t, llmAcquired, "LLM acquire during external hold")
imageAcquired := make(chan struct{})
go func() {
if err := lk.AcquireImage(ctx); err != nil {
t.Error(err)
}
close(imageAcquired)
}()
assertBlocked(t, imageAcquired, "image acquire during external hold")
lk.ClearExternal()
// The queued image job wins over the LLM waiter (image priority).
waitFor(t, imageAcquired, "image acquire after ClearExternal")
assertBlocked(t, llmAcquired, "LLM acquire while image active")
lk.ReleaseImage()
waitFor(t, llmAcquired, "LLM acquire after image release")
lk.ReleaseLLM()
}
func TestExternalHoldDoesNotPreempt(t *testing.T) {
lk := New(nil)
ctx := context.Background()
if err := lk.AcquireLLM(ctx); err != nil {
t.Fatal(err)
}
lk.SetExternal("game.exe")
// In-flight LLM work keeps the llm state; the hold blocks new grants.
if state, _, _ := lk.Snapshot(); state != StateLLM {
t.Fatalf("state=%s, want llm while work in flight", state)
}
if lk.TryAcquireLLM() {
t.Fatal("TryAcquireLLM succeeded during external hold")
}
lk.ReleaseLLM()
if state, _, _ := lk.Snapshot(); state != StateExternal {
t.Fatalf("state=%s, want external after drain", state)
}
lk.ClearExternal()
if state, _, _ := lk.Snapshot(); state != StateIdle {
t.Fatalf("state=%s, want idle", state)
}
}
+1 -1
View File
@@ -89,7 +89,7 @@ func (m *Metrics) Render(w io.Writer, state string, llmInflight int, imagePendin
fmt.Fprint(w, `# HELP gpu_turnstile_state Current GPU state (1 for the active state).
# TYPE gpu_turnstile_state gauge
`)
for _, s := range []string{"idle", "llm", "image"} {
for _, s := range []string{"idle", "llm", "image", "external"} {
v := 0
if s == state {
v = 1
+62 -3
View File
@@ -26,6 +26,7 @@ import (
"gpu-turnstile/internal/lock"
"gpu-turnstile/internal/metrics"
"gpu-turnstile/internal/ollama"
"gpu-turnstile/internal/supervise"
)
// defaultCaptureLimit bounds how much of a /prompt response body is
@@ -44,6 +45,11 @@ type Config struct {
Metrics *metrics.Metrics
Log *slog.Logger
// ComfySup, when non-nil, is the managed ComfyUI process: any ComfyUI
// request starts it on demand, /prompt additionally waits for
// readiness before taking the GPU lock. nil = unmanaged upstream.
ComfySup *supervise.Process
// LogColor enables ANSI colors in per-request log lines. Ignored when
// the log level is above INFO (request lines are not emitted at all).
LogColor bool
@@ -438,7 +444,7 @@ func isLLMRequest(r *http.Request) bool {
return r.Method == http.MethodPost && llmPaths[r.URL.Path]
}
// OllamaHandler serves the Ollama-facing listener.
// OllamaHandler serves the listener for Ollama-compatible clients.
func (s *Server) OllamaHandler() http.Handler {
return s.logRequests("ollama", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
@@ -458,10 +464,14 @@ func (s *Server) OllamaHandler() http.Handler {
if s.busyMode == "reject" {
if !s.cfg.Lock.TryAcquireLLM() {
s.cfg.Metrics.ObserveLockWait("llm", time.Since(start).Seconds())
msg := "GPU busy: image job active or queued"
if holder := s.cfg.Lock.External(); holder != "" {
msg = "GPU busy: " + holder
}
s.log.Info("llm request rejected; GPU busy",
"path", r.URL.Path, "status", s.busyStatus)
w.Header().Set("Retry-After", strconv.Itoa(s.busyRetryAfter))
http.Error(w, "GPU busy: image job active or queued", s.busyStatus)
http.Error(w, msg, s.busyStatus)
return
}
s.cfg.Metrics.ObserveLockWait("llm", time.Since(start).Seconds())
@@ -485,7 +495,7 @@ func (s *Server) OllamaHandler() http.Handler {
}))
}
// ComfyHandler serves the ComfyUI-facing listener.
// ComfyHandler serves the listener for ComfyUI clients.
func (s *Server) ComfyHandler() http.Handler {
return s.logRequests("comfy", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
@@ -500,6 +510,21 @@ func (s *Server) ComfyHandler() http.Handler {
s.handlePrompt(w, r)
return
}
if s.cfg.ComfySup != nil {
// Any other ComfyUI request also wakes the managed server; the
// retry backoff bridges the time it needs to come up. While a
// foreign process holds the GPU we refuse to spawn it — the
// request gets the busy answer instead of fighting for VRAM.
if holder := s.cfg.Lock.External(); holder != "" && !s.cfg.ComfySup.Running() {
w.Header().Set("Retry-After", strconv.Itoa(s.busyRetryAfter))
http.Error(w, "GPU busy: "+holder, http.StatusServiceUnavailable)
return
}
if err := s.cfg.ComfySup.EnsureRunning(); err != nil {
http.Error(w, fmt.Sprintf("cannot start ComfyUI: %v", err), http.StatusBadGateway)
return
}
}
s.comfyProxy.ServeHTTP(w, r)
}))
}
@@ -539,6 +564,23 @@ func (w *captureWriter) Unwrap() http.ResponseWriter { return w.ResponseWriter }
func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) {
log := s.log.With("op", "image")
// Normally the managed server is brought up *before* taking the GPU
// lock: torch can take a minute to load, and LLM traffic should keep
// flowing in the meantime. While a foreign process holds the GPU, LLM
// traffic is blocked anyway and a fresh ComfyUI would fight it for
// VRAM — so the lock comes first in that case.
comfyFirst := s.cfg.ComfySup != nil && s.cfg.Lock.External() == ""
if comfyFirst {
if err := s.cfg.ComfySup.EnsureRunning(); err != nil {
http.Error(w, fmt.Sprintf("cannot start ComfyUI: %v", err), http.StatusBadGateway)
return
}
if err := s.cfg.ComfySup.WaitReady(r.Context()); err != nil {
http.Error(w, fmt.Sprintf("ComfyUI did not become ready: %v", err), http.StatusBadGateway)
return
}
}
start := time.Now()
if err := s.cfg.Lock.AcquireImage(r.Context()); err != nil {
if errors.Is(err, context.DeadlineExceeded) {
@@ -549,6 +591,19 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) {
s.cfg.Metrics.ObserveLockWait("image", time.Since(start).Seconds())
log.Info("image lock acquired")
if s.cfg.ComfySup != nil && !comfyFirst {
if err := s.cfg.ComfySup.EnsureRunning(); err != nil {
s.cfg.Lock.ReleaseImage()
http.Error(w, fmt.Sprintf("cannot start ComfyUI: %v", err), http.StatusBadGateway)
return
}
if err := s.cfg.ComfySup.WaitReady(r.Context()); err != nil {
s.cfg.Lock.ReleaseImage()
http.Error(w, fmt.Sprintf("ComfyUI did not become ready: %v", err), http.StatusBadGateway)
return
}
}
if s.cfg.Ollama != nil {
uctx, ucancel := context.WithTimeout(r.Context(), s.cfg.UnloadTimeout)
elapsed, uerr := s.cfg.Ollama.UnloadAll(uctx)
@@ -608,6 +663,10 @@ func (s *Server) finishImageJob(promptID string) {
s.cfg.Lock.ReleaseImage()
log.Info("image lock released")
if s.cfg.ComfySup != nil {
// The idle clock starts when the job ends, not when it began.
s.cfg.ComfySup.NoteActivity()
}
if s.cfg.Ollama != nil && s.cfg.WarmModel != "" {
if state, _, _ := s.cfg.Lock.Snapshot(); state == lock.StateIdle {
+2 -2
View File
@@ -46,8 +46,8 @@ type fakes struct {
rec *recorder
ollama *httptest.Server
comfy *httptest.Server
server *httptest.Server // Ollama-facing gpu-turnstile listener
comfySrv *httptest.Server // ComfyUI-facing gpu-turnstile listener
server *httptest.Server // gpu-turnstile listener for Ollama-compatible clients
comfySrv *httptest.Server // gpu-turnstile listener for ComfyUI clients
freeCh chan struct{}
chatCh chan struct{}
historyMu sync.Mutex
+63
View File
@@ -0,0 +1,63 @@
// Shared env-file handling for the installers (Windows and Linux).
package service
import (
"fmt"
"os"
"strings"
"gpu-turnstile/internal/config"
)
// syncEnvFile ensures the installed env file at path is a current
// installer-written sample. A missing file is created; a file without a
// CFG_VER line (hand-written or from before versioning) is invalid and
// replaced by a fresh sample after a .bak backup; a file written by an
// older version gets newly added settings appended via config.SyncSample.
// logPath activates the LOG_FILE line (Windows); empty leaves it commented.
func syncEnvFile(path, version, logPath string) error {
data, err := os.ReadFile(path)
switch {
case os.IsNotExist(err):
return writeEnvFile(path, config.SampleEnv(version, logPath))
case err != nil:
return fmt.Errorf("read %s: %w", path, err)
}
values, perr := config.ParseEnvFile(strings.NewReader(string(data)))
if perr == nil && values["CFG_VER"] != "" {
synced, changed := config.SyncSample(string(data), version, logPath)
if !changed {
return nil
}
return writeEnvFile(path, synced)
}
// No readable CFG_VER: the file is invalid — keep a backup and start
// from a fresh sample.
if err := os.WriteFile(path+".bak", data, 0o644); err != nil {
return fmt.Errorf("back up %s: %w", path, err)
}
return writeEnvFile(path, config.SampleEnv(version, logPath))
}
func writeEnvFile(path, content string) error {
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
return fmt.Errorf("write %s: %w", path, err)
}
return nil
}
// configuredValue reads one key from the env file the service will load, so
// the installers can adapt the sandbox to it (ACL grants, unit directives).
// "" when unset or unreadable.
func configuredValue(configPath, key string) string {
f, err := os.Open(configPath)
if err != nil {
return ""
}
defer f.Close()
values, err := config.ParseEnvFile(f)
if err != nil {
return ""
}
return values[key]
}
+76 -10
View File
@@ -6,6 +6,7 @@
package service
import (
"bytes"
"context"
"fmt"
"io"
@@ -61,8 +62,14 @@ func Run(run func(ctx context.Context) error) error {
// filesystem is read-only except StateDirectory (the install dir, so
// self-updates can rewrite the binary), and the usual no-privilege-escalation
// directives apply. The proxy needs nothing but outbound TCP/UDP and the
// notify socket, so it loses nothing.
func renderUnit(exePath, configPath string) string {
// notify socket, so it loses nothing. A managed ComfyUI (comfyDir) gets a
// BindPaths hole through ProtectHome/ProtectSystem: it reads its venv and
// writes output/temp/user data under COMFY_DIR.
func renderUnit(exePath, configPath, comfyDir string) string {
bind := ""
if comfyDir != "" {
bind = "BindPaths=" + comfyDir + "\n"
}
return fmt.Sprintf(`[Unit]
Description=gpu-turnstile GPU arbitration proxy for Ollama and ComfyUI
After=network-online.target
@@ -77,7 +84,7 @@ RestartSec=5s
DynamicUser=yes
StateDirectory=%s
ProtectSystem=strict
%sProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
NoNewPrivileges=yes
@@ -99,7 +106,7 @@ SystemCallErrorNumber=EPERM
[Install]
WantedBy=multi-user.target
`, exePath, configPath, Name)
`, exePath, configPath, Name, bind)
}
// copyFile copies src to dst, creating dst with the given mode.
@@ -124,14 +131,22 @@ func copyFile(src, dst string, mode os.FileMode) error {
// sure /etc/gpu-turnstile.env exists (copied from the given config file if
// provided), writes the hardened unit, then enables and starts it. With
// copyBin=false the current executable location and config path are
// registered as-is instead. Needs root.
// registered as-is instead. When the config sets COMFY_DIR, the unit gets a
// BindPaths= for it so the sandboxed service can reach the managed ComfyUI
// even under /home. Needs root.
//
// Re-running install converges an existing unit instead of failing: it is
// stopped first if active, the installed binary is replaced only when the
// content differs, the unit file is rewritten (followed by daemon-reload)
// only when it changed, and the service is started again only if it was
// active before.
//
// The binary lives in the StateDirectory rather than /usr/local/sbin on
// purpose: replacing a running binary needs write access to its
// *directory*, and granting the sandboxed service user write access to a
// shared system directory would let a compromised service overwrite other
// binaries. /var/lib/gpu-turnstile is exclusively ours.
func Install(configPath string, copyBin bool) error {
func Install(configPath string, copyBin bool, version string) error {
exe, err := os.Executable()
if err != nil {
return err
@@ -139,6 +154,16 @@ func Install(configPath string, copyBin bool) error {
if abs, absErr := filepath.Abs(exe); absErr == nil {
exe = abs
}
unit := Name + ".service"
_, statErr := os.Stat(unitPath)
fresh := os.IsNotExist(statErr)
wasRunning := exec.Command("systemctl", "is-active", "--quiet", unit).Run() == nil
if wasRunning {
if out, err := exec.Command("systemctl", "stop", unit).CombinedOutput(); err != nil {
return fmt.Errorf("systemctl stop (run as root): %w (%s)", err, out)
}
}
cfg := etcConfig
if copyBin {
if err := os.MkdirAll(stateDir, 0o755); err != nil {
@@ -146,31 +171,72 @@ func Install(configPath string, copyBin bool) error {
}
installedExe := filepath.Join(stateDir, Name)
if exe != installedExe {
if same, _ := sameFileContent(exe, installedExe); !same {
if err := copyFile(exe, installedExe, 0o755); err != nil {
return fmt.Errorf("install binary to %s: %w", installedExe, err)
}
}
}
exe = installedExe
if _, err := os.Stat(etcConfig); os.IsNotExist(err) && configPath != "" {
// Missing config is not fatal: the service fails fast with a
// clear "no consumer URL" error until the user writes one.
copyFile(configPath, etcConfig, 0o644) //nolint:errcheck // best effort
}
// Missing configs get a fully commented sample (LOG_FILE stays
// commented — stderr goes to the journal on Linux);
// installer-written ones from older versions get new settings
// appended; anything without CFG_VER is invalid and gets replaced
// (backup kept as .bak).
if err := syncEnvFile(etcConfig, version, ""); err != nil {
return err
}
} else if configPath != "" {
if abs, absErr := filepath.Abs(configPath); absErr == nil {
cfg = abs
}
}
if err := os.WriteFile(unitPath, []byte(renderUnit(exe, cfg)), 0o644); err != nil {
rendered := renderUnit(exe, cfg, configuredValue(cfg, "COMFY_DIR"))
if old, _ := os.ReadFile(unitPath); string(old) != rendered {
if err := os.WriteFile(unitPath, []byte(rendered), 0o644); err != nil {
return fmt.Errorf("write %s (run as root): %w", unitPath, err)
}
if out, err := exec.Command("systemctl", "daemon-reload").CombinedOutput(); err != nil {
return fmt.Errorf("systemctl daemon-reload: %w (%s)", err, out)
}
if out, err := exec.Command("systemctl", "enable", "--now", Name+".service").CombinedOutput(); err != nil {
}
if fresh {
if out, err := exec.Command("systemctl", "enable", "--now", unit).CombinedOutput(); err != nil {
return fmt.Errorf("systemctl enable --now: %w (%s)", err, out)
}
return nil
}
if exec.Command("systemctl", "is-enabled", "--quiet", unit).Run() != nil {
if out, err := exec.Command("systemctl", "enable", unit).CombinedOutput(); err != nil {
return fmt.Errorf("systemctl enable: %w (%s)", err, out)
}
}
if wasRunning {
if out, err := exec.Command("systemctl", "start", unit).CombinedOutput(); err != nil {
return fmt.Errorf("systemctl start: %w (%s)", err, out)
}
}
return nil
}
// sameFileContent reports whether two files hold identical bytes. A missing
// destination is simply "different".
func sameFileContent(a, b string) (bool, error) {
ba, err := os.ReadFile(a)
if err != nil {
return false, err
}
bb, err := os.ReadFile(b)
if os.IsNotExist(err) {
return false, nil
}
if err != nil {
return false, err
}
return bytes.Equal(ba, bb), nil
}
// Remove stops and disables the service and deletes the unit file and the
+9 -1
View File
@@ -8,7 +8,7 @@ import (
)
func TestRenderUnit(t *testing.T) {
unit := renderUnit("/var/lib/gpu-turnstile/gpu-turnstile", "/etc/gpu-turnstile.env")
unit := renderUnit("/var/lib/gpu-turnstile/gpu-turnstile", "/etc/gpu-turnstile.env", "")
for _, want := range []string{
"Type=notify",
"WatchdogSec=30s",
@@ -26,4 +26,12 @@ func TestRenderUnit(t *testing.T) {
t.Fatalf("unit missing %q:\n%s", want, unit)
}
}
if strings.Contains(unit, "BindPaths") {
t.Fatalf("unit without COMFY_DIR must not bind anything:\n%s", unit)
}
unit = renderUnit("/var/lib/gpu-turnstile/gpu-turnstile", "/etc/gpu-turnstile.env", "/home/gpu/ComfyUI")
if !strings.Contains(unit, "BindPaths=/home/gpu/ComfyUI\n") {
t.Fatalf("unit with COMFY_DIR must bind it:\n%s", unit)
}
}
+1 -1
View File
@@ -29,7 +29,7 @@ func Run(run func(ctx context.Context) error) error {
}
// Install is unsupported on non-Windows, non-Linux platforms.
func Install(string, bool) error { return errUnsupported }
func Install(string, bool, string) error { return errUnsupported }
// Remove is unsupported on non-Windows, non-Linux platforms.
func Remove() error { return errUnsupported }
+211 -67
View File
@@ -8,6 +8,7 @@
package service
import (
"bytes"
"context"
"errors"
"fmt"
@@ -23,8 +24,6 @@ import (
"golang.org/x/sys/windows"
"golang.org/x/sys/windows/svc"
"golang.org/x/sys/windows/svc/mgr"
"gpu-turnstile/internal/config"
)
// Name is the Windows service name.
@@ -105,11 +104,19 @@ func (h *handler) Execute(_ []string, requests <-chan svc.ChangeRequest, status
// the service after 5s on failure — this is also what brings up a staged
// update after the updater exits with a non-zero code. After registering,
// the virtual account is granted modify access to the install and data
// directories (self-updates rewrite the exe), and read access to the
// config file if it lives elsewhere. The grants must come after
// CreateService: the virtual account's SID only exists once the service is
// registered.
func Install(configPath string, copyBin bool) error {
// directories (self-updates rewrite the exe), read access to the
// config file if it lives elsewhere, and — when the config sets COMFY_DIR —
// recursive modify access to the managed ComfyUI's install tree, which may
// live inside a user profile the account otherwise cannot enter. The grants
// must come after CreateService: the virtual account's SID only exists once
// the service is registered.
//
// Re-running install on an existing service converges instead of failing:
// the service is stopped first if running (so the binary can be replaced),
// the installed binary is refreshed only when the content differs, the
// registration is updated only where it drifted, and the service is
// started again only if it was running before.
func Install(configPath string, copyBin bool, version string) error {
exe, err := os.Executable()
if err != nil {
return err
@@ -123,33 +130,65 @@ func Install(configPath string, copyBin bool) error {
}
}
installDir, _ := installDirs()
if copyBin && !strings.EqualFold(filepath.Dir(exe), installDir) {
if err := os.MkdirAll(installDir, 0o755); err != nil {
return fmt.Errorf("create %s: %w", installDir, err)
}
installedExe := filepath.Join(installDir, "gpu-turnstile.exe")
if err := copyFile(exe, installedExe); err != nil {
return fmt.Errorf("copy binary to %s: %w", installedExe, err)
}
exe = installedExe
targetCfg := filepath.Join(installDir, "gpu-turnstile.env")
if configPath != "" && !strings.EqualFold(configPath, targetCfg) {
if _, statErr := os.Stat(targetCfg); os.IsNotExist(statErr) {
copyFile(configPath, targetCfg) //nolint:errcheck // best effort
}
configPath = targetCfg
}
}
m, err := mgr.Connect()
if err != nil {
return fmt.Errorf("connect to service manager (run as administrator): %w", err)
}
defer m.Disconnect()
// An existing service is converged, not an error. Stop it first so the
// binary copy can be replaced, and remember whether to start it again.
var s *mgr.Service
wasRunning := false
if existing, openErr := m.OpenService(Name); openErr == nil {
s = existing
defer s.Close()
if st, qErr := s.Query(); qErr == nil &&
(st.State == svc.Running || st.State == svc.StartPending) {
wasRunning = true
fmt.Println("stopping the running gpu-turnstile service")
if err := stopAndWait(s); err != nil {
return err
}
}
}
installDir, dataDir := installDirs()
if copyBin {
targetCfg := filepath.Join(installDir, "gpu-turnstile.env")
if !strings.EqualFold(filepath.Dir(exe), installDir) {
if err := os.MkdirAll(installDir, 0o755); err != nil {
return fmt.Errorf("create %s: %w", installDir, err)
}
installedExe := filepath.Join(installDir, "gpu-turnstile.exe")
if same, _ := sameFileContent(exe, installedExe); !same {
fmt.Printf("installing %s\n", installedExe)
if err := copyFile(exe, installedExe); err != nil {
return fmt.Errorf("copy binary to %s: %w", installedExe, err)
}
}
exe = installedExe
if configPath != "" && !strings.EqualFold(configPath, targetCfg) {
if _, statErr := os.Stat(targetCfg); os.IsNotExist(statErr) {
copyFile(configPath, targetCfg) //nolint:errcheck // best effort
}
}
}
// A service has no console: without LOG_FILE the output vanishes, so
// the sample pre-wires it to ProgramData. Missing configs get a
// fresh sample; installer-written ones from older versions get new
// settings appended; anything without CFG_VER is invalid and gets
// replaced (backup kept as .bak).
logPath := filepath.Join(dataDir, "gpu-turnstile.log")
if err := syncEnvFile(targetCfg, version, logPath); err != nil {
return err
}
configPath = targetCfg
}
binPath := fmt.Sprintf(`"%s" -config "%s"`, exe, configPath)
s, err := m.CreateService(Name, binPath, mgr.Config{
if s == nil {
s, err = m.CreateService(Name, binPath, mgr.Config{
StartType: mgr.StartAutomatic,
DisplayName: "gpu-turnstile",
Description: "GPU arbitration proxy for Ollama and ComfyUI",
@@ -160,14 +199,10 @@ func Install(configPath string, copyBin bool) error {
}
defer s.Close()
restart := mgr.RecoveryAction{Type: mgr.ServiceRestart, Delay: 5 * time.Second}
if err := s.SetRecoveryActions([]mgr.RecoveryAction{restart, restart, restart}, 24*60*60); err != nil {
return fmt.Errorf("set recovery actions: %w", err)
if err := ensureRecovery(s); err != nil {
s.Delete() // roll back so a retry starts clean
return err
}
if err := s.SetRecoveryActionsOnNonCrashFailures(true); err != nil {
return fmt.Errorf("set failure actions flag: %w", err)
}
if err := grantAll(exe, configPath); err != nil {
s.Delete() // roll back so a retry starts clean
return err
@@ -175,8 +210,113 @@ func Install(configPath string, copyBin bool) error {
// Best effort: start now instead of waiting for the next boot. A
// missing config (no consumer URLs) fails the start; the service stays
// registered and can be started once the config exists.
fmt.Println("starting the gpu-turnstile service")
s.Start()
return nil
}
// Existing service: update the registration only where it drifted.
cur, err := s.Config()
if err != nil {
return fmt.Errorf("query service config: %w", err)
}
const displayName = "gpu-turnstile"
const description = "GPU arbitration proxy for Ollama and ComfyUI"
if cur.BinaryPathName != binPath ||
cur.StartType != mgr.StartAutomatic ||
cur.ServiceStartName != virtualAccount ||
cur.DisplayName != displayName ||
cur.Description != description {
upd := cur
upd.BinaryPathName = binPath
upd.StartType = mgr.StartAutomatic
upd.ServiceStartName = virtualAccount
upd.DisplayName = displayName
upd.Description = description
if err := s.UpdateConfig(upd); err != nil {
return fmt.Errorf("update service config: %w", err)
}
}
if err := ensureRecovery(s); err != nil {
return err
}
// Idempotent: re-assert the virtual account's ACLs (granting an
// existing ACE is a no-op).
if err := grantAll(exe, configPath); err != nil {
return err
}
if wasRunning {
fmt.Println("starting the gpu-turnstile service")
if err := s.Start(); err != nil {
return fmt.Errorf("start service: %w", err)
}
}
return nil
}
// ensureRecovery makes sure the service restarts 5s after a failure — the
// mechanism that also brings up a staged update. The current actions are
// queried first so an up-to-date service is left untouched.
func ensureRecovery(s *mgr.Service) error {
want := mgr.RecoveryAction{Type: mgr.ServiceRestart, Delay: 5 * time.Second}
actions, err := s.RecoveryActions()
onFailure, flagErr := s.RecoveryActionsOnNonCrashFailures()
if err == nil && flagErr == nil && onFailure && len(actions) == 3 {
ok := true
for _, a := range actions {
if a.Type != want.Type || a.Delay != want.Delay {
ok = false
}
}
if ok {
return nil
}
}
if err := s.SetRecoveryActions([]mgr.RecoveryAction{want, want, want}, 24*60*60); err != nil {
return fmt.Errorf("set recovery actions: %w", err)
}
if err := s.SetRecoveryActionsOnNonCrashFailures(true); err != nil {
return fmt.Errorf("set failure actions flag: %w", err)
}
return nil
}
// stopAndWait stops the service and waits up to 30s for the stopped state.
func stopAndWait(s *mgr.Service) error {
if _, err := s.Control(svc.Stop); err != nil {
return fmt.Errorf("stop service: %w", err)
}
deadline := time.Now().Add(30 * time.Second)
for {
st, err := s.Query()
if err != nil {
return fmt.Errorf("query service: %w", err)
}
if st.State == svc.Stopped {
return nil
}
if time.Now().After(deadline) {
return fmt.Errorf("service did not stop within 30s")
}
time.Sleep(300 * time.Millisecond)
}
}
// sameFileContent reports whether two files hold identical bytes. A missing
// destination is simply "different".
func sameFileContent(a, b string) (bool, error) {
ba, err := os.ReadFile(a)
if err != nil {
return false, err
}
bb, err := os.ReadFile(b)
if os.IsNotExist(err) {
return false, nil
}
if err != nil {
return false, err
}
return bytes.Equal(ba, bb), nil
}
// copyFile copies src to dst (0755 on the new file).
@@ -218,7 +358,7 @@ func grantAll(exe, configPath string) error {
return err
}
}
if logFile := configuredLogFile(configPath); logFile != "" {
if logFile := configuredValue(configPath, "LOG_FILE"); logFile != "" {
dir := filepath.Dir(logFile)
if err := os.MkdirAll(dir, 0o755); err == nil {
if err := grantAccess(dir, "(OI)(CI)(M)"); err != nil {
@@ -226,6 +366,17 @@ func grantAll(exe, configPath string) error {
}
}
}
// A managed ComfyUI whose install lives somewhere the virtual account
// may not go (a user profile) needs an explicit grant — recursively,
// since ComfyUI also writes output/temp/user data next to its code. A
// missing directory is skipped: the startup warning covers it.
if comfyDir := configuredValue(configPath, "COMFY_DIR"); comfyDir != "" {
if _, err := os.Stat(comfyDir); err == nil {
if err := grantAccessTree(comfyDir, "(OI)(CI)(M)"); err != nil {
return err
}
}
}
return nil
}
@@ -330,28 +481,35 @@ func RelaunchElevated(args []string) (int, error) {
// grantAccess gives the virtual account the icacls permission set (e.g.
// "(OI)(CI)(M)") on path.
func grantAccess(path, perms string) error {
out, err := exec.Command("icacls", path, "/grant", virtualAccount+":"+perms).CombinedOutput()
return runIcacls(path, perms, false)
}
// grantAccessTree is grantAccess with /T: the ACE is applied to the
// existing tree, not just inherited by children created later. Needed when
// the tree already exists, e.g. a ComfyUI install in a user profile. On a
// large tree (a venv has tens of thousands of files) this takes minutes,
// so it says what it is doing instead of looking hung.
func grantAccessTree(path, perms string) error {
return runIcacls(path, perms, true)
}
func runIcacls(path, perms string, recursive bool) error {
args := []string{path, "/grant", virtualAccount + ":" + perms}
if recursive {
fmt.Printf("granting %s modify access to %s (large trees can take minutes)\n", virtualAccount, path)
args = append(args, "/T")
}
start := time.Now()
out, err := exec.Command("icacls", args...).CombinedOutput()
if err != nil {
return fmt.Errorf("grant %s access to %s: %w (%s)", virtualAccount, path, err, strings.TrimSpace(string(out)))
}
if recursive {
fmt.Printf("access granted in %s\n", time.Since(start).Round(time.Second))
}
return nil
}
// configuredLogFile reads LOG_FILE from the config file so the installer
// can pre-create and ACL the log directory. "" when unset or unreadable.
func configuredLogFile(configPath string) string {
f, err := os.Open(configPath)
if err != nil {
return ""
}
defer f.Close()
values, err := config.ParseEnvFile(f)
if err != nil {
return ""
}
return values["LOG_FILE"]
}
// RestartIfRunning restarts the service when it is installed and running
// (used after a forced update staged a new binary). Reports whether a
// restart happened. A service that is not installed or not running is not
@@ -374,22 +532,8 @@ func RestartIfRunning() (bool, error) {
if st.State != svc.Running && st.State != svc.StartPending {
return false, nil
}
if _, err := s.Control(svc.Stop); err != nil {
return false, fmt.Errorf("stop service: %w", err)
}
deadline := time.Now().Add(30 * time.Second)
for {
st, err = s.Query()
if err != nil {
return false, fmt.Errorf("query service: %w", err)
}
if st.State == svc.Stopped {
break
}
if time.Now().After(deadline) {
return false, fmt.Errorf("service did not stop within 30s")
}
time.Sleep(300 * time.Millisecond)
if err := stopAndWait(s); err != nil {
return false, err
}
if err := s.Start(); err != nil {
return false, fmt.Errorf("start service: %w", err)
+314
View File
@@ -0,0 +1,314 @@
// Package supervise runs an upstream server (ComfyUI) as a managed child
// process: started on demand when a request needs it, stopped after an
// idle timeout so the GPU memory it holds is freed, and stopped with the
// parent. Crashes are logged; the next request respawns it.
package supervise
import (
"context"
"fmt"
"io"
"log/slog"
"net/url"
"os"
"os/exec"
"path/filepath"
"runtime"
"strings"
"sync"
"time"
)
// ComfyLayout returns the interpreter and script path of a standard ComfyUI
// venv install rooted at dir for the given GOOS: .venv\Scripts\python.exe
// on Windows, .venv/bin/python elsewhere. The script is main.py — either in
// a ComfyUI subdirectory or directly under dir, whichever exists (the
// subdirectory form wins ties and is the default when neither exists yet,
// so the caller's missing-file warning points at the documented layout).
func ComfyLayout(goos, dir string) (python, script string) {
if goos == "windows" {
python = filepath.Join(dir, ".venv", "Scripts", "python.exe")
} else {
python = filepath.Join(dir, ".venv", "bin", "python")
}
script = filepath.Join(dir, "ComfyUI", "main.py")
if _, err := os.Stat(script); err != nil {
if _, err := os.Stat(filepath.Join(dir, "main.py")); err == nil {
script = filepath.Join(dir, "main.py")
}
}
return python, script
}
// DefaultComfyCommand builds the launch command for the standard venv
// layout (see ComfyLayout): the script is passed relative to dir so dir
// stays the working directory, and --port is taken from comfyURL when the
// URL carries one.
func DefaultComfyCommand(goos, dir, comfyURL string) string {
python, script := ComfyLayout(goos, dir)
rel, err := filepath.Rel(dir, script)
if err != nil {
rel = script
}
cmd := `"` + python + `" ` + rel
if u, err := url.Parse(comfyURL); err == nil && u.Port() != "" {
cmd += " --port " + u.Port()
}
return cmd
}
// Process is one managed child process.
type Process struct {
name string
argv []string
dir string
probe func(context.Context) error
startTimeout time.Duration
log *slog.Logger
mu sync.Mutex
cmd *exec.Cmd
stopping bool
ready bool
external bool // someone else serves the port; not our process
lastActivity time.Time
}
// New parses cmdLine (double quotes group arguments containing spaces) and
// prepares a managed process. probe reports whether the server answers
// (e.g. the comfy client's Probe); startTimeout bounds WaitReady. dir is
// the child's working directory; empty inherits ours.
func New(name, cmdLine, dir string, probe func(context.Context) error, startTimeout time.Duration, log *slog.Logger) (*Process, error) {
argv, err := splitCommandLine(cmdLine)
if err != nil {
return nil, fmt.Errorf("%s command: %w", name, err)
}
if len(argv) == 0 {
return nil, fmt.Errorf("%s command is empty", name)
}
if log == nil {
log = slog.Default()
}
if startTimeout <= 0 {
startTimeout = 2 * time.Minute
}
return &Process{name: name, argv: argv, dir: dir, probe: probe, startTimeout: startTimeout, log: log}, nil
}
// splitCommandLine splits a command line on whitespace, treating
// double-quoted sections as one argument (quotes removed). Backslashes are
// literal — this matches Windows paths.
func splitCommandLine(s string) ([]string, error) {
var argv []string
var cur strings.Builder
inQuote := false
have := false
flush := func() {
if have {
argv = append(argv, cur.String())
cur.Reset()
have = false
}
}
for _, r := range s {
switch {
case r == '"':
inQuote = !inQuote
have = true
case (r == ' ' || r == '\t') && !inQuote:
flush()
default:
cur.WriteRune(r)
have = true
}
}
if inQuote {
return nil, fmt.Errorf("unterminated quote in %q", s)
}
flush()
return argv, nil
}
// Running reports whether the child process is currently alive.
func (p *Process) Running() bool {
p.mu.Lock()
defer p.mu.Unlock()
return p.cmd != nil
}
// Ready reports whether the server has answered a probe since its last
// (re)start. Health checks use it to tell "starting up" from "outage".
func (p *Process) Ready() bool {
p.mu.Lock()
defer p.mu.Unlock()
return p.ready
}
// MarkReady records that the server answered.
func (p *Process) MarkReady() {
p.mu.Lock()
p.ready = true
p.mu.Unlock()
}
// NoteActivity resets the idle clock; called for every request served.
func (p *Process) NoteActivity() {
p.mu.Lock()
p.lastActivity = time.Now()
p.mu.Unlock()
}
// EnsureRunning starts the child if it is not running. It returns as soon
// as the process is spawned; readiness is WaitReady's job (and the proxy's
// retry backoff bridges the gap for plain proxied requests). When the URL
// already answers — e.g. the ComfyUI desktop app grabbed the port — no
// child is spawned: the external server is used as-is, and the idle
// watcher never touches it (it only kills its own child).
func (p *Process) EnsureRunning() error {
p.mu.Lock()
defer p.mu.Unlock()
p.lastActivity = time.Now()
if p.cmd != nil {
return nil
}
pctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
err := p.probe(pctx)
cancel()
if err == nil {
p.ready = true
if !p.external {
p.external = true
p.log.Info(p.name + " is already served externally; not spawning a managed instance")
}
return nil
}
p.external = false
cmd := exec.Command(p.argv[0], p.argv[1:]...)
cmd.Dir = p.dir
stdout, err := cmd.StdoutPipe()
if err != nil {
return err
}
stderr, err := cmd.StderrPipe()
if err != nil {
return err
}
if err := cmd.Start(); err != nil {
return fmt.Errorf("start %s: %w", p.name, err)
}
p.cmd = cmd
p.stopping = false
p.ready = false
go p.pipeLog(stdout)
go p.pipeLog(stderr)
go func() {
err := cmd.Wait()
p.mu.Lock()
p.cmd = nil
p.ready = false
intentional := p.stopping
p.mu.Unlock()
if intentional {
p.log.Info(p.name + " stopped")
} else {
p.log.Warn(p.name+" exited unexpectedly; the next request restarts it", "err", err)
}
}()
p.log.Info(p.name+" starting", "pid", cmd.Process.Pid, "cmd", strings.Join(p.argv, " "))
return nil
}
// WaitReady blocks until the probe succeeds, ctx ends, or the start
// timeout passes.
func (p *Process) WaitReady(ctx context.Context) error {
ctx, cancel := context.WithTimeout(ctx, p.startTimeout)
defer cancel()
for {
pctx, pcancel := context.WithTimeout(ctx, 5*time.Second)
err := p.probe(pctx)
pcancel()
if err == nil {
p.NoteActivity()
p.MarkReady()
return nil
}
select {
case <-ctx.Done():
return fmt.Errorf("%s did not become ready: %w", p.name, ctx.Err())
case <-time.After(500 * time.Millisecond):
}
}
}
// Stop kills the child process (the whole tree on Windows) if running.
func (p *Process) Stop() {
p.mu.Lock()
cmd := p.cmd
if cmd == nil {
p.mu.Unlock()
return
}
p.stopping = true
p.ready = false
p.mu.Unlock()
stopTree(cmd)
}
// WatchIdle stops the child after idleTimeout without activity, but only
// when gpuIdle reports the GPU lock is free (no active or pending work).
// Returns when ctx ends.
func (p *Process) WatchIdle(ctx context.Context, idleTimeout time.Duration, gpuIdle func() bool) {
ticker := time.NewTicker(5 * time.Second)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
}
p.mu.Lock()
idleFor := time.Since(p.lastActivity)
running := p.cmd != nil
p.mu.Unlock()
if running && idleFor > idleTimeout && gpuIdle() {
p.log.Info(p.name+" idle; stopping to free the GPU", "idle_for", idleFor.Round(time.Second))
p.Stop()
}
}
}
// pipeLog forwards one child output stream to the log at INFO, line by
// line, prefixed with the process name.
func (p *Process) pipeLog(r io.Reader) {
buf := make([]byte, 4096)
var line string
for {
n, err := r.Read(buf)
line += string(buf[:n])
for {
i := strings.IndexByte(line, '\n')
if i < 0 {
break
}
p.log.Info(p.name + ": " + strings.TrimRight(line[:i], "\r"))
line = line[i+1:]
}
if err != nil {
if strings.TrimSpace(line) != "" {
p.log.Info(p.name + ": " + line)
}
return
}
}
}
// stopTree kills cmd's process, including its children on Windows (python
// launchers tend to spawn some). The Wait goroutine reaps it.
func stopTree(cmd *exec.Cmd) {
if runtime.GOOS == "windows" {
exec.Command("taskkill", "/T", "/F", "/PID",
fmt.Sprint(cmd.Process.Pid)).Run() //nolint:errcheck // best effort
return
}
cmd.Process.Kill() //nolint:errcheck // best effort
}
+241
View File
@@ -0,0 +1,241 @@
package supervise
import (
"context"
"errors"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestSplitCommandLine(t *testing.T) {
cases := []struct {
in string
want []string
}{
{`python main.py --port 8188`, []string{"python", "main.py", "--port", "8188"}},
{`"C:\Program Files\py\python.exe" main.py`, []string{`C:\Program Files\py\python.exe`, "main.py"}},
{` spaced out `, []string{"spaced", "out"}},
{`a "b c" d`, []string{"a", "b c", "d"}},
{"", nil},
}
for _, c := range cases {
got, err := splitCommandLine(c.in)
if err != nil {
t.Errorf("splitCommandLine(%q): %v", c.in, err)
continue
}
if len(got) != len(c.want) {
t.Errorf("splitCommandLine(%q) = %v, want %v", c.in, got, c.want)
continue
}
for i := range got {
if got[i] != c.want[i] {
t.Errorf("splitCommandLine(%q) = %v, want %v", c.in, got, c.want)
break
}
}
}
if _, err := splitCommandLine(`"unterminated`); err == nil {
t.Error("expected error for unterminated quote")
}
}
// TestHelperProcess is the child executed by the lifecycle tests: it just
// sleeps. The env marker is set only around the spawn, so in the normal
// test run this returns immediately.
func TestHelperProcess(t *testing.T) {
if os.Getenv("GO_HELPER_PROCESS") != "1" {
return
}
time.Sleep(30 * time.Second)
os.Exit(0)
}
func newHelper(t *testing.T, name string) *Process {
t.Helper()
// The probe always fails: nothing external serves the port, so
// EnsureRunning spawns the helper child.
p, err := New(name, `"`+os.Args[0]+`" -test.run=TestHelperProcess`, "",
func(context.Context) error { return errors.New("nothing there") }, 5*time.Second, slog.Default())
if err != nil {
t.Fatal(err)
}
return p
}
// startHelper spawns the child with the marker set; exec.Command inherits
// the environment at spawn time, so it can be unset right after.
func startHelper(t *testing.T, p *Process) {
t.Helper()
os.Setenv("GO_HELPER_PROCESS", "1")
defer os.Unsetenv("GO_HELPER_PROCESS")
if err := p.EnsureRunning(); err != nil {
t.Fatal(err)
}
}
func waitStopped(t *testing.T, p *Process, timeout time.Duration) {
t.Helper()
deadline := time.Now().Add(timeout)
for p.Running() && time.Now().Before(deadline) {
time.Sleep(50 * time.Millisecond)
}
if p.Running() {
t.Fatal("process still running")
}
}
func TestEnsureRunningAndStop(t *testing.T) {
p := newHelper(t, "helper")
if p.Running() {
t.Fatal("Running before start")
}
startHelper(t, p)
if !p.Running() {
t.Fatal("not Running after EnsureRunning")
}
if err := p.EnsureRunning(); err != nil {
t.Fatal("second EnsureRunning must be a no-op")
}
p.Stop()
waitStopped(t, p, 5*time.Second)
}
func TestEnsureRunningPrefersExternalServer(t *testing.T) {
// The port is already served (e.g. the ComfyUI desktop app): no child
// is spawned, the supervisor reports ready, and Stop is a no-op.
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
}))
defer srv.Close()
probe := func(ctx context.Context) error {
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, srv.URL, nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
resp.Body.Close()
return nil
}
p, err := New("external", `"`+os.Args[0]+`"`, "", probe, 5*time.Second, slog.Default())
if err != nil {
t.Fatal(err)
}
if err := p.EnsureRunning(); err != nil {
t.Fatal(err)
}
if p.Running() {
t.Fatal("spawned a child even though the port is already served")
}
if !p.Ready() {
t.Fatal("external server should count as ready")
}
p.Stop() // must not touch the external server
}
func TestWaitReady(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
}))
defer srv.Close()
probe := func(ctx context.Context) error {
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, srv.URL, nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
resp.Body.Close()
return nil
}
p, err := New("ready", `"`+os.Args[0]+`"`, "", probe, 5*time.Second, slog.Default())
if err != nil {
t.Fatal(err)
}
if err := p.WaitReady(context.Background()); err != nil {
t.Fatal(err)
}
failing, err := New("failing", `"`+os.Args[0]+`"`, "",
func(context.Context) error { return errors.New("no") }, 500*time.Millisecond, slog.Default())
if err != nil {
t.Fatal(err)
}
if err := failing.WaitReady(context.Background()); err == nil {
t.Fatal("expected timeout error from WaitReady")
}
}
func TestWatchIdleStops(t *testing.T) {
p := newHelper(t, "idle")
startHelper(t, p)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go p.WatchIdle(ctx, 200*time.Millisecond, func() bool { return true })
waitStopped(t, p, 10*time.Second)
}
func TestWatchIdleRespectsBusyGPU(t *testing.T) {
p := newHelper(t, "busy")
startHelper(t, p)
defer p.Stop()
ctx, cancel := context.WithCancel(context.Background())
go p.WatchIdle(ctx, 100*time.Millisecond, func() bool { return false })
time.Sleep(600 * time.Millisecond)
cancel()
if !p.Running() {
t.Fatal("process was stopped while the GPU was busy")
}
}
func TestComfyLayoutAndDefaultCommand(t *testing.T) {
// Nested layout (ComfyUI/main.py under dir) wins.
dir := t.TempDir()
nested := filepath.Join(dir, "ComfyUI", "main.py")
if err := os.MkdirAll(filepath.Dir(nested), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(nested, []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
python, script := ComfyLayout("windows", dir)
if want := filepath.Join(dir, ".venv", "Scripts", "python.exe"); python != want {
t.Errorf("python = %s, want %s", python, want)
}
if script != nested {
t.Errorf("script = %s, want %s", script, nested)
}
cmd := DefaultComfyCommand("windows", dir, "http://127.0.0.1:8189")
want := `"` + filepath.Join(dir, ".venv", "Scripts", "python.exe") + `" ` + filepath.Join("ComfyUI", "main.py") + " --port 8189"
if cmd != want {
t.Errorf("cmd = %q, want %q", cmd, want)
}
// Flat layout (main.py directly under dir) is found too.
flat := t.TempDir()
if err := os.WriteFile(filepath.Join(flat, "main.py"), []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
if _, script := ComfyLayout("linux", flat); script != filepath.Join(flat, "main.py") {
t.Errorf("flat script = %s", script)
}
cmd = DefaultComfyCommand("linux", flat, "http://comfy.internal")
if strings.Contains(cmd, "--port") {
t.Errorf("cmd = %q, want no --port for a port-less URL", cmd)
}
if !strings.HasSuffix(cmd, `" main.py`) {
t.Errorf("cmd = %q, want quoted python + relative main.py", cmd)
}
// Neither exists yet: default to the documented nested form so the
// startup warning points there.
empty := t.TempDir()
if _, script := ComfyLayout("windows", empty); script != filepath.Join(empty, "ComfyUI", "main.py") {
t.Errorf("missing-layout script = %s", script)
}
}
+34 -9
View File
@@ -26,11 +26,15 @@ import (
// maxAssetSize bounds release asset downloads.
const maxAssetSize = 512 << 20
// Updater checks one Gitea repository for newer releases.
// Updater checks one Gitea repository for releases.
type Updater struct {
Repo string // e.g. https://git.rambossek.at/PUBLIC/gpu-turnstile
Asset string // e.g. gpu-turnstile.exe
Version string // current version, e.g. v0.1.2 ("dev" disables updates)
Version string // current binary version, e.g. v0.1.2 ("dev" cannot be compared)
// Desired is the APP_VER policy: "dev" disables updates, "stable" (or
// empty) tracks the latest release, anything else is an exact vX.Y.Z
// release tag to pin — staging it even when that means a downgrade.
Desired string
Log *slog.Logger
Client *http.Client
}
@@ -167,15 +171,20 @@ func CleanupOld(exePath string) {
os.Remove(exePath + ".new")
}
// Check performs a single update check. staged is true when a newer,
// Check performs a single update check. staged is true when a
// signature-verified binary has been swapped into place at exePath; the
// caller should then restart the process. A nil error with staged=false
// means "no action" (up to date, disabled, or dev build); a non-nil error
// means the check failed and the running binary is untouched.
// means "no action" (up to date, APP_VER=dev, or no embedded public key);
// a non-nil error means the check failed and the running binary is
// untouched.
func (u *Updater) Check(ctx context.Context, exePath string) (staged bool, err error) {
log := u.logger()
if u.Version == "" || u.Version == "dev" {
log.Debug("auto-update: dev build, skipping")
desired := u.Desired
if desired == "" {
desired = "stable"
}
if desired == "dev" {
log.Debug("auto-update: APP_VER=dev, skipping")
return false, nil
}
if publicKeyPEM == "" {
@@ -187,14 +196,29 @@ func (u *Updater) Check(ctx context.Context, exePath string) (staged bool, err e
return false, err
}
body, err := u.get(ctx, api+"/releases/latest")
pinned := desired != "stable"
endpoint := api + "/releases/latest"
if pinned {
endpoint = api + "/releases/tags/" + desired
}
body, err := u.get(ctx, endpoint)
if err != nil {
return false, fmt.Errorf("fetch latest release: %w", err)
return false, fmt.Errorf("fetch release: %w", err)
}
var rel release
if err := json.Unmarshal(body, &rel); err != nil {
return false, fmt.Errorf("parse release: %w", err)
}
if pinned {
// Pin mode: any difference from the target tag means stage it —
// including downgrades and replacing a dev binary.
if u.Version == rel.TagName {
log.Debug("auto-update: already on pinned version", "version", u.Version)
return false, nil
}
} else if u.Version != "" && u.Version != "dev" {
// Stable mode: only strictly newer releases count; a dev binary
// cannot be compared and is always replaced by the latest release.
newer, err := newerVersion(u.Version, rel.TagName)
if err != nil {
return false, err
@@ -203,6 +227,7 @@ func (u *Updater) Check(ctx context.Context, exePath string) (staged bool, err e
log.Debug("auto-update: up to date", "version", u.Version, "latest", rel.TagName)
return false, nil
}
}
urls := make(map[string]string, len(rel.Assets))
for _, a := range rel.Assets {
+62 -5
View File
@@ -43,7 +43,7 @@ func newFakeGitea(t *testing.T, tag string, assetContent []byte) *fakeGitea {
sign := func() []byte { return ed25519.Sign(priv, f.asset) }
mux := http.NewServeMux()
mux.HandleFunc("/api/v1/repos/o/r/releases/latest", func(w http.ResponseWriter, r *http.Request) {
serveRelease := func(w http.ResponseWriter, r *http.Request) {
assets := []map[string]string{
{"name": "gpu-turnstile.exe", "browser_download_url": f.srv.URL + "/dl/exe"},
{"name": "gpu-turnstile.exe.sha256", "browser_download_url": f.srv.URL + "/dl/sha"},
@@ -52,7 +52,9 @@ func newFakeGitea(t *testing.T, tag string, assetContent []byte) *fakeGitea {
assets = append(assets, map[string]string{"name": "gpu-turnstile.exe.sig", "browser_download_url": f.srv.URL + "/dl/sig"})
}
json.NewEncoder(w).Encode(map[string]any{"tag_name": f.tag, "assets": assets})
})
}
mux.HandleFunc("/api/v1/repos/o/r/releases/latest", serveRelease)
mux.HandleFunc("/api/v1/repos/o/r/releases/tags/"+f.tag, serveRelease)
mux.HandleFunc("/dl/exe", func(w http.ResponseWriter, r *http.Request) { w.Write(f.asset) })
mux.HandleFunc("/dl/sig", func(w http.ResponseWriter, r *http.Request) {
sig := sign()
@@ -73,6 +75,12 @@ func (f *fakeGitea) updater(version string) *Updater {
return &Updater{Repo: f.srv.URL + "/o/r", Asset: "gpu-turnstile.exe", Version: version}
}
func (f *fakeGitea) updaterDesired(version, desired string) *Updater {
u := f.updater(version)
u.Desired = desired
return u
}
func fakeExe(t *testing.T) string {
t.Helper()
exe := filepath.Join(t.TempDir(), "gpu-turnstile.exe")
@@ -153,12 +161,61 @@ func TestCheckSkipsWithoutPublicKey(t *testing.T) {
}
}
func TestCheckSkipsDevBuild(t *testing.T) {
func TestCheckDevBuildGetsStable(t *testing.T) {
// A dev binary cannot be compared; APP_VER=stable replaces it with the
// latest release.
f := newFakeGitea(t, "v9.9.9", []byte("new-binary"))
withPublicKey(t, f.pubPEM)
staged, err := f.updater("dev").Check(context.Background(), fakeExe(t))
exe := fakeExe(t)
staged, err := f.updater("dev").Check(context.Background(), exe)
if err != nil {
t.Fatal(err)
}
if !staged {
t.Fatal("expected dev binary to be replaced by the latest release")
}
content, _ := os.ReadFile(exe)
if string(content) != "new-binary" {
t.Fatalf("exe content = %q", content)
}
}
func TestCheckDesiredDevDisables(t *testing.T) {
f := newFakeGitea(t, "v9.9.9", []byte("new-binary"))
withPublicKey(t, f.pubPEM)
for _, version := range []string{"dev", "v0.1.2"} {
staged, err := f.updaterDesired(version, "dev").Check(context.Background(), fakeExe(t))
if err != nil || staged {
t.Fatalf("staged=%v err=%v, want no action for dev build", staged, err)
t.Fatalf("version %s: staged=%v err=%v, want no action with APP_VER=dev", version, staged, err)
}
}
}
func TestCheckPinned(t *testing.T) {
// Pin mode stages the exact tag — up or down — and skips when the
// binary already matches.
for _, version := range []string{"v0.1.2", "v9.9.9", "dev"} {
f := newFakeGitea(t, "v0.5.0", []byte("pinned-binary"))
withPublicKey(t, f.pubPEM)
exe := fakeExe(t)
staged, err := f.updaterDesired(version, "v0.5.0").Check(context.Background(), exe)
if err != nil {
t.Fatalf("version %s: %v", version, err)
}
if !staged {
t.Fatalf("version %s: expected pinned v0.5.0 to be staged", version)
}
content, _ := os.ReadFile(exe)
if string(content) != "pinned-binary" {
t.Fatalf("version %s: exe content = %q", version, content)
}
}
f := newFakeGitea(t, "v0.5.0", []byte("pinned-binary"))
withPublicKey(t, f.pubPEM)
staged, err := f.updaterDesired("v0.5.0", "v0.5.0").Check(context.Background(), fakeExe(t))
if err != nil || staged {
t.Fatalf("staged=%v err=%v, want no action when already on the pinned version", staged, err)
}
}