# gpu-turnstile

GPU arbitration proxy for Ollama + ComfyUI. One consumer GPU is shared by an
LLM server (Ollama) and an image generator (ComfyUI); gpu-turnstile sits in
front of both and guarantees the GPU is always in exactly one of four states:
`idle`, `llm` (N ≥ 1 Ollama requests in flight), `image` (exactly one
ComfyUI job, Ollama models unloaded), or `external` (a foreign process such
as a game holds the GPU). See [SPEC.md](SPEC.md) for the full
design.

gpu-turnstile listens on the ports the services normally use; the actual
services run one port higher (Ollama on 11435, ComfyUI on 8189).

```
LiteLLM / Open WebUI ──► :11434 ─┐                           ┌─► Ollama  :11435
                                 ├── gpu-turnstile (1 lock) ──┤
Open WebUI / n8n ────► :8188  ───┘                           └─► ComfyUI :8189
```

- LLM endpoints (`/api/generate`, `/api/chat`, `/api/embed`, `/v1/*`) take the
  LLM lock: concurrent requests allowed, but blocked while an image job is
  active or waiting (image priority). Blocked requests either hang until the
  lock is free (`LLM_BUSY_MODE=wait`, default) or fail immediately with 503
  (or 429) + `Retry-After` (`LLM_BUSY_MODE=reject`) — the latter lets routers
  like LiteLLM cool down and retry instead of holding a hung connection.
- `POST /prompt` on the ComfyUI listener takes the image lock: new LLM
  requests block, in-flight LLMs drain, Ollama models are unloaded, the prompt
  is forwarded, and the lock is held until the job finishes and ComfyUI frees
  its VRAM.
- Everything else (including websockets and all streaming) passes through
  transparently and unbuffered.

Each consumer is enabled by setting its URL (`OLLAMA_URL`, `COMFY_URL`) and
disabled by leaving it empty — at least one is required. With only Ollama
the proxy is a pass-through (no image jobs can arrive); with only ComfyUI
the Ollama unload/warm steps are skipped. A third, URL-less consumer —
detection of foreign GPU holders such as games — is enabled by `GAME_PROCS`
and/or `GPU_FOREIGN_VRAM_MB` (see below).

## Configuration

Configuration comes from environment variables and/or an `.env`-style
config file (`KEY=VALUE` lines, `#` comments). File lookup order:
`-config <path>` flag, then `GPU_TURNSTILE_CONFIG`, then
`gpu-turnstile.env` next to the executable. Process environment variables
override file values. Invalid values fail at startup.

| Var | Default | Meaning |
|---|---|---|
| `LISTEN_OLLAMA` | `:11434` | Listener for Ollama-compatible clients |
| `LISTEN_COMFY` | `:8188` | Listener for ComfyUI clients |
| `OLLAMA_URL` | _(empty = disabled)_ | Ollama upstream; set to enable the Ollama consumer |
| `COMFY_URL` | _(empty = disabled)_ | ComfyUI upstream; set to enable the ComfyUI consumer |
| `UNLOAD_TIMEOUT` | `60s` | Wait for Ollama to unload before an image job |
| `JOB_TIMEOUT` | `15m` | Wait for a ComfyUI job to finish |
| `LLM_WAIT_TIMEOUT` | `10m` | Max lock wait for an LLM request before 503 (wait mode) |
| `LLM_BUSY_MODE` | `wait` | `wait` = hold blocked LLM requests; `reject` = fail them immediately |
| `LLM_BUSY_STATUS` | `503` | HTTP status for rejected LLM requests in reject mode (400–599, e.g. 429) |
| `BUSY_RETRY_AFTER` | `30` | Seconds sent as `Retry-After` on busy responses (both modes) |
| `WARM_MODEL` | _(empty)_ | Model to reload after an image job (off by default) |
| `COMFY_CMD` | _(derived from `COMFY_DIR`; both empty = unmanaged)_ | Supervise ComfyUI: start on demand, stop when idle to free VRAM. Requires `COMFY_URL` |
| `COMFY_DIR` | _(empty)_ | Standard venv install root: set alone to supervise ComfyUI with the derived command (`.venv` + `main.py`); also the working directory for `COMFY_CMD` |
| `COMFY_IDLE_TIMEOUT` | `5m` | Stop the managed ComfyUI after this long idle |
| `COMFY_START_TIMEOUT` | `2m` | Max wait for the managed ComfyUI to come up |
| `GAME_PROCS` | _(empty = disabled)_ | Process names (comma-separated); while any runs, the GPU counts as held: requests wait, Ollama unloads, managed ComfyUI stops |
| `GPU_FOREIGN_VRAM_MB` | `0` (disabled) | Also treat the GPU as held when a non-ignored process uses more VRAM than this (needs nvidia-smi) |
| `GPU_IGNORE_PROCS` | `ollama,ollama app,ollama_llama_server,python,pythonw` | Process names never counted as foreign GPU users |
| `GAME_POLL_INTERVAL` | `15s` | How often game/VRAM detection runs (don't go below ~10s — nvidia-smi polls keep the GPU awake) |
| `LOGLEVEL` | `warn` | `info` logs every request (colored arrows in text mode), `debug` adds lock transitions. `LOG_LEVEL` works as an alias |
| `LOG_FORMAT` | `text` | `json` for structured JSON logs |
| `LOG_FILE` | _(empty)_ | Append logs to this file instead of stderr |
| `UNLOAD_POLL_INTERVAL` | `500ms` | `/api/ps` poll interval while unloading |
| `HISTORY_POLL_INTERVAL` | `1s` | `/history/<id>` poll interval while a job runs |
| `PROBE_TIMEOUT` | `5s` | Startup probe of both upstreams (also per-probe health check timeout) |
| `HEALTH_INTERVAL` | `30s` | Periodic upstream probe; down/recovered changes are logged |
| `FREE_TIMEOUT` | `30s` | `POST /free` call after an image job |
| `WARM_TIMEOUT` | `2m` | Warm-model reload after an image job |
| `SHUTDOWN_TIMEOUT` | `10s` | Graceful shutdown on SIGINT/SIGTERM |
| `BACKOFF_INITIAL` | `1s` | First retry wait when an upstream refuses a connection |
| `BACKOFF_MAX` | `60s` | Cap for the exponential retry backoff |
| `PROMPT_CAPTURE_LIMIT` | `65536` | Bytes of the `/prompt` response buffered to find `prompt_id` (pass-through is unaffected) |
| `AUTO_UPDATE` | `true` | Poll the Gitea releases API for signed updates |
| `UPDATE_INTERVAL` | `6h` | Auto-update check interval |
| `UPDATE_REPO` | `https://git.rambossek.at/PUBLIC/gpu-turnstile` | Repository checked for releases |
| `UPDATE_ASSET` | `gpu-turnstile.exe` | Release asset to download |
| `APP_VER` | `stable` | `dev` disables updates, `stable` tracks latest, or pin an exact `vX.Y.Z` |

## Observability

- `GET /healthz` (both listeners): `{"state":"idle|llm|image|external","llm_inflight":N,"image_pending":B}`
- `GET /metrics` (both listeners): Prometheus text format — `gpu_turnstile_state`,
  `gpu_turnstile_llm_inflight`, `gpu_turnstile_image_pending`,
  `gpu_turnstile_image_jobs_total`, `gpu_turnstile_lock_wait_seconds`
  (histogram, `kind="llm|image"`), `gpu_turnstile_unload_seconds`.
- Logs: startup logs the version and every setting (visible even at the
  default `warn` level). With `LOGLEVEL=info` or `debug`, every request
  logs a `-->` incoming line and a `<--` response line with status and
  duration — ANSI-colored (cyan incoming; green/yellow/red by status
  class) in text mode, which renders in `docker compose logs` on Windows
  Terminal. Set `NO_COLOR` to disable colors.

## Managed ComfyUI (`COMFY_CMD` / `COMFY_DIR`)

Don't want ComfyUI running 24/7 (it holds VRAM even when idle — and the
Desktop app kills its server when you close it)? gpu-turnstile can supervise
it: the first request starts it, it stops again after `COMFY_IDLE_TIMEOUT`
(default 5m) without work, freeing the GPU for games or the LLM.

The easy way — point `COMFY_DIR` at a standard venv install (a folder with
`.venv` and `main.py`, or `.venv` and `ComfyUI\main.py`) and the launch
command is derived from it, including `--port` from `COMFY_URL`:

```
COMFY_URL=http://127.0.0.1:8189
COMFY_DIR=C:\ComfyUI
```

For other layouts, spell the command out yourself (run it once manually to
confirm it works):

```
COMFY_URL=http://127.0.0.1:8189
COMFY_CMD="C:\ComfyUI\.venv\Scripts\python.exe" main.py --port 8189
COMFY_DIR=C:\ComfyUI
```

`POST /prompt` waits for the server to answer before taking the GPU lock
(LLM traffic keeps flowing while torch loads); crashes are logged and the
next request respawns. Shutting gpu-turnstile down stops the child too.

Running the ComfyUI Desktop app alongside is safe: if something already
answers on the port, gpu-turnstile just uses it instead of spawning
(and never kills it — it only ever stops its own child). If the managed
instance already holds the port when you open the desktop app, the
desktop's server is the one that fails to bind.

One catch when gpu-turnstile runs as a service: the sandboxed service
account may not enter your user profile, so a ComfyUI install under
`C:\Users\...` (or `/home/...`) fails with "Access is denied".
`--install-service` fixes that automatically — it grants
`NT SERVICE\gpu-turnstile` recursive access to `COMFY_DIR` on Windows and
adds a `BindPaths=` to the systemd unit on Linux. Re-run it after changing
`COMFY_DIR`; or grant by hand from an admin shell:
`icacls "<COMFY_DIR>" /grant "NT SERVICE\gpu-turnstile:(OI)(CI)M" /T`.

## Game detection

Want to game on the same GPU without Ollama/ComfyUI squatting on the VRAM?
gpu-turnstile can watch for foreign GPU holders and, while one is active,
make LLM/image requests wait (or 503, per `LLM_BUSY_MODE`), unload Ollama's
models and stop the managed ComfyUI so the game gets the memory. Two
detection paths, each optional, polled every `GAME_POLL_INTERVAL` (15s):

```
GAME_PROCS=cyberpunk2077.exe,bg3.exe     # the reliable way on Windows
GPU_FOREIGN_VRAM_MB=1024                 # catch-all via nvidia-smi
```

`GAME_PROCS` matches running process names (case-insensitive, `.exe`
optional). `GPU_FOREIGN_VRAM_MB` asks nvidia-smi which processes hold GPU
memory and treats anything not in `GPU_IGNORE_PROCS` above the threshold as
foreign — handy as a catch-all, but note that under Windows' WDDM driver
graphics-only games may not show up in nvidia-smi's per-process list, so
name your games in `GAME_PROCS` there; on Linux both paths work. When the
game exits, requests resume automatically.

## Build and run

```sh
go build ./cmd/gpu-turnstile
OLLAMA_URL=http://127.0.0.1:11435 COMFY_URL=http://127.0.0.1:8189 ./gpu-turnstile
```

Running the binary with no arguments in a terminal prints the help screen
(same as `-h`/`--help`); without a terminal (services, containers) a bare
invocation starts the proxy.

### Run natively on Windows (current primary deployment)

Download `gpu-turnstile.exe` from a release and install it as a Windows
service — no admin shell needed, a UAC prompt appears automatically and
the elevated child does the work (its window waits for Enter so you can
read the result):

```sh
gpu-turnstile.exe --install-service        # installs into Program Files, auto-start
gpu-turnstile.exe --install-service --no-copy   # register in place instead
gpu-turnstile.exe --remove-service
```

Layout: `C:\Program Files\gpu-turnstile\` holds the exe and
`gpu-turnstile.env`, logs go to `C:\ProgramData\gpu-turnstile\`. If you
install without a config, the installer writes a sample env file with every
setting commented and explained — only `LOG_FILE` is active (a service has
no console). Your own `LOG_FILE` setting is always kept. The service always
runs
as the virtual account `NT SERVICE\gpu-turnstile` (low-privilege,
per-service, no password); the installer automatically grants it write
access to the install and data directories — nothing else to do.

Re-running `--install-service` is safe: it stops a running service,
replaces the installed binary only if it changed, fixes the registration
only where it drifted, and restarts the service only if it was running.

### Run natively on Linux (systemd)

The same binary works on Linux. Install it as a systemd service as root:

```sh
gpu-turnstile --install-service            # installs into /var/lib/gpu-turnstile, enables + starts
gpu-turnstile --install-service --no-copy  # register in place instead
gpu-turnstile --remove-service
```

The unit (`/etc/systemd/system/gpu-turnstile.service`) is `Type=notify`:
`systemctl start` blocks until the listeners are actually bound, a 30 s
watchdog restarts the process if it wedges, and logs land in the journal
(`journalctl -u gpu-turnstile -f`) unless `LOG_FILE` is set. Install
copies the binary to `/var/lib/gpu-turnstile/` and the config to
`/etc/gpu-turnstile.env` (edit that one after installing). The service
runs sandboxed with `DynamicUser=yes` — a transient low-privilege UID,
read-only filesystem except its install dir (so self-update keeps
working), no capabilities, syscall-filtered: same least-privilege idea as
the Windows virtual account. The notify integration is a no-op in
containers and interactive shells.

**Auto-update is on by default**: the binary checks the repo's latest
release on startup and every `UPDATE_INTERVAL`, verifies the Ed25519
signature of the download against the public key embedded at build time,
and — once the GPU lock is idle — restarts the service onto the new
version. `APP_VER` controls the target: `dev` disables updates, `stable`
(the default) tracks the latest release, and an exact `vX.Y.Z` pins that
release (even as a downgrade or to replace a dev build). Disable entirely
with `AUTO_UPDATE=false`. Releases are signed by CI with
OpenSSL; the matching public key lives in `internal/update/pubkey.go`
(one-time setup: `openssl genpkey -algorithm ed25519 -out private.pem`,
`openssl pkey -in private.pem -pubout -out public.pem`; private key goes
to the `RELEASE_SIGNING_KEY` repo secret, public key is committed).
`gpu-turnstile --force-update` checks immediately, stages the new binary
and restarts the running service (elevating via UAC only if needed).

### Docker

```sh
docker build -t gpu-turnstile .
docker run --rm -p 11434:11434 -p 8188:8188 \
  -e OLLAMA_URL=http://<workstation-ip>:11435 \
  -e COMFY_URL=http://<workstation-ip>:8189 \
  gpu-turnstile
```

Releases are built by Gitea Actions (`.gitea/workflows/ci.yml`): every push
runs `go vet` and `go test -race`, and pushing a semantic-version tag
`vX.Y.Z` publishes the container image
(`git.rambossek.at/<owner>/gpu-turnstile:vX.Y.Z` plus `:latest`) and a
signed Windows binary attached to a Gitea release. Nothing is built from
branches.

The registry login needs one repository secret (Settings → Actions →
Secrets): `REGISTRY_TOKEN` — an access token with `write:package` scope.
The automatic `GITEA_TOKEN` cannot push packages.

## Development

```sh
go vet ./...
go test -race ./...
```

Go 1.23+; the only external dependency is `golang.org/x/sys` (Windows
service integration, unused in the Linux build). Layout:

```
cmd/gpu-turnstile/main.go  wiring, config, listeners, service + updater
internal/lock/             two-mode lock (LLM readers / image writer, FIFO)
internal/ollama/           ps / unload / warm client
internal/comfy/            history poll / free client
internal/proxy/            handlers for both listeners
internal/metrics/          Prometheus exposition, no dependencies
internal/config/           env + .env file configuration
internal/update/           signed auto-updater
internal/service/          Windows service integration
```