mram
e363c9e4e4
- internal/config: .env-style config file (gpu-turnstile.env next to the exe, -config flag or GPU_TURNSTILE_CONFIG); process env overrides file. - internal/service: Windows service via golang.org/x/sys/windows/svc — graceful SCM stop, 'service install/remove' commands, restart-on-failure recovery (also applies staged updates). First external dependency, Windows-only; Linux/Docker build unaffected (go.mod stays at 1.23). - internal/update: polls the Gitea releases API, verifies the Ed25519 signature of the downloaded binary against an embedded public key (openssl-signed by CI), swaps it in next to the running exe, and once the GPU lock is idle exits with code 3 so service recovery restarts onto the new version. Dev builds and empty pubkey never update. - CI: tag builds additionally produce gpu-turnstile.exe + .sig + .sha256 attached to a Gitea release. - LOG_FILE env var so the service has somewhere to log.
151 lines
6.9 KiB
Markdown
151 lines
6.9 KiB
Markdown
# gpu-turnstile
|
|
|
|
GPU arbitration proxy for Ollama + ComfyUI. One consumer GPU is shared by an
|
|
LLM server (Ollama) and an image generator (ComfyUI); gpu-turnstile sits in
|
|
front of both and guarantees the GPU is always in exactly one of three states:
|
|
`idle`, `llm` (N ≥ 1 Ollama requests in flight), or `image` (exactly one
|
|
ComfyUI job, Ollama models unloaded). See [SPEC.md](SPEC.md) for the full
|
|
design.
|
|
|
|
gpu-turnstile listens on the ports the services normally use; the actual
|
|
services run one port higher (Ollama on 11435, ComfyUI on 8189).
|
|
|
|
```
|
|
LiteLLM / Open WebUI ──► :11434 ─┐ ┌─► Ollama :11435
|
|
├── gpu-turnstile (1 lock) ──┤
|
|
Open WebUI / n8n ────► :8188 ───┘ └─► ComfyUI :8189
|
|
```
|
|
|
|
- LLM endpoints (`/api/generate`, `/api/chat`, `/api/embed`, `/v1/*`) take the
|
|
LLM lock: concurrent requests allowed, but blocked while an image job is
|
|
active or waiting (image priority).
|
|
- `POST /prompt` on the ComfyUI listener takes the image lock: new LLM
|
|
requests block, in-flight LLMs drain, Ollama models are unloaded, the prompt
|
|
is forwarded, and the lock is held until the job finishes and ComfyUI frees
|
|
its VRAM.
|
|
- Everything else (including websockets and all streaming) passes through
|
|
transparently and unbuffered.
|
|
|
|
## Configuration
|
|
|
|
Configuration comes from environment variables and/or an `.env`-style
|
|
config file (`KEY=VALUE` lines, `#` comments). File lookup order:
|
|
`-config <path>` flag, then `GPU_TURNSTILE_CONFIG`, then
|
|
`gpu-turnstile.env` next to the executable. Process environment variables
|
|
override file values. Invalid values fail at startup.
|
|
|
|
| Var | Default | Meaning |
|
|
|---|---|---|
|
|
| `LISTEN_OLLAMA` | `:11434` | Ollama-facing listener |
|
|
| `LISTEN_COMFY` | `:8188` | ComfyUI-facing listener |
|
|
| `OLLAMA_URL` | `http://127.0.0.1:11435` | Ollama upstream |
|
|
| `COMFY_URL` | `http://127.0.0.1:8189` | ComfyUI upstream |
|
|
| `UNLOAD_TIMEOUT` | `60s` | Wait for Ollama to unload before an image job |
|
|
| `JOB_TIMEOUT` | `15m` | Wait for a ComfyUI job to finish |
|
|
| `LLM_WAIT_TIMEOUT` | `10m` | Max lock wait for an LLM request before 503 |
|
|
| `WARM_MODEL` | _(empty)_ | Model to reload after an image job (off by default) |
|
|
| `LOGLEVEL` | `warn` | `info` logs every request (colored arrows in text mode), `debug` adds lock transitions. `LOG_LEVEL` works as an alias |
|
|
| `LOG_FORMAT` | `text` | `json` for structured JSON logs |
|
|
| `LOG_FILE` | _(empty)_ | Append logs to this file instead of stderr |
|
|
| `UNLOAD_POLL_INTERVAL` | `500ms` | `/api/ps` poll interval while unloading |
|
|
| `HISTORY_POLL_INTERVAL` | `1s` | `/history/<id>` poll interval while a job runs |
|
|
| `PROBE_TIMEOUT` | `5s` | Startup probe of both upstreams |
|
|
| `FREE_TIMEOUT` | `30s` | `POST /free` call after an image job |
|
|
| `WARM_TIMEOUT` | `2m` | Warm-model reload after an image job |
|
|
| `SHUTDOWN_TIMEOUT` | `10s` | Graceful shutdown on SIGINT/SIGTERM |
|
|
| `BACKOFF_INITIAL` | `1s` | First retry wait when an upstream refuses a connection |
|
|
| `BACKOFF_MAX` | `60s` | Cap for the exponential retry backoff |
|
|
| `PROMPT_CAPTURE_LIMIT` | `65536` | Bytes of the `/prompt` response buffered to find `prompt_id` (pass-through is unaffected) |
|
|
| `AUTO_UPDATE` | `true` | Poll the Gitea releases API for signed updates |
|
|
| `UPDATE_INTERVAL` | `6h` | Auto-update check interval |
|
|
| `UPDATE_REPO` | `https://git.rambossek.at/PUBLIC/gpu-turnstile` | Repository checked for releases |
|
|
| `UPDATE_ASSET` | `gpu-turnstile.exe` | Release asset to download |
|
|
|
|
## Observability
|
|
|
|
- `GET /healthz` (both listeners): `{"state":"idle|llm|image","llm_inflight":N,"image_pending":B}`
|
|
- `GET /metrics` (Ollama listener): Prometheus text format — `gpu_turnstile_state`,
|
|
`gpu_turnstile_llm_inflight`, `gpu_turnstile_image_pending`,
|
|
`gpu_turnstile_image_jobs_total`, `gpu_turnstile_lock_wait_seconds`
|
|
(histogram, `kind="llm|image"`), `gpu_turnstile_unload_seconds`.
|
|
- Logs: startup logs the version and every setting (visible even at the
|
|
default `warn` level). With `LOGLEVEL=info` or `debug`, every request
|
|
logs a `-->` incoming line and a `<--` response line with status and
|
|
duration — ANSI-colored (cyan incoming; green/yellow/red by status
|
|
class) in text mode, which renders in `docker compose logs` on Windows
|
|
Terminal. Set `NO_COLOR` to disable colors.
|
|
|
|
## Build and run
|
|
|
|
```sh
|
|
go build ./cmd/gpu-turnstile
|
|
./gpu-turnstile
|
|
```
|
|
|
|
### Run natively on Windows (primary deployment)
|
|
|
|
Download `gpu-turnstile.exe` from a release, put a `gpu-turnstile.env`
|
|
next to it, and run it — or install it as a Windows service from an
|
|
elevated shell:
|
|
|
|
```sh
|
|
gpu-turnstile.exe service install # auto-start service, recovery = restart
|
|
gpu-turnstile.exe service remove
|
|
```
|
|
|
|
The service uses the config file (services have no convenient
|
|
environment); set `LOG_FILE` in it since there is no console.
|
|
|
|
**Auto-update is on by default**: the binary checks the repo's latest
|
|
release on startup and every `UPDATE_INTERVAL`, verifies the Ed25519
|
|
signature of the download against the public key embedded at build time,
|
|
and — once the GPU lock is idle — restarts the service onto the new
|
|
version. Disable with `AUTO_UPDATE=false`. Releases are signed by CI with
|
|
OpenSSL; the matching public key lives in `internal/update/pubkey.go`
|
|
(one-time setup: `openssl genpkey -algorithm ed25519 -out private.pem`,
|
|
`openssl pkey -in private.pem -pubout -out public.pem`; private key goes
|
|
to the `SIGNING_KEY` repo secret, public key is committed).
|
|
|
|
### Docker
|
|
|
|
```sh
|
|
docker build -t gpu-turnstile .
|
|
docker run --rm -p 11434:11434 -p 8188:8188 \
|
|
-e OLLAMA_URL=http://<workstation-ip>:11435 \
|
|
-e COMFY_URL=http://<workstation-ip>:8189 \
|
|
gpu-turnstile
|
|
```
|
|
|
|
Releases are built by Gitea Actions (`.gitea/workflows/ci.yml`): every push
|
|
runs `go vet` and `go test -race`, and pushing a semantic-version tag
|
|
`vX.Y.Z` publishes the container image
|
|
(`git.rambossek.at/<owner>/gpu-turnstile:vX.Y.Z` plus `:latest`) and a
|
|
signed Windows binary attached to a Gitea release. Nothing is built from
|
|
branches.
|
|
|
|
The registry login needs one repository secret (Settings → Actions →
|
|
Secrets): `REGISTRY_TOKEN` — an access token with `write:package` scope.
|
|
The automatic `GITEA_TOKEN` cannot push packages.
|
|
|
|
## Development
|
|
|
|
```sh
|
|
go vet ./...
|
|
go test -race ./...
|
|
```
|
|
|
|
Go 1.23+; the only external dependency is `golang.org/x/sys` (Windows
|
|
service integration, unused in the Linux build). Layout:
|
|
|
|
```
|
|
cmd/gpu-turnstile/main.go wiring, config, listeners, service + updater
|
|
internal/lock/ two-mode lock (LLM readers / image writer, FIFO)
|
|
internal/ollama/ ps / unload / warm client
|
|
internal/comfy/ history poll / free client
|
|
internal/proxy/ handlers for both listeners
|
|
internal/metrics/ Prometheus exposition, no dependencies
|
|
internal/config/ env + .env file configuration
|
|
internal/update/ signed auto-updater
|
|
internal/service/ Windows service integration
|
|
```
|