mram
20d5439b5f
A refused dial (service down/restarting) is retried with a wait that doubles from BACKOFF_INITIAL (1s) up to BACKOFF_MAX (60s) until the upstream answers or the client disconnects. Handles the Windows WSA errno (10061) as well as POSIX ECONNREFUSED. compose.yaml.example now uses host.docker.internal like the working local deployment.
106 lines
4.4 KiB
Markdown
106 lines
4.4 KiB
Markdown
# gpu-turnstile
|
|
|
|
GPU arbitration proxy for Ollama + ComfyUI. One consumer GPU is shared by an
|
|
LLM server (Ollama) and an image generator (ComfyUI); gpu-turnstile sits in
|
|
front of both and guarantees the GPU is always in exactly one of three states:
|
|
`idle`, `llm` (N ≥ 1 Ollama requests in flight), or `image` (exactly one
|
|
ComfyUI job, Ollama models unloaded). See [SPEC.md](SPEC.md) for the full
|
|
design.
|
|
|
|
gpu-turnstile listens on the ports the services normally use; the actual
|
|
services run one port higher (Ollama on 11435, ComfyUI on 8189).
|
|
|
|
```
|
|
LiteLLM / Open WebUI ──► :11434 ─┐ ┌─► Ollama :11435
|
|
├── gpu-turnstile (1 lock) ──┤
|
|
Open WebUI / n8n ────► :8188 ───┘ └─► ComfyUI :8189
|
|
```
|
|
|
|
- LLM endpoints (`/api/generate`, `/api/chat`, `/api/embed`, `/v1/*`) take the
|
|
LLM lock: concurrent requests allowed, but blocked while an image job is
|
|
active or waiting (image priority).
|
|
- `POST /prompt` on the ComfyUI listener takes the image lock: new LLM
|
|
requests block, in-flight LLMs drain, Ollama models are unloaded, the prompt
|
|
is forwarded, and the lock is held until the job finishes and ComfyUI frees
|
|
its VRAM.
|
|
- Everything else (including websockets and all streaming) passes through
|
|
transparently and unbuffered.
|
|
|
|
## Configuration
|
|
|
|
All configuration is via environment variables; invalid values fail at
|
|
startup.
|
|
|
|
| Var | Default | Meaning |
|
|
|---|---|---|
|
|
| `LISTEN_OLLAMA` | `:11434` | Ollama-facing listener |
|
|
| `LISTEN_COMFY` | `:8188` | ComfyUI-facing listener |
|
|
| `OLLAMA_URL` | `http://127.0.0.1:11435` | Ollama upstream |
|
|
| `COMFY_URL` | `http://127.0.0.1:8189` | ComfyUI upstream |
|
|
| `UNLOAD_TIMEOUT` | `60s` | Wait for Ollama to unload before an image job |
|
|
| `JOB_TIMEOUT` | `15m` | Wait for a ComfyUI job to finish |
|
|
| `LLM_WAIT_TIMEOUT` | `10m` | Max lock wait for an LLM request before 503 |
|
|
| `WARM_MODEL` | _(empty)_ | Model to reload after an image job (off by default) |
|
|
| `LOG_LEVEL` | `info` | `debug` logs every lock transition |
|
|
| `LOG_FORMAT` | `text` | `json` for structured JSON logs |
|
|
| `UNLOAD_POLL_INTERVAL` | `500ms` | `/api/ps` poll interval while unloading |
|
|
| `HISTORY_POLL_INTERVAL` | `1s` | `/history/<id>` poll interval while a job runs |
|
|
| `PROBE_TIMEOUT` | `5s` | Startup probe of both upstreams |
|
|
| `FREE_TIMEOUT` | `30s` | `POST /free` call after an image job |
|
|
| `WARM_TIMEOUT` | `2m` | Warm-model reload after an image job |
|
|
| `SHUTDOWN_TIMEOUT` | `10s` | Graceful shutdown on SIGINT/SIGTERM |
|
|
| `BACKOFF_INITIAL` | `1s` | First retry wait when an upstream refuses a connection |
|
|
| `BACKOFF_MAX` | `60s` | Cap for the exponential retry backoff |
|
|
| `PROMPT_CAPTURE_LIMIT` | `65536` | Bytes of the `/prompt` response buffered to find `prompt_id` (pass-through is unaffected) |
|
|
|
|
## Observability
|
|
|
|
- `GET /healthz` (both listeners): `{"state":"idle|llm|image","llm_inflight":N,"image_pending":B}`
|
|
- `GET /metrics` (Ollama listener): Prometheus text format — `gpu_turnstile_state`,
|
|
`gpu_turnstile_llm_inflight`, `gpu_turnstile_image_pending`,
|
|
`gpu_turnstile_image_jobs_total`, `gpu_turnstile_lock_wait_seconds`
|
|
(histogram, `kind="llm|image"`), `gpu_turnstile_unload_seconds`.
|
|
|
|
## Build and run
|
|
|
|
```sh
|
|
go build ./cmd/gpu-turnstile
|
|
./gpu-turnstile
|
|
```
|
|
|
|
```sh
|
|
docker build -t gpu-turnstile .
|
|
docker run --rm -p 11434:11434 -p 8188:8188 \
|
|
-e OLLAMA_URL=http://<workstation-ip>:11435 \
|
|
-e COMFY_URL=http://<workstation-ip>:8189 \
|
|
gpu-turnstile
|
|
```
|
|
|
|
Releases are built by Gitea Actions (`.gitea/workflows/ci.yml`): every push
|
|
runs `go vet` and `go test -race`, and pushing a semantic-version tag
|
|
`vX.Y.Z` builds and publishes
|
|
`git.rambossek.at/<owner>/gpu-turnstile:vX.Y.Z` (and updates `:latest`).
|
|
No images are built from branches.
|
|
|
|
The registry login needs one repository secret (Settings → Actions →
|
|
Secrets): `REGISTRY_TOKEN` — an access token with `write:package` scope.
|
|
The automatic `GITEA_TOKEN` cannot push packages.
|
|
|
|
## Development
|
|
|
|
```sh
|
|
go vet ./...
|
|
go test -race ./...
|
|
```
|
|
|
|
Stdlib only, Go 1.23+. Layout:
|
|
|
|
```
|
|
cmd/gpu-turnstile/main.go wiring, config, listeners
|
|
internal/lock/ two-mode lock (LLM readers / image writer, FIFO)
|
|
internal/ollama/ ps / unload / warm client
|
|
internal/comfy/ history poll / free client
|
|
internal/proxy/ handlers for both listeners
|
|
internal/metrics/ Prometheus exposition, no dependencies
|
|
```
|