PUBLIC/gpu-turnstile
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Rename to gpu-turnstile; swap port roles

Browse Source 
The proxy now listens on the standard service ports (Ollama :11434,
ComfyUI :8188) and the actual services move one port up (:11435, :8189).
Metric prefix is now gpu_turnstile_.
This commit is contained in:
mram
2026-09-20 18:10:22 +02:00
parent 065c294a96
commit db11da8307
10 changed files with 113 additions and 105 deletions
Download Patch File Download Diff File Expand all files Collapse all files
SPEC.md
+35 -32
View File
@@ -1,4 +1,4 @@
# gpulock — GPU arbitration proxy for Ollama + ComfyUI
# gpu-turnstile — GPU arbitration proxy for Ollama + ComfyUI
## Problem
@@ -16,8 +16,9 @@ at any moment the GPU is in exactly one of three states:
- `llm` — N ≥ 1 Ollama inference requests in flight (concurrency allowed)
- `image` — exactly one ComfyUI job in flight, Ollama models unloaded
Clients (LiteLLM, Open WebUI, n8n) point at gpulock instead of at the services.
gpulock is transparent for everything that does not touch the GPU.
Clients (LiteLLM, Open WebUI, n8n) point at gpu-turnstile instead of at the
services. gpu-turnstile is transparent for everything that does not touch the
GPU.
## Non-goals
@@ -30,15 +31,16 @@ gpulock is transparent for everything that does not touch the GPU.
## Architecture
```
LiteLLM / Open WebUI ──► :11435 ─┐ ┌─► Ollama :11434
├── gpulock (1 lock) ──┤
Open WebUI / n8n ────► :8189 ───┘ └─► ComfyUI :8188
LiteLLM / Open WebUI ──► :11434 ─┐ ┌─► Ollama :11435
├── gpu-turnstile (1 lock) ──┤
Open WebUI / n8n ────► :8188 ───┘ └─► ComfyUI :8189
```
Two listeners, one process, one lock. Each listener is an
`httputil.ReverseProxy` to its upstream. Websocket upgrades (ComfyUI `/ws`)
and streaming bodies (Ollama NDJSON / SSE) must pass through unbuffered
(`FlushInterval = -1`).
gpu-turnstile listens on the ports the services normally use; the actual
services run one port higher. Two listeners, one process, one lock. Each
listener is an `httputil.ReverseProxy` to its upstream. Websocket upgrades
(ComfyUI `/ws`) and streaming bodies (Ollama NDJSON / SSE) must pass through
unbuffered (`FlushInterval = -1`).
### Lock semantics
@@ -58,7 +60,7 @@ are LLM requests and the single "writer" is an image job):
### Endpoint classification
Ollama listener (`:11435` → `OLLAMA_URL`):
Ollama listener (`:11434` → `OLLAMA_URL`):
| Path | Handling |
|---|---|
@@ -66,7 +68,7 @@ Ollama listener (`:11435` → `OLLAMA_URL`):
| `POST /v1/chat/completions`, `/v1/completions`, `/v1/embeddings` | LLM lock |
| everything else (`/api/tags`, `/api/ps`, `/api/show`, `/api/version`, `/v1/models`, `/api/pull`, …) | pass-through, no lock |
ComfyUI listener (`:8189` → `COMFY_URL`):
ComfyUI listener (`:8188` → `COMFY_URL`):
| Path | Handling |
|---|---|
@@ -100,10 +102,10 @@ load time. Off by default.
| Var | Default | Meaning |
|---|---|---|
| `LISTEN_OLLAMA` | `:11435` | Ollama-facing listener |
| `LISTEN_COMFY` | `:8189` | ComfyUI-facing listener |
| `OLLAMA_URL` | `http://127.0.0.1:11434` | upstream |
| `COMFY_URL` | `http://127.0.0.1:8188` | upstream |
| `LISTEN_OLLAMA` | `:11434` | Ollama-facing listener |
| `LISTEN_COMFY` | `:8188` | ComfyUI-facing listener |
| `OLLAMA_URL` | `http://127.0.0.1:11435` | upstream |
| `COMFY_URL` | `http://127.0.0.1:8189` | upstream |
| `UNLOAD_TIMEOUT` | `60s` | wait for Ollama to unload |
| `JOB_TIMEOUT` | `15m` | wait for ComfyUI job |
| `LLM_WAIT_TIMEOUT` | `10m` | max time an LLM request waits for the lock before 503 |
@@ -119,9 +121,9 @@ start (`/api/version`, `/system_stats`); failure is logged, not fatal.
`{"state":"idle|llm|image","llm_inflight":N,"image_pending":B}`.
- `GET /metrics` on the Ollama listener: Prometheus text format, no external
dependency needed:
`gpulock_state{state="…"} 1`, `gpulock_llm_inflight`,
`gpulock_image_jobs_total`, `gpulock_lock_wait_seconds` (histogram, label
`kind="llm|image"`), `gpulock_unload_seconds`.
`gpu_turnstile_state{state="…"} 1`, `gpu_turnstile_llm_inflight`,
`gpu_turnstile_image_jobs_total`, `gpu_turnstile_lock_wait_seconds`
(histogram, label `kind="llm|image"`), `gpu_turnstile_unload_seconds`.
- Structured logs (`log/slog`, JSON when `LOG_FORMAT=json`), one line per
state transition and per image job phase with `prompt_id`.
@@ -143,8 +145,8 @@ start (`/api/version`, `/system_stats`); failure is logged, not fatal.
## Repository layout
```
gpulock/
cmd/gpulock/main.go # wiring, config, listeners
gpu-turnstile/
cmd/gpu-turnstile/main.go # wiring, config, listeners
internal/lock/lock.go # two-mode lock + tests
internal/ollama/client.go # ps / unload / warm
internal/comfy/client.go # history poll / free
@@ -176,30 +178,31 @@ are new.
- Go 1.23+, stdlib only. `CGO_ENABLED=0`, `-ldflags="-s -w"`, version from
`git describe` injected via `-X main.version=`.
- Dockerfile: multi-stage, final image `gcr.io/distroless/static` (or
`scratch`), non-root user, `EXPOSE 8189 11435`, `ENTRYPOINT ["/gpulock"]`.
`scratch`), non-root user, `EXPOSE 8188 11434`,
`ENTRYPOINT ["/gpu-turnstile"]`.
- `.gitea/workflows/ci.yml` (Gitea Actions):
1. on push and tag: `go vet`, `go test -race ./...`, `golangci-lint` if
available in the runner image
2. build image with buildx, tags `:sha-<short>` and `:latest` on main,
`:<tag>` on tags
3. push to the Gitea registry `git.rambossek.at/<owner>/gpulock` using the
workflow token (`${{ secrets.GITEA_TOKEN }}` / `gitea.actor`)
3. push to the Gitea registry `git.rambossek.at/<owner>/gpu-turnstile`
using the workflow token (`${{ secrets.GITEA_TOKEN }}` / `gitea.actor`)
- Release: a git tag `vX.Y.Z` produces the versioned image; the Open WebUI
compose pins that tag.
## Deployment (target)
```yaml
gpulock:
image: git.rambossek.at/<owner>/gpulock:v0.1.0
gpu-turnstile:
image: git.rambossek.at/<owner>/gpu-turnstile:v0.1.0
environment:
OLLAMA_URL: http://<workstation-ip>:11434
COMFY_URL: http://<workstation-ip>:8188
OLLAMA_URL: http://<workstation-ip>:11435
COMFY_URL: http://<workstation-ip>:8189
networks: [internal]
```
LiteLLM `api_base` → `http://gpulock:11435`; Open WebUI
`COMFYUI_BASE_URL` → `http://gpulock:8189`. Nothing else talks to the
LiteLLM `api_base` → `http://gpu-turnstile:11434`; Open WebUI
`COMFYUI_BASE_URL` → `http://gpu-turnstile:8188`. Nothing else talks to the
workstation directly.
## Open questions
@@ -207,5 +210,5 @@ workstation directly.
- Should embedding requests (`/api/embed`, `/v1/embeddings`) count as LLM
traffic for the lock? They do in this spec (they hold VRAM); reconsider if
RAG indexing starves image jobs for too long.
- Whether to add a `POST /gpulock/release` admin endpoint to force-reset the
lock without restarting. Cheap to add; decide once it's been stuck once.
- Whether to add a `POST /gpu-turnstile/release` admin endpoint to force-reset
the lock without restarting. Cheap to add; decide once it's been stuck once.