Rename to gpu-turnstile; swap port roles
The proxy now listens on the standard service ports (Ollama :11434, ComfyUI :8188) and the actual services move one port up (:11435, :8189). Metric prefix is now gpu_turnstile_.
This commit is contained in:
+2
-2
@@ -1,2 +1,2 @@
|
|||||||
/gpulock
|
/gpu-turnstile
|
||||||
/gpulock.exe
|
/gpu-turnstile.exe
|
||||||
|
|||||||
+4
-4
@@ -4,10 +4,10 @@ COPY go.mod ./
|
|||||||
COPY cmd ./cmd
|
COPY cmd ./cmd
|
||||||
COPY internal ./internal
|
COPY internal ./internal
|
||||||
ARG VERSION=dev
|
ARG VERSION=dev
|
||||||
RUN CGO_ENABLED=0 go build -ldflags="-s -w -X main.version=${VERSION}" -o /gpulock ./cmd/gpulock
|
RUN CGO_ENABLED=0 go build -ldflags="-s -w -X main.version=${VERSION}" -o /gpu-turnstile ./cmd/gpu-turnstile
|
||||||
|
|
||||||
FROM gcr.io/distroless/static:nonroot
|
FROM gcr.io/distroless/static:nonroot
|
||||||
COPY --from=build /gpulock /gpulock
|
COPY --from=build /gpu-turnstile /gpu-turnstile
|
||||||
EXPOSE 8189 11435
|
EXPOSE 8188 11434
|
||||||
USER nonroot
|
USER nonroot
|
||||||
ENTRYPOINT ["/gpulock"]
|
ENTRYPOINT ["/gpu-turnstile"]
|
||||||
|
|||||||
@@ -1,15 +1,19 @@
|
|||||||
# gpulock
|
# gpu-turnstile
|
||||||
|
|
||||||
GPU arbitration proxy for Ollama + ComfyUI. One consumer GPU is shared by an
|
GPU arbitration proxy for Ollama + ComfyUI. One consumer GPU is shared by an
|
||||||
LLM server (Ollama) and an image generator (ComfyUI); gpulock sits in front of
|
LLM server (Ollama) and an image generator (ComfyUI); gpu-turnstile sits in
|
||||||
both and guarantees the GPU is always in exactly one of three states: `idle`,
|
front of both and guarantees the GPU is always in exactly one of three states:
|
||||||
`llm` (N ≥ 1 Ollama requests in flight), or `image` (exactly one ComfyUI job,
|
`idle`, `llm` (N ≥ 1 Ollama requests in flight), or `image` (exactly one
|
||||||
Ollama models unloaded). See [SPEC.md](SPEC.md) for the full design.
|
ComfyUI job, Ollama models unloaded). See [SPEC.md](SPEC.md) for the full
|
||||||
|
design.
|
||||||
|
|
||||||
|
gpu-turnstile listens on the ports the services normally use; the actual
|
||||||
|
services run one port higher (Ollama on 11435, ComfyUI on 8189).
|
||||||
|
|
||||||
```
|
```
|
||||||
LiteLLM / Open WebUI ──► :11435 ─┐ ┌─► Ollama :11434
|
LiteLLM / Open WebUI ──► :11434 ─┐ ┌─► Ollama :11435
|
||||||
├── gpulock (1 lock) ──┤
|
├── gpu-turnstile (1 lock) ──┤
|
||||||
Open WebUI / n8n ────► :8189 ───┘ └─► ComfyUI :8188
|
Open WebUI / n8n ────► :8188 ───┘ └─► ComfyUI :8189
|
||||||
```
|
```
|
||||||
|
|
||||||
- LLM endpoints (`/api/generate`, `/api/chat`, `/api/embed`, `/v1/*`) take the
|
- LLM endpoints (`/api/generate`, `/api/chat`, `/api/embed`, `/v1/*`) take the
|
||||||
@@ -29,10 +33,10 @@ startup.
|
|||||||
|
|
||||||
| Var | Default | Meaning |
|
| Var | Default | Meaning |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `LISTEN_OLLAMA` | `:11435` | Ollama-facing listener |
|
| `LISTEN_OLLAMA` | `:11434` | Ollama-facing listener |
|
||||||
| `LISTEN_COMFY` | `:8189` | ComfyUI-facing listener |
|
| `LISTEN_COMFY` | `:8188` | ComfyUI-facing listener |
|
||||||
| `OLLAMA_URL` | `http://127.0.0.1:11434` | Ollama upstream |
|
| `OLLAMA_URL` | `http://127.0.0.1:11435` | Ollama upstream |
|
||||||
| `COMFY_URL` | `http://127.0.0.1:8188` | ComfyUI upstream |
|
| `COMFY_URL` | `http://127.0.0.1:8189` | ComfyUI upstream |
|
||||||
| `UNLOAD_TIMEOUT` | `60s` | Wait for Ollama to unload before an image job |
|
| `UNLOAD_TIMEOUT` | `60s` | Wait for Ollama to unload before an image job |
|
||||||
| `JOB_TIMEOUT` | `15m` | Wait for a ComfyUI job to finish |
|
| `JOB_TIMEOUT` | `15m` | Wait for a ComfyUI job to finish |
|
||||||
| `LLM_WAIT_TIMEOUT` | `10m` | Max lock wait for an LLM request before 503 |
|
| `LLM_WAIT_TIMEOUT` | `10m` | Max lock wait for an LLM request before 503 |
|
||||||
@@ -43,29 +47,29 @@ startup.
|
|||||||
## Observability
|
## Observability
|
||||||
|
|
||||||
- `GET /healthz` (both listeners): `{"state":"idle|llm|image","llm_inflight":N,"image_pending":B}`
|
- `GET /healthz` (both listeners): `{"state":"idle|llm|image","llm_inflight":N,"image_pending":B}`
|
||||||
- `GET /metrics` (Ollama listener): Prometheus text format — `gpulock_state`,
|
- `GET /metrics` (Ollama listener): Prometheus text format — `gpu_turnstile_state`,
|
||||||
`gpulock_llm_inflight`, `gpulock_image_pending`, `gpulock_image_jobs_total`,
|
`gpu_turnstile_llm_inflight`, `gpu_turnstile_image_pending`,
|
||||||
`gpulock_lock_wait_seconds` (histogram, `kind="llm|image"`),
|
`gpu_turnstile_image_jobs_total`, `gpu_turnstile_lock_wait_seconds`
|
||||||
`gpulock_unload_seconds`.
|
(histogram, `kind="llm|image"`), `gpu_turnstile_unload_seconds`.
|
||||||
|
|
||||||
## Build and run
|
## Build and run
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
go build ./cmd/gpulock
|
go build ./cmd/gpu-turnstile
|
||||||
./gpulock
|
./gpu-turnstile
|
||||||
```
|
```
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
docker build -t gpulock .
|
docker build -t gpu-turnstile .
|
||||||
docker run --rm -p 11435:11435 -p 8189:8189 \
|
docker run --rm -p 11434:11434 -p 8188:8188 \
|
||||||
-e OLLAMA_URL=http://<workstation-ip>:11434 \
|
-e OLLAMA_URL=http://<workstation-ip>:11435 \
|
||||||
-e COMFY_URL=http://<workstation-ip>:8188 \
|
-e COMFY_URL=http://<workstation-ip>:8189 \
|
||||||
gpulock
|
gpu-turnstile
|
||||||
```
|
```
|
||||||
|
|
||||||
Releases are built by Gitea Actions (`.gitea/workflows/ci.yml`): pushes run
|
Releases are built by Gitea Actions (`.gitea/workflows/ci.yml`): pushes run
|
||||||
`go vet` and `go test -race` and publish
|
`go vet` and `go test -race` and publish
|
||||||
`git.rambossek.at/<owner>/gpulock:sha-<short>`; `main` additionally gets
|
`git.rambossek.at/<owner>/gpu-turnstile:sha-<short>`; `main` additionally gets
|
||||||
`:latest`, and a git tag `vX.Y.Z` produces the versioned image.
|
`:latest`, and a git tag `vX.Y.Z` produces the versioned image.
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
@@ -78,7 +82,7 @@ go test -race ./...
|
|||||||
Stdlib only, Go 1.23+. Layout:
|
Stdlib only, Go 1.23+. Layout:
|
||||||
|
|
||||||
```
|
```
|
||||||
cmd/gpulock/main.go wiring, config, listeners
|
cmd/gpu-turnstile/main.go wiring, config, listeners
|
||||||
internal/lock/ two-mode lock (LLM readers / image writer, FIFO)
|
internal/lock/ two-mode lock (LLM readers / image writer, FIFO)
|
||||||
internal/ollama/ ps / unload / warm client
|
internal/ollama/ ps / unload / warm client
|
||||||
internal/comfy/ history poll / free client
|
internal/comfy/ history poll / free client
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
# gpulock — GPU arbitration proxy for Ollama + ComfyUI
|
# gpu-turnstile — GPU arbitration proxy for Ollama + ComfyUI
|
||||||
|
|
||||||
## Problem
|
## Problem
|
||||||
|
|
||||||
@@ -16,8 +16,9 @@ at any moment the GPU is in exactly one of three states:
|
|||||||
- `llm` — N ≥ 1 Ollama inference requests in flight (concurrency allowed)
|
- `llm` — N ≥ 1 Ollama inference requests in flight (concurrency allowed)
|
||||||
- `image` — exactly one ComfyUI job in flight, Ollama models unloaded
|
- `image` — exactly one ComfyUI job in flight, Ollama models unloaded
|
||||||
|
|
||||||
Clients (LiteLLM, Open WebUI, n8n) point at gpulock instead of at the services.
|
Clients (LiteLLM, Open WebUI, n8n) point at gpu-turnstile instead of at the
|
||||||
gpulock is transparent for everything that does not touch the GPU.
|
services. gpu-turnstile is transparent for everything that does not touch the
|
||||||
|
GPU.
|
||||||
|
|
||||||
## Non-goals
|
## Non-goals
|
||||||
|
|
||||||
@@ -30,15 +31,16 @@ gpulock is transparent for everything that does not touch the GPU.
|
|||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
```
|
```
|
||||||
LiteLLM / Open WebUI ──► :11435 ─┐ ┌─► Ollama :11434
|
LiteLLM / Open WebUI ──► :11434 ─┐ ┌─► Ollama :11435
|
||||||
├── gpulock (1 lock) ──┤
|
├── gpu-turnstile (1 lock) ──┤
|
||||||
Open WebUI / n8n ────► :8189 ───┘ └─► ComfyUI :8188
|
Open WebUI / n8n ────► :8188 ───┘ └─► ComfyUI :8189
|
||||||
```
|
```
|
||||||
|
|
||||||
Two listeners, one process, one lock. Each listener is an
|
gpu-turnstile listens on the ports the services normally use; the actual
|
||||||
`httputil.ReverseProxy` to its upstream. Websocket upgrades (ComfyUI `/ws`)
|
services run one port higher. Two listeners, one process, one lock. Each
|
||||||
and streaming bodies (Ollama NDJSON / SSE) must pass through unbuffered
|
listener is an `httputil.ReverseProxy` to its upstream. Websocket upgrades
|
||||||
(`FlushInterval = -1`).
|
(ComfyUI `/ws`) and streaming bodies (Ollama NDJSON / SSE) must pass through
|
||||||
|
unbuffered (`FlushInterval = -1`).
|
||||||
|
|
||||||
### Lock semantics
|
### Lock semantics
|
||||||
|
|
||||||
@@ -58,7 +60,7 @@ are LLM requests and the single "writer" is an image job):
|
|||||||
|
|
||||||
### Endpoint classification
|
### Endpoint classification
|
||||||
|
|
||||||
Ollama listener (`:11435` → `OLLAMA_URL`):
|
Ollama listener (`:11434` → `OLLAMA_URL`):
|
||||||
|
|
||||||
| Path | Handling |
|
| Path | Handling |
|
||||||
|---|---|
|
|---|---|
|
||||||
@@ -66,7 +68,7 @@ Ollama listener (`:11435` → `OLLAMA_URL`):
|
|||||||
| `POST /v1/chat/completions`, `/v1/completions`, `/v1/embeddings` | LLM lock |
|
| `POST /v1/chat/completions`, `/v1/completions`, `/v1/embeddings` | LLM lock |
|
||||||
| everything else (`/api/tags`, `/api/ps`, `/api/show`, `/api/version`, `/v1/models`, `/api/pull`, …) | pass-through, no lock |
|
| everything else (`/api/tags`, `/api/ps`, `/api/show`, `/api/version`, `/v1/models`, `/api/pull`, …) | pass-through, no lock |
|
||||||
|
|
||||||
ComfyUI listener (`:8189` → `COMFY_URL`):
|
ComfyUI listener (`:8188` → `COMFY_URL`):
|
||||||
|
|
||||||
| Path | Handling |
|
| Path | Handling |
|
||||||
|---|---|
|
|---|---|
|
||||||
@@ -100,10 +102,10 @@ load time. Off by default.
|
|||||||
|
|
||||||
| Var | Default | Meaning |
|
| Var | Default | Meaning |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `LISTEN_OLLAMA` | `:11435` | Ollama-facing listener |
|
| `LISTEN_OLLAMA` | `:11434` | Ollama-facing listener |
|
||||||
| `LISTEN_COMFY` | `:8189` | ComfyUI-facing listener |
|
| `LISTEN_COMFY` | `:8188` | ComfyUI-facing listener |
|
||||||
| `OLLAMA_URL` | `http://127.0.0.1:11434` | upstream |
|
| `OLLAMA_URL` | `http://127.0.0.1:11435` | upstream |
|
||||||
| `COMFY_URL` | `http://127.0.0.1:8188` | upstream |
|
| `COMFY_URL` | `http://127.0.0.1:8189` | upstream |
|
||||||
| `UNLOAD_TIMEOUT` | `60s` | wait for Ollama to unload |
|
| `UNLOAD_TIMEOUT` | `60s` | wait for Ollama to unload |
|
||||||
| `JOB_TIMEOUT` | `15m` | wait for ComfyUI job |
|
| `JOB_TIMEOUT` | `15m` | wait for ComfyUI job |
|
||||||
| `LLM_WAIT_TIMEOUT` | `10m` | max time an LLM request waits for the lock before 503 |
|
| `LLM_WAIT_TIMEOUT` | `10m` | max time an LLM request waits for the lock before 503 |
|
||||||
@@ -119,9 +121,9 @@ start (`/api/version`, `/system_stats`); failure is logged, not fatal.
|
|||||||
`{"state":"idle|llm|image","llm_inflight":N,"image_pending":B}`.
|
`{"state":"idle|llm|image","llm_inflight":N,"image_pending":B}`.
|
||||||
- `GET /metrics` on the Ollama listener: Prometheus text format, no external
|
- `GET /metrics` on the Ollama listener: Prometheus text format, no external
|
||||||
dependency needed:
|
dependency needed:
|
||||||
`gpulock_state{state="…"} 1`, `gpulock_llm_inflight`,
|
`gpu_turnstile_state{state="…"} 1`, `gpu_turnstile_llm_inflight`,
|
||||||
`gpulock_image_jobs_total`, `gpulock_lock_wait_seconds` (histogram, label
|
`gpu_turnstile_image_jobs_total`, `gpu_turnstile_lock_wait_seconds`
|
||||||
`kind="llm|image"`), `gpulock_unload_seconds`.
|
(histogram, label `kind="llm|image"`), `gpu_turnstile_unload_seconds`.
|
||||||
- Structured logs (`log/slog`, JSON when `LOG_FORMAT=json`), one line per
|
- Structured logs (`log/slog`, JSON when `LOG_FORMAT=json`), one line per
|
||||||
state transition and per image job phase with `prompt_id`.
|
state transition and per image job phase with `prompt_id`.
|
||||||
|
|
||||||
@@ -143,8 +145,8 @@ start (`/api/version`, `/system_stats`); failure is logged, not fatal.
|
|||||||
## Repository layout
|
## Repository layout
|
||||||
|
|
||||||
```
|
```
|
||||||
gpulock/
|
gpu-turnstile/
|
||||||
cmd/gpulock/main.go # wiring, config, listeners
|
cmd/gpu-turnstile/main.go # wiring, config, listeners
|
||||||
internal/lock/lock.go # two-mode lock + tests
|
internal/lock/lock.go # two-mode lock + tests
|
||||||
internal/ollama/client.go # ps / unload / warm
|
internal/ollama/client.go # ps / unload / warm
|
||||||
internal/comfy/client.go # history poll / free
|
internal/comfy/client.go # history poll / free
|
||||||
@@ -176,30 +178,31 @@ are new.
|
|||||||
- Go 1.23+, stdlib only. `CGO_ENABLED=0`, `-ldflags="-s -w"`, version from
|
- Go 1.23+, stdlib only. `CGO_ENABLED=0`, `-ldflags="-s -w"`, version from
|
||||||
`git describe` injected via `-X main.version=`.
|
`git describe` injected via `-X main.version=`.
|
||||||
- Dockerfile: multi-stage, final image `gcr.io/distroless/static` (or
|
- Dockerfile: multi-stage, final image `gcr.io/distroless/static` (or
|
||||||
`scratch`), non-root user, `EXPOSE 8189 11435`, `ENTRYPOINT ["/gpulock"]`.
|
`scratch`), non-root user, `EXPOSE 8188 11434`,
|
||||||
|
`ENTRYPOINT ["/gpu-turnstile"]`.
|
||||||
- `.gitea/workflows/ci.yml` (Gitea Actions):
|
- `.gitea/workflows/ci.yml` (Gitea Actions):
|
||||||
1. on push and tag: `go vet`, `go test -race ./...`, `golangci-lint` if
|
1. on push and tag: `go vet`, `go test -race ./...`, `golangci-lint` if
|
||||||
available in the runner image
|
available in the runner image
|
||||||
2. build image with buildx, tags `:sha-<short>` and `:latest` on main,
|
2. build image with buildx, tags `:sha-<short>` and `:latest` on main,
|
||||||
`:<tag>` on tags
|
`:<tag>` on tags
|
||||||
3. push to the Gitea registry `git.rambossek.at/<owner>/gpulock` using the
|
3. push to the Gitea registry `git.rambossek.at/<owner>/gpu-turnstile`
|
||||||
workflow token (`${{ secrets.GITEA_TOKEN }}` / `gitea.actor`)
|
using the workflow token (`${{ secrets.GITEA_TOKEN }}` / `gitea.actor`)
|
||||||
- Release: a git tag `vX.Y.Z` produces the versioned image; the Open WebUI
|
- Release: a git tag `vX.Y.Z` produces the versioned image; the Open WebUI
|
||||||
compose pins that tag.
|
compose pins that tag.
|
||||||
|
|
||||||
## Deployment (target)
|
## Deployment (target)
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
gpulock:
|
gpu-turnstile:
|
||||||
image: git.rambossek.at/<owner>/gpulock:v0.1.0
|
image: git.rambossek.at/<owner>/gpu-turnstile:v0.1.0
|
||||||
environment:
|
environment:
|
||||||
OLLAMA_URL: http://<workstation-ip>:11434
|
OLLAMA_URL: http://<workstation-ip>:11435
|
||||||
COMFY_URL: http://<workstation-ip>:8188
|
COMFY_URL: http://<workstation-ip>:8189
|
||||||
networks: [internal]
|
networks: [internal]
|
||||||
```
|
```
|
||||||
|
|
||||||
LiteLLM `api_base` → `http://gpulock:11435`; Open WebUI
|
LiteLLM `api_base` → `http://gpu-turnstile:11434`; Open WebUI
|
||||||
`COMFYUI_BASE_URL` → `http://gpulock:8189`. Nothing else talks to the
|
`COMFYUI_BASE_URL` → `http://gpu-turnstile:8188`. Nothing else talks to the
|
||||||
workstation directly.
|
workstation directly.
|
||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
@@ -207,5 +210,5 @@ workstation directly.
|
|||||||
- Should embedding requests (`/api/embed`, `/v1/embeddings`) count as LLM
|
- Should embedding requests (`/api/embed`, `/v1/embeddings`) count as LLM
|
||||||
traffic for the lock? They do in this spec (they hold VRAM); reconsider if
|
traffic for the lock? They do in this spec (they hold VRAM); reconsider if
|
||||||
RAG indexing starves image jobs for too long.
|
RAG indexing starves image jobs for too long.
|
||||||
- Whether to add a `POST /gpulock/release` admin endpoint to force-reset the
|
- Whether to add a `POST /gpu-turnstile/release` admin endpoint to force-reset
|
||||||
lock without restarting. Cheap to add; decide once it's been stuck once.
|
the lock without restarting. Cheap to add; decide once it's been stuck once.
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
// gpulock is a GPU arbitration proxy that sits in front of Ollama and
|
// gpu-turnstile is a GPU arbitration proxy that sits in front of Ollama and
|
||||||
// ComfyUI and guarantees only one of them uses the GPU at a time.
|
// ComfyUI and guarantees only one of them uses the GPU at a time.
|
||||||
package main
|
package main
|
||||||
|
|
||||||
@@ -52,10 +52,10 @@ func envDuration(getenv func(string) string, name string, dst *time.Duration) er
|
|||||||
|
|
||||||
func loadConfig(getenv func(string) string) (config, error) {
|
func loadConfig(getenv func(string) string) (config, error) {
|
||||||
cfg := config{
|
cfg := config{
|
||||||
listenOllama: ":11435",
|
listenOllama: ":11434",
|
||||||
listenComfy: ":8189",
|
listenComfy: ":8188",
|
||||||
ollamaURL: "http://127.0.0.1:11434",
|
ollamaURL: "http://127.0.0.1:11435",
|
||||||
comfyURL: "http://127.0.0.1:8188",
|
comfyURL: "http://127.0.0.1:8189",
|
||||||
unloadTimeout: time.Minute,
|
unloadTimeout: time.Minute,
|
||||||
jobTimeout: 15 * time.Minute,
|
jobTimeout: 15 * time.Minute,
|
||||||
llmWaitTimeout: 10 * time.Minute,
|
llmWaitTimeout: 10 * time.Minute,
|
||||||
@@ -107,7 +107,7 @@ func loadConfig(getenv func(string) string) (config, error) {
|
|||||||
func main() {
|
func main() {
|
||||||
cfg, err := loadConfig(os.Getenv)
|
cfg, err := loadConfig(os.Getenv)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintf(os.Stderr, "gpulock: %v\n", err)
|
fmt.Fprintf(os.Stderr, "gpu-turnstile: %v\n", err)
|
||||||
os.Exit(1)
|
os.Exit(1)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -119,7 +119,7 @@ func main() {
|
|||||||
log := slog.New(handler)
|
log := slog.New(handler)
|
||||||
slog.SetDefault(log)
|
slog.SetDefault(log)
|
||||||
|
|
||||||
log.Info("starting gpulock",
|
log.Info("starting gpu-turnstile",
|
||||||
"version", version,
|
"version", version,
|
||||||
"listen_ollama", cfg.listenOllama,
|
"listen_ollama", cfg.listenOllama,
|
||||||
"listen_comfy", cfg.listenComfy,
|
"listen_comfy", cfg.listenComfy,
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
// Package comfy is a minimal client for the ComfyUI endpoints gpulock needs
|
// Package comfy is a minimal client for the ComfyUI endpoints gpu-turnstile
|
||||||
// after a prompt has been accepted: polling /history and freeing VRAM.
|
// needs after a prompt has been accepted: polling /history and freeing VRAM.
|
||||||
package comfy
|
package comfy
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
|||||||
+21
-21
@@ -1,4 +1,4 @@
|
|||||||
// Package metrics provides the Prometheus text exposition for gpulock
|
// Package metrics provides the Prometheus text exposition for gpu-turnstile
|
||||||
// without any external dependencies.
|
// without any external dependencies.
|
||||||
package metrics
|
package metrics
|
||||||
|
|
||||||
@@ -51,7 +51,7 @@ func trimLabel(label string) string {
|
|||||||
return label
|
return label
|
||||||
}
|
}
|
||||||
|
|
||||||
// Metrics holds all gpulock metric values.
|
// Metrics holds all gpu-turnstile metric values.
|
||||||
type Metrics struct {
|
type Metrics struct {
|
||||||
llmWait *histogram
|
llmWait *histogram
|
||||||
imageWait *histogram
|
imageWait *histogram
|
||||||
@@ -86,36 +86,36 @@ func (m *Metrics) IncImageJobs() { m.imageJobs.Add(1) }
|
|||||||
|
|
||||||
// Render writes the Prometheus text exposition for the given lock snapshot.
|
// Render writes the Prometheus text exposition for the given lock snapshot.
|
||||||
func (m *Metrics) Render(w io.Writer, state string, llmInflight int, imagePending bool) {
|
func (m *Metrics) Render(w io.Writer, state string, llmInflight int, imagePending bool) {
|
||||||
fmt.Fprint(w, `# HELP gpulock_state Current GPU state (1 for the active state).
|
fmt.Fprint(w, `# HELP gpu_turnstile_state Current GPU state (1 for the active state).
|
||||||
# TYPE gpulock_state gauge
|
# TYPE gpu_turnstile_state gauge
|
||||||
`)
|
`)
|
||||||
for _, s := range []string{"idle", "llm", "image"} {
|
for _, s := range []string{"idle", "llm", "image"} {
|
||||||
v := 0
|
v := 0
|
||||||
if s == state {
|
if s == state {
|
||||||
v = 1
|
v = 1
|
||||||
}
|
}
|
||||||
fmt.Fprintf(w, "gpulock_state{state=%q} %d\n", s, v)
|
fmt.Fprintf(w, "gpu_turnstile_state{state=%q} %d\n", s, v)
|
||||||
}
|
}
|
||||||
pending := 0
|
pending := 0
|
||||||
if imagePending {
|
if imagePending {
|
||||||
pending = 1
|
pending = 1
|
||||||
}
|
}
|
||||||
fmt.Fprintf(w, `# HELP gpulock_llm_inflight LLM requests currently in flight.
|
fmt.Fprintf(w, `# HELP gpu_turnstile_llm_inflight LLM requests currently in flight.
|
||||||
# TYPE gpulock_llm_inflight gauge
|
# TYPE gpu_turnstile_llm_inflight gauge
|
||||||
gpulock_llm_inflight %d
|
gpu_turnstile_llm_inflight %d
|
||||||
# HELP gpulock_image_pending Whether an image job is active or waiting.
|
# HELP gpu_turnstile_image_pending Whether an image job is active or waiting.
|
||||||
# TYPE gpulock_image_pending gauge
|
# TYPE gpu_turnstile_image_pending gauge
|
||||||
gpulock_image_pending %d
|
gpu_turnstile_image_pending %d
|
||||||
# HELP gpulock_image_jobs_total Image jobs accepted by ComfyUI.
|
# HELP gpu_turnstile_image_jobs_total Image jobs accepted by ComfyUI.
|
||||||
# TYPE gpulock_image_jobs_total counter
|
# TYPE gpu_turnstile_image_jobs_total counter
|
||||||
gpulock_image_jobs_total %d
|
gpu_turnstile_image_jobs_total %d
|
||||||
# HELP gpulock_lock_wait_seconds Time spent waiting to acquire the GPU lock.
|
# HELP gpu_turnstile_lock_wait_seconds Time spent waiting to acquire the GPU lock.
|
||||||
# TYPE gpulock_lock_wait_seconds histogram
|
# TYPE gpu_turnstile_lock_wait_seconds histogram
|
||||||
`, llmInflight, pending, m.imageJobs.Load())
|
`, llmInflight, pending, m.imageJobs.Load())
|
||||||
m.llmWait.write(w, "gpulock_lock_wait_seconds", `kind="llm",`)
|
m.llmWait.write(w, "gpu_turnstile_lock_wait_seconds", `kind="llm",`)
|
||||||
m.imageWait.write(w, "gpulock_lock_wait_seconds", `kind="image",`)
|
m.imageWait.write(w, "gpu_turnstile_lock_wait_seconds", `kind="image",`)
|
||||||
fmt.Fprint(w, `# HELP gpulock_unload_seconds Time spent unloading Ollama models before an image job.
|
fmt.Fprint(w, `# HELP gpu_turnstile_unload_seconds Time spent unloading Ollama models before an image job.
|
||||||
# TYPE gpulock_unload_seconds histogram
|
# TYPE gpu_turnstile_unload_seconds histogram
|
||||||
`)
|
`)
|
||||||
m.unload.write(w, "gpulock_unload_seconds", "")
|
m.unload.write(w, "gpu_turnstile_unload_seconds", "")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
// Package ollama is a minimal client for the Ollama management endpoints
|
// Package ollama is a minimal client for the Ollama management endpoints
|
||||||
// gpulock needs: listing loaded models, unloading them, and warming a model.
|
// gpu-turnstile needs: listing loaded models, unloading them, and warming a
|
||||||
|
// model.
|
||||||
package ollama
|
package ollama
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
// Package proxy contains the HTTP handlers for both gpulock listeners:
|
// Package proxy contains the HTTP handlers for both gpu-turnstile listeners:
|
||||||
// reverse proxies to Ollama and ComfyUI with GPU lock arbitration in front
|
// reverse proxies to Ollama and ComfyUI with GPU lock arbitration in front
|
||||||
// of the endpoints that load models.
|
// of the endpoints that load models.
|
||||||
package proxy
|
package proxy
|
||||||
@@ -43,7 +43,7 @@ type Config struct {
|
|||||||
WarmModel string
|
WarmModel string
|
||||||
}
|
}
|
||||||
|
|
||||||
// Server serves both gpulock listeners.
|
// Server serves both gpu-turnstile listeners.
|
||||||
type Server struct {
|
type Server struct {
|
||||||
cfg Config
|
cfg Config
|
||||||
log *slog.Logger
|
log *slog.Logger
|
||||||
|
|||||||
@@ -40,13 +40,13 @@ func (r *recorder) index(e string) int {
|
|||||||
return -1
|
return -1
|
||||||
}
|
}
|
||||||
|
|
||||||
// fakes wires up fake Ollama and ComfyUI upstreams plus the gpulock server.
|
// fakes wires up fake Ollama and ComfyUI upstreams plus the gpu-turnstile server.
|
||||||
type fakes struct {
|
type fakes struct {
|
||||||
rec *recorder
|
rec *recorder
|
||||||
ollama *httptest.Server
|
ollama *httptest.Server
|
||||||
comfy *httptest.Server
|
comfy *httptest.Server
|
||||||
server *httptest.Server // Ollama-facing gpulock listener
|
server *httptest.Server // Ollama-facing gpu-turnstile listener
|
||||||
comfySrv *httptest.Server // ComfyUI-facing gpulock listener
|
comfySrv *httptest.Server // ComfyUI-facing gpu-turnstile listener
|
||||||
freeCh chan struct{}
|
freeCh chan struct{}
|
||||||
chatCh chan struct{}
|
chatCh chan struct{}
|
||||||
historyMu sync.Mutex
|
historyMu sync.Mutex
|
||||||
@@ -279,7 +279,7 @@ func TestStreamingNotBuffered(t *testing.T) {
|
|||||||
}
|
}
|
||||||
defer resp.Body.Close()
|
defer resp.Body.Close()
|
||||||
|
|
||||||
// If gpulock buffered the stream, this read would never complete while
|
// If gpu-turnstile buffered the stream, this read would never complete while
|
||||||
// the gate is closed.
|
// the gate is closed.
|
||||||
line, err := bufio.NewReader(resp.Body).ReadString('\n')
|
line, err := bufio.NewReader(resp.Body).ReadString('\n')
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -321,11 +321,11 @@ func TestHealthzAndMetrics(t *testing.T) {
|
|||||||
resp.Body.Close()
|
resp.Body.Close()
|
||||||
text := string(body)
|
text := string(body)
|
||||||
for _, want := range []string{
|
for _, want := range []string{
|
||||||
`gpulock_state{state="idle"} 1`,
|
`gpu_turnstile_state{state="idle"} 1`,
|
||||||
"gpulock_llm_inflight 0",
|
"gpu_turnstile_llm_inflight 0",
|
||||||
"gpulock_image_jobs_total 0",
|
"gpu_turnstile_image_jobs_total 0",
|
||||||
`gpulock_lock_wait_seconds_bucket{kind="llm",le="+Inf"} 0`,
|
`gpu_turnstile_lock_wait_seconds_bucket{kind="llm",le="+Inf"} 0`,
|
||||||
"gpulock_unload_seconds_count{} 0",
|
"gpu_turnstile_unload_seconds_count{} 0",
|
||||||
} {
|
} {
|
||||||
if !strings.Contains(text, want) {
|
if !strings.Contains(text, want) {
|
||||||
t.Fatalf("metrics missing %q", want)
|
t.Fatalf("metrics missing %q", want)
|
||||||
|
|||||||
Reference in New Issue
Block a user