Game detection: foreign GPU holders take an external lock hold (GAME_PROCS, GPU_FOREIGN_VRAM_MB)

SPEC.md

@@ -10,11 +10,12 @@ becomes very slow; on Linux it would OOM instead.
 ## Goal
 
 A single Go binary that sits in front of **both** services and guarantees that
-at any moment the GPU is in exactly one of three states:
+at any moment the GPU is in exactly one of four states:
 
 - `idle` — nothing in flight
 - `llm` — N ≥ 1 Ollama inference requests in flight (concurrency allowed)
 - `image` — exactly one ComfyUI job in flight, Ollama models unloaded
+- `external` — a foreign process (e.g. a game) holds the GPU; new work waits
 
 Clients (LiteLLM, Open WebUI, n8n) point at gpu-turnstile instead of at the
 services. gpu-turnstile is transparent for everything that does not touch the
@@ -53,9 +54,10 @@ consumer gets no listener, no startup probe, and no lock participation:
   blocks since no image jobs can arrive.
 - **Only `COMFY_URL`**: image jobs are tracked and ComfyUI's VRAM is freed
   afterwards, but the Ollama unload and warm-reload steps are skipped.
-- Future consumers (e.g. detecting a local game holding VRAM) plug into the
-  same lock the same way: enabled by their config knob, excluded when
-  absent.
+- **Game detection** is a third, optional consumer without a URL: enabled by
+  `GAME_PROCS` and/or `GPU_FOREIGN_VRAM_MB` it watches for foreign processes
+  holding the GPU (see below) and plugs into the same lock the same way —
+  excluded when both knobs are unset.
 
 ### Lock semantics
 
@@ -75,6 +77,11 @@ are LLM requests and the single "writer" is an image job):
   requests start), waits until n == 0, sets state := `image`. Released after
   the ComfyUI job finished and models were freed.
 - Concurrent image jobs queue FIFO behind each other.
+- **External hold**: game detection calls `SetExternal(holder)` while a
+  foreign process holds the GPU. New LLM and image grants block (same busy
+  handling as above) until `ClearExternal()`; in-flight work is not
+  preempted, it drains. Snapshot reports state `external` once nothing else
+  is in flight.
 - All waits are context-aware: a client that disconnects while waiting is
   removed from the queue.
 
@@ -148,6 +155,41 @@ process instead of expecting an always-on server:
   skips the intentionally-stopped/starting states; a failed probe while
   the process is alive and was previously ready is logged as DOWN.
 
+## Game detection (foreign GPU holders)
+
+Games and other foreign GPU users sit outside the URL-based consumer model —
+nothing proxies through gpu-turnstile for them. Two independent detection
+paths, polled every `GAME_POLL_INTERVAL` (default 5 s); either one being
+configured enables the feature:
+
+- **Process watch list** (`GAME_PROCS`, comma-separated, case-insensitive,
+  `.exe` optional): while any listed process runs, the GPU counts as held.
+  This is the reliable path on Windows.
+- **Foreign VRAM threshold** (`GPU_FOREIGN_VRAM_MB`): `nvidia-smi
+  --query-compute-apps` lists per-process GPU memory; any process not in
+  `GPU_IGNORE_PROCS` (default: Ollama and python — ComfyUI runs under python)
+  holding more than the threshold counts as a foreign holder. Needs
+  nvidia-smi on the PATH (absent: logged once, path disabled) and works best
+  on Linux — under Windows' WDDM driver, graphics-only games may not appear
+  in the per-process list.
+
+While a holder is detected, gpu-turnstile:
+
+1. takes the external hold on the lock (`SetExternal`), so new LLM/image
+   requests wait or are rejected per `LLM_BUSY_MODE`; busy responses name the
+   holder,
+2. once in-flight work has drained, **frees VRAM for the foreign process**:
+   the managed ComfyUI is stopped (an external server on its port is never
+   touched) and Ollama's resident models are unloaded,
+3. refuses to spawn the managed ComfyUI; ComfyUI requests that would need a
+   spawn are answered 503 + `Retry-After` (a running desktop instance keeps
+   being proxied),
+4. logs the transitions at WARN ("GPU held by an external process …" /
+   "released the GPU; resuming") and reports state `external` in `/healthz`
+   and `/metrics`.
+
+When the holder disappears, the hold is lifted and queued requests proceed.
+
 ## Configuration (env)
 
 Configuration comes from environment variables and/or an `.env`-style
@@ -173,6 +215,10 @@ override file values. A missing file is fine; a malformed one is fatal.
 | `COMFY_DIR` | `` | working directory for `COMFY_CMD` |
 | `COMFY_IDLE_TIMEOUT` | `5m` | stop the managed ComfyUI after this long without requests or jobs |
 | `COMFY_START_TIMEOUT` | `2m` | how long a request waits for the managed ComfyUI to come up |
+| `GAME_PROCS` | _(empty = disabled)_ | comma-separated process names (case-insensitive, `.exe` optional); while any runs, the GPU counts as held by it: requests wait, Ollama unloads, the managed ComfyUI stops |
+| `GPU_FOREIGN_VRAM_MB` | `0` (disabled) | also treat the GPU as held when a process not in `GPU_IGNORE_PROCS` uses more VRAM than this; needs nvidia-smi |
+| `GPU_IGNORE_PROCS` | `ollama,ollama app,ollama_llama_server,python,pythonw` | process names never counted as foreign GPU users |
+| `GAME_POLL_INTERVAL` | `5s` | how often game/VRAM detection runs |
 | `LOGLEVEL` | `warn` | `info` logs every request (colored arrows in text mode), `debug` adds lock transitions. `LOG_LEVEL` is accepted as an alias |
 | `LOG_FORMAT` | `text` | `json` for structured JSON logs |
 | `LOG_FILE` | `` | append logs to this file instead of stderr (useful as a service) |