Extend retry backoff; rework logging (LOGLEVEL, request lines, colors)

Backoff now covers all dial-phase errors (refused, timeout, DNS), TLS
handshake failures, and 5xx responses with replayable bodies; streamed
POSTs are never replayed to avoid duplicate work. First retry of an
episode logs at WARN, subsequent attempts at INFO.

LOGLEVEL (LOG_LEVEL kept as alias) now defaults to warn: startup logs
version plus every setting; INFO adds one line per incoming request and
per response with status/duration, ANSI-colored in text mode (bypasses
slog's escaping so colors render in docker compose logs); NO_COLOR or
LOG_FORMAT=json disables colors.
This commit is contained in:
mram
2026-09-20 20:00:57 +02:00
parent 20d5439b5f
commit 42e1386811
5 changed files with 312 additions and 42 deletions
+16 -7
View File
@@ -112,7 +112,7 @@ load time. Off by default.
| `JOB_TIMEOUT` | `15m` | wait for ComfyUI job |
| `LLM_WAIT_TIMEOUT` | `10m` | max time an LLM request waits for the lock before 503 |
| `WARM_MODEL` | `` | optional model to reload after an image job |
| `LOG_LEVEL` | `info` | `debug` logs every lock transition |
| `LOGLEVEL` | `warn` | `info` logs every request (colored arrows in text mode), `debug` adds lock transitions. `LOG_LEVEL` is accepted as an alias |
| `UNLOAD_POLL_INTERVAL` | `500ms` | `/api/ps` poll interval while unloading |
| `HISTORY_POLL_INTERVAL` | `1s` | `/history/<id>` poll interval while a job runs |
| `PROBE_TIMEOUT` | `5s` | startup probe of both upstreams |
@@ -136,7 +136,13 @@ start (`/api/version`, `/system_stats`); failure is logged, not fatal.
`gpu_turnstile_image_jobs_total`, `gpu_turnstile_lock_wait_seconds`
(histogram, label `kind="llm|image"`), `gpu_turnstile_unload_seconds`.
- Structured logs (`log/slog`, JSON when `LOG_FORMAT=json`), one line per
state transition and per image job phase with `prompt_id`.
state transition and per image job phase with `prompt_id`. Startup logs
the version and every setting (visible even at the default `warn`
level). With `LOGLEVEL=info` or `debug`, every request logs a `-->`
incoming line and a `<--` response line with status and duration —
ANSI-colored (cyan incoming; green/yellow/red by status class) in text
mode, which renders in `docker compose logs` on Windows Terminal. Set
`NO_COLOR` to disable colors.
## Edge cases to handle
@@ -148,11 +154,14 @@ start (`/api/version`, `/system_stats`); failure is logged, not fatal.
`JOB_TIMEOUT` releases the lock; log at warn.
- Ollama unreachable during unload: continue with the image job; the whole
point is not to block users on a misbehaving neighbour.
- Upstream connection refused while proxying (service down or restarting):
retry with exponential backoff — `BACKOFF_INITIAL`, doubling per attempt,
capped at `BACKOFF_MAX` — until the upstream answers or the client
disconnects. Retrying a refused connection is safe: no request bytes were
sent. Other upstream errors are not retried.
- Upstream unreachable while proxying (connection refused, dial timeout,
DNS failure, TLS handshake error): retry with exponential backoff —
`BACKOFF_INITIAL`, doubling per attempt, capped at `BACKOFF_MAX` — until
the upstream answers or the client disconnects. These are safe to retry:
the request never reached the upstream application. 5xx responses are
retried the same way, but only when the request body can be replayed
(GETs, or bodies with `GetBody`); streamed POSTs are never replayed to
avoid duplicate work such as a double-enqueued ComfyUI prompt.
- `POST /prompt` with a body that ComfyUI rejects (400): lock released
immediately, body passed back.
- Websocket `/ws` connections are long-lived and never take the lock.