Extend retry backoff; rework logging (LOGLEVEL, request lines, colors)
Backoff now covers all dial-phase errors (refused, timeout, DNS), TLS handshake failures, and 5xx responses with replayable bodies; streamed POSTs are never replayed to avoid duplicate work. First retry of an episode logs at WARN, subsequent attempts at INFO. LOGLEVEL (LOG_LEVEL kept as alias) now defaults to warn: startup logs version plus every setting; INFO adds one line per incoming request and per response with status/duration, ANSI-colored in text mode (bypasses slog's escaping so colors render in docker compose logs); NO_COLOR or LOG_FORMAT=json disables colors.
This commit is contained in:
@@ -112,7 +112,7 @@ load time. Off by default.
|
||||
| `JOB_TIMEOUT` | `15m` | wait for ComfyUI job |
|
||||
| `LLM_WAIT_TIMEOUT` | `10m` | max time an LLM request waits for the lock before 503 |
|
||||
| `WARM_MODEL` | `` | optional model to reload after an image job |
|
||||
| `LOG_LEVEL` | `info` | `debug` logs every lock transition |
|
||||
| `LOGLEVEL` | `warn` | `info` logs every request (colored arrows in text mode), `debug` adds lock transitions. `LOG_LEVEL` is accepted as an alias |
|
||||
| `UNLOAD_POLL_INTERVAL` | `500ms` | `/api/ps` poll interval while unloading |
|
||||
| `HISTORY_POLL_INTERVAL` | `1s` | `/history/<id>` poll interval while a job runs |
|
||||
| `PROBE_TIMEOUT` | `5s` | startup probe of both upstreams |
|
||||
@@ -136,7 +136,13 @@ start (`/api/version`, `/system_stats`); failure is logged, not fatal.
|
||||
`gpu_turnstile_image_jobs_total`, `gpu_turnstile_lock_wait_seconds`
|
||||
(histogram, label `kind="llm|image"`), `gpu_turnstile_unload_seconds`.
|
||||
- Structured logs (`log/slog`, JSON when `LOG_FORMAT=json`), one line per
|
||||
state transition and per image job phase with `prompt_id`.
|
||||
state transition and per image job phase with `prompt_id`. Startup logs
|
||||
the version and every setting (visible even at the default `warn`
|
||||
level). With `LOGLEVEL=info` or `debug`, every request logs a `-->`
|
||||
incoming line and a `<--` response line with status and duration —
|
||||
ANSI-colored (cyan incoming; green/yellow/red by status class) in text
|
||||
mode, which renders in `docker compose logs` on Windows Terminal. Set
|
||||
`NO_COLOR` to disable colors.
|
||||
|
||||
## Edge cases to handle
|
||||
|
||||
@@ -148,11 +154,14 @@ start (`/api/version`, `/system_stats`); failure is logged, not fatal.
|
||||
`JOB_TIMEOUT` releases the lock; log at warn.
|
||||
- Ollama unreachable during unload: continue with the image job; the whole
|
||||
point is not to block users on a misbehaving neighbour.
|
||||
- Upstream connection refused while proxying (service down or restarting):
|
||||
retry with exponential backoff — `BACKOFF_INITIAL`, doubling per attempt,
|
||||
capped at `BACKOFF_MAX` — until the upstream answers or the client
|
||||
disconnects. Retrying a refused connection is safe: no request bytes were
|
||||
sent. Other upstream errors are not retried.
|
||||
- Upstream unreachable while proxying (connection refused, dial timeout,
|
||||
DNS failure, TLS handshake error): retry with exponential backoff —
|
||||
`BACKOFF_INITIAL`, doubling per attempt, capped at `BACKOFF_MAX` — until
|
||||
the upstream answers or the client disconnects. These are safe to retry:
|
||||
the request never reached the upstream application. 5xx responses are
|
||||
retried the same way, but only when the request body can be replayed
|
||||
(GETs, or bodies with `GetBody`); streamed POSTs are never replayed to
|
||||
avoid duplicate work such as a double-enqueued ComfyUI prompt.
|
||||
- `POST /prompt` with a body that ComfyUI rejects (400): lock released
|
||||
immediately, body passed back.
|
||||
- Websocket `/ws` connections are long-lived and never take the lock.
|
||||
|
||||
Reference in New Issue
Block a user