Add LLM busy modes: wait (hang) or reject with Retry-After
LLM_BUSY_MODE=reject answers blocked LLM requests immediately with LLM_BUSY_STATUS (default 503, 429 works) and Retry-After, so routers like LiteLLM can cool down and retry instead of holding a hung connection. The default wait mode now also sends Retry-After when LLM_WAIT_TIMEOUT expires. Document the service account (LocalSystem default, NT SERVICE virtual-account hardening) and the Program Files / ProgramData install layout.
This commit is contained in:
@@ -83,6 +83,9 @@ func TestLoadErrors(t *testing.T) {
|
||||
{"AUTO_UPDATE", "maybe"},
|
||||
{"LOGLEVEL", "shouty"},
|
||||
{"LOG_FORMAT", "yaml"},
|
||||
{"LLM_BUSY_MODE", "bogus"},
|
||||
{"LLM_BUSY_STATUS", "200"},
|
||||
{"BUSY_RETRY_AFTER", "0"},
|
||||
} {
|
||||
_, err := Load(func(k string) string {
|
||||
if k == tc.key {
|
||||
|
||||
Reference in New Issue
Block a user