Add Linux systemd support and --install-service/--remove-service flags

The service package now has a Linux implementation alongside the Windows
one: systemd unit install/remove (/etc/systemd/system), readiness
notification (READY=1 via go-systemd) sent only after the listeners are
bound, a 30s watchdog, and STOPPING on shutdown. Listeners are pre-bound
so port conflicts fail fast and the readiness signal is truthful. The
notify calls are no-ops without NOTIFY_SOCKET (containers, shells) and
on non-Linux builds. --install-service/--remove-service work on both
platforms; the 'service install|remove' subcommand remains as an alias.

README.md

@@ -94,15 +94,15 @@ go build ./cmd/gpu-turnstile
 ./gpu-turnstile
 ```
 
-### Run natively on Windows (primary deployment)
+### Run natively on Windows (current primary deployment)
 
 Download `gpu-turnstile.exe` from a release, put a `gpu-turnstile.env`
 next to it, and run it — or install it as a Windows service from an
 elevated shell:
 
 ```sh
-gpu-turnstile.exe service install          # auto-start service, recovery = restart
-gpu-turnstile.exe service remove
+gpu-turnstile.exe --install-service        # auto-start service, recovery = restart
+gpu-turnstile.exe --remove-service
 ```
 
 The service uses the config file (services have no convenient
@@ -115,6 +115,23 @@ the install directory for self-updates. For least privilege, run it as the
 virtual account `NT SERVICE\gpu-turnstile` and grant write access to just
 those two directories.
 
+### Run natively on Linux (systemd)
+
+The same binary works on Linux. Install it as a systemd service as root:
+
+```sh
+gpu-turnstile --install-service            # writes + enables + starts the unit
+gpu-turnstile --remove-service
+```
+
+The unit (`/etc/systemd/system/gpu-turnstile.service`) is `Type=notify`:
+`systemctl start` blocks until the listeners are actually bound, a 30 s
+watchdog restarts the process if it wedges, and logs land in the journal
+(`journalctl -u gpu-turnstile -f`) unless `LOG_FILE` is set. Put the
+config in a `gpu-turnstile.env` next to the binary (or pass
+`-config /path` during install). The notify integration is a no-op in
+containers and interactive shells.
+
 **Auto-update is on by default**: the binary checks the repo's latest
 release on startup and every `UPDATE_INTERVAL`, verifies the Ed25519
 signature of the download against the public key embedded at build time,

SPEC.md

@@ -163,14 +163,21 @@ Startup fails fast on unparsable values and when neither consumer URL is
 set. Enabled upstreams are probed once at start (`/api/version`,
 `/system_stats`); failure is logged, not fatal.
 
-## Native Windows deployment
+## Native deployment (Windows and Linux)
 
-The binary runs natively on Windows (the primary deployment) as well as in
-Docker.
+The binary runs natively on Windows (the current primary deployment) and on
+Linux with systemd (the future GPU server), as well as in Docker.
 
-- `gpu-turnstile.exe service install [-config path]` registers an
-  auto-start Windows service (needs an elevated shell). Recovery actions
-  restart it 5 s after any failure. `service remove` uninstalls.
+Service management is the same on both platforms:
+`gpu-turnstile --install-service [-config path]` registers and starts an
+auto-start service; `--remove-service` stops and unregisters it (both need
+an elevated/root shell). The legacy form `gpu-turnstile service
+install|remove` does the same thing.
+
+### Windows
+
+- `--install-service` registers a Windows service; recovery actions restart
+  it 5 s after any failure.
 - **Layout**: install to `C:\Program Files\gpu-turnstile\` (exe plus
   `gpu-turnstile.env`); logs belong in `C:\ProgramData\gpu-turnstile\` via
   `LOG_FILE`. The service must be able to write its install directory for
@@ -182,6 +189,27 @@ Docker.
   log directories only (no network logon, no user profile).
 - Use a config file (above) for the service — Windows services have no
   convenient environment. Logs go to `LOG_FILE` since there is no console.
+
+### Linux (systemd)
+
+- `--install-service` writes `/etc/systemd/system/gpu-turnstile.service`
+  with `ExecStart` pointing at the current executable and the `-config`
+  file, then runs `systemctl daemon-reload` and `enable --now`. The unit
+  runs as root (it must be able to overwrite its own binary for
+  self-updates); harden with `ProtectSystem=strict` plus a writable
+  `ReadWritePaths` if desired.
+- The unit is `Type=notify`: the binary sends `READY=1` via
+  `github.com/coreos/go-systemd` only after the listeners are bound, so
+  `systemctl start` blocks until the proxy accepts connections. A 30 s
+  watchdog (`WatchdogSec=`) is pinged as long as the process runs; three
+  missed pings make systemd restart it. `STOPPING=1` is sent on shutdown.
+  All notify calls are no-ops when `NOTIFY_SOCKET` is unset (containers,
+  interactive shells), and the whole integration is Linux-only — Windows
+  builds carry no-op stubs.
+- Logs go to the journal (`journalctl -u gpu-turnstile`) or to `LOG_FILE`.
+- **Auto-update** works the same as on Windows: `Restart=on-failure` with
+  `RestartSec=5s` brings up the staged binary after the updater exits with
+  code 3.
 - **Auto-update**: on startup and every `UPDATE_INTERVAL`, the binary
   checks `UPDATE_REPO`'s latest release; if its tag is a newer `vX.Y.Z`,
   it downloads `UPDATE_ASSET` plus its `.sig` (and `.sha256` when present)
@@ -250,7 +278,7 @@ gpu-turnstile/
   internal/metrics/          # Prometheus exposition
   internal/config/           # env + .env file configuration
   internal/update/           # signed auto-updater (public key in pubkey.go)
-  internal/service/          # Windows service integration
+  internal/service/          # Windows SCM + Linux systemd (notify/watchdog) integration
   Dockerfile
   .gitea/workflows/ci.yml
   README.md
@@ -278,8 +306,9 @@ are new.
 
 ## Build and CI
 
-- Go 1.23+, `golang.org/x/sys` is the only external dependency (Windows
-  service integration; not used in the Linux build). `CGO_ENABLED=0`,
+- Go 1.23+, two external dependencies: `golang.org/x/sys` (Windows service
+  integration) and `github.com/coreos/go-systemd` (systemd notify/watchdog,
+  Linux build only). `CGO_ENABLED=0`,
   `-ldflags="-s -w"`, version from `git describe` injected via
   `-X main.version=`.
 - Dockerfile: multi-stage, final image `gcr.io/distroless/static` (or

cmd/gpu-turnstile/main.go

@@ -8,6 +8,7 @@ import (
 	"fmt"
 	"io"
 	"log/slog"
+	"net"
 	"net/http"
 	"os"
 	"os/signal"
@@ -34,12 +35,22 @@ var version = "dev"
 const exitCodeUpdate = 3
 
 func main() {
-	configPath, args := splitConfigFlag(os.Args[1:])
+	configPath, install, remove, args := parseFlags(os.Args[1:])
+	switch {
+	case install && remove:
+		fmt.Fprintf(os.Stderr, "gpu-turnstile: --install-service and --remove-service are mutually exclusive\n")
+		os.Exit(2)
+	case install:
+		os.Exit(serviceCommand(configPath, []string{"install"}))
+	case remove:
+		os.Exit(serviceCommand(configPath, []string{"remove"}))
+	}
 	if len(args) > 0 && args[0] == "service" {
 		os.Exit(serviceCommand(configPath, args[1:]))
 	}
 	if len(args) > 0 {
-		fmt.Fprintf(os.Stderr, "usage: gpu-turnstile [-config path] | gpu-turnstile service install|remove [-config path]\n")
+		fmt.Fprintf(os.Stderr, "usage: gpu-turnstile [-config path] [--install-service | --remove-service]\n")
+		fmt.Fprintf(os.Stderr, "       gpu-turnstile service install|remove [-config path]\n")
 		os.Exit(2)
 	}
 
@@ -70,10 +81,10 @@ func main() {
 	}
 }
 
-// splitConfigFlag extracts -config <path> (or -config=<path>) from args.
-func splitConfigFlag(args []string) (string, []string) {
-	var configPath string
-	rest := args[:0]
+// parseFlags extracts -config <path> (or -config=<path>) and the
+// --install-service / --remove-service switches from args.
+func parseFlags(args []string) (configPath string, install, remove bool, rest []string) {
+	rest = args[:0]
 	for i := 0; i < len(args); i++ {
 		switch {
 		case args[i] == "-config" && i+1 < len(args):
@@ -81,11 +92,15 @@ func splitConfigFlag(args []string) (string, []string) {
 			i++
 		case strings.HasPrefix(args[i], "-config="):
 			configPath = strings.TrimPrefix(args[i], "-config=")
+		case args[i] == "--install-service" || args[i] == "-install-service":
+			install = true
+		case args[i] == "--remove-service" || args[i] == "-remove-service":
+			remove = true
 		default:
 			rest = append(rest, args[i])
 		}
 	}
-	return configPath, rest
+	return configPath, install, remove, rest
 }
 
 // defaultConfigPath returns gpu-turnstile.env next to the executable.
@@ -282,21 +297,41 @@ func run(ctx context.Context, cfg config.Config, log *slog.Logger, logOut io.Wri
 	}
 	probeCancel()
 
+	// Bind the listeners up front so a port conflict fails fast and the
+	// readiness notification below really means "accepting connections".
 	var servers []*http.Server
+	var listeners []net.Listener
+	bind := func(addr string, handler http.Handler, consumer string) error {
+		ln, err := net.Listen("tcp", addr)
+		if err != nil {
+			return fmt.Errorf("listen %s on %s: %w", consumer, addr, err)
+		}
+		servers = append(servers, &http.Server{Addr: addr, Handler: handler})
+		listeners = append(listeners, ln)
+		log.Warn("listening", "consumer", consumer, "addr", addr)
+		return nil
+	}
 	if ollamaClient != nil {
-		servers = append(servers, &http.Server{Addr: cfg.ListenOllama, Handler: srv.OllamaHandler()})
-		log.Warn("listening", "consumer", "ollama", "addr", cfg.ListenOllama)
+		if err := bind(cfg.ListenOllama, srv.OllamaHandler(), "ollama"); err != nil {
+			return err
+		}
 	}
 	if comfyClient != nil {
-		servers = append(servers, &http.Server{Addr: cfg.ListenComfy, Handler: srv.ComfyHandler()})
-		log.Warn("listening", "consumer", "comfy", "addr", cfg.ListenComfy)
+		if err := bind(cfg.ListenComfy, srv.ComfyHandler(), "comfy"); err != nil {
+			return err
+		}
 	}
 
 	errCh := make(chan error, len(servers))
-	for _, s := range servers {
-		go func(s *http.Server) { errCh <- s.ListenAndServe() }(s)
+	for i := range servers {
+		go func(s *http.Server, ln net.Listener) { errCh <- s.Serve(ln) }(servers[i], listeners[i])
 	}
 
+	// Tell systemd we are up and start the watchdog pings; both are no-ops
+	// when not running under a notify/watchdog unit.
+	service.NotifyReady()
+	service.StartWatchdog(ctx)
+
 	if cfg.AutoUpdate {
 		go updateLoop(ctx, cfg, log, lk, isService)
 	}

go.mod

@@ -3,3 +3,5 @@ module gpu-turnstile
 go 1.23
 
 require golang.org/x/sys v0.29.0
+
+require github.com/coreos/go-systemd/v22 v22.7.0

go.sum

@@ -1,2 +1,4 @@
+github.com/coreos/go-systemd/v22 v22.7.0 h1:LAEzFkke61DFROc7zNLX/WA2i5J8gYqe0rSj9KI28KA=
+github.com/coreos/go-systemd/v22 v22.7.0/go.mod h1:xNUYtjHu2EDXbsxz1i41wouACIwT7Ybq9o0BQhMwD0w=
 golang.org/x/sys v0.29.0 h1:TPYlXGxvx1MGTn2GiZDhnjPA9wZzZeGKHHmKhHYvgaU=
 golang.org/x/sys v0.29.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=

internal/service/notify_linux.go

@@ -0,0 +1,46 @@
+//go:build linux
+
+package service
+
+import (
+	"context"
+	"os"
+	"strconv"
+	"time"
+
+	"github.com/coreos/go-systemd/v22/daemon"
+)
+
+// NotifyReady tells systemd the service is up (Type=notify). It is a no-op
+// when NOTIFY_SOCKET is unset, e.g. in a container or interactive shell.
+func NotifyReady() {
+	daemon.SdNotify(false, daemon.SdNotifyReady)
+}
+
+// NotifyStopping tells systemd the service is shutting down.
+func NotifyStopping() {
+	daemon.SdNotify(false, daemon.SdNotifyStopping)
+}
+
+// StartWatchdog pings the systemd watchdog every half of WATCHDOG_USEC
+// until ctx is cancelled. It is a no-op unless systemd started the process
+// with a watchdog configured (WatchdogSec= in the unit).
+func StartWatchdog(ctx context.Context) {
+	usec, err := strconv.Atoi(os.Getenv("WATCHDOG_USEC"))
+	if err != nil || usec <= 0 {
+		return
+	}
+	interval := time.Duration(usec) * time.Microsecond / 2
+	go func() {
+		ticker := time.NewTicker(interval)
+		defer ticker.Stop()
+		for {
+			select {
+			case <-ctx.Done():
+				return
+			case <-ticker.C:
+				daemon.SdNotify(false, daemon.SdNotifyWatchdog)
+			}
+		}
+	}()
+}

internal/service/notify_other.go

@@ -0,0 +1,14 @@
+//go:build !linux
+
+package service
+
+import "context"
+
+// NotifyReady is a no-op outside Linux (no systemd notify socket).
+func NotifyReady() {}
+
+// NotifyStopping is a no-op outside Linux.
+func NotifyStopping() {}
+
+// StartWatchdog is a no-op outside Linux.
+func StartWatchdog(context.Context) {}

internal/service/service_linux.go

@@ -0,0 +1,90 @@
+//go:build linux
+
+// Package service integrates gpu-turnstile with systemd on Linux: running
+// under a unit with readiness notification and watchdog, plus
+// install/remove helpers that manage a system unit.
+package service
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"os/exec"
+	"os/signal"
+	"path/filepath"
+	"syscall"
+)
+
+// Name matches the Windows service name; the systemd unit is Name + ".service".
+const Name = "gpu-turnstile"
+
+// unitPath is where Install writes the unit file.
+const unitPath = "/etc/systemd/system/" + Name + ".service"
+
+// IsService reports whether the process was started by systemd.
+func IsService() bool { return os.Getenv("INVOCATION_ID") != "" }
+
+// Run executes run with SIGINT/SIGTERM cancellation (which is how systemctl
+// stop signals the process) and tells systemd when the shutdown begins.
+func Run(run func(ctx context.Context) error) error {
+	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
+	defer stop()
+	defer NotifyStopping()
+	return run(ctx)
+}
+
+// renderUnit builds the systemd unit: Type=notify so systemctl start blocks
+// until the listeners are bound, a 30s watchdog, and restart-on-failure
+// with a 5s delay — which is also what brings up a staged update after the
+// updater exits with a non-zero code.
+func renderUnit(exePath, configPath string) string {
+	return fmt.Sprintf(`[Unit]
+Description=gpu-turnstile GPU arbitration proxy for Ollama and ComfyUI
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=notify
+WatchdogSec=30s
+ExecStart=%q -config %q
+Restart=on-failure
+RestartSec=5s
+
+[Install]
+WantedBy=multi-user.target
+`, exePath, configPath)
+}
+
+// Install writes the unit for the current executable and the given config
+// file, then enables and starts it. Needs root.
+func Install(configPath string) error {
+	exe, err := os.Executable()
+	if err != nil {
+		return err
+	}
+	if abs, absErr := filepath.Abs(exe); absErr == nil {
+		exe = abs
+	}
+	if err := os.WriteFile(unitPath, []byte(renderUnit(exe, configPath)), 0o644); err != nil {
+		return fmt.Errorf("write %s (run as root): %w", unitPath, err)
+	}
+	if out, err := exec.Command("systemctl", "daemon-reload").CombinedOutput(); err != nil {
+		return fmt.Errorf("systemctl daemon-reload: %w (%s)", err, out)
+	}
+	if out, err := exec.Command("systemctl", "enable", "--now", Name+".service").CombinedOutput(); err != nil {
+		return fmt.Errorf("systemctl enable --now: %w (%s)", err, out)
+	}
+	return nil
+}
+
+// Remove stops and disables the service and deletes the unit file.
+func Remove() error {
+	exec.Command("systemctl", "disable", "--now", Name+".service").Run() // ignore: may not exist
+	if err := os.Remove(unitPath); err != nil && !os.IsNotExist(err) {
+		return fmt.Errorf("remove %s: %w", unitPath, err)
+	}
+	if out, err := exec.Command("systemctl", "daemon-reload").CombinedOutput(); err != nil {
+		return fmt.Errorf("systemctl daemon-reload: %w (%s)", err, out)
+	}
+	return nil
+}

internal/service/service_linux_test.go

@@ -0,0 +1,23 @@
+//go:build linux
+
+package service
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestRenderUnit(t *testing.T) {
+	unit := renderUnit("/usr/local/bin/gpu-turnstile", "/etc/gpu-turnstile.env")
+	for _, want := range []string{
+		"Type=notify",
+		"WatchdogSec=30s",
+		`ExecStart="/usr/local/bin/gpu-turnstile" -config "/etc/gpu-turnstile.env"`,
+		"Restart=on-failure",
+		"WantedBy=multi-user.target",
+	} {
+		if !strings.Contains(unit, want) {
+			t.Fatalf("unit missing %q:\n%s", want, unit)
+		}
+	}
+}

internal/service/service_other.go

@@ -1,8 +1,8 @@
-//go:build !windows
+//go:build !windows && !linux
 
-// Package service provides the non-Windows stubs for the Windows service
-// integration. Run falls back to plain signal handling; install/remove
-// are unsupported.
+// Package service provides the stubs for platforms without service
+// integration (Windows uses the SCM, Linux uses systemd). Run falls back
+// to plain signal handling; install/remove are unsupported.
 package service
 
 import (
@@ -15,7 +15,7 @@ import (
 // Name matches the Windows service name.
 const Name = "gpu-turnstile"
 
-var errUnsupported = errors.New("service management is only supported on Windows")
+var errUnsupported = errors.New("service management is only supported on Windows and Linux (systemd)")
 
 // IsService is always false on non-Windows platforms.
 func IsService() bool { return false }