echolot/server/README.md

# echolot-server

The probe server ([spec](../docs/probe-protocol.md)). Pure Go, stdlib only, GPL-3.0-or-later.

**Skeleton status:** control plane (enroll / profile / sessions with the spec's HKDF key
schedule), UDP data plane (ECHO with observation block, TIMESYNC, HMAC gate, anti-replay,
anti-amplification — wire format covered by tests). Not yet: TCP/TLS echo, STUN, canary DNS,
actions, observations API, admin UI beyond token minting.

## Deployment requirements

**No reverse proxy, no 80/443 — by design.** Traefik/nginx on the same host are fine; this
server never touches their ports, and putting it *behind* them would break two protocol
properties:

- Clients trust the control plane **only** via the SPKI pin from enrollment (self-signed is
  first-class). A proxy terminates TLS with its own rotating ACME cert → pins break. The pin
  model exists so no real certificate is ever needed.
- On the data plane, **the observed source address/port/TTL/DSCP *is* the measurement**. Any
  proxy or NAT layer (including Docker's) substitutes its own — hence host networking.

What a target host actually needs:

| Port | Proto | Purpose | Notes |
|---|---|---|---|
| 8443 | tcp | control plane (pinned HTTPS) | any port — it travels in the enrollment QR + profile |
| 8442 | udp | UDP probe data plane | any port, profile-driven |
| 8441 | tcp | TCP/TLS echo | any port (not yet implemented) |
| 3478 | udp | STUN | keep standard: vanilla RFC 5389 for tool interop; rarely contended |
| 8444 | tcp | admin | loopback-only by design — reach via SSH tunnel |

All configurable via `ECHOLOT_*_LISTEN`. Plus:

1. **A public IP on the host** (v4, ideally also v6 — v6 topology issues are half of what
   clients want to measure). Behind NAT, plain port-forwards work.
2. **Second IP (optional):** full RFC 5780 NAT-behavior discovery (`stun-5780`) needs an
   alternate reply address; without it the profile advertises `stun-basic` and clients degrade
   gracefully.
3. **Delegated DNS subzone (for `canary-dns`):** set `ECHOLOT_DNS_LISTEN` (udp+tcp/53 on the
   service IPs) and `ECHOLOT_CANARY_ZONE` (e.g. `c.echo-lot.app`), then delegate the zone to
   this host in your DNS provider:
   ```
   c.echo-lot.app.  NS  fmr-1.echo-lot.app.
   c.echo-lot.app.  NS  fmr-2.echo-lot.app.
   ```
   The server is authoritative for that zone only, serving the spec §6.1 reference records
   (frozen in `internal/canarydns/dns_reference.go`) plus per-query `<nonce>.<session>.<zone>`
   lookups it logs. Binding :53 on the public IPs is fine even with systemd-resolved (it only
   claims 127.0.0.53). Absent config → capability simply not advertised.
4. **Outbound freedom** for connect-back / delayed-echo actions — no extra inbound ports;
   generated traffic goes only to the session's observed source.

Deliberately out of scope here: an echo listener on 443 (to detect port-based egress filtering)
— that genuinely needs 443 and belongs on a dedicated IP, not on a host running a reverse proxy.

## Host tuning (measurement fidelity)

A measurement server must not let the kernel distort what clients observe. Apply the
recommended sysctls and the daemon will confirm the host is clean:

```sh
sudo cp deploy/99-echolot-sysctl.conf /etc/sysctl.d/ && sudo sysctl --system
```

The daemon **self-tests at startup and via `GET /admin/selftest`** (localhost):
- **sysctl audit** — flags settings that would distort results (RA acceptance on a static host,
  ICMP redirects, ICMP rate-limiting of the server's own errors, disabled TCP options).
- **egress-MTU self-proof** — DF-probes external anchors (`ECHOLOT_MTU_PROBE_TARGETS`,
  default 1.1.1.1 + a v6 anchor) and reads the discovered path MTU. If the server's *own* uplink
  can't carry 1500, client MTU results would measure this server, not the client — so the profile
  exposes `server_selftest.mtu_ok` and the log warns loudly.

Both signals ride in `GET /v1/profile` as `server_selftest` so a client can trust — or skip —
MTU testing accordingly.

## Run in Docker (config via env)

```sh
docker compose up -d          # see compose.yaml — network_mode: host is required
```

Host networking is not negotiable: behind Docker NAT the server would observe the proxy's
source addresses and TTLs instead of the client's — falsifying exactly what it measures.
Container mode is autodetected (`/.dockerenv` etc.); `--docker` / `ECHOLOT_DOCKER=1` forces it.
In this mode systemd install and self-update are refused — update by pulling a new image tag.

## Run native (systemd)

```sh
go build -o /usr/local/bin/echolot-server ./cmd/echolot-server
sudo /usr/local/bin/echolot-server --install-systemd     # writes unit, enables, starts
sudo /usr/local/bin/echolot-server --uninstall-systemd
```

Config precedence: flags > `ECHOLOT_*` env > defaults. Every flag has an env twin
(`--udp-listen` ↔ `ECHOLOT_UDP_LISTEN`). Host config lives in `/etc/echolot-server.env`
(seeded by `--install-systemd`, never overwritten).

**Multi-IP hosts:** listen specs are comma-separated, and you should bind explicit addresses —
a wildcard bind would also claim management-only IPs:

```sh
ECHOLOT_CONTROL_LISTEN=203.0.113.10:8443,[2001:db8::10]:8443
ECHOLOT_UDP_LISTEN=203.0.113.10:8442,203.0.113.11:8442,[2001:db8::10]:8442,[2001:db8::11]:8442
```

Passing `--self-update-api` to `--install-systemd` additionally installs a daily randomized
self-update timer (`echolot-server-update.timer`) that restarts the service after a successful
update.

### Self-update (opt-in, native only)

```sh
echolot-server --self-update \
  --self-update-api https://git.example.net/api/v1/repos/owner/repo
```

Fetches the newest `server-v*` release asset for this OS/arch and atomically replaces the
binary; systemd's `Restart=` brings up the new version. Run it from a systemd timer for
unattended updates.

Releases are trusted by signature, not by host: CI signs `SHA256SUMS` with an ed25519 key that
exists only in its secret store (`RELEASE_SIGNING_KEY`), and the updater verifies
`SHA256SUMS.sig` against the public key baked into the binary before believing any checksum —
an unsigned or re-signed release is refused, so a compromised Gitea can withhold updates but not
inject one. Running your own release pipeline? Mint a keypair with
`go run ./cmd/release-sign -gen`, set the secret, and point `ECHOLOT_SELF_UPDATE_PUBKEY` (or
`--self-update-pubkey`) at your public key.

## First contact

```sh
# 1. mint an enrollment token (admin listener is loopback-only; authenticates as the
#    break-glass admin — set that once with --set-admin-password)
curl -s -u admin:<password> -H 'Accept: application/json' \
  -X POST 'http://127.0.0.1:8444/admin/enroll-tokens?note=phone'
# → { "token": "…", "expires_in_s": 86400, "enroll_uri": "echolot://enroll?…" }
# 2. device enrolls with it (normally via the echolot:// QR code)
curl -sk -X POST https://<host>:8443/v1/enroll -H 'Authorization: Bearer <token>'
# 3. device fetches its profile
curl -sk https://<host>:8443/v1/profile -H 'Authorization: Bearer <credential>'
```

The SPKI pin clients must verify is logged at startup (`pin-sha256`).

## Dev relay (adb endpoint)

Development scaffolding, not protocol: it appears in no capability list and in no measurement
document, and `docs/probe-protocol.md` does not describe it.

It exists because **mDNS does not cross subnets**. Android's wireless debugging advertises adbd's
port over mDNS and rotates that port every few minutes, so a developer on another subnet cannot
discover it at all. An Echolot instance running on the test LAN can — and relays it here.

```
POST /v1/devtools/adb-endpoint     control plane, device credential (same Bearer as /v1/sessions)
  {"host":"10.13.102.128","port":45305,"device_name":"TB330FU","note":"…"}
GET  /admin/adb-endpoints          admin UI, session cookie or HTTP Basic
```

Read it back from the developer's machine (the admin listener is loopback-only, so over the same
SSH tunnel as everything else):

```sh
curl -s -u admin:<password> -H 'Accept: application/json' \
  http://127.0.0.1:8444/admin/adb-endpoints
# → [{"device":"…","host":"10.13.102.128","port":45305,"device_name":"TB330FU",
#     "reported_at":"…","source_ip":"…","age_s":37}]
```

The same list is a card on the admin dashboard, so it is usable without curl.

The newest report per submitting device wins — a rotated port makes the previous one wrong, not
historical. The device id comes from the credential and `source_ip` from the connection, so neither
is something a body can claim. `ECHOLOT_ADB_ENDPOINT_RETENTION_H` (default 24) bounds how long an
entry lives; `0` keeps it until the device replaces it. The window exists because the value is a
LAN address that stops being true within minutes: keeping it afterwards discloses the inside of
someone's network in exchange for nothing.

**Why it lives on these two listeners and nowhere else.** Its predecessor was a separate Python
service wildcard-bound to `0.0.0.0:443`. That silently occupied port 443 on the addresses reserved
for measurement — voiding the IPv4 interception proof for as long as it ran — and it accepted a port
report from anyone who could reach it. So: no new listener, no new port, no wildcard bind, and
nothing unauthenticated. The submit side is rate-limited by the existing §2.5 actions bucket
(`ECHOLOT_RATE_ACTIONS_PER_MIN`).

## Development

```sh
go test ./...   # includes wire-format tests for the UDP data plane
go vet ./...
```

CI (`.gitea/workflows/build-server.yml`): tests on every push touching `server/`;
tagging `server-v1.2.3` builds + pushes the container image to the Gitea registry and
attaches static linux amd64/arm64 binaries (+ signed SHA256SUMS) to a release — the same
artifacts `--self-update` consumes.

## TLS for the admin UI

The binary terminates TLS itself; there is no reverse proxy in the design. It already serves TLS
for the control plane, so this is reuse rather than new machinery, and it keeps the "one process,
one config file" property. A proxy would also invite someone to eventually front the control plane
too — which would break SPKI pinning, because clients pin *that* certificate's key.

```
ECHOLOT_ADMIN_LISTEN=[2001:db8::2]:443
ECHOLOT_ADMIN_TLS_CERT=/etc/echolot/admin.pem
ECHOLOT_ADMIN_TLS_KEY=/etc/echolot/admin.key
ECHOLOT_ADMIN_BASE_URL=https://admin.example.net
```

Certificates come from any ACME client. **DNS-01 is the one to use here**: it needs no inbound
port 80, which matters on a host where 80 is awkward or already spoken for.

```sh
acme.sh --issue --dns dns_cf -d admin.example.net \
  --key-file       /etc/echolot/admin.key \
  --fullchain-file /etc/echolot/admin.pem
```

**No reload hook is needed.** The certificate is re-read when the files change, so a renewal that
drops new files in place is picked up on the next handshake. That is deliberate: a reload hook is
the part of a renewal setup that quietly stops working, months later, and is noticed only once the
certificate has already expired. A torn write — renewal tools write cert and key separately — keeps
the previous certificate rather than failing the listener.

Serving the admin UI in plaintext on a non-loopback address is refused: the session cookie is a
bearer credential for everything the server can do, and the OIDC authorization code arrives in a
URL. Bind to loopback and use an SSH tunnel (`ssh -L 8444:localhost:8444 host`), supply a
certificate, or set `ECHOLOT_ADMIN_INSECURE=1` if you mean it.

The control-plane certificate is deliberately *not* hot-reloaded. Clients pin its public key, so
replacing it is a rotation an operator should have to think about, not something that happens
because a file changed.