echolot/docs/probe-protocol.md

# Echolot Probe Protocol — Spec v1 (draft 1)

Project name: **Echolot** (decided 2026-07-29; domain: echo-lot.app). Status: draft for review, 2026-07-29. Companion to `measurement-schema.md` (evidence fields reference observations defined here).

Two planes: a **control plane** (HTTPS + JSON, key-pinned) and a **data plane** (binary UDP probe protocol, TCP/TLS/HTTP echo endpoints, standard STUN, standard DNS on the canary zone).

## 1. Transport & trust

- Control plane: HTTPS. The client validates the server certificate **only** against the SPKI pin from enrollment (`pin-sha256`, RFC 7469 computation). CA validation is not required; self-signed is first-class. Key rotation: profile may list `next_pins[]` ahead of rotation.
- All control endpoints are under `/v1/`. Version negotiation is by capability list (§2.3), never by sniffing version numbers.
- Data-plane authenticity: per-session HMAC key (§3), truncated HMAC on every UDP probe packet.

Default ports (all configurable): control HTTPS **8443**; UDP probe **8442** (same number on v4 and v6); STUN **3478** (+ alternate address per RFC 5780 — requires a second IP); TCP echo **8441**; canary DNS: standard 53 on the delegated zone.

## 2. Control plane

### 2.1 Enrollment

Bootstrap payload (QR / paste string):

```
echolot://enroll?v=1&u=<control-URL, urlencoded>&p=pin-sha256:<b64 SPKI hash>&t=<enrollment token>
```

```
POST /v1/enroll        Authorization: Bearer <enrollment-token>
→ 200 { "device_credential": "<random 256-bit, b64url>",
        "device_id": "uuid",
        "profile": { ... §2.2 ... } }
```

Enrollment tokens are single-use with expiry, created in the admin UI, scoped `enroll`. The device credential is a long-lived bearer secret, scoped `run-tests`; it is also the HKDF input for session keys. Revocation = deleting the device in the admin UI.

### 2.2 Profile

```
GET /v1/profile        Authorization: Bearer <device-credential>
```

```json
{
  "profile_version": 7,
  "name": "homelab",
  "server_version": "0.4.0",
  "capabilities": ["udp-probe", "stun-5780", "canary-dns", "connect-back",
                    "delayed-echo", "big-send", "tls-echo", "throughput", "ntp"],
  "targets": [
    { "id": "vie-1", "location": "Vienna",
      "ip4": "203.0.113.10", "ip6": "2001:db8::10",
      "udp_port": 8442, "tcp_port": 8441, "stun_port": 3478,
      "stun_alt": { "ip4": "203.0.113.11", "port": 3478 } }
  ],
  "canary_zone": "c.probe.example.net",
  "recursive_resolver": { "ip4": "203.0.113.10", "dot": true, "doh_url": "https://..." },
  "pins": ["pin-sha256:..."], "next_pins": [],
  "limits": { "max_kbps": 50000, "max_session_s": 900 }
}
```

The app re-fetches the profile at the start of every run (falling back to the cached copy offline) and records `profile_version` in the measurement.

### 2.3 Capabilities (v1 registry)

`udp-probe`, `stun-basic`, `stun-5780`, `canary-dns`, `recursive-dns`, `connect-back`, `delayed-echo`, `big-send`, `frag-send`, `tls-echo`, `http-echo`, `throughput`, `ntp`. A server omits what it can't offer (e.g. `stun-5780` without a second IP degrades to `stun-basic`). Clients must skip, and record as `unsupported`, any test whose capability is absent. Unknown capability strings are ignored.

### 2.4 Sessions

Every measurement run opens one session per target:

```
POST /v1/sessions      Authorization: Bearer <device-credential>
  { "target": "vie-1" }
→ 201 { "session_id": "<opaque, 8-byte hex prefix used on the wire>",
        "key_salt": "<b64, 16 bytes>",
        "epoch": "server wall-clock RFC3339 at session start",
        "expires_s": 900 }
```

Session key: `HKDF-SHA256(ikm = device_credential, salt = key_salt, info = "echolot-v1/" + session_id)` → 32 bytes. Both sides derive it; it never crosses the wire.

```
GET  /v1/sessions/{id}/observations     → §6
POST /v1/sessions/{id}/actions          → §5 (connect-back, delayed echo, big/frag send)
DELETE /v1/sessions/{id}
```

### 2.5 Rate limiting & abuse

Per-credential and per-source-IP token buckets on: session creation, actions, UDP packets, bytes. `429` on control plane; silent drop on data plane (probes must tolerate loss anyway). All reflected/generated traffic goes **only** to the session's observed source address (or, for connect-back, the source address of the session-creating request). Data-plane responses to unauthenticated packets are never larger than the request (§3.4).

## 3. UDP probe protocol

### 3.1 Packet header (fixed 32 bytes, network byte order)

```
offset  size  field
0       4     magic  "ELT1"
4       1     type
5       1     flags
6       2     payload_len
8       8     session_prefix  (first 8 bytes of session_id)
16      4     seq
20      8     t_ns    (sender timestamp, ns since session epoch, sender's clock)
28      4     hmac32  (first 4 bytes of HMAC-SHA256(session_key, header[0..28] || payload))
```

Padding payload is arbitrary bytes counted by `payload_len`; total datagram size is what MTU/train tests vary. `hmac32` is an anti-abuse gate, not cryptographic integrity for hostile networks — 4 bytes is enough to make blind reflection/replay impractical at the allowed rates; on-path attackers are out of scope for the data plane (tampering is what the *tests* detect). Server drops packets with unknown session prefix, bad HMAC, expired session, or seq replay outside a 1024-wide window.

### 3.2 Types

| type | dir | purpose |
|---|---|---|
| 0x01 ECHO_REQ / 0x02 ECHO_RESP | c→s / s→c | RTT, loss, reordering; RESP carries observation block (§3.3) |
| 0x03 TRAIN_DATA | c→s | upstream train, no per-packet response |
| 0x04 TRAIN_REPORT_REQ / 0x05 TRAIN_REPORT | c→s / s→c | server's received-view of an upstream train (columnar, may span multiple RESP datagrams) |
| 0x06 DOWNTRAIN_DATA | s→c | downstream train (scheduled via §5 action) |
| 0x07 TIMESYNC_REQ / 0x08 TIMESYNC_RESP | c→s / s→c | 4-timestamp exchange: client t1 in REQ; RESP carries t2 (rx) and t3 (tx); t4 on receipt. Feeds `time.server_offset` |
| 0x09 MTU_PROBE / 0x0A MTU_ACK | c→s / s→c | client sends DF-flagged sizes; ACK reports size received (ACK is small — never amplifies) |
| 0x0B DELAYED_ECHO | s→c | single packet sent T seconds after the action request (NAT mapping lifetime) |

### 3.3 Observation block (in ECHO_RESP, appended per-packet)

Server reports what it saw on the corresponding request: `t_rx_ns`, `t_tx_ns` (server clock, session epoch), observed source IP + port (detects NAT rebinding mid-flow), received TTL/hop-limit, DSCP, ECN bits, received size. This is the raw material for `train.udp_updown`, `sec.dscp_ecn_survival`, and TTL-based path-length evidence in the measurement schema.

### 3.4 Anti-amplification rule (normative)

For any datagram whose HMAC does not verify: no response ever. For verified packets: ECHO_RESP/MTU_ACK/TRAIN_REPORT responses are ≤ request size unless the session is in an **asymmetric grant** created via an authenticated control-plane action (§5), which sets an explicit byte budget and rate. DOWNTRAIN and big/frag sends exist only under such grants.

## 4. TCP, TLS and HTTP endpoints

- **TCP echo (8441):** after connect, server sends one JSON line: observed source IP/port, negotiated MSS (from `TCP_INFO`), timestamps/window-scale options seen — this is the `mtu.mss_observed` evidence. Then byte-echo until FIN. TLS variant on the same port via ALPN `elt-echo` (capability `tls-echo`): server additionally returns the ClientHello it received, raw + JA4, before echoing — the `sec.clienthello_echo` evidence.
- **HTTP echo:** `POST /v1/echo` on the control listener: returns exact received request bytes (headers + body) base64-wrapped in JSON, plus observed TLS parameters. Detects header injection/stripping/proxying (`sec.http_echo`). A plain-HTTP variant on a configurable port (default off) tests plaintext-path tampering.
- **TLS reference:** `GET /v1/tls-reference?host=<name>` returns the full cert chain the server itself serves, DER+base64, so the app can compare an out-of-band copy against what a direct handshake yielded (`sec.tls_reference`).
- **STUN:** unmodified RFC 5389/5780 on 3478; second address enables full behavior discovery. No custom framing — interop with existing tooling is a feature.

## 5. Actions (authenticated asymmetric operations)

`POST /v1/sessions/{id}/actions` with one of:

```json
{ "action": "downtrain", "count": 1000, "size_bytes": 64, "interval_us": 20000, "dscp": 46 }
{ "action": "big_send",  "sizes_bytes": [1400, 1472, 1500, 1600, 2000], "df": true }
{ "action": "frag_send", "size_bytes": 2000, "family": "4 | 6" }
{ "action": "delayed_echo", "delay_s": 30 }
{ "action": "connect_back", "protocol": "tcp | udp", "port": 40123 }
{ "action": "throughput", "direction": "up | down", "streams": 4, "duration_s": 10 }
```

Rules: destination is always the session's observed source address; every action is bounded by `limits` from the profile; the response includes an `action_id` echoed in resulting data-plane packets (in payload) so evidence can be correlated. `delayed_echo` is the NAT-mapping-lifetime primitive: client binary-searches `delay_s` over repeated actions on fresh sockets.

## 6. Observations API

```
GET /v1/sessions/{id}/observations
```

Returns everything the server witnessed for this session, merged by the app into measurement evidence:

```json
{
  "udp": { "packets_seen": ..., "trains": [ { columnar per-seq view } ] },
  "tcp": [ { "connected_at", "src", "mss", "options": [...] } ],
  "http": [ { "raw_request_b64", "tls": {...}, "ja4": "..." } ],
  "dns_canary": [
    { "qname": "x7f3a.sess1.c.probe.example.net", "at": "...",
      "resolver_ip": "198.51.100.7", "resolver_asn": 64500,
      "transport": "udp", "edns": { "present": true, "bufsize": 1232, "flags": ["do"] },
      "case_preserved": true, "qname_minimized": false }
  ],
  "connect_back": [ { "action_id", "result": "connected | refused | timeout", "rtt_ms" } ]
}
```

Canary DNS: the app generates `<nonce>.<session-prefix>.<canary_zone>` names and resolves them through the resolver under test (system/provider resolver, user-configured override resolvers, or the server's reference recursive); the authoritative server records who actually asked and how. Query-log retention is a server-config value surfaced in the admin UI (privacy default: 24 h). The `ecs` field in canary observations records any EDNS Client Subnet option the resolver forwarded.

### 6.1 Reference records (normative, capability `canary-dns`)

The server MUST serve these fixed records under the canary zone; TTLs and RDATA are defined by this spec (not configurable), so clients have ground truth without a side channel. Used by `dns.ttl_integrity` and `dns.answer_integrity` (measurement schema §6.4).

| name | type | TTL | RDATA |
|---|---|---|---|
| `ttl-5.<zone>` | A / AAAA / TXT | 5 | `192.0.2.5` / `2001:db8::5` / `"echolot-ref ttl=5"` |
| `ttl-60.<zone>` | A / AAAA / TXT | 60 | `192.0.2.60` / `2001:db8::60` / `"echolot-ref ttl=60"` |
| `ttl-3600.<zone>` | A / AAAA / TXT | 3600 | `192.0.2.36` / `2001:db8::3600` / `"echolot-ref ttl=3600"` |
| `ttl-86400.<zone>` | A / AAAA / TXT | 86400 | `192.0.2.86` / `2001:db8::8640` / `"echolot-ref ttl=86400"` |
| `many-rr.<zone>` | A | 300 | exactly 8 A records in defined order (order/stripping check) |
| `big-txt.<zone>` | TXT | 300 | ~1800 bytes (EDNS bufsize / TCP-fallback check) |

Note: exact RDATA constants to be frozen in the implementation's `dns_reference.go` and mirrored in the app; the table above fixes the *names and TTLs*; RDATA must be deterministic, documentation-range addresses. Cache-miss variants: `<nonce>.miss.<zone>` wildcard answers with TTL 3600 and RDATA encoding the nonce (per-query ground truth that can never be pre-cached).

## 7. Server admin UI (scope note)

Same daemon, separate listener (default localhost-only): health + self-test (are both IPs live, is the canary zone delegated correctly, is UDP reachable from outside — tested via a public echolot "mirror" if configured), enrollment token management (create/expire/scope), device list + revocation, retention settings, QR rendering (client-side JS). Out of scope for this spec beyond the endpoints above.

## 8. Cross-references to the measurement schema

- Observation-block fields (§3.3) appear as `*_seen_by_server` columns in `train` evidence (schema §6.2).
- `TIMESYNC` (§3.2) produces the `time.server_offset` test; without it, cross-clock fields must not be compared (schema §6.2 note).
- Capability strings (§2.3) are copied verbatim into `server_sessions[].capabilities` (schema §5); tests skipped for missing capability get `status: "unsupported"`.
- `session_id` maps to `server_sessions[].session_id`; `action_id`s appear in test `params`.

## 9. Open items

1. Whether TRAIN_REPORT should also stream during long trains (partial reports every N packets) for live UI feedback — leaning yes, same type with a `flags` bit.
2. Throughput methodology (fixed streams vs BBR-style ramp) — decide with `perf.*` test design.
3. Server mirror/self-test protocol between two echolot servers (nice for operators; postpone).
4. IPv6 flow-label control for ECMP-variance tracing — needs `IPV6_FLOWINFO` sockopt verification on Android first (capability-prober item).