Initial commit: capability prober + design docs
Monorepo root for Echolot. Contains the no-root capability prober (Kotlin/Compose, app.echo_lot.prober) and the four design docs that act as the contract for the production app and the Go server. LICENSE is deliberately absent — still undecided, see docs/build-status.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,35 @@
|
||||
# Echolot — build status & next steps
|
||||
|
||||
Last updated: 2026-07-29.
|
||||
|
||||
## Decided
|
||||
- Name **Echolot**; domain echo-lot.app; scheme `echolot://`; namespace `app.echo_lot.*`
|
||||
(hyphen→underscore; appIds/packages can't contain hyphens). Prober appId `app.echo_lot.prober`.
|
||||
- Stack: native **Kotlin + Jetpack Compose**, no Flutter.
|
||||
- Tiers: `app` (no root), **`shizuku` in v1** (wireless-ADB pairing), `root` future.
|
||||
- License: still undecided (GPLv3 vs Apache-2.0) — needed before repos go public.
|
||||
|
||||
## Specs (in `docs/`, alongside this file)
|
||||
- `feature-catalog-and-feasibility.md`, `measurement-schema.md`, `probe-protocol.md`. Considered draft-complete and reviewed by the user.
|
||||
|
||||
## Capability prober — DELIVERED (as zip, 2026-07-29)
|
||||
Full Kotlin/Compose project scaffolded: `app.echo_lot.prober`, minSdk 26 / target+compile 35,
|
||||
AGP 8.7.3, Kotlin 2.0.21, Compose BOM 2024.10, Shizuku api+provider 13.1.5, kotlinx-serialization.
|
||||
Probes implemented: `link.snapshot`, `icmp.ping4/6` (unprivileged ICMP datagram), `sockopt.matrix`
|
||||
(TTL/TOS/RECVERR/MTU_DISCOVER), `trace.errqueue_reachable`, `multinetwork.request_and_bind`,
|
||||
`local.mdns_discover`, `peer.ble_advertise`, `shizuku.command_battery` (ip neigh / ip -6 route /
|
||||
ip addr / ip monitor / dumpsys network_stack DHCP+IpClient / dumpsys wifi). JSON export via share
|
||||
intent; results carry verdict + raw evidence.
|
||||
|
||||
Could NOT be compiled in the cloud sandbox: dl.google.com (Google Maven) and services.gradle.org
|
||||
are proxy-blocked, and no device is reachable for on-device runs. Build + iterate locally
|
||||
(Android Studio / Claude Code). Wrapper is pinned to Gradle 8.14.3.
|
||||
|
||||
## Next steps
|
||||
1. Build locally, run on several physical devices (varied Android versions/vendors), collect the
|
||||
JSON reports — especially the real per-device Shizuku dump formats.
|
||||
2. If `trace.errqueue_reachable` = PARTIAL, add a C-over-JNI errqueue shim (`recvmsg`+cmsg parse)
|
||||
as a `:native` module and a real `traceroute.udp4` probe.
|
||||
3. Start the Go server skeleton (enrollment + profile + sessions + UDP echo with observation
|
||||
blocks + canary-DNS reference records) per probe-protocol.md.
|
||||
4. Fold confirmed capabilities into the production `core-probe` / `core-shizuku` modules.
|
||||
@@ -0,0 +1,148 @@
|
||||
# Echolot (Android Network Debug App) — Feature Catalog & Feasibility (v0.1 draft)
|
||||
|
||||
Status: brainstorm/design input, 2026-07-29. Premises: F/OSS, no root in v1, Wi-Fi + cellular + USB ethernet, own server infrastructure available, audience = network engineers, detailed archived/exportable measurements with a summary score.
|
||||
|
||||
Decisions so far: name = **Echolot** (domain echo-lot.app, scheme echolot://, suggested appId at.rambossek.echolot); **Shizuku support in v1** (not v1.x); **native Kotlin + Jetpack Compose** (no Flutter — all value is in platform-level APIs; Flutter would be a Dart skin over a full Kotlin app and complicates F-Droid reproducible builds); **peer mode with BLE out-of-band control channel** is a headline feature.
|
||||
|
||||
## 1. What no-root Android allows and forbids
|
||||
|
||||
The whole design hangs on this. Verified against Linux/Android capability rules; re-check details against the current Android release before implementation.
|
||||
|
||||
**Allowed without root:**
|
||||
|
||||
- UDP/TCP sockets on ephemeral ports, all the usual `setsockopt`s: `IP_TTL`, `IP_RECVERR` (+ `MSG_ERRQUEUE`), `IP_MTU_DISCOVER` (DF bit / probe mode), `IP_TOS`/DSCP, ECN bits via `IP_TOS`, `SO_TIMESTAMP(NS)`.
|
||||
- ICMP echo: `socket(AF_INET, SOCK_DGRAM, IPPROTO_ICMP)` and the ICMPv6 equivalent work for any UID on Android (`ping_group_range` is opened system-wide). Echo only — the kernel filters other ICMP types.
|
||||
- Traceroute: UDP (or ICMP-echo) probes with increasing TTL + `IP_RECVERR`; time-exceeded senders are delivered on the error queue. No raw socket needed.
|
||||
- Multicast send/receive with `CHANGE_WIFI_MULTICAST_STATE` + `MulticastLock` → mDNS, SSDP, LLMNR.
|
||||
- Platform info: `ConnectivityManager`/`LinkProperties` (addresses, prefixes, routes, gateways, DNS servers, MTU, NAT64 prefix, private-DNS state, captive-portal state incl. RFC 8908 API), `WifiManager`/`WifiInfo` (RSSI, link speed, channel/band, Wi-Fi standard, security type; SSID/BSSID require fine-location permission), `TelephonyManager` (cell type, signal, APN-ish info), deprecated-but-working `DhcpInfo`.
|
||||
- Multi-network: `requestNetwork` for `TRANSPORT_CELLULAR`, `TRANSPORT_WIFI`, `TRANSPORT_ETHERNET` concurrently; `Network.bindSocket()` per probe. USB ethernet adapters supported by the kernel appear as ethernet networks.
|
||||
- `VpnService`: capture/inspect the device's own traffic without root (PCAPdroid model).
|
||||
|
||||
**Shizuku tier (v1, no root — ADB-shell privileges via wireless-ADB pairing):**
|
||||
|
||||
The `shell` user's SELinux domain is far less restricted than an app's. Verified capabilities to build on (each needs per-release verification; build as version-tagged parsers with graceful degradation, and record in every report which tier produced which evidence):
|
||||
|
||||
- `ip neigh` → ARP/ND table: gateway MAC monitoring, ARP-spoof heuristics (moves from root-only to Shizuku tier).
|
||||
- `ip -6 route` / `ip addr` with lifetimes → RA-derived routes (`proto ra`, expiry), address valid/preferred lifetimes.
|
||||
- `ip monitor` → live netlink stream of addr/route/neigh changes (watch DHCP renewals and RA churn in real time).
|
||||
- `svc wifi disable|enable`, `cmd wifi` → trigger reconnect/re-lease programmatically, observe via `ip monitor` + dumpsys.
|
||||
- `dumpsys network_stack` / `dumpsys wifi` → IpClient logs: DHCP OFFER/ACK details, lease params, provisioning events.
|
||||
- `logcat` → network-stack log lines.
|
||||
|
||||
**Not possible without root (→ future root-only module):**
|
||||
|
||||
- Raw/packet sockets: own DHCP packets, ARP, RA sniffing, promiscuous/monitor capture, ESP/GRE probes.
|
||||
- Binding ports < 1024 (this alone kills a real DHCP client — replies go to UDP 68).
|
||||
- `/proc/net/*` (blocked since Android 10), netlink `RTM_GETLINK` (blocked since 11) → no ARP/neighbor table, no MAC addresses (own interface reports `02:00:00:00:00:00`).
|
||||
- Wi-Fi monitor mode, beacon-level analysis.
|
||||
|
||||
**Platform gotchas to design around:** Wi-Fi scan throttling (≈4 scans/2 min foreground); fine-location permission (+ location services ON) required for SSID/BSSID/scan results; per-SSID MAC randomization; Doze/background limits → long tests need a foreground service, monitor mode uses WorkManager with coarse intervals; `NetworkInterface` enumeration is partially restricted for modern target SDKs → treat `LinkProperties` per `Network` as the source of truth.
|
||||
|
||||
## 2. The original wishlist, item by item
|
||||
|
||||
| Item | Feasibility (no root) | Approach |
|
||||
|---|---|---|
|
||||
| v4/v6 config readout | Full | `LinkProperties` per network: addresses, prefixes, routes, DNS, MTU, NAT64 prefix. Analyze: address scope/type, privacy extensions in use, prefix sanity, gateway in subnet, DNS reachability. |
|
||||
| Own DHCP requests + option analysis | **Blocked** | Needs port 68 or raw socket (server replies always go to UDP 68, never the requester's ephemeral port — so send-only broadcast is useless). VpnService does NOT help: TUN is a virtual L3 interface; link-scoped traffic (DHCP/ARP/ND/RA) is handled by IpClient directly on wlan0 and never routed through the VPN, and writing into the TUN only feeds your own stack. v1 fallback: read `DhcpInfo` (server, lease) + diff `LinkProperties` across lease renewals; flag server-outside-subnet, weird lease times, gateway/DNS flapping. Middle tier (v1.x): Shizuku/ADB-shell → parse `dumpsys network_stack` / `dumpsys wifi` IpClient logs (real observed DHCP OFFER/ACK details, lease params, RA provisioning events) — unstable format, needs per-release parser + graceful degradation; verify with a prototype spike. Full crafted-packet prober → root module. |
|
||||
| IPv6 RA inspection | **Blocked** (raw ICMPv6) | Indirect: multiple v6 default routes with different link-local next-hops, prefixes from unrelated aggregates, default-route churn → "multiple RA sources suspected". M/O flags and lifetimes not readable. |
|
||||
| Multiple DHCP servers / RAs | Partial | Above heuristics only; reliable detection needs sniffing (root module). Peer mode (see 3.7) can help: a second device may hold a lease from the other server — diff their views. |
|
||||
| Packet loss with patterns | Full | UDP trains to own server, sequence numbers + timestamps both directions; report per-direction loss, burstiness (Gilbert-Elliott fit), reordering, duplication, jitter distribution — not just a percentage. |
|
||||
| Multiple IP networks on one segment (VLAN leakage) | Good | mDNS/SSDP/LLMNR inventory: announcements revealing addresses in foreign prefixes; own subnet vs observed peers. Plus DHCP-flap heuristics. |
|
||||
| NAT analysis | Full | RFC 5780 STUN (needs 2 IPs on server): mapping + filtering behavior, hairpinning, port preservation/allocation pattern. Mapping lifetime measurement (idle-timeout binary search, UDP and TCP separately). CGNAT: traceroute into 100.64.0.0/10, STUN-mapped vs interface addr. Inbound: server connects back to mapped port. |
|
||||
| MTU measurement | Full (with server) | Both directions with DF via `IP_PMTUDISC_PROBE` + binary search; MSS clamping (server reports SYN MSS seen); PMTU blackhole detection (large-DF timeout while small passes); v6 fragmentation behavior; DNS large-response/TCP-fallback test. |
|
||||
| IDS/MITM detection | Good (with server) | Echo-comparison principle: server returns exactly what it received — HTTP request bytes, TLS ClientHello (JA4), DSCP/ECN, remaining TTL, source port. App diffs against what it sent. Catches transparent proxies, header injection/stripping, TLS interception (also: cert-chain pinning against out-of-band-fetched chain), DPI resets (SNI-based RST vs timeout), DSCP bleaching. ARP-spoof detection: not possible app-only, but feasible in the Shizuku tier via `ip neigh` polling (gateway MAC change detection). |
|
||||
| DNS analysis + canary zone | Full | Own resolver implementation (UDP/TCP/DoT/DoH). User-configurable **override resolvers**: full dns.* battery runs once per resolver, in parallel, incl. TTL-integrity (clamping/freezing/serve-stale vs spec-defined reference records) and answer-integrity (RDATA/flags/order/ECS vs ground truth); `dns.compare` diffs all resolvers. Canary zone with unique per-run labels → authoritative server logs actual querying resolver, source AS, EDNS/0x20/DNSSEC behavior, minimization. Client side: provider vs own-server resolver diff, interception test (DNS query to a known non-DNS IP:53 — any answer = transparent redirect), NXDOMAIN wildcarding, DNSSEC validation (known-bad zone must SERVFAIL), rebind protection (does resolver strip answers pointing to RFC1918?), AAAA filtering, DNS64 synthesis correctness, latency/cache behavior. |
|
||||
|
||||
## 3. Additional test catalog (suggestions)
|
||||
|
||||
### 3.1 IPv6 & dual-stack
|
||||
- Broken-v6 detection: v6 configured but blackholed (very common). Compare v4 vs v6 for every test; measure real Happy-Eyeballs behavior and fallback delay.
|
||||
- NAT64/DNS64/464XLAT/CLAT detection and correctness.
|
||||
- v6 privacy/temporary address usage; source-address selection sanity.
|
||||
|
||||
### 3.2 Path & routing
|
||||
- Paris-style traceroute (fixed flow tuple) v4+v6, UDP and ICMP modes; ECMP path variance by varying flow label/ports.
|
||||
- Remaining-TTL analysis at server → hop-count asymmetry, transparent-proxy hint (TTL reset).
|
||||
- Egress identity: public IP, reverse DNS, ASN/geo (server-side lookup), consistency across protocols.
|
||||
|
||||
### 3.3 Transport & middleboxes
|
||||
- ECN negotiation and survival; DSCP bleaching/remarking (server echo).
|
||||
- Outbound port/protocol sweep: 25, 465, 587, 110/143/993, 22, 53-TCP/UDP, 123, 445, 3389, 1194, 500/4500, 51820, 8080/8443, QUIC 443/UDP. Distinguish RST vs timeout vs proxy-intercepted.
|
||||
- "Is UDP usable at all" (hotel/guest networks); QUIC vs TCP-443 performance delta.
|
||||
- Rate-limit/shaping detection: ramp test, token-bucket signature; per-port throttling comparison.
|
||||
- IPv4 fragmentation and v6 fragment-header survival (server sends fragmented; client reports).
|
||||
|
||||
### 3.4 Performance
|
||||
- Throughput (multi-stream TCP + paced UDP), both directions.
|
||||
- Bufferbloat: latency under load, RPM-style / Waveform-style grade.
|
||||
- Idle→active latency step on cellular (RRC state transitions).
|
||||
- Gateway RTT vs first-external-hop RTT vs server RTT → segment isolation ("is it my Wi-Fi, my router, or my ISP").
|
||||
|
||||
### 3.5 Wi-Fi layer (within API limits)
|
||||
- RSSI/link-speed/channel/band/standard logging over time; roaming log (BSSID transitions, gap duration) in a walk-around mode.
|
||||
- Same-SSID environment scan: BSSIDs, bands, security types; security-downgrade and evil-twin heuristics (same SSID, different security).
|
||||
- Channel congestion snapshot from scan results (co-channel BSS count).
|
||||
|
||||
### 3.6 Gateway & LAN services
|
||||
- UPnP-IGD / NAT-PMP / PCP probe: discover, query external IP, attempt (and immediately release) a mapping — capability report + security finding.
|
||||
- mDNS/SSDP/LLMNR device inventory; duplicate names, foreign-subnet announcements.
|
||||
- Captive-portal analysis: RFC 8908/8910 API data + multi-URL probe matrix (HTTP/HTTPS/DNS behavior while captive).
|
||||
- NTP: offset/latency to pool + own server; is 123/UDP intercepted?
|
||||
|
||||
### 3.7 Peer mode (two or more devices running the app)
|
||||
Covers what no server can see: AP client isolation, LAN loss/jitter/throughput, multicast/IGMP delivery, cross-VLAN reachability, band-steering comparison (one device per band), roaming behavior diffing, and "do we even see the same network" (diff both devices' LinkProperties and leases — helps with the multi-DHCP question).
|
||||
|
||||
Design: **control plane over BLE GATT** (out-of-band — coordinating over the medium under test is self-defeating; when Wi-Fi is broken is when coordination matters most). Small CBOR messages: orchestration, result summaries. Data plane over the network under test; bulk result exchange syncs later over the network or the server. Third rendezvous path via the server when both devices have cellular. Clock sync for one-way delay: offset handshake over the LAN when usable (BLE jitter too high for precise sync — coarse coordination only). >2 devices: star topology, one coordinator. Permissions: BLUETOOTH_SCAN/CONNECT/ADVERTISE (Android 12+) with `neverForLocation`.
|
||||
|
||||
### 3.8 Modes & UX features
|
||||
- Continuous monitor mode: lightweight periodic checks, outage/change timeline ("what broke at 03:12").
|
||||
- Diff view: compare any two archived measurements (before/after, Wi-Fi vs cellular vs USB-eth, site A vs site B).
|
||||
- Simultaneous multi-path run: same battery over Wi-Fi + cellular + ethernet at once via per-network socket binding.
|
||||
- Scheduled runs via WorkManager (battery-honest).
|
||||
|
||||
### 3.9 Later (v1.5/v2)
|
||||
- VpnService capture module: own-traffic pcap, per-app flows, DNS-leak analysis. Sees only routed L3 app traffic — no DHCP/ARP/ND/RA. Conflicts with real VPNs → separate opt-in.
|
||||
- Root module: real DHCP prober (all options, multiple-offer capture), RA sniffer (flags, lifetimes, RDNSS), ARP/ND table + spoof detection, monitor-mode Wi-Fi where hardware allows, ESP/GRE reachability.
|
||||
- Anonymized export (strip/pseudonymize addresses, SSIDs, hostnames consistently within a report).
|
||||
|
||||
## 4. Server-side design sketch
|
||||
|
||||
One self-contained daemon (Go or Rust; open-source alongside the app), deployable on multiple VMs/locations. Needs at least one host with **two public IPv4 addresses** (RFC 5780 STUN) and native v6.
|
||||
|
||||
Components: control API (TLS, versioned, session tokens; sessions correlate all probes of one run); UDP echo/pattern responder with receive-timestamps and observed-header report (TTL, DSCP, ECN, source port, size); TCP endpoints (MSS/window report, connect-back tester); STUN (RFC 5780); authoritative DNS for the canary zone with query logging exposed through the control API; recursive resolver for comparison; HTTPS echo endpoint (returns exact request bytes + ClientHello JA4 + observed TLS params) and reference cert-chain endpoint; large/fragmented-packet senders; NTP responder optional.
|
||||
|
||||
Abuse controls from day one (it is an F/OSS-published traffic reflector): session-token gating, per-IP rate limits, no amplification (responses ≤ requests unless session-authenticated), connect-back only to the session's source address.
|
||||
|
||||
### 4.1 Self-hosting, enrollment & profiles (decided direction)
|
||||
|
||||
Anyone can deploy the server in their own infrastructure. Configuration of the app happens via a **bootstrap QR / paste-string** — same payload, two transports:
|
||||
|
||||
```
|
||||
echolot://enroll?v=1&u=<control-URL>&p=pin-sha256:<SPKI-hash>&t=<enrollment-token>
|
||||
```
|
||||
|
||||
- QR carries only: control endpoint, server SPKI pin, one-time enrollment token. App verifies TLS against the **pin, not CAs** (self-signed homelab certs work), presents token, downloads the full profile: probe targets per location, STUN addresses, canary zone, long-lived credential, `capabilities` list.
|
||||
- Trust anchor is transferred out-of-band (camera↔screen) → the reference channel is immune to CA-level/on-path interference — exactly the property the MITM/tampering tests need. Enrollment doubles as trust bootstrap.
|
||||
- Profile is refreshed from the control server on each run → server-side updates without re-scanning; multi-location = one control server handing out all targets, or multiple independent profiles aggregated in the app.
|
||||
- Capability negotiation via the profile's `capabilities` list (`stun-5780`, `canary-dns`, `connect-back`, `fragmented-send`, …) — third parties will run mismatched server versions forever; degrade gracefully per capability, not per version number.
|
||||
- App side: CameraX + **ZXing** for scanning (NOT ML Kit — proprietary, would disqualify from F-Droid main repo). Custom URI scheme registered so any scanner app opens the enrollment.
|
||||
- Server side: same single daemon serves a small admin web UI (localhost or separate admin port, admin password set at install): health/self-test, enrollment-token management (expiry, scopes enroll vs run-tests, per-token rate limits), QR rendered client-side in JS. Reinforces Go single-static-binary + `embed` for the web UI; cross-compile for ARM (VPSes, RPis).
|
||||
|
||||
## 5. App architecture notes
|
||||
|
||||
- **Decided: Kotlin + coroutines, Jetpack Compose UI. No Flutter** — all differentiating functionality is platform-level (Os sockets, Network callbacks, VpnService, NsdManager, Shizuku Binder, foreground services); Flutter would mean platform channels around a full Kotlin app plus F-Droid build friction.
|
||||
- Module layout: `core-probe` (measurement engine, no UI deps → JUnit/CLI-testable, desktop-portable), `core-shizuku` (privileged tier, version-tagged parsers), `core-server` (protocol client), `core-peer` (BLE control plane + peer tests), `feature-*` per test family, thin `app` (Compose).
|
||||
- Sockets via `android.system.Os` for ~95%; plan a small C-over-JNI shim (a few hundred lines, deliberately not Rust — lower contributor bar) for the corners with patchy `Os` coverage: `MSG_ERRQUEUE` reads, cmsg parsing, `SO_TIMESTAMPING`.
|
||||
- Storage: Room; measurement = schema-versioned JSON (or CBOR) containing raw evidence (every packet summary, every reply, timestamps) + derived findings + per-category verdicts. Export via `ACTION_SEND`/SAF: full JSON + rendered human-readable report (HTML/PDF).
|
||||
- Scoring: per-category traffic lights that expand into findings, each with severity, plain-language explanation, and raw evidence links. Engineers distrust opaque scores — always show the reasoning chain.
|
||||
- Permissions: INTERNET, ACCESS_NETWORK_STATE, ACCESS_WIFI_STATE, ACCESS_FINE_LOCATION (Wi-Fi identifiers/scans), CHANGE_WIFI_MULTICAST_STATE, POST_NOTIFICATIONS, FOREGROUND_SERVICE(+type). Explain the location permission prominently — it's the one users will question.
|
||||
|
||||
## 6. Open decisions
|
||||
|
||||
(Decided: app framework = Kotlin/Compose; Shizuku tier in v1; peer mode with BLE control plane.)
|
||||
|
||||
1. Server language/stack: leaning Go (single static binary, embedded admin UI, ARM cross-compile) — confirm vs Rust or composing existing components (e.g. coturn for STUN).
|
||||
2. minSdk: 26 keeps old spare phones usable; 29+ simplifies (cleaner APIs, but scan throttling and /proc/net loss apply anyway). Suggest 26–28 floor, decide per API audit.
|
||||
3. License: GPLv3 vs Apache-2.0 (server and app can differ).
|
||||
4. Measurement format: JSON vs CBOR; whether to define the schema first (recommended — it drives both app and server).
|
||||
5. Canary-zone domain + how much server-side logging to retain (privacy statement needed even for a debug tool).
|
||||
@@ -0,0 +1,285 @@
|
||||
# Echolot Measurement Schema — Spec v1 (draft 1)
|
||||
|
||||
Project name: **Echolot** (decided 2026-07-29; domain: echo-lot.app). Status: draft for review, 2026-07-29.
|
||||
|
||||
This document defines the JSON format of one **measurement run** — the unit that is archived on-device, diffed against other runs, and exported/shared. It is the contract between the probe engine, the server's observation API, the report renderer, and (later) the anonymizer.
|
||||
|
||||
## 1. Design rules
|
||||
|
||||
1. **Observation and interpretation are separated.** `tests[]` contain raw evidence and computed metrics; `findings[]` contain interpretation with references back to evidence. A reader must be able to re-derive every finding from the evidence alone.
|
||||
2. **Schema-versioned.** `schema_version` is semver. Minor = additive only. Consumers must ignore unknown fields (forward compat) and must not require fields added after their version.
|
||||
3. **Every value is attributable.** Each test records which network it ran on, which trust tier produced it (`app` / `shizuku` / `root`), and which server session (if any) was involved.
|
||||
4. **Two clocks.** Wall-clock timestamps are RFC 3339 UTC with millisecond precision, only for human correlation. All measurement math uses `*_mono_ns`: nanoseconds since `run.clock.mono_origin` (an arbitrary per-run epoch from the monotonic clock). Never mix the two.
|
||||
5. **Units in field names.** `_ms`, `_ns`, `_bytes`, `_bps`, `_pct`. No unitless numbers for physical quantities.
|
||||
6. **Anonymization-ready.** Every sensitive scalar is declared with a logical type in the schema registry (§8): `ip4`, `ip6`, `mac`, `fqdn`, `ssid`, `bssid`, `imsi-ish`. The v2 anonymizer walks the schema, not the data, and transforms consistently within a document (prefix-preserving for IPs).
|
||||
7. **Big arrays go columnar.** Packet-train evidence is stored as parallel arrays (§6.2), not arrays of objects — a 10 000-packet train stays in the hundreds of kB.
|
||||
8. **IDs.** `run.id` and `tests[].id` are UUIDv7 (time-ordered). Cross-references use these IDs plus optional JSON Pointers for precision.
|
||||
|
||||
## 2. Top-level document
|
||||
|
||||
```json
|
||||
{
|
||||
"schema": "echolot/measurement",
|
||||
"schema_version": "1.0.0",
|
||||
"run": { ... }, // §3
|
||||
"networks": [ ... ], // §4
|
||||
"server_sessions": [ ... ], // §5
|
||||
"tests": [ ... ], // §6
|
||||
"findings": [ ... ], // §7
|
||||
"summary": { ... } // §7.3
|
||||
}
|
||||
```
|
||||
|
||||
Export encoding: UTF-8 JSON, gzip for files (`.echolot.json.gz`), share intent uses MIME `application/gzip` plus a rendered HTML report alongside.
|
||||
|
||||
## 3. `run` — context of the whole measurement
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "0198c5f2-...-uuidv7",
|
||||
"trigger": "manual | scheduled | monitor | peer",
|
||||
"started_at": "2026-07-29T14:03:21.114Z",
|
||||
"ended_at": "2026-07-29T14:07:44.902Z",
|
||||
"clock": {
|
||||
"mono_origin_wall": "2026-07-29T14:03:21.114Z",
|
||||
"ntp_offset_ms": -12.4, // optional, if NTP test ran
|
||||
"ntp_offset_source": "test-uuid" // evidence ref
|
||||
},
|
||||
"app": { "version": "0.3.1", "build": 310, "git": "a1b2c3d", "flavor": "fdroid" },
|
||||
"device": {
|
||||
"manufacturer": "Google", "model": "Pixel 8a",
|
||||
"android_sdk": 35, "android_release": "15", "security_patch": "2026-06-05"
|
||||
},
|
||||
"tiers": { "app": true, "shizuku": true, "root": false },
|
||||
"profiles_used": ["profile-uuid", ...],
|
||||
"notes": "free-text user annotation"
|
||||
}
|
||||
```
|
||||
|
||||
`tiers` records what was *available*; each test records what it *used*.
|
||||
|
||||
## 4. `networks[]` — one entry per Android `Network` in play
|
||||
|
||||
A run may exercise several networks simultaneously (Wi-Fi + cellular + USB ethernet). Everything is a snapshot at run start; a `changes[]` list captures mid-run deltas.
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "net-1",
|
||||
"transport": "wifi | cellular | ethernet | vpn | other",
|
||||
"interface": "wlan0",
|
||||
"link": {
|
||||
"mtu": 1500,
|
||||
"addresses": [
|
||||
{ "addr": "192.0.2.23", "prefix_len": 24, "scope": "global",
|
||||
"flags": ["temporary"], "valid_lft_s": 3541, "pref_lft_s": 3541 }
|
||||
],
|
||||
"routes": [
|
||||
{ "dst": "0.0.0.0/0", "gateway": "192.0.2.1", "iface": "wlan0",
|
||||
"proto": "dhcp | ra | static | unknown", "expires_s": 1799 }
|
||||
],
|
||||
"dns": {
|
||||
"servers": ["192.0.2.1"],
|
||||
"private_dns_mode": "off | opportunistic | strict",
|
||||
"private_dns_hostname": null,
|
||||
"search_domains": ["lan"],
|
||||
"nat64_prefix": null
|
||||
},
|
||||
"dhcp": { "server": "192.0.2.1", "lease_s": 3600 },
|
||||
"captive_portal": { "detected": false, "api_url": null, "venue_url": null }
|
||||
},
|
||||
"wifi": {
|
||||
"ssid": "example-net", "bssid": "aa:bb:cc:dd:ee:ff",
|
||||
"rssi_dbm": -54, "link_speed_mbps": 573, "frequency_mhz": 5240,
|
||||
"channel_width_mhz": 80, "standard": "11ax",
|
||||
"security": "wpa3-sae", "mac_randomization": true
|
||||
},
|
||||
"cellular": { "rat": "nr-nsa", "operator": "...", "band": "n78" },
|
||||
"changes": [
|
||||
{ "at_mono_ns": 91000000000, "kind": "lost | gained | link_changed",
|
||||
"detail": { /* new link snapshot or diff */ } }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`routes[].proto` and lifetime fields are Shizuku-tier data (`ip route`/`ip addr`); app-tier snapshots leave them absent — absence means "not observed", never "not present".
|
||||
|
||||
## 5. `server_sessions[]`
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "sess-1",
|
||||
"profile_id": "profile-uuid",
|
||||
"profile_name": "homelab",
|
||||
"control_url": "https://probe.example.net:8443",
|
||||
"server_version": "0.4.0",
|
||||
"capabilities": ["udp-probe", "stun-5780", "canary-dns", "connect-back", "delayed-echo", "big-send", "tls-echo"],
|
||||
"session_id": "opaque-server-issued",
|
||||
"target": { "ip4": "203.0.113.10", "ip6": "2001:db8::10", "udp_port": 8442 }
|
||||
}
|
||||
```
|
||||
|
||||
`capabilities` here records what the server offered *at run time*; test-level `status: "unsupported"` records what was consequently skipped.
|
||||
|
||||
## 6. `tests[]` — the generic result envelope
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "uuidv7",
|
||||
"type": "traceroute.udp4", // registry, §6.1
|
||||
"network_ref": "net-1",
|
||||
"session_ref": "sess-1", // null for purely local tests
|
||||
"tier": "app | shizuku | root",
|
||||
"started_mono_ns": 12000000,
|
||||
"ended_mono_ns": 4530000000,
|
||||
"status": "ok | failed | unsupported | skipped | partial",
|
||||
"error": { "code": "timeout", "detail": "..." }, // when failed/partial
|
||||
"params": { ... }, // exact inputs, type-specific
|
||||
"evidence": { ... }, // raw observations, type-specific
|
||||
"metrics": { ... } // derived numbers, type-specific
|
||||
}
|
||||
```
|
||||
|
||||
Rules: `params` must contain everything needed to reproduce the test. `evidence` is append-only raw truth. `metrics` must be recomputable from `evidence` (renderer and diff view use `metrics`; auditors use `evidence`).
|
||||
|
||||
### 6.1 Test type registry (v1)
|
||||
|
||||
Dotted names, family first. Initial registry; additions are minor version bumps.
|
||||
|
||||
| Family | Types |
|
||||
|---|---|
|
||||
| `link` | `link.snapshot`, `link.dhcp_renewal_watch`, `link.ip_monitor` (shizuku) |
|
||||
| `icmp` | `icmp.ping4`, `icmp.ping6` (targets: gateway, first-hop-external, server, public refs) |
|
||||
| `trace` | `traceroute.udp4`, `traceroute.udp6`, `traceroute.icmp4`, `traceroute.icmp6` |
|
||||
| `train` | `train.udp_updown` (loss/jitter/reorder/dup, both directions) |
|
||||
| `mtu` | `mtu.pmtud_up`, `mtu.pmtud_down`, `mtu.blackhole`, `mtu.mss_observed`, `mtu.frag_delivery` |
|
||||
| `nat` | `nat.stun_5780`, `nat.mapping_lifetime_udp`, `nat.mapping_lifetime_tcp`, `nat.hairpin`, `nat.connect_back`, `nat.cgnat_detect` |
|
||||
| `dns` | `dns.resolver_inventory`, `dns.canary`, `dns.interception`, `dns.ttl_integrity`, `dns.answer_integrity`, `dns.dnssec`, `dns.nxdomain_wildcard`, `dns.rebind_filter`, `dns.aaaa_filter`, `dns.dns64`, `dns.compare` |
|
||||
| `sec` | `sec.tls_reference`, `sec.clienthello_echo`, `sec.http_echo`, `sec.sni_filter`, `sec.dscp_ecn_survival`, `sec.arp_watch` (shizuku) |
|
||||
| `port` | `port.reach_sweep` (outbound), `port.udp_usability` |
|
||||
| `perf` | `perf.throughput_tcp`, `perf.throughput_udp`, `perf.bufferbloat`, `perf.rrc_latency` |
|
||||
| `v6` | `v6.dualstack_compare`, `v6.happy_eyeballs`, `v6.brokenness`, `v6.nat64_clat` |
|
||||
| `wifi` | `wifi.environment_scan`, `wifi.roam_log`, `wifi.signal_log` |
|
||||
| `local` | `local.mdns_inventory`, `local.ssdp_inventory`, `local.llmnr_inventory`, `local.gateway_services` (UPnP-IGD/NAT-PMP/PCP), `local.ntp` |
|
||||
| `peer` | `peer.reachability`, `peer.isolation`, `peer.multicast`, `peer.lan_train`, `peer.lease_diff` |
|
||||
| `time` | `time.server_offset` (4-timestamp exchange, feeds one-way metrics) |
|
||||
|
||||
### 6.2 Evidence conventions for packet trains
|
||||
|
||||
Columnar parallel arrays, one index per probe packet. Missing observations are `null` at that index.
|
||||
|
||||
```json
|
||||
"evidence": {
|
||||
"epoch_mono_ns": 12000000,
|
||||
"seq": [0, 1, 2, 3],
|
||||
"t_tx_ns": [0, 20000000, 40000000, 60000000], // relative to epoch
|
||||
"t_srv_rx_ns":[8123456, 28090000, null, 68240000], // server clock, session epoch
|
||||
"t_srv_tx_ns":[8180000, 28150000, null, 68300000],
|
||||
"t_rx_ns": [16500000, 36400000, null, 76800000],
|
||||
"size_bytes": [64, 64, 64, 64],
|
||||
"dscp_sent": 46, "dscp_seen_by_server": [0, 0, null, 0],
|
||||
"ecn_sent": 1, "ecn_seen_by_server": [1, 1, null, 1],
|
||||
"ttl_seen_by_server": [54, 54, null, 54]
|
||||
}
|
||||
```
|
||||
|
||||
Server-side observations come from the observation API (probe-protocol §6) and are merged in by the app; their clock is the server session epoch — only differences within the same clock are meaningful unless a `time.server_offset` test provides the mapping.
|
||||
|
||||
### 6.3 Traceroute evidence
|
||||
|
||||
```json
|
||||
"evidence": {
|
||||
"flow": { "src_port": 40123, "dst_port": 8442, "fixed_tuple": true },
|
||||
"hops": [
|
||||
{ "ttl": 1, "probes": [
|
||||
{ "reply_from": "192.0.2.1", "rtt_ns": 1830000, "icmp": "ttl-exceeded", "reply_ttl": 64 },
|
||||
{ "reply_from": null, "rtt_ns": null, "icmp": null }
|
||||
]}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 6.4 DNS test conventions
|
||||
|
||||
**Resolver targeting.** Every `dns.*` test carries a `params.resolver` object identifying the resolver under test:
|
||||
|
||||
```json
|
||||
"resolver": {
|
||||
"source": "system | manual | server-recursive",
|
||||
"address": "9.9.9.9", "port": 53,
|
||||
"transport": "do53-udp | do53-tcp | dot | doh",
|
||||
"doh_url": null
|
||||
}
|
||||
```
|
||||
|
||||
The resolver set for a run is: all system resolvers from `LinkProperties` (per network), plus any **manually configured override resolvers** (user-entered, stored per profile or ad-hoc), plus the server's reference recursive resolver. When multiple resolvers are in scope, the full `dns.*` battery is instantiated **once per (test type × resolver) pair** and the instances run concurrently; `dns.compare` then diffs answer sets, TTL behavior, latency, and filtering verdicts across all of them. Override resolvers are tested even if unreachable from the current network (evidence of *that* is itself useful).
|
||||
|
||||
**`dns.ttl_integrity` methodology.** Uses the spec-defined reference records in the canary zone (probe-protocol §6.1), which have fixed, known TTLs (5 s … 7 d):
|
||||
|
||||
1. Cache-miss query (unique nonce label): returned TTL must equal the authoritative TTL → detects clamping (min/max caps) and static rewriting on first answer.
|
||||
2. Re-query after delay *d*: TTL must have decreased by ≈ *d* → detects TTL freezing/reset-on-every-answer (middleboxes that always return the original TTL).
|
||||
3. Query after expiry of a short-TTL record: detects serve-stale and over-caching.
|
||||
4. Evidence records, per resolver and per reference record: authoritative TTL, returned TTLs with query times, computed clamp floor/ceiling estimates.
|
||||
|
||||
**`dns.answer_integrity`.** Compares the full response against spec-defined ground truth for the reference records: RDATA values, record order, flags (AA/RA/AD), case preservation (0x20), EDNS handling, CNAME flattening, stripped/injected additional records, ECS forwarding (visible in server-side canary observations). Any delta is evidence for a finding; TTL deltas are delegated to `dns.ttl_integrity`.
|
||||
|
||||
## 7. Findings, verdicts, summary
|
||||
|
||||
### 7.1 `findings[]`
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "uuidv7",
|
||||
"code": "dns.interception.transparent_redirect", // stable registry, like lint rules
|
||||
"category": "dns", // §7.2 list
|
||||
"severity": "info | low | medium | high | critical",
|
||||
"confidence": "high | medium | low",
|
||||
"network_ref": "net-1",
|
||||
"title": "DNS queries are transparently redirected",
|
||||
"description": "Plain-language explanation of what was observed and why it matters.",
|
||||
"evidence_refs": [
|
||||
{ "test": "test-uuid", "pointer": "/evidence/answers/2" }
|
||||
],
|
||||
"recommendation": "optional plain-language next step"
|
||||
}
|
||||
```
|
||||
|
||||
Finding **codes** are a stable, documented registry (`findings-registry.md`, to be written; grows continuously). A finding with no `evidence_refs` is invalid.
|
||||
|
||||
### 7.2 Categories
|
||||
|
||||
`connectivity`, `dns`, `nat`, `mtu`, `ipv6`, `security`, `performance`, `local`, `wifi`. Fixed in v1; each maps to one traffic light.
|
||||
|
||||
### 7.3 `summary`
|
||||
|
||||
```json
|
||||
{
|
||||
"overall": "green | yellow | red | inconclusive",
|
||||
"categories": {
|
||||
"dns": { "verdict": "red", "worst_finding": "finding-uuid", "tests_run": 9, "tests_failed": 0 },
|
||||
"mtu": { "verdict": "green", "worst_finding": null, "tests_run": 5, "tests_failed": 0 }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Verdict derivation is deterministic and fixed in this spec: category = worst severity among its findings (`critical|high → red`, `medium|low → yellow`, `info/none → green`); `inconclusive` when > 50 % of the category's tests are `failed`/`unsupported`. Overall = worst category, except `inconclusive` only if all are. The UI must always allow drilling from a light to the findings to the raw evidence — no unexplained scores.
|
||||
|
||||
## 8. Logical type registry (anonymization contract)
|
||||
|
||||
The JSON Schema (machine-readable companion, `measurement.schema.json`, generated from this doc) annotates string fields with `x-echolot-type`:
|
||||
|
||||
| type | example fields | v2 anonymizer transform |
|
||||
|---|---|---|
|
||||
| `ip4`, `ip6` | addresses, routes, hops, DNS answers | prefix-preserving pseudonymization, consistent per document; well-known/reserved ranges kept verbatim |
|
||||
| `mac`, `bssid` | wifi, arp_watch | OUI kept, NIC part pseudonymized |
|
||||
| `fqdn` | DNS names, reverse lookups | per-label pseudonyms, public-suffix kept |
|
||||
| `ssid` | wifi | pseudonym |
|
||||
| `opaque-id` | session ids, tokens | redacted |
|
||||
|
||||
Free-text fields (`notes`, `error.detail`, dump excerpts from Shizuku parsers) cannot be safely auto-anonymized; the exporter flags them for manual review.
|
||||
|
||||
## 9. Open items
|
||||
|
||||
1. Findings registry document — start alongside the first implemented tests.
|
||||
2. Whether Shizuku raw-dump excerpts (dumpsys/ip output) are embedded in `evidence` verbatim (auditable, but large and hard to anonymize) or parsed-only with an optional "attach raw dumps" toggle. Proposal: toggle, default on for local archive, default off for export.
|
||||
3. Peer-mode documents: each device produces its own run; the coordinator embeds the peer's findings summary and cross-references by `run.id`. Full merge format deferred.
|
||||
4. Size guardrails: soft cap 20 MB uncompressed per run; trains beyond that downsample evidence (keep aggregates + first/last N + all anomalies) and record `"evidence_truncated": true`.
|
||||
@@ -0,0 +1,207 @@
|
||||
# Echolot Probe Protocol — Spec v1 (draft 1)
|
||||
|
||||
Project name: **Echolot** (decided 2026-07-29; domain: echo-lot.app). Status: draft for review, 2026-07-29. Companion to `measurement-schema.md` (evidence fields reference observations defined here).
|
||||
|
||||
Two planes: a **control plane** (HTTPS + JSON, key-pinned) and a **data plane** (binary UDP probe protocol, TCP/TLS/HTTP echo endpoints, standard STUN, standard DNS on the canary zone).
|
||||
|
||||
## 1. Transport & trust
|
||||
|
||||
- Control plane: HTTPS. The client validates the server certificate **only** against the SPKI pin from enrollment (`pin-sha256`, RFC 7469 computation). CA validation is not required; self-signed is first-class. Key rotation: profile may list `next_pins[]` ahead of rotation.
|
||||
- All control endpoints are under `/v1/`. Version negotiation is by capability list (§2.3), never by sniffing version numbers.
|
||||
- Data-plane authenticity: per-session HMAC key (§3), truncated HMAC on every UDP probe packet.
|
||||
|
||||
Default ports (all configurable): control HTTPS **8443**; UDP probe **8442** (same number on v4 and v6); STUN **3478** (+ alternate address per RFC 5780 — requires a second IP); TCP echo **8441**; canary DNS: standard 53 on the delegated zone.
|
||||
|
||||
## 2. Control plane
|
||||
|
||||
### 2.1 Enrollment
|
||||
|
||||
Bootstrap payload (QR / paste string):
|
||||
|
||||
```
|
||||
echolot://enroll?v=1&u=<control-URL, urlencoded>&p=pin-sha256:<b64 SPKI hash>&t=<enrollment token>
|
||||
```
|
||||
|
||||
```
|
||||
POST /v1/enroll Authorization: Bearer <enrollment-token>
|
||||
→ 200 { "device_credential": "<random 256-bit, b64url>",
|
||||
"device_id": "uuid",
|
||||
"profile": { ... §2.2 ... } }
|
||||
```
|
||||
|
||||
Enrollment tokens are single-use with expiry, created in the admin UI, scoped `enroll`. The device credential is a long-lived bearer secret, scoped `run-tests`; it is also the HKDF input for session keys. Revocation = deleting the device in the admin UI.
|
||||
|
||||
### 2.2 Profile
|
||||
|
||||
```
|
||||
GET /v1/profile Authorization: Bearer <device-credential>
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"profile_version": 7,
|
||||
"name": "homelab",
|
||||
"server_version": "0.4.0",
|
||||
"capabilities": ["udp-probe", "stun-5780", "canary-dns", "connect-back",
|
||||
"delayed-echo", "big-send", "tls-echo", "throughput", "ntp"],
|
||||
"targets": [
|
||||
{ "id": "vie-1", "location": "Vienna",
|
||||
"ip4": "203.0.113.10", "ip6": "2001:db8::10",
|
||||
"udp_port": 8442, "tcp_port": 8441, "stun_port": 3478,
|
||||
"stun_alt": { "ip4": "203.0.113.11", "port": 3478 } }
|
||||
],
|
||||
"canary_zone": "c.probe.example.net",
|
||||
"recursive_resolver": { "ip4": "203.0.113.10", "dot": true, "doh_url": "https://..." },
|
||||
"pins": ["pin-sha256:..."], "next_pins": [],
|
||||
"limits": { "max_kbps": 50000, "max_session_s": 900 }
|
||||
}
|
||||
```
|
||||
|
||||
The app re-fetches the profile at the start of every run (falling back to the cached copy offline) and records `profile_version` in the measurement.
|
||||
|
||||
### 2.3 Capabilities (v1 registry)
|
||||
|
||||
`udp-probe`, `stun-basic`, `stun-5780`, `canary-dns`, `recursive-dns`, `connect-back`, `delayed-echo`, `big-send`, `frag-send`, `tls-echo`, `http-echo`, `throughput`, `ntp`. A server omits what it can't offer (e.g. `stun-5780` without a second IP degrades to `stun-basic`). Clients must skip, and record as `unsupported`, any test whose capability is absent. Unknown capability strings are ignored.
|
||||
|
||||
### 2.4 Sessions
|
||||
|
||||
Every measurement run opens one session per target:
|
||||
|
||||
```
|
||||
POST /v1/sessions Authorization: Bearer <device-credential>
|
||||
{ "target": "vie-1" }
|
||||
→ 201 { "session_id": "<opaque, 8-byte hex prefix used on the wire>",
|
||||
"key_salt": "<b64, 16 bytes>",
|
||||
"epoch": "server wall-clock RFC3339 at session start",
|
||||
"expires_s": 900 }
|
||||
```
|
||||
|
||||
Session key: `HKDF-SHA256(ikm = device_credential, salt = key_salt, info = "echolot-v1/" + session_id)` → 32 bytes. Both sides derive it; it never crosses the wire.
|
||||
|
||||
```
|
||||
GET /v1/sessions/{id}/observations → §6
|
||||
POST /v1/sessions/{id}/actions → §5 (connect-back, delayed echo, big/frag send)
|
||||
DELETE /v1/sessions/{id}
|
||||
```
|
||||
|
||||
### 2.5 Rate limiting & abuse
|
||||
|
||||
Per-credential and per-source-IP token buckets on: session creation, actions, UDP packets, bytes. `429` on control plane; silent drop on data plane (probes must tolerate loss anyway). All reflected/generated traffic goes **only** to the session's observed source address (or, for connect-back, the source address of the session-creating request). Data-plane responses to unauthenticated packets are never larger than the request (§3.4).
|
||||
|
||||
## 3. UDP probe protocol
|
||||
|
||||
### 3.1 Packet header (fixed 32 bytes, network byte order)
|
||||
|
||||
```
|
||||
offset size field
|
||||
0 4 magic "ELT1"
|
||||
4 1 type
|
||||
5 1 flags
|
||||
6 2 payload_len
|
||||
8 8 session_prefix (first 8 bytes of session_id)
|
||||
16 4 seq
|
||||
20 8 t_ns (sender timestamp, ns since session epoch, sender's clock)
|
||||
28 4 hmac32 (first 4 bytes of HMAC-SHA256(session_key, header[0..28] || payload))
|
||||
```
|
||||
|
||||
Padding payload is arbitrary bytes counted by `payload_len`; total datagram size is what MTU/train tests vary. `hmac32` is an anti-abuse gate, not cryptographic integrity for hostile networks — 4 bytes is enough to make blind reflection/replay impractical at the allowed rates; on-path attackers are out of scope for the data plane (tampering is what the *tests* detect). Server drops packets with unknown session prefix, bad HMAC, expired session, or seq replay outside a 1024-wide window.
|
||||
|
||||
### 3.2 Types
|
||||
|
||||
| type | dir | purpose |
|
||||
|---|---|---|
|
||||
| 0x01 ECHO_REQ / 0x02 ECHO_RESP | c→s / s→c | RTT, loss, reordering; RESP carries observation block (§3.3) |
|
||||
| 0x03 TRAIN_DATA | c→s | upstream train, no per-packet response |
|
||||
| 0x04 TRAIN_REPORT_REQ / 0x05 TRAIN_REPORT | c→s / s→c | server's received-view of an upstream train (columnar, may span multiple RESP datagrams) |
|
||||
| 0x06 DOWNTRAIN_DATA | s→c | downstream train (scheduled via §5 action) |
|
||||
| 0x07 TIMESYNC_REQ / 0x08 TIMESYNC_RESP | c→s / s→c | 4-timestamp exchange: client t1 in REQ; RESP carries t2 (rx) and t3 (tx); t4 on receipt. Feeds `time.server_offset` |
|
||||
| 0x09 MTU_PROBE / 0x0A MTU_ACK | c→s / s→c | client sends DF-flagged sizes; ACK reports size received (ACK is small — never amplifies) |
|
||||
| 0x0B DELAYED_ECHO | s→c | single packet sent T seconds after the action request (NAT mapping lifetime) |
|
||||
|
||||
### 3.3 Observation block (in ECHO_RESP, appended per-packet)
|
||||
|
||||
Server reports what it saw on the corresponding request: `t_rx_ns`, `t_tx_ns` (server clock, session epoch), observed source IP + port (detects NAT rebinding mid-flow), received TTL/hop-limit, DSCP, ECN bits, received size. This is the raw material for `train.udp_updown`, `sec.dscp_ecn_survival`, and TTL-based path-length evidence in the measurement schema.
|
||||
|
||||
### 3.4 Anti-amplification rule (normative)
|
||||
|
||||
For any datagram whose HMAC does not verify: no response ever. For verified packets: ECHO_RESP/MTU_ACK/TRAIN_REPORT responses are ≤ request size unless the session is in an **asymmetric grant** created via an authenticated control-plane action (§5), which sets an explicit byte budget and rate. DOWNTRAIN and big/frag sends exist only under such grants.
|
||||
|
||||
## 4. TCP, TLS and HTTP endpoints
|
||||
|
||||
- **TCP echo (8441):** after connect, server sends one JSON line: observed source IP/port, negotiated MSS (from `TCP_INFO`), timestamps/window-scale options seen — this is the `mtu.mss_observed` evidence. Then byte-echo until FIN. TLS variant on the same port via ALPN `elt-echo` (capability `tls-echo`): server additionally returns the ClientHello it received, raw + JA4, before echoing — the `sec.clienthello_echo` evidence.
|
||||
- **HTTP echo:** `POST /v1/echo` on the control listener: returns exact received request bytes (headers + body) base64-wrapped in JSON, plus observed TLS parameters. Detects header injection/stripping/proxying (`sec.http_echo`). A plain-HTTP variant on a configurable port (default off) tests plaintext-path tampering.
|
||||
- **TLS reference:** `GET /v1/tls-reference?host=<name>` returns the full cert chain the server itself serves, DER+base64, so the app can compare an out-of-band copy against what a direct handshake yielded (`sec.tls_reference`).
|
||||
- **STUN:** unmodified RFC 5389/5780 on 3478; second address enables full behavior discovery. No custom framing — interop with existing tooling is a feature.
|
||||
|
||||
## 5. Actions (authenticated asymmetric operations)
|
||||
|
||||
`POST /v1/sessions/{id}/actions` with one of:
|
||||
|
||||
```json
|
||||
{ "action": "downtrain", "count": 1000, "size_bytes": 64, "interval_us": 20000, "dscp": 46 }
|
||||
{ "action": "big_send", "sizes_bytes": [1400, 1472, 1500, 1600, 2000], "df": true }
|
||||
{ "action": "frag_send", "size_bytes": 2000, "family": "4 | 6" }
|
||||
{ "action": "delayed_echo", "delay_s": 30 }
|
||||
{ "action": "connect_back", "protocol": "tcp | udp", "port": 40123 }
|
||||
{ "action": "throughput", "direction": "up | down", "streams": 4, "duration_s": 10 }
|
||||
```
|
||||
|
||||
Rules: destination is always the session's observed source address; every action is bounded by `limits` from the profile; the response includes an `action_id` echoed in resulting data-plane packets (in payload) so evidence can be correlated. `delayed_echo` is the NAT-mapping-lifetime primitive: client binary-searches `delay_s` over repeated actions on fresh sockets.
|
||||
|
||||
## 6. Observations API
|
||||
|
||||
```
|
||||
GET /v1/sessions/{id}/observations
|
||||
```
|
||||
|
||||
Returns everything the server witnessed for this session, merged by the app into measurement evidence:
|
||||
|
||||
```json
|
||||
{
|
||||
"udp": { "packets_seen": ..., "trains": [ { columnar per-seq view } ] },
|
||||
"tcp": [ { "connected_at", "src", "mss", "options": [...] } ],
|
||||
"http": [ { "raw_request_b64", "tls": {...}, "ja4": "..." } ],
|
||||
"dns_canary": [
|
||||
{ "qname": "x7f3a.sess1.c.probe.example.net", "at": "...",
|
||||
"resolver_ip": "198.51.100.7", "resolver_asn": 64500,
|
||||
"transport": "udp", "edns": { "present": true, "bufsize": 1232, "flags": ["do"] },
|
||||
"case_preserved": true, "qname_minimized": false }
|
||||
],
|
||||
"connect_back": [ { "action_id", "result": "connected | refused | timeout", "rtt_ms" } ]
|
||||
}
|
||||
```
|
||||
|
||||
Canary DNS: the app generates `<nonce>.<session-prefix>.<canary_zone>` names and resolves them through the resolver under test (system/provider resolver, user-configured override resolvers, or the server's reference recursive); the authoritative server records who actually asked and how. Query-log retention is a server-config value surfaced in the admin UI (privacy default: 24 h). The `ecs` field in canary observations records any EDNS Client Subnet option the resolver forwarded.
|
||||
|
||||
### 6.1 Reference records (normative, capability `canary-dns`)
|
||||
|
||||
The server MUST serve these fixed records under the canary zone; TTLs and RDATA are defined by this spec (not configurable), so clients have ground truth without a side channel. Used by `dns.ttl_integrity` and `dns.answer_integrity` (measurement schema §6.4).
|
||||
|
||||
| name | type | TTL | RDATA |
|
||||
|---|---|---|---|
|
||||
| `ttl-5.<zone>` | A / AAAA / TXT | 5 | `192.0.2.5` / `2001:db8::5` / `"echolot-ref ttl=5"` |
|
||||
| `ttl-60.<zone>` | A / AAAA / TXT | 60 | `192.0.2.60` / `2001:db8::60` / `"echolot-ref ttl=60"` |
|
||||
| `ttl-3600.<zone>` | A / AAAA / TXT | 3600 | `192.0.2.36` / `2001:db8::3600` / `"echolot-ref ttl=3600"` |
|
||||
| `ttl-86400.<zone>` | A / AAAA / TXT | 86400 | `192.0.2.86` / `2001:db8::8640` / `"echolot-ref ttl=86400"` |
|
||||
| `many-rr.<zone>` | A | 300 | exactly 8 A records in defined order (order/stripping check) |
|
||||
| `big-txt.<zone>` | TXT | 300 | ~1800 bytes (EDNS bufsize / TCP-fallback check) |
|
||||
|
||||
Note: exact RDATA constants to be frozen in the implementation's `dns_reference.go` and mirrored in the app; the table above fixes the *names and TTLs*; RDATA must be deterministic, documentation-range addresses. Cache-miss variants: `<nonce>.miss.<zone>` wildcard answers with TTL 3600 and RDATA encoding the nonce (per-query ground truth that can never be pre-cached).
|
||||
|
||||
## 7. Server admin UI (scope note)
|
||||
|
||||
Same daemon, separate listener (default localhost-only): health + self-test (are both IPs live, is the canary zone delegated correctly, is UDP reachable from outside — tested via a public echolot "mirror" if configured), enrollment token management (create/expire/scope), device list + revocation, retention settings, QR rendering (client-side JS). Out of scope for this spec beyond the endpoints above.
|
||||
|
||||
## 8. Cross-references to the measurement schema
|
||||
|
||||
- Observation-block fields (§3.3) appear as `*_seen_by_server` columns in `train` evidence (schema §6.2).
|
||||
- `TIMESYNC` (§3.2) produces the `time.server_offset` test; without it, cross-clock fields must not be compared (schema §6.2 note).
|
||||
- Capability strings (§2.3) are copied verbatim into `server_sessions[].capabilities` (schema §5); tests skipped for missing capability get `status: "unsupported"`.
|
||||
- `session_id` maps to `server_sessions[].session_id`; `action_id`s appear in test `params`.
|
||||
|
||||
## 9. Open items
|
||||
|
||||
1. Whether TRAIN_REPORT should also stream during long trains (partial reports every N packets) for live UI feedback — leaning yes, same type with a `flags` bit.
|
||||
2. Throughput methodology (fixed streams vs BBR-style ramp) — decide with `perf.*` test design.
|
||||
3. Server mirror/self-test protocol between two echolot servers (nice for operators; postpone).
|
||||
4. IPv6 flow-label control for ECMP-variance tracing — needs `IPV6_FLOWINFO` sockopt verification on Android first (capability-prober item).
|
||||
Reference in New Issue
Block a user