echolot/docs/measurement-schema.md

# Echolot Measurement Schema — Spec v1 (draft 1)

Project name: **Echolot** (decided 2026-07-29; domain: echo-lot.app). Status: draft for review, 2026-07-29.

This document defines the JSON format of one **measurement run** — the unit that is archived on-device, diffed against other runs, and exported/shared. It is the contract between the probe engine, the server's observation API, the report renderer, and (later) the anonymizer.

## 1. Design rules

1. **Observation and interpretation are separated.** `tests[]` contain raw evidence and computed metrics; `findings[]` contain interpretation with references back to evidence. A reader must be able to re-derive every finding from the evidence alone.
2. **Schema-versioned.** `schema_version` is semver. Minor = additive only. Consumers must ignore unknown fields (forward compat) and must not require fields added after their version.
3. **Every value is attributable.** Each test records which network it ran on, which trust tier produced it (`app` / `shizuku` / `root`), and which server session (if any) was involved.
4. **Two clocks.** Wall-clock timestamps are RFC 3339 UTC with millisecond precision, only for human correlation. All measurement math uses `*_mono_ns`: nanoseconds since `run.clock.mono_origin` (an arbitrary per-run epoch from the monotonic clock). Never mix the two.
5. **Units in field names.** `_ms`, `_ns`, `_bytes`, `_bps`, `_pct`. No unitless numbers for physical quantities.
6. **Anonymization-ready.** Every sensitive scalar is declared with a logical type in the schema registry (§8): `ip4`, `ip6`, `mac`, `fqdn`, `ssid`, `bssid`, `imsi-ish`. The v2 anonymizer walks the schema, not the data, and transforms consistently within a document (prefix-preserving for IPs).
7. **Big arrays go columnar.** Packet-train evidence is stored as parallel arrays (§6.2), not arrays of objects — a 10 000-packet train stays in the hundreds of kB.
8. **IDs.** `run.id` and `tests[].id` are UUIDv7 (time-ordered). Cross-references use these IDs plus optional JSON Pointers for precision.

## 2. Top-level document

```json
{
  "schema": "echolot/measurement",
  "schema_version": "1.0.0",
  "run": { ... },              // §3
  "networks": [ ... ],         // §4
  "server_sessions": [ ... ],  // §5
  "tests": [ ... ],            // §6
  "findings": [ ... ],         // §7
  "summary": { ... }           // §7.3
}
```

Export encoding: UTF-8 JSON, gzip for files (`.echolot.json.gz`), share intent uses MIME `application/gzip` plus a rendered HTML report alongside.

## 3. `run` — context of the whole measurement

```json
{
  "id": "0198c5f2-...-uuidv7",
  "trigger": "manual | scheduled | monitor | peer",
  "mode": "short | long",
  "started_at": "2026-07-29T14:03:21.114Z",
  "ended_at": "2026-07-29T14:07:44.902Z",
  "clock": {
    "mono_origin_wall": "2026-07-29T14:03:21.114Z",
    "ntp_offset_ms": -12.4,            // optional, if NTP test ran
    "ntp_offset_source": "test-uuid"   // evidence ref
  },
  "app": { "version": "0.3.1", "build": 310, "git": "a1b2c3d", "flavor": "fdroid" },
  "device": {
    "manufacturer": "Google", "model": "Pixel 8a",
    "android_sdk": 35, "android_release": "15", "security_patch": "2026-06-05"
  },
  "tiers": { "app": true, "shizuku": true, "root": false },
  "profiles_used": ["profile-uuid", ...],
  "constraints": {
    "vpn_active": true,
    "per_network_blocked": true,
    "unmeasured_networks": ["net-0", "net-1"]
  },
  "notes": "free-text user annotation"
}
```

`tiers` records what was *available*; each test records what it *used*.

`mode` records how long the run watched, and it exists because **it changes what a reader may
conclude from absence**. A `short` run is a sequence of one-shot probes — each looks at the network
for a second or two and moves on — which characterises the network's *configuration* well and is
structurally blind to anything intermittent. A `long` run starts continuous listeners at t=0, runs
the same battery beside them, and keeps sampling until its window closes; the window's length is
recorded in the `params` of the tests the listeners produce, not here.

The consequence is asymmetric and matters more than the field looks. A finding is worth the same in
either mode: a drop that was observed, was observed. Silence is not. "No link changes were seen" is
evidence of a stable link after five minutes of watching and is evidence of nothing at all after a
thirty-second run, in which a link could drop and return between two consecutive probes without
leaving a mark anywhere in the document. Consumers — a diff between two runs, a dashboard counting
how often a fault occurs, a person reading one report — must therefore not treat the absence of a
time-dependent finding in a `short` run as its refutation, and must not compare the two modes as if
they had asked the same question. `connectivity.link_flapping` is the first finding that only a
`long` run can reach; `networks[].changes[]` (§4) is likewise populated only by a long run's
listener, and an empty `changes[]` in a short run means "not watched", never "nothing happened".

Absent `mode` means `short`: it was added after the first documents were written, and every one of
them was a battery of one-shot probes.

`constraints` records what was *prevented*. A constrained run is neither a failed run nor a normal
one, and the distinction has to survive into the data: a run taken through a VPN has the same shape
and the same green verdict as a clean run of a healthy network, so without this a reader — or a
server aggregating thousands of them — cannot tell that almost nothing was measured. The known case
is `per_network_blocked`: Android refuses `Network.bindSocket()` on the underlying networks while a
VPN holds the default route, so every per-network test measures the tunnel or nothing at all, and
any conclusion about the link underneath is unfounded. Consumers should treat findings from a
constrained run as scoped to what was actually reachable, and `unmeasured_networks` names the rest.

## 4. `networks[]` — one entry per Android `Network` in play

A run may exercise several networks simultaneously (Wi-Fi + cellular + USB ethernet). Everything is a snapshot at run start; a `changes[]` list captures mid-run deltas.

```json
{
  "id": "net-1",
  "transport": "wifi | cellular | ethernet | vpn | other",
  "interface": "wlan0",
  "link": {
    "mtu": 1500,
    "addresses": [
      { "addr": "192.0.2.23", "prefix_len": 24, "scope": "global",
        "flags": ["temporary"], "valid_lft_s": 3541, "pref_lft_s": 3541 }
    ],
    "routes": [
      { "dst": "0.0.0.0/0", "gateway": "192.0.2.1", "iface": "wlan0",
        "proto": "dhcp | ra | static | unknown", "expires_s": 1799 }
    ],
    "dns": {
      "servers": ["192.0.2.1"],
      "private_dns_mode": "off | opportunistic | strict",
      "private_dns_hostname": null,
      "search_domains": ["lan"],
      "nat64_prefix": null
    },
    "dhcp": { "server": "192.0.2.1", "lease_s": 3600 },
    "captive_portal": { "detected": false, "api_url": null, "venue_url": null }
  },
  "wifi": {
    "ssid": "example-net", "bssid": "aa:bb:cc:dd:ee:ff",
    "rssi_dbm": -54, "link_speed_mbps": 573, "frequency_mhz": 5240,
    "channel_width_mhz": 80, "standard": "11ax",
    "security": "wpa3-sae", "mac_randomization": true
  },
  "cellular": { "rat": "nr-nsa", "operator": "...", "band": "n78" },
  "changes": [
    { "at_mono_ns": 91000000000, "kind": "lost | gained | link_changed",
      "detail": { /* new link snapshot or diff */ } }
  ],
  "app_usable": true
}
```

`routes[].proto` and lifetime fields are Shizuku-tier data (`ip route`/`ip addr`); app-tier snapshots leave them absent — absence means "not observed", never "not present".

`app_usable` records whether an ordinary app may send on this network at all. Android lists the
carrier's special-purpose networks — IMS/VoLTE, MMS, XCAP — alongside the real ones, and they
carry neither `INTERNET` nor `NOT_RESTRICTED`; binding to one needs
`CONNECTIVITY_USE_RESTRICTED_NETWORKS`, which is signature-level and unobtainable for a normal
app. Those networks are therefore permanently unmeasurable, and that is a property of Android's
permission model rather than of the link. They stay in `networks[]` because they are genuinely
present — an interface silently missing from the inventory is its own kind of lie — but a
consumer must not read the absence of tests against them as a fault, and they are **not**
`constraints.unmeasured_networks` (§3): nothing was prevented, the run was never entitled to
measure them.

## 5. `server_sessions[]`

```json
{
  "id": "sess-1",
  "profile_id": "profile-uuid",
  "profile_name": "homelab",
  "control_url": "https://probe.example.net:8443",
  "server_version": "0.4.0",
  "capabilities": ["udp-probe", "stun-5780", "canary-dns", "connect-back", "delayed-echo", "big-send", "tls-echo"],
  "session_id": "opaque-server-issued",
  "target": { "ip4": "203.0.113.10", "ip6": "2001:db8::10", "udp_port": 8442 }
}
```

`capabilities` here records what the server offered *at run time*; test-level `status: "unsupported"` records what was consequently skipped.

## 6. `tests[]` — the generic result envelope

```json
{
  "id": "uuidv7",
  "type": "traceroute.udp4",        // registry, §6.1
  "network_ref": "net-1",
  "session_ref": "sess-1",           // null for purely local tests
  "tier": "app | shizuku | root",
  "started_mono_ns": 12000000,
  "ended_mono_ns": 4530000000,
  "status": "ok | failed | unsupported | skipped | partial",
  "error": { "code": "timeout", "detail": "..." },   // when failed/partial
  "params": { ... },                 // exact inputs, type-specific
  "evidence": { ... },               // raw observations, type-specific
  "metrics": { ... }                 // derived numbers, type-specific
}
```

Rules: `params` must contain everything needed to reproduce the test. `evidence` is append-only raw truth. `metrics` must be recomputable from `evidence` (renderer and diff view use `metrics`; auditors use `evidence`).

### 6.1 Test type registry (v1)

Dotted names, family first. Initial registry; additions are minor version bumps.

| Family | Types |
|---|---|
| `link` | `link.snapshot`, `link.dhcp_renewal_watch`, `link.ip_monitor` (shizuku) |
| `icmp` | `icmp.ping4`, `icmp.ping6` (targets: gateway, first-hop-external, server, public refs) |
| `trace` | `traceroute.udp4`, `traceroute.udp6`, `traceroute.icmp4`, `traceroute.icmp6` |
| `train` | `train.udp_updown` (loss/jitter/reorder/dup, both directions) |
| `mtu` | `mtu.pmtud_up`, `mtu.pmtud_down`, `mtu.blackhole`, `mtu.mss_observed`, `mtu.frag_delivery` |
| `nat` | `nat.stun_5780`, `nat.mapping_lifetime_udp`, `nat.mapping_lifetime_tcp`, `nat.hairpin`, `nat.connect_back`, `nat.cgnat_detect` |
| `dns` | `dns.resolver_inventory`, `dns.canary`, `dns.interception`, `dns.ttl_integrity`, `dns.answer_integrity`, `dns.dnssec`, `dns.nxdomain_wildcard`, `dns.rebind_filter`, `dns.aaaa_filter`, `dns.dns64`, `dns.compare` |
| `sec` | `sec.tls_reference`, `sec.clienthello_echo`, `sec.http_echo`, `sec.sni_filter`, `sec.dscp_ecn_survival`, `sec.arp_watch` (shizuku) |
| `port` | `port.reach_sweep` (outbound), `port.udp_usability` |
| `perf` | `perf.throughput_tcp`, `perf.throughput_udp`, `perf.bufferbloat`, `perf.rrc_latency` |
| `v6` | `v6.dualstack_compare`, `v6.happy_eyeballs`, `v6.brokenness`, `v6.nat64_clat` |
| `wifi` | `wifi.environment_scan`, `wifi.roam_log`, `wifi.signal_log` |
| `local` | `local.mdns_inventory`, `local.ssdp_inventory`, `local.llmnr_inventory`, `local.gateway_services` (UPnP-IGD/NAT-PMP/PCP), `local.ntp` |
| `peer` | `peer.reachability`, `peer.isolation`, `peer.multicast`, `peer.lan_train`, `peer.lease_diff` |
| `time` | `time.server_offset` (4-timestamp exchange, feeds one-way metrics) |

### 6.2 Evidence conventions for packet trains

Columnar parallel arrays, one index per probe packet. Missing observations are `null` at that index.

```json
"evidence": {
  "epoch_mono_ns": 12000000,
  "seq":        [0, 1, 2, 3],
  "t_tx_ns":    [0, 20000000, 40000000, 60000000],   // relative to epoch
  "t_srv_rx_ns":[8123456, 28090000, null, 68240000], // server clock, session epoch
  "t_srv_tx_ns":[8180000, 28150000, null, 68300000],
  "t_rx_ns":    [16500000, 36400000, null, 76800000],
  "size_bytes": [64, 64, 64, 64],
  "dscp_sent": 46, "dscp_seen_by_server": [0, 0, null, 0],
  "ecn_sent": 1,  "ecn_seen_by_server": [1, 1, null, 1],
  "ttl_seen_by_server": [54, 54, null, 54]
}
```

Server-side observations come from the observation API (probe-protocol §6) and are merged in by the app; their clock is the server session epoch — only differences within the same clock are meaningful unless a `time.server_offset` test provides the mapping.

### 6.3 Traceroute evidence

```json
"evidence": {
  "flow": { "src_port": 40123, "dst_port": 8442, "fixed_tuple": true },
  "hops": [
    { "ttl": 1, "probes": [
      { "reply_from": "192.0.2.1", "rtt_ns": 1830000, "icmp": "ttl-exceeded", "reply_ttl": 64 },
      { "reply_from": null,        "rtt_ns": null,    "icmp": null }
    ]}
  ]
}
```

### 6.4 DNS test conventions

**Resolver targeting.** Every `dns.*` test carries a `params.resolver` object identifying the resolver under test:

```json
"resolver": {
  "source": "system | manual | server-recursive",
  "address": "9.9.9.9", "port": 53,
  "transport": "do53-udp | do53-tcp | dot | doh",
  "doh_url": null
}
```

The resolver set for a run is: all system resolvers from `LinkProperties` (per network), plus any **manually configured override resolvers** (user-entered, stored per profile or ad-hoc), plus the server's reference recursive resolver. When multiple resolvers are in scope, the full `dns.*` battery is instantiated **once per (test type × resolver) pair** and the instances run concurrently; `dns.compare` then diffs answer sets, TTL behavior, latency, and filtering verdicts across all of them. Override resolvers are tested even if unreachable from the current network (evidence of *that* is itself useful).

**`dns.ttl_integrity` methodology.** Uses the spec-defined reference records in the canary zone (probe-protocol §6.1), which have fixed, known TTLs (5 s … 7 d):

1. Cache-miss query (unique nonce label): returned TTL must equal the authoritative TTL → detects clamping (min/max caps) and static rewriting on first answer.
2. Re-query after delay *d*: TTL must have decreased by ≈ *d* → detects TTL freezing/reset-on-every-answer (middleboxes that always return the original TTL).
3. Query after expiry of a short-TTL record: detects serve-stale and over-caching.
4. Evidence records, per resolver and per reference record: authoritative TTL, returned TTLs with query times, computed clamp floor/ceiling estimates.

**`dns.answer_integrity`.** Compares the full response against spec-defined ground truth for the reference records: RDATA values, record order, flags (AA/RA/AD), case preservation (0x20), EDNS handling, CNAME flattening, stripped/injected additional records, ECS forwarding (visible in server-side canary observations). Any delta is evidence for a finding; TTL deltas are delegated to `dns.ttl_integrity`.

## 7. Findings, verdicts, summary

### 7.1 `findings[]`

```json
{
  "id": "uuidv7",
  "code": "dns.interception.transparent_redirect",   // stable registry, like lint rules
  "category": "dns",                                  // §7.2 list
  "severity": "info | low | medium | high | critical",
  "confidence": "high | medium | low",
  "network_ref": "net-1",
  "title": "DNS queries are transparently redirected",
  "description": "Plain-language explanation of what was observed and why it matters.",
  "evidence_refs": [
    { "test": "test-uuid", "pointer": "/evidence/answers/2" }
  ],
  "recommendation": "optional plain-language next step"
}
```

Finding **codes** are a stable, documented registry (`findings-registry.md`, to be written; grows continuously). A finding with no `evidence_refs` is invalid.

### 7.2 Categories

`connectivity`, `dns`, `nat`, `mtu`, `ipv6`, `security`, `performance`, `local`, `wifi`. Fixed in v1; each maps to one traffic light.

### 7.3 `summary`

```json
{
  "overall": "green | yellow | red | inconclusive",
  "categories": {
    "dns":  { "verdict": "red", "worst_finding": "finding-uuid", "tests_run": 9, "tests_failed": 0 },
    "mtu":  { "verdict": "green", "worst_finding": null, "tests_run": 5, "tests_failed": 0 }
  }
}
```

Verdict derivation is deterministic and fixed in this spec: category = worst severity among its findings (`critical|high → red`, `medium|low → yellow`, `info/none → green`); `inconclusive` when > 50 % of the category's tests are `failed`/`unsupported`. Overall = worst category, except `inconclusive` only if all are. The UI must always allow drilling from a light to the findings to the raw evidence — no unexplained scores.

## 8. Logical type registry (anonymization contract)

The JSON Schema (machine-readable companion, `measurement.schema.json`, generated from this doc) annotates string fields with `x-echolot-type`:

| type | example fields | v2 anonymizer transform |
|---|---|---|
| `ip4`, `ip6` | addresses, routes, hops, DNS answers | prefix-preserving pseudonymization, consistent per document; well-known/reserved ranges kept verbatim. **Exception: ULA (`fc00::/7`) has its whole prefix pseudonymized as a unit.** It resembles RFC1918 but is not analogous: a ULA global ID is 40 random bits, unique to one network by construction (RFC 4193), so the prefix *is* the identifier, whereas `192.168.0.0/16` is shared by millions of networks and identifies none. Pseudonymizing it as a unit keeps "these hosts are on one subnet" while dropping "this is that subnet". |
| `mac`, `bssid` | wifi, arp_watch | OUI kept, NIC part pseudonymized |
| `fqdn` | DNS names, reverse lookups | per-label pseudonyms, public-suffix kept |
| `ssid` | wifi | pseudonym |
| `opaque-id` | session ids, tokens | redacted |

Free-text fields (`notes`, `error.detail`, dump excerpts from Shizuku parsers) cannot be safely auto-anonymized; the exporter flags them for manual review.

## 9. Open items

1. ~~Findings registry document~~ — done: `findings-registry.md`, kept in step with
   `FindingRegistry.kt` by a test that fails when the two disagree.
2. Whether Shizuku raw-dump excerpts (dumpsys/ip output) are embedded in `evidence` verbatim (auditable, but large and hard to anonymize) or parsed-only with an optional "attach raw dumps" toggle. Proposal: toggle, default on for local archive, default off for export.
3. Peer-mode documents: each device produces its own run; the coordinator embeds the peer's findings summary and cross-references by `run.id`. Full merge format deferred.
4. Size guardrails: soft cap 20 MB uncompressed per run; trains beyond that downsample evidence (keep aggregates + first/last N + all anomalies) and record `"evidence_truncated": true`.