echolot/docs/measurement-schema.md

Echolot Measurement Schema — Spec v1 (draft 1)

Project name: Echolot (decided 2026-07-29; domain: echo-lot.app). Status: draft for review, 2026-07-29.

This document defines the JSON format of one measurement run — the unit that is archived on-device, diffed against other runs, and exported/shared. It is the contract between the probe engine, the server's observation API, the report renderer, and (later) the anonymizer.

1. Design rules

1. Observation and interpretation are separated. tests[] contain raw evidence and computed metrics; findings[] contain interpretation with references back to evidence. A reader must be able to re-derive every finding from the evidence alone.
2. Schema-versioned. schema_version is semver. Minor = additive only. Consumers must ignore unknown fields (forward compat) and must not require fields added after their version.
3. Every value is attributable. Each test records which network it ran on, which trust tier produced it (app / shizuku / root), and which server session (if any) was involved.
4. Two clocks. Wall-clock timestamps are RFC 3339 UTC with millisecond precision, only for human correlation. All measurement math uses *_mono_ns: nanoseconds since run.clock.mono_origin (an arbitrary per-run epoch from the monotonic clock). Never mix the two.
5. Units in field names. _ms, _ns, _bytes, _bps, _pct. No unitless numbers for physical quantities.
6. Anonymization-ready. Every sensitive scalar is declared with a logical type in the schema registry (§8): ip4, ip6, mac, fqdn, ssid, bssid, imsi-ish. The v2 anonymizer walks the schema, not the data, and transforms consistently within a document (prefix-preserving for IPs).
7. Big arrays go columnar. Packet-train evidence is stored as parallel arrays (§6.2), not arrays of objects — a 10 000-packet train stays in the hundreds of kB.
8. IDs. run.id and tests[].id are UUIDv7 (time-ordered). Cross-references use these IDs plus optional JSON Pointers for precision.

2. Top-level document

{
  "schema": "echolot/measurement",
  "schema_version": "1.0.0",
  "run": { ... },              // §3
  "networks": [ ... ],         // §4
  "server_sessions": [ ... ],  // §5
  "tests": [ ... ],            // §6
  "findings": [ ... ],         // §7
  "summary": { ... }           // §7.3
}

Export encoding: UTF-8 JSON, gzip for files (.echolot.json.gz), share intent uses MIME application/gzip plus a rendered HTML report alongside.

3. run — context of the whole measurement

{
  "id": "0198c5f2-...-uuidv7",
  "trigger": "manual | scheduled | monitor | peer",
  "mode": "short | long",
  "started_at": "2026-07-29T14:03:21.114Z",
  "ended_at": "2026-07-29T14:07:44.902Z",
  "clock": {
    "mono_origin_wall": "2026-07-29T14:03:21.114Z",
    "ntp_offset_ms": -12.4,            // optional, if NTP test ran
    "ntp_offset_source": "test-uuid"   // evidence ref
  },
  "app": { "version": "0.3.1", "build": 310, "git": "a1b2c3d", "flavor": "fdroid" },
  "device": {
    "manufacturer": "Google", "model": "Pixel 8a",
    "android_sdk": 35, "android_release": "15", "security_patch": "2026-06-05"
  },
  "tiers": { "app": true, "shizuku": true, "root": false },
  "profiles_used": ["profile-uuid", ...],
  "constraints": {
    "vpn_active": true,
    "per_network_blocked": true,
    "unmeasured_networks": ["net-0", "net-1"]
  },
  "notes": "free-text user annotation"
}

tiers records what was available; each test records what it used.

mode records how long the run watched, and it exists because it changes what a reader may conclude from absence. A short run is a sequence of one-shot probes — each looks at the network for a second or two and moves on — which characterises the network's configuration well and is structurally blind to anything intermittent. A long run starts continuous listeners at t=0, runs the same battery beside them, and keeps sampling until its window closes; the window's length is recorded in the params of the tests the listeners produce, not here.

The consequence is asymmetric and matters more than the field looks. A finding is worth the same in either mode: a drop that was observed, was observed. Silence is not. "No link changes were seen" is evidence of a stable link after five minutes of watching and is evidence of nothing at all after a thirty-second run, in which a link could drop and return between two consecutive probes without leaving a mark anywhere in the document. Consumers — a diff between two runs, a dashboard counting how often a fault occurs, a person reading one report — must therefore not treat the absence of a time-dependent finding in a short run as its refutation, and must not compare the two modes as if they had asked the same question. connectivity.link_flapping is the first finding that only a long run can reach; networks[].changes[] (§4) is likewise populated only by a long run's listener, and an empty changes[] in a short run means "not watched", never "nothing happened".

Absent mode means short: it was added after the first documents were written, and every one of them was a battery of one-shot probes.

constraints records what was prevented. A constrained run is neither a failed run nor a normal one, and the distinction has to survive into the data: a run taken through a VPN has the same shape and the same green verdict as a clean run of a healthy network, so without this a reader — or a server aggregating thousands of them — cannot tell that almost nothing was measured. The known case is per_network_blocked: Android refuses Network.bindSocket() on the underlying networks while a VPN holds the default route, so every per-network test measures the tunnel or nothing at all, and any conclusion about the link underneath is unfounded. Consumers should treat findings from a constrained run as scoped to what was actually reachable, and unmeasured_networks names the rest.

4. networks[] — one entry per Android Network in play

A run may exercise several networks simultaneously (Wi-Fi + cellular + USB ethernet). Everything is a snapshot at run start; a changes[] list captures mid-run deltas.

{
  "id": "net-1",
  "transport": "wifi | cellular | ethernet | vpn | other",
  "interface": "wlan0",
  "link": {
    "mtu": 1500,
    "addresses": [
      { "addr": "192.0.2.23", "prefix_len": 24, "scope": "global",
        "flags": ["temporary"], "valid_lft_s": 3541, "pref_lft_s": 3541 }
    ],
    "routes": [
      { "dst": "0.0.0.0/0", "gateway": "192.0.2.1", "iface": "wlan0",
        "proto": "dhcp | ra | static | unknown", "expires_s": 1799 }
    ],
    "dns": {
      "servers": ["192.0.2.1"],
      "private_dns_mode": "off | opportunistic | strict",
      "private_dns_hostname": null,
      "search_domains": ["lan"],
      "nat64_prefix": null
    },
    "dhcp": { "server": "192.0.2.1", "lease_s": 3600 },
    "captive_portal": { "detected": false, "api_url": null, "venue_url": null }
  },
  "wifi": {
    "ssid": "example-net", "bssid": "aa:bb:cc:dd:ee:ff",
    "rssi_dbm": -54, "link_speed_mbps": 573, "frequency_mhz": 5240,
    "channel_width_mhz": 80, "standard": "11ax",
    "security": "wpa3-sae", "mac_randomization": true
  },
  "cellular": { "rat": "nr-nsa", "operator": "...", "band": "n78" },
  "changes": [
    { "at_mono_ns": 91000000000, "kind": "lost | gained | link_changed",
      "detail": { /* new link snapshot or diff */ } }
  ],
  "app_usable": true
}

routes[].proto and lifetime fields are Shizuku-tier data (ip route/ip addr); app-tier snapshots leave them absent — absence means "not observed", never "not present".

app_usable records whether an ordinary app may send on this network at all. Android lists the carrier's special-purpose networks — IMS/VoLTE, MMS, XCAP — alongside the real ones, and they carry neither INTERNET nor NOT_RESTRICTED; binding to one needs CONNECTIVITY_USE_RESTRICTED_NETWORKS, which is signature-level and unobtainable for a normal app. Those networks are therefore permanently unmeasurable, and that is a property of Android's permission model rather than of the link. They stay in networks[] because they are genuinely present — an interface silently missing from the inventory is its own kind of lie — but a consumer must not read the absence of tests against them as a fault, and they are not constraints.unmeasured_networks (§3): nothing was prevented, the run was never entitled to measure them.

5. server_sessions[]

{
  "id": "sess-1",
  "profile_id": "profile-uuid",
  "profile_name": "homelab",
  "control_url": "https://probe.example.net:8443",
  "server_version": "0.4.0",
  "capabilities": ["udp-probe", "stun-5780", "canary-dns", "connect-back", "delayed-echo", "big-send", "tls-echo"],
  "session_id": "opaque-server-issued",
  "target": { "ip4": "203.0.113.10", "ip6": "2001:db8::10", "udp_port": 8442 }
}

capabilities here records what the server offered at run time; test-level status: "unsupported" records what was consequently skipped.

6. tests[] — the generic result envelope

{
  "id": "uuidv7",
  "type": "traceroute.udp4",        // registry, §6.1
  "network_ref": "net-1",
  "session_ref": "sess-1",           // null for purely local tests
  "tier": "app | shizuku | root",
  "started_mono_ns": 12000000,
  "ended_mono_ns": 4530000000,
  "status": "ok | failed | unsupported | skipped | partial",
  "error": { "code": "timeout", "detail": "..." },   // when failed/partial
  "params": { ... },                 // exact inputs, type-specific
  "evidence": { ... },               // raw observations, type-specific
  "metrics": { ... }                 // derived numbers, type-specific
}

Rules: params must contain everything needed to reproduce the test. evidence is append-only raw truth. metrics must be recomputable from evidence (renderer and diff view use metrics; auditors use evidence).

6.1 Test type registry (v1)

Dotted names, family first. Initial registry; additions are minor version bumps.

Family	Types
link	link.snapshot, link.dhcp_renewal_watch, link.ip_monitor (shizuku)
icmp	icmp.ping4, icmp.ping6 (targets: gateway, first-hop-external, server, public refs)
trace	traceroute.udp4, traceroute.udp6, traceroute.icmp4, traceroute.icmp6
train	train.udp_updown (loss/jitter/reorder/dup, both directions)
mtu	mtu.pmtud_up, mtu.pmtud_down, mtu.blackhole, mtu.mss_observed, mtu.frag_delivery
nat	nat.stun_5780, nat.mapping_lifetime_udp, nat.mapping_lifetime_tcp, nat.hairpin, nat.connect_back, nat.cgnat_detect
dns	dns.resolver_inventory, dns.canary, dns.interception, dns.ttl_integrity, dns.answer_integrity, dns.dnssec, dns.nxdomain_wildcard, dns.rebind_filter, dns.aaaa_filter, dns.dns64, dns.compare
sec	sec.tls_reference, sec.clienthello_echo, sec.http_echo, sec.sni_filter, sec.dscp_ecn_survival, sec.arp_watch (shizuku)
port	port.reach_sweep (outbound), port.udp_usability
perf	perf.throughput_tcp, perf.throughput_udp, perf.bufferbloat, perf.rrc_latency
v6	v6.dualstack_compare, v6.happy_eyeballs, v6.brokenness, v6.nat64_clat
wifi	wifi.environment_scan, wifi.roam_log, wifi.signal_log
local	local.mdns_inventory, local.ssdp_inventory, local.llmnr_inventory, local.gateway_services (UPnP-IGD/NAT-PMP/PCP), local.ntp
peer	peer.reachability, peer.isolation, peer.multicast, peer.lan_train, peer.lease_diff
time	time.server_offset (4-timestamp exchange, feeds one-way metrics)

6.2 Evidence conventions for packet trains

Columnar parallel arrays, one index per probe packet. Missing observations are null at that index.

"evidence": {
  "epoch_mono_ns": 12000000,
  "seq":        [0, 1, 2, 3],
  "t_tx_ns":    [0, 20000000, 40000000, 60000000],   // relative to epoch
  "t_srv_rx_ns":[8123456, 28090000, null, 68240000], // server clock, session epoch
  "t_srv_tx_ns":[8180000, 28150000, null, 68300000],
  "t_rx_ns":    [16500000, 36400000, null, 76800000],
  "size_bytes": [64, 64, 64, 64],
  "dscp_sent": 46, "dscp_seen_by_server": [0, 0, null, 0],
  "ecn_sent": 1,  "ecn_seen_by_server": [1, 1, null, 1],
  "ttl_seen_by_server": [54, 54, null, 54]
}

Server-side observations come from the observation API (probe-protocol §6) and are merged in by the app; their clock is the server session epoch — only differences within the same clock are meaningful unless a time.server_offset test provides the mapping.

6.3 Traceroute evidence

"evidence": {
  "flow": { "src_port": 40123, "dst_port": 8442, "fixed_tuple": true },
  "hops": [
    { "ttl": 1, "probes": [
      { "reply_from": "192.0.2.1", "rtt_ns": 1830000, "icmp": "ttl-exceeded", "reply_ttl": 64 },
      { "reply_from": null,        "rtt_ns": null,    "icmp": null }
    ]}
  ]
}

6.4 DNS test conventions

Resolver targeting. Every dns.* test carries a params.resolver object identifying the resolver under test:

"resolver": {
  "source": "system | manual | server-recursive",
  "address": "9.9.9.9", "port": 53,
  "transport": "do53-udp | do53-tcp | dot | doh",
  "doh_url": null
}

The resolver set for a run is: all system resolvers from LinkProperties (per network), plus any manually configured override resolvers (user-entered, stored per profile or ad-hoc), plus the server's reference recursive resolver. When multiple resolvers are in scope, the full dns.* battery is instantiated once per (test type × resolver) pair and the instances run concurrently; dns.compare then diffs answer sets, TTL behavior, latency, and filtering verdicts across all of them. Override resolvers are tested even if unreachable from the current network (evidence of that is itself useful).

dns.ttl_integrity methodology. Uses the spec-defined reference records in the canary zone (probe-protocol §6.1), which have fixed, known TTLs (5 s … 7 d):

1. Cache-miss query (unique nonce label): returned TTL must equal the authoritative TTL → detects clamping (min/max caps) and static rewriting on first answer.
2. Re-query after delay d: TTL must have decreased by ≈ d → detects TTL freezing/reset-on-every-answer (middleboxes that always return the original TTL).
3. Query after expiry of a short-TTL record: detects serve-stale and over-caching.
4. Evidence records, per resolver and per reference record: authoritative TTL, returned TTLs with query times, computed clamp floor/ceiling estimates.

dns.answer_integrity. Compares the full response against spec-defined ground truth for the reference records: RDATA values, record order, flags (AA/RA/AD), case preservation (0x20), EDNS handling, CNAME flattening, stripped/injected additional records, ECS forwarding (visible in server-side canary observations). Any delta is evidence for a finding; TTL deltas are delegated to dns.ttl_integrity.

7. Findings, verdicts, summary

7.1 findings[]

{
  "id": "uuidv7",
  "code": "dns.interception.transparent_redirect",   // stable registry, like lint rules
  "category": "dns",                                  // §7.2 list
  "severity": "info | low | medium | high | critical",
  "confidence": "high | medium | low",
  "network_ref": "net-1",
  "title": "DNS queries are transparently redirected",
  "description": "Plain-language explanation of what was observed and why it matters.",
  "evidence_refs": [
    { "test": "test-uuid", "pointer": "/evidence/answers/2" }
  ],
  "recommendation": "optional plain-language next step"
}

Finding codes are a stable, documented registry (findings-registry.md, to be written; grows continuously). A finding with no evidence_refs is invalid.

7.2 Categories

connectivity, dns, nat, mtu, ipv6, security, performance, local, wifi. Fixed in v1; each maps to one traffic light.

7.3 summary

{
  "overall": "green | yellow | red | inconclusive",
  "categories": {
    "dns":  { "verdict": "red", "worst_finding": "finding-uuid", "tests_run": 9, "tests_failed": 0 },
    "mtu":  { "verdict": "green", "worst_finding": null, "tests_run": 5, "tests_failed": 0 }
  }
}

Verdict derivation is deterministic and fixed in this spec: category = worst severity among its findings (critical|high → red, medium|low → yellow, info/none → green); inconclusive when > 50 % of the category's tests are failed/unsupported. Overall = worst category, except inconclusive only if all are. The UI must always allow drilling from a light to the findings to the raw evidence — no unexplained scores.

8. Logical type registry (anonymization contract)

The JSON Schema (machine-readable companion, measurement.schema.json, generated from this doc) annotates string fields with x-echolot-type:

type	example fields	v2 anonymizer transform
ip4, ip6	addresses, routes, hops, DNS answers	prefix-preserving pseudonymization, consistent per document; well-known/reserved ranges kept verbatim. Exception: ULA (fc00::/7) has its whole prefix pseudonymized as a unit. It resembles RFC1918 but is not analogous: a ULA global ID is 40 random bits, unique to one network by construction (RFC 4193), so the prefix is the identifier, whereas 192.168.0.0/16 is shared by millions of networks and identifies none. Pseudonymizing it as a unit keeps "these hosts are on one subnet" while dropping "this is that subnet".
mac, bssid	wifi, arp_watch	OUI kept, NIC part pseudonymized
fqdn	DNS names, reverse lookups	per-label pseudonyms, public-suffix kept
ssid	wifi	pseudonym
opaque-id	session ids, tokens	redacted

Free-text fields (notes, error.detail, dump excerpts from Shizuku parsers) cannot be safely auto-anonymized; the exporter flags them for manual review.

9. Open items

1. Findings registry document — done: findings-registry.md, kept in step with FindingRegistry.kt by a test that fails when the two disagree.
2. Whether Shizuku raw-dump excerpts (dumpsys/ip output) are embedded in evidence verbatim (auditable, but large and hard to anonymize) or parsed-only with an optional "attach raw dumps" toggle. Proposal: toggle, default on for local archive, default off for export.
3. Peer-mode documents: each device produces its own run; the coordinator embeds the peer's findings summary and cross-references by run.id. Full merge format deferred.
4. Size guardrails: soft cap 20 MB uncompressed per run; trains beyond that downsample evidence (keep aggregates + first/last N + all anomalies) and record "evidence_truncated": true.