mrambossekandClaude Fable 5
e7afc2210f
A finding code is the stable half of a result - what a dashboard groups by and
what someone greps a year of archived runs for. That only holds if a code means
exactly one thing forever, which fifteen ad-hoc string literals cannot promise.
By the time this was written the failure had happened twice:
- Two emitters independently produced connectivity.downstream_loss and
connectivity.loss_downstream for the same claim. Nothing objected. Anyone
aggregating either would have silently seen half their data.
- Two codes sat under nat.* while being declared Category.CONNECTIVITY.
nat.udp_unreachable is not about NAT, and the prefix decides the category,
which decides which verdict light the finding rolls up into. Renamed while
that is still cheap.
Codes are now typed FindingSpecs carrying category and default severity;
emitters reference the spec rather than retyping the string, so a typo is a
compile error and two call sites cannot disagree about a finding's category.
docs/findings-registry.md is the contract and a test reads it, failing when the
document and the code disagree on which codes exist or how severe they are.
Documentation that drifts from its implementation is worse than none, because it
still looks authoritative. The check reads table rows only, so the prose can go
on explaining which codes were retired and why.
Closes open item 1 of measurement-schema.md section 9.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
287 lines
15 KiB
Markdown
287 lines
15 KiB
Markdown
# Echolot Measurement Schema — Spec v1 (draft 1)
|
||
|
||
Project name: **Echolot** (decided 2026-07-29; domain: echo-lot.app). Status: draft for review, 2026-07-29.
|
||
|
||
This document defines the JSON format of one **measurement run** — the unit that is archived on-device, diffed against other runs, and exported/shared. It is the contract between the probe engine, the server's observation API, the report renderer, and (later) the anonymizer.
|
||
|
||
## 1. Design rules
|
||
|
||
1. **Observation and interpretation are separated.** `tests[]` contain raw evidence and computed metrics; `findings[]` contain interpretation with references back to evidence. A reader must be able to re-derive every finding from the evidence alone.
|
||
2. **Schema-versioned.** `schema_version` is semver. Minor = additive only. Consumers must ignore unknown fields (forward compat) and must not require fields added after their version.
|
||
3. **Every value is attributable.** Each test records which network it ran on, which trust tier produced it (`app` / `shizuku` / `root`), and which server session (if any) was involved.
|
||
4. **Two clocks.** Wall-clock timestamps are RFC 3339 UTC with millisecond precision, only for human correlation. All measurement math uses `*_mono_ns`: nanoseconds since `run.clock.mono_origin` (an arbitrary per-run epoch from the monotonic clock). Never mix the two.
|
||
5. **Units in field names.** `_ms`, `_ns`, `_bytes`, `_bps`, `_pct`. No unitless numbers for physical quantities.
|
||
6. **Anonymization-ready.** Every sensitive scalar is declared with a logical type in the schema registry (§8): `ip4`, `ip6`, `mac`, `fqdn`, `ssid`, `bssid`, `imsi-ish`. The v2 anonymizer walks the schema, not the data, and transforms consistently within a document (prefix-preserving for IPs).
|
||
7. **Big arrays go columnar.** Packet-train evidence is stored as parallel arrays (§6.2), not arrays of objects — a 10 000-packet train stays in the hundreds of kB.
|
||
8. **IDs.** `run.id` and `tests[].id` are UUIDv7 (time-ordered). Cross-references use these IDs plus optional JSON Pointers for precision.
|
||
|
||
## 2. Top-level document
|
||
|
||
```json
|
||
{
|
||
"schema": "echolot/measurement",
|
||
"schema_version": "1.0.0",
|
||
"run": { ... }, // §3
|
||
"networks": [ ... ], // §4
|
||
"server_sessions": [ ... ], // §5
|
||
"tests": [ ... ], // §6
|
||
"findings": [ ... ], // §7
|
||
"summary": { ... } // §7.3
|
||
}
|
||
```
|
||
|
||
Export encoding: UTF-8 JSON, gzip for files (`.echolot.json.gz`), share intent uses MIME `application/gzip` plus a rendered HTML report alongside.
|
||
|
||
## 3. `run` — context of the whole measurement
|
||
|
||
```json
|
||
{
|
||
"id": "0198c5f2-...-uuidv7",
|
||
"trigger": "manual | scheduled | monitor | peer",
|
||
"started_at": "2026-07-29T14:03:21.114Z",
|
||
"ended_at": "2026-07-29T14:07:44.902Z",
|
||
"clock": {
|
||
"mono_origin_wall": "2026-07-29T14:03:21.114Z",
|
||
"ntp_offset_ms": -12.4, // optional, if NTP test ran
|
||
"ntp_offset_source": "test-uuid" // evidence ref
|
||
},
|
||
"app": { "version": "0.3.1", "build": 310, "git": "a1b2c3d", "flavor": "fdroid" },
|
||
"device": {
|
||
"manufacturer": "Google", "model": "Pixel 8a",
|
||
"android_sdk": 35, "android_release": "15", "security_patch": "2026-06-05"
|
||
},
|
||
"tiers": { "app": true, "shizuku": true, "root": false },
|
||
"profiles_used": ["profile-uuid", ...],
|
||
"notes": "free-text user annotation"
|
||
}
|
||
```
|
||
|
||
`tiers` records what was *available*; each test records what it *used*.
|
||
|
||
## 4. `networks[]` — one entry per Android `Network` in play
|
||
|
||
A run may exercise several networks simultaneously (Wi-Fi + cellular + USB ethernet). Everything is a snapshot at run start; a `changes[]` list captures mid-run deltas.
|
||
|
||
```json
|
||
{
|
||
"id": "net-1",
|
||
"transport": "wifi | cellular | ethernet | vpn | other",
|
||
"interface": "wlan0",
|
||
"link": {
|
||
"mtu": 1500,
|
||
"addresses": [
|
||
{ "addr": "192.0.2.23", "prefix_len": 24, "scope": "global",
|
||
"flags": ["temporary"], "valid_lft_s": 3541, "pref_lft_s": 3541 }
|
||
],
|
||
"routes": [
|
||
{ "dst": "0.0.0.0/0", "gateway": "192.0.2.1", "iface": "wlan0",
|
||
"proto": "dhcp | ra | static | unknown", "expires_s": 1799 }
|
||
],
|
||
"dns": {
|
||
"servers": ["192.0.2.1"],
|
||
"private_dns_mode": "off | opportunistic | strict",
|
||
"private_dns_hostname": null,
|
||
"search_domains": ["lan"],
|
||
"nat64_prefix": null
|
||
},
|
||
"dhcp": { "server": "192.0.2.1", "lease_s": 3600 },
|
||
"captive_portal": { "detected": false, "api_url": null, "venue_url": null }
|
||
},
|
||
"wifi": {
|
||
"ssid": "example-net", "bssid": "aa:bb:cc:dd:ee:ff",
|
||
"rssi_dbm": -54, "link_speed_mbps": 573, "frequency_mhz": 5240,
|
||
"channel_width_mhz": 80, "standard": "11ax",
|
||
"security": "wpa3-sae", "mac_randomization": true
|
||
},
|
||
"cellular": { "rat": "nr-nsa", "operator": "...", "band": "n78" },
|
||
"changes": [
|
||
{ "at_mono_ns": 91000000000, "kind": "lost | gained | link_changed",
|
||
"detail": { /* new link snapshot or diff */ } }
|
||
]
|
||
}
|
||
```
|
||
|
||
`routes[].proto` and lifetime fields are Shizuku-tier data (`ip route`/`ip addr`); app-tier snapshots leave them absent — absence means "not observed", never "not present".
|
||
|
||
## 5. `server_sessions[]`
|
||
|
||
```json
|
||
{
|
||
"id": "sess-1",
|
||
"profile_id": "profile-uuid",
|
||
"profile_name": "homelab",
|
||
"control_url": "https://probe.example.net:8443",
|
||
"server_version": "0.4.0",
|
||
"capabilities": ["udp-probe", "stun-5780", "canary-dns", "connect-back", "delayed-echo", "big-send", "tls-echo"],
|
||
"session_id": "opaque-server-issued",
|
||
"target": { "ip4": "203.0.113.10", "ip6": "2001:db8::10", "udp_port": 8442 }
|
||
}
|
||
```
|
||
|
||
`capabilities` here records what the server offered *at run time*; test-level `status: "unsupported"` records what was consequently skipped.
|
||
|
||
## 6. `tests[]` — the generic result envelope
|
||
|
||
```json
|
||
{
|
||
"id": "uuidv7",
|
||
"type": "traceroute.udp4", // registry, §6.1
|
||
"network_ref": "net-1",
|
||
"session_ref": "sess-1", // null for purely local tests
|
||
"tier": "app | shizuku | root",
|
||
"started_mono_ns": 12000000,
|
||
"ended_mono_ns": 4530000000,
|
||
"status": "ok | failed | unsupported | skipped | partial",
|
||
"error": { "code": "timeout", "detail": "..." }, // when failed/partial
|
||
"params": { ... }, // exact inputs, type-specific
|
||
"evidence": { ... }, // raw observations, type-specific
|
||
"metrics": { ... } // derived numbers, type-specific
|
||
}
|
||
```
|
||
|
||
Rules: `params` must contain everything needed to reproduce the test. `evidence` is append-only raw truth. `metrics` must be recomputable from `evidence` (renderer and diff view use `metrics`; auditors use `evidence`).
|
||
|
||
### 6.1 Test type registry (v1)
|
||
|
||
Dotted names, family first. Initial registry; additions are minor version bumps.
|
||
|
||
| Family | Types |
|
||
|---|---|
|
||
| `link` | `link.snapshot`, `link.dhcp_renewal_watch`, `link.ip_monitor` (shizuku) |
|
||
| `icmp` | `icmp.ping4`, `icmp.ping6` (targets: gateway, first-hop-external, server, public refs) |
|
||
| `trace` | `traceroute.udp4`, `traceroute.udp6`, `traceroute.icmp4`, `traceroute.icmp6` |
|
||
| `train` | `train.udp_updown` (loss/jitter/reorder/dup, both directions) |
|
||
| `mtu` | `mtu.pmtud_up`, `mtu.pmtud_down`, `mtu.blackhole`, `mtu.mss_observed`, `mtu.frag_delivery` |
|
||
| `nat` | `nat.stun_5780`, `nat.mapping_lifetime_udp`, `nat.mapping_lifetime_tcp`, `nat.hairpin`, `nat.connect_back`, `nat.cgnat_detect` |
|
||
| `dns` | `dns.resolver_inventory`, `dns.canary`, `dns.interception`, `dns.ttl_integrity`, `dns.answer_integrity`, `dns.dnssec`, `dns.nxdomain_wildcard`, `dns.rebind_filter`, `dns.aaaa_filter`, `dns.dns64`, `dns.compare` |
|
||
| `sec` | `sec.tls_reference`, `sec.clienthello_echo`, `sec.http_echo`, `sec.sni_filter`, `sec.dscp_ecn_survival`, `sec.arp_watch` (shizuku) |
|
||
| `port` | `port.reach_sweep` (outbound), `port.udp_usability` |
|
||
| `perf` | `perf.throughput_tcp`, `perf.throughput_udp`, `perf.bufferbloat`, `perf.rrc_latency` |
|
||
| `v6` | `v6.dualstack_compare`, `v6.happy_eyeballs`, `v6.brokenness`, `v6.nat64_clat` |
|
||
| `wifi` | `wifi.environment_scan`, `wifi.roam_log`, `wifi.signal_log` |
|
||
| `local` | `local.mdns_inventory`, `local.ssdp_inventory`, `local.llmnr_inventory`, `local.gateway_services` (UPnP-IGD/NAT-PMP/PCP), `local.ntp` |
|
||
| `peer` | `peer.reachability`, `peer.isolation`, `peer.multicast`, `peer.lan_train`, `peer.lease_diff` |
|
||
| `time` | `time.server_offset` (4-timestamp exchange, feeds one-way metrics) |
|
||
|
||
### 6.2 Evidence conventions for packet trains
|
||
|
||
Columnar parallel arrays, one index per probe packet. Missing observations are `null` at that index.
|
||
|
||
```json
|
||
"evidence": {
|
||
"epoch_mono_ns": 12000000,
|
||
"seq": [0, 1, 2, 3],
|
||
"t_tx_ns": [0, 20000000, 40000000, 60000000], // relative to epoch
|
||
"t_srv_rx_ns":[8123456, 28090000, null, 68240000], // server clock, session epoch
|
||
"t_srv_tx_ns":[8180000, 28150000, null, 68300000],
|
||
"t_rx_ns": [16500000, 36400000, null, 76800000],
|
||
"size_bytes": [64, 64, 64, 64],
|
||
"dscp_sent": 46, "dscp_seen_by_server": [0, 0, null, 0],
|
||
"ecn_sent": 1, "ecn_seen_by_server": [1, 1, null, 1],
|
||
"ttl_seen_by_server": [54, 54, null, 54]
|
||
}
|
||
```
|
||
|
||
Server-side observations come from the observation API (probe-protocol §6) and are merged in by the app; their clock is the server session epoch — only differences within the same clock are meaningful unless a `time.server_offset` test provides the mapping.
|
||
|
||
### 6.3 Traceroute evidence
|
||
|
||
```json
|
||
"evidence": {
|
||
"flow": { "src_port": 40123, "dst_port": 8442, "fixed_tuple": true },
|
||
"hops": [
|
||
{ "ttl": 1, "probes": [
|
||
{ "reply_from": "192.0.2.1", "rtt_ns": 1830000, "icmp": "ttl-exceeded", "reply_ttl": 64 },
|
||
{ "reply_from": null, "rtt_ns": null, "icmp": null }
|
||
]}
|
||
]
|
||
}
|
||
```
|
||
|
||
### 6.4 DNS test conventions
|
||
|
||
**Resolver targeting.** Every `dns.*` test carries a `params.resolver` object identifying the resolver under test:
|
||
|
||
```json
|
||
"resolver": {
|
||
"source": "system | manual | server-recursive",
|
||
"address": "9.9.9.9", "port": 53,
|
||
"transport": "do53-udp | do53-tcp | dot | doh",
|
||
"doh_url": null
|
||
}
|
||
```
|
||
|
||
The resolver set for a run is: all system resolvers from `LinkProperties` (per network), plus any **manually configured override resolvers** (user-entered, stored per profile or ad-hoc), plus the server's reference recursive resolver. When multiple resolvers are in scope, the full `dns.*` battery is instantiated **once per (test type × resolver) pair** and the instances run concurrently; `dns.compare` then diffs answer sets, TTL behavior, latency, and filtering verdicts across all of them. Override resolvers are tested even if unreachable from the current network (evidence of *that* is itself useful).
|
||
|
||
**`dns.ttl_integrity` methodology.** Uses the spec-defined reference records in the canary zone (probe-protocol §6.1), which have fixed, known TTLs (5 s … 7 d):
|
||
|
||
1. Cache-miss query (unique nonce label): returned TTL must equal the authoritative TTL → detects clamping (min/max caps) and static rewriting on first answer.
|
||
2. Re-query after delay *d*: TTL must have decreased by ≈ *d* → detects TTL freezing/reset-on-every-answer (middleboxes that always return the original TTL).
|
||
3. Query after expiry of a short-TTL record: detects serve-stale and over-caching.
|
||
4. Evidence records, per resolver and per reference record: authoritative TTL, returned TTLs with query times, computed clamp floor/ceiling estimates.
|
||
|
||
**`dns.answer_integrity`.** Compares the full response against spec-defined ground truth for the reference records: RDATA values, record order, flags (AA/RA/AD), case preservation (0x20), EDNS handling, CNAME flattening, stripped/injected additional records, ECS forwarding (visible in server-side canary observations). Any delta is evidence for a finding; TTL deltas are delegated to `dns.ttl_integrity`.
|
||
|
||
## 7. Findings, verdicts, summary
|
||
|
||
### 7.1 `findings[]`
|
||
|
||
```json
|
||
{
|
||
"id": "uuidv7",
|
||
"code": "dns.interception.transparent_redirect", // stable registry, like lint rules
|
||
"category": "dns", // §7.2 list
|
||
"severity": "info | low | medium | high | critical",
|
||
"confidence": "high | medium | low",
|
||
"network_ref": "net-1",
|
||
"title": "DNS queries are transparently redirected",
|
||
"description": "Plain-language explanation of what was observed and why it matters.",
|
||
"evidence_refs": [
|
||
{ "test": "test-uuid", "pointer": "/evidence/answers/2" }
|
||
],
|
||
"recommendation": "optional plain-language next step"
|
||
}
|
||
```
|
||
|
||
Finding **codes** are a stable, documented registry (`findings-registry.md`, to be written; grows continuously). A finding with no `evidence_refs` is invalid.
|
||
|
||
### 7.2 Categories
|
||
|
||
`connectivity`, `dns`, `nat`, `mtu`, `ipv6`, `security`, `performance`, `local`, `wifi`. Fixed in v1; each maps to one traffic light.
|
||
|
||
### 7.3 `summary`
|
||
|
||
```json
|
||
{
|
||
"overall": "green | yellow | red | inconclusive",
|
||
"categories": {
|
||
"dns": { "verdict": "red", "worst_finding": "finding-uuid", "tests_run": 9, "tests_failed": 0 },
|
||
"mtu": { "verdict": "green", "worst_finding": null, "tests_run": 5, "tests_failed": 0 }
|
||
}
|
||
}
|
||
```
|
||
|
||
Verdict derivation is deterministic and fixed in this spec: category = worst severity among its findings (`critical|high → red`, `medium|low → yellow`, `info/none → green`); `inconclusive` when > 50 % of the category's tests are `failed`/`unsupported`. Overall = worst category, except `inconclusive` only if all are. The UI must always allow drilling from a light to the findings to the raw evidence — no unexplained scores.
|
||
|
||
## 8. Logical type registry (anonymization contract)
|
||
|
||
The JSON Schema (machine-readable companion, `measurement.schema.json`, generated from this doc) annotates string fields with `x-echolot-type`:
|
||
|
||
| type | example fields | v2 anonymizer transform |
|
||
|---|---|---|
|
||
| `ip4`, `ip6` | addresses, routes, hops, DNS answers | prefix-preserving pseudonymization, consistent per document; well-known/reserved ranges kept verbatim |
|
||
| `mac`, `bssid` | wifi, arp_watch | OUI kept, NIC part pseudonymized |
|
||
| `fqdn` | DNS names, reverse lookups | per-label pseudonyms, public-suffix kept |
|
||
| `ssid` | wifi | pseudonym |
|
||
| `opaque-id` | session ids, tokens | redacted |
|
||
|
||
Free-text fields (`notes`, `error.detail`, dump excerpts from Shizuku parsers) cannot be safely auto-anonymized; the exporter flags them for manual review.
|
||
|
||
## 9. Open items
|
||
|
||
1. ~~Findings registry document~~ — done: `findings-registry.md`, kept in step with
|
||
`FindingRegistry.kt` by a test that fails when the two disagree.
|
||
2. Whether Shizuku raw-dump excerpts (dumpsys/ip output) are embedded in `evidence` verbatim (auditable, but large and hard to anonymize) or parsed-only with an optional "attach raw dumps" toggle. Proposal: toggle, default on for local archive, default off for export.
|
||
3. Peer-mode documents: each device produces its own run; the coordinator embeds the peer's findings summary and cross-references by `run.id`. Full merge format deferred.
|
||
4. Size guardrails: soft cap 20 MB uncompressed per run; trains beyond that downsample evidence (keep aggregates + first/last N + all anomalies) and record `"evidence_truncated": true`.
|