EchoLot/echolot
1
0
Fork 0
Code Pull Requests Actions Packages Projects Releases 31 Wiki Activity

privacy: fix a real leak - global IPv6 addresses were uploaded verbatim

Browse Source 
Setting out to build the machine-readable schema, the first step was checking
whether the anonymizer covers the fields the schema declares sensitive. It did
not, and five identifying values were going out at the `balanced` level:

  networks[].link.addresses[].addr   the device's own global IPv6 address
  networks[].link.routes[].gateway   the ISP allocation
  networks[].link.dns.servers[]      the configured resolver
  private_dns_hostname               an internal hostname
  search_domains[]                   the internal domain

The settings screen describes that level as pseudonymizing addresses.

Root cause: classification keyed on field names, and the schema's actual names
were never added to the table. Every existing test passed, because each checked
a field somebody had remembered to write a case for - an unfalsifiable design
for a privacy control.

So beyond adding the names, classification now falls back to the *value* when
the name is unknown: anything shaped like an IPv4/IPv6 address or a MAC is
treated as one. Hostnames deliberately are not inferred by shape, since
train.udp_updown is indistinguishable from a domain and mangling a test type
would corrupt the document to protect nothing.

LeakTest is the guard, and is written to fail for fields nobody thought of: it
plants identifying values wherever one can occur and asserts none survive. It
also pins that RFC1918 addresses stay readable, so it cannot pass by
over-redacting. Route prefixes and :: needed care - 0.0.0.0/0 must stay itself
or a routing table becomes unreadable for no privacy gain.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
mrambossek
2026-08-01 14:45:52 +02:00
co-authored by Claude Fable 5
parent e7afc2210f
commit 172afb421d
4 changed files with 469 additions and 237 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/build-status.md
+45
View File
@@ -848,3 +848,48 @@ itself to table rows, so the prose can keep explaining which codes were retired
Six tests: uniqueness, declared-vs-listed, prefix↔category agreement, naming convention, a Six tests: uniqueness, declared-vs-listed, prefix↔category agreement, naming convention, a
word-order-anagram check (the shape the duplication actually took), and the document agreement. word-order-anagram check (the shape the duplication actually took), and the document agreement.
### A real privacy leak, found by starting on the machine-readable schema (2026-08-01)
The intent was `measurement.schema.json` (§8's promised companion). The first step — checking
whether the anonymizer actually covers the fields the schema declares as sensitive — found that it
did not, so that became the work.
**At the `balanced` level, five identifying values were being uploaded verbatim:**
| value | field | why it matters |
|---|---|---|
| `2001:…::150` | `networks[].link.addresses[].addr` | the device's own global IPv6 address — a strong, geolocatable device identifier |
| `2a02:…::1` | `networks[].link.routes[].gateway` | identifies the ISP allocation |
| `203.0.113.77` | `networks[].link.dns.servers[]` | the configured resolver |
| `nas.example.lan` | `private_dns_hostname` | an internal hostname |
| `example.lan` | `search_domains[]` | the internal domain |
The settings screen describes that level as pseudonymizing addresses. It was not.
**Root cause:** classification keyed on field *names*, and the schema's actual names (`addr`,
`gateway`, `dst`, `servers`, `search_domains`, `private_dns_hostname`) had never been added to the
table. Not a subtle bug — just an unfalsifiable design. The existing tests all passed, because each
one checked a field somebody had remembered to write a case for.
**Two fixes, one of them structural:**
1. The missing names were added.
2. More importantly, a **shape-based backstop**: when a field name is unrecognised, the *value* is
inspected, and anything shaped like an IPv4/IPv6 address or a MAC is treated as one. A name
table can only protect fields someone thought of, which is precisely the wrong property for a
privacy control. Hostnames are deliberately *not* inferred by shape — `train.udp_updown` is
indistinguishable from a domain, and mangling a test type would corrupt the document to protect
nothing.
`LeakTest` is the new guard and is written to fail for fields nobody has considered: it plants
identifying values wherever one can actually occur and asserts none survive, rather than checking
a list of known cases. It also pins that RFC1918 addresses still come through readable, so the
test cannot pass by over-redacting everything.
Route prefixes and the unspecified address needed care in the transform: `0.0.0.0/0` and `::/0`
must stay themselves, or a routing table becomes unreadable for no privacy gain.
**Still outstanding:** `measurement.schema.json` itself. Worth noting what this episode implies for
it — much of a document's payload lives in `evidence`/`metrics`/`params`, which are per-test-type
`JsonObject` by design and therefore *outside* any schema. A schema-driven anonymizer would have
less coverage there than the name-plus-shape one now does, so the schema should be built for
validation and external tooling, not as a replacement for the classifier.
echolot-app/core-privacy/src/main/kotlin/app/echo_lot/privacy/Anonymizer.kt
+256 -237
View File
@@ -1,237 +1,256 @@
// SPDX-FileCopyrightText: 2026 Echolot contributors // SPDX-FileCopyrightText: 2026 Echolot contributors
// SPDX-License-Identifier: GPL-3.0-or-later // SPDX-License-Identifier: GPL-3.0-or-later
// Package privacy implements the anonymization contract of measurement-schema.md §8. // Package privacy implements the anonymization contract of measurement-schema.md §8.
// //
// The threat model is specific. An engineer running their own server wants the full document — // The threat model is specific. An engineer running their own server wants the full document —
// SSIDs and MACs are what make a run useful a week later. Someone measuring against a stranger's // SSIDs and MACs are what make a run useful a week later. Someone measuring against a stranger's
// server wants the numbers to survive and the identifiers not to. So this is a *transform*, not a // server wants the numbers to survive and the identifiers not to. So this is a *transform*, not a
// filter: the output is still a valid measurement document with the same tests, metrics and // filter: the output is still a valid measurement document with the same tests, metrics and
// findings; only the identifying scalars change, and consistently, so "same SSID as last run" is // findings; only the identifying scalars change, and consistently, so "same SSID as last run" is
// still answerable from pseudonyms alone. // still answerable from pseudonyms alone.
// //
// Two properties are load-bearing and are what the tests pin: // Two properties are load-bearing and are what the tests pin:
// - Consistency within a document: one input value always maps to one pseudonym, so // - Consistency within a document: one input value always maps to one pseudonym, so
// correlations inside a run survive. // correlations inside a run survive.
// - No consistency *across* documents unless the user asks for it: the salt is per-run by // - No consistency *across* documents unless the user asks for it: the salt is per-run by
// default, so pseudonyms cannot be used to track a device between uploads. A stable salt is // default, so pseudonyms cannot be used to track a device between uploads. A stable salt is
// opt-in (`Salt.stable`) for people diffing their own history on their own server. // opt-in (`Salt.stable`) for people diffing their own history on their own server.
package app.echo_lot.privacy package app.echo_lot.privacy
import kotlinx.serialization.json.* import kotlinx.serialization.json.*
import java.security.MessageDigest import java.security.MessageDigest
import java.util.Locale import java.util.Locale
/** How much to strip. Ordered: FULL < BALANCED < STRICT. Wire values match the server's. */ /** How much to strip. Ordered: FULL < BALANCED < STRICT. Wire values match the server's. */
enum class PrivacyLevel(val wire: String) { enum class PrivacyLevel(val wire: String) {
/** Nothing removed. The right choice for your own server. */ /** Nothing removed. The right choice for your own server. */
FULL("full"), FULL("full"),
/** /**
* Identifiers pseudonymized, neighbour inventory dropped. Topology and timing survive: * Identifiers pseudonymized, neighbour inventory dropped. Topology and timing survive:
* you can still see that the gateway is a MikroTik at a /24 boundary with 3 % loss, but not * you can still see that the gateway is a MikroTik at a /24 boundary with 3 % loss, but not
* which MikroTik, on which SSID, next to whose Chromecast. * which MikroTik, on which SSID, next to whose Chromecast.
*/ */
BALANCED("balanced"), BALANCED("balanced"),
/** /**
* Numbers only: tests keep their metrics and status, evidence is dropped, findings keep their * Numbers only: tests keep their metrics and status, evidence is dropped, findings keep their
* codes and severities but lose descriptions (which quote real names). What is left cannot * codes and severities but lose descriptions (which quote real names). What is left cannot
* identify a network, and is still enough for aggregate "how common is this fault" work. * identify a network, and is still enough for aggregate "how common is this fault" work.
*/ */
STRICT("strict"); STRICT("strict");
companion object { companion object {
fun fromWire(s: String?): PrivacyLevel = fun fromWire(s: String?): PrivacyLevel =
entries.firstOrNull { it.wire == s?.lowercase(Locale.ROOT) } ?: FULL entries.firstOrNull { it.wire == s?.lowercase(Locale.ROOT) } ?: FULL
/** The stricter of two levels — used to honour a server's minimum. */ /** The stricter of two levels — used to honour a server's minimum. */
fun max(a: PrivacyLevel, b: PrivacyLevel): PrivacyLevel = if (a.ordinal >= b.ordinal) a else b fun max(a: PrivacyLevel, b: PrivacyLevel): PrivacyLevel = if (a.ordinal >= b.ordinal) a else b
} }
} }
/** /**
* The pseudonymization salt. Per-run by default: a fresh random salt means the same SSID uploaded * The pseudonymization salt. Per-run by default: a fresh random salt means the same SSID uploaded
* twice yields two different pseudonyms, so an upload endpoint cannot link runs to a device. * twice yields two different pseudonyms, so an upload endpoint cannot link runs to a device.
* A stable salt trades that away for cross-run diffing and is only appropriate on a server you * A stable salt trades that away for cross-run diffing and is only appropriate on a server you
* own — the app makes that an explicit choice, not a default. * own — the app makes that an explicit choice, not a default.
*/ */
class Salt private constructor(internal val bytes: ByteArray, val stable: Boolean) { class Salt private constructor(internal val bytes: ByteArray, val stable: Boolean) {
companion object { companion object {
fun perRun(random: ByteArray): Salt = Salt(random.copyOf(), stable = false) fun perRun(random: ByteArray): Salt = Salt(random.copyOf(), stable = false)
fun stable(secret: ByteArray): Salt = Salt(secret.copyOf(), stable = true) fun stable(secret: ByteArray): Salt = Salt(secret.copyOf(), stable = true)
} }
} }
/** /**
* Transforms a measurement document to [level]. * Transforms a measurement document to [level].
* *
* Field classification is by JSON key name, because the schema names things consistently * Field classification is by JSON key name, because the schema names things consistently
* (`ssid`, `bssid`, `mac`, `ip4`, `ip6`, `fqdn`, …) and a name-driven pass is auditable by * (`ssid`, `bssid`, `mac`, `ip4`, `ip6`, `fqdn`, …) and a name-driven pass is auditable by
* reading one table. Anything unrecognized is treated as identifying when it is a string inside * reading one table. Anything unrecognized is treated as identifying when it is a string inside
* a known-sensitive container, and left alone otherwise — see [Classification]. * a known-sensitive container, and left alone otherwise — see [Classification].
*/ */
class Anonymizer(private val level: PrivacyLevel, private val salt: Salt) { class Anonymizer(private val level: PrivacyLevel, private val salt: Salt) {
private val cache = HashMap<String, String>() private val cache = HashMap<String, String>()
fun anonymize(doc: JsonObject): JsonObject { fun anonymize(doc: JsonObject): JsonObject {
if (level == PrivacyLevel.FULL) return stamp(doc) if (level == PrivacyLevel.FULL) return stamp(doc)
val walked = walkObject(doc, path = emptyList()) val walked = walkObject(doc, path = emptyList())
val out = if (level == PrivacyLevel.STRICT) strip(walked) else walked val out = if (level == PrivacyLevel.STRICT) strip(walked) else walked
return stamp(out) return stamp(out)
} }
/** Records what was done, so a reader of the archived/uploaded document is never guessing. */ /** Records what was done, so a reader of the archived/uploaded document is never guessing. */
private fun stamp(doc: JsonObject): JsonObject { private fun stamp(doc: JsonObject): JsonObject {
val run = doc["run"]?.jsonObject ?: return doc val run = doc["run"]?.jsonObject ?: return doc
val privacy = buildJsonObject { val privacy = buildJsonObject {
put("anonymization", level.wire) put("anonymization", level.wire)
put("salt", if (salt.stable) "stable" else "per_run") put("salt", if (salt.stable) "stable" else "per_run")
} }
return JsonObject(doc + ("run" to JsonObject(run + ("privacy" to privacy)))) return JsonObject(doc + ("run" to JsonObject(run + ("privacy" to privacy))))
} }
// ---- the tree walk ------------------------------------------------------------------- // ---- the tree walk -------------------------------------------------------------------
private fun walkObject(obj: JsonObject, path: List<String>): JsonObject = buildJsonObject { private fun walkObject(obj: JsonObject, path: List<String>): JsonObject = buildJsonObject {
for ((k, v) in obj) { for ((k, v) in obj) {
val childPath = path + k val childPath = path + k
when { when {
Classification.dropAtBalanced(childPath) -> Unit // omit entirely Classification.dropAtBalanced(childPath) -> Unit // omit entirely
else -> put(k, walk(k, v, childPath)) else -> put(k, walk(k, v, childPath))
} }
} }
} }
private fun walk(key: String, v: JsonElement, path: List<String>): JsonElement = when (v) { private fun walk(key: String, v: JsonElement, path: List<String>): JsonElement = when (v) {
is JsonObject -> walkObject(v, path) is JsonObject -> walkObject(v, path)
is JsonArray -> JsonArray(v.map { walk(key, it, path) }) is JsonArray -> JsonArray(v.map { walk(key, it, path) })
is JsonPrimitive -> is JsonPrimitive ->
if (v.isString) transform(Classification.typeOf(key, path), v.content).let(::JsonPrimitive) if (v.isString) {
else v // Name first (it is precise), then shape (it is exhaustive). A field nobody
} // classified must not be a field that leaks.
val type = Classification.typeOf(key, path) ?: Classification.inferFromValue(v.content)
private fun transform(type: LogicalType?, value: String): String = when (type) { JsonPrimitive(transform(type, v.content))
null -> value } else {
LogicalType.SSID -> pseudo("ssid", value) { "net-" + it.take(6) } v
LogicalType.MAC, LogicalType.BSSID -> macPreservingOui(value) }
LogicalType.IP4 -> ip4(value) }
LogicalType.IP6 -> ip6(value)
LogicalType.FQDN -> fqdn(value) private fun transform(type: LogicalType?, value: String): String = when (type) {
LogicalType.OPAQUE_ID -> "redacted" null -> value
LogicalType.FREETEXT -> "[removed: may contain identifying text]" LogicalType.SSID -> pseudo("ssid", value) { "net-" + it.take(6) }
} LogicalType.MAC, LogicalType.BSSID -> macPreservingOui(value)
LogicalType.IP4 -> ip4(value)
// ---- per-type transforms ------------------------------------------------------------- LogicalType.IP6 -> ip6(value)
LogicalType.FQDN -> fqdn(value)
/** LogicalType.OPAQUE_ID -> "redacted"
* Keeps the OUI (which vendor) and pseudonymizes the NIC part (which unit). Vendor is the LogicalType.FREETEXT -> "[removed: may contain identifying text]"
* diagnostically valuable half — "the RA comes from a MikroTik" survives, "…from THAT }
* MikroTik" does not.
*/ // ---- per-type transforms -------------------------------------------------------------
private fun macPreservingOui(value: String): String {
val sep = if (value.contains('-')) '-' else ':' /**
val parts = value.split(sep) * Keeps the OUI (which vendor) and pseudonymizes the NIC part (which unit). Vendor is the
if (parts.size != 6 || parts.any { it.length != 2 }) return pseudo("mac", value) { "mac-" + it.take(8) } * diagnostically valuable half — "the RA comes from a MikroTik" survives, "…from THAT
val nic = pseudo("mac", value) { it } * MikroTik" does not.
return (parts.take(3) + listOf(nic.substring(0, 2), nic.substring(2, 4), nic.substring(4, 6))) */
.joinToString(sep.toString()) private fun macPreservingOui(value: String): String {
.lowercase(Locale.ROOT) val sep = if (value.contains('-')) '-' else ':'
} val parts = value.split(sep)
if (parts.size != 6 || parts.any { it.length != 2 }) return pseudo("mac", value) { "mac-" + it.take(8) }
/** val nic = pseudo("mac", value) { it }
* Prefix-preserving within the same class, with reserved ranges kept verbatim: RFC1918 and return (parts.take(3) + listOf(nic.substring(0, 2), nic.substring(2, 4), nic.substring(4, 6)))
* CGNAT addresses say something about the topology and nothing about the person, and a run .joinToString(sep.toString())
* where 192.168.1.1 became a random public address would be actively misleading to read. .lowercase(Locale.ROOT)
* Public addresses keep only their /16 so the network is still locatable at ISP granularity. }
*/
private fun ip4(value: String): String { /**
val o = value.split(".") * Prefix-preserving within the same class, with reserved ranges kept verbatim: RFC1918 and
if (o.size != 4 || o.any { it.toIntOrNull() == null }) return value * CGNAT addresses say something about the topology and nothing about the person, and a run
val n = o.map { it.toInt() } * where 192.168.1.1 became a random public address would be actively misleading to read.
val reserved = n[0] == 10 || * Public addresses keep only their /16 so the network is still locatable at ISP granularity.
(n[0] == 172 && n[1] in 16..31) || */
(n[0] == 192 && n[1] == 168) || private fun ip4(value: String): String {
(n[0] == 169 && n[1] == 254) || // A route destination carries a prefix length; pseudonymize the address and put it back,
(n[0] == 100 && n[1] in 64..127) || // or "0.0.0.0/0" turns into nonsense and the routing table becomes unreadable.
n[0] == 127 || n[0] == 0 || n[0] >= 224 value.substringAfter('/', "").takeIf { it.isNotEmpty() && value.contains('/') }?.let { len ->
if (reserved) return value return ip4(value.substringBefore('/')) + "/" + len
val h = pseudo("ip4", value) { it } }
return "${n[0]}.${n[1]}.${h.substring(0, 2).toInt(16)}.${h.substring(2, 4).toInt(16)}" val o = value.split(".")
} if (o.size != 4 || o.any { it.toIntOrNull() == null }) return value
val n = o.map { it.toInt() }
/** val reserved = n[0] == 10 ||
* IPv6 keeps the scope and the first 32 bits (so 2001:db8:… still reads as global unicast in (n[0] == 172 && n[1] in 16..31) ||
* the same allocation) and pseudonymizes the rest — the interface identifier is the part that (n[0] == 192 && n[1] == 168) ||
* is a device fingerprint, especially with EUI-64. (n[0] == 169 && n[1] == 254) ||
*/ (n[0] == 100 && n[1] in 64..127) ||
private fun ip6(value: String): String { n[0] == 127 || n[0] == 0 || n[0] >= 224
val v = value.lowercase(Locale.ROOT) if (reserved) return value
if (v == "::1" || v == "::" || v.startsWith("fe80:") || v.startsWith("ff")) return v val h = pseudo("ip4", value) { it }
val groups = v.substringBefore('%').split(":") return "${n[0]}.${n[1]}.${h.substring(0, 2).toInt(16)}.${h.substring(2, 4).toInt(16)}"
if (groups.size < 3) return v }
val h = pseudo("ip6", value) { it }
return "${groups[0]}:${groups[1]}:${h.substring(0, 4)}:${h.substring(4, 8)}::${h.substring(8, 12)}" /**
} * IPv6 keeps the scope and the first 32 bits (so 2001:db8:… still reads as global unicast in
* the same allocation) and pseudonymizes the rest — the interface identifier is the part that
/** * is a device fingerprint, especially with EUI-64.
* Per-label pseudonyms with the public suffix kept, so "it resolved somewhere under .local" */
* or "…under example.com" survives without naming the host. The suffix list is deliberately private fun ip6(value: String): String {
* short: guessing wrong keeps *more* pseudonymized, never less. // Dotted quads reach here through the family-agnostic field names (addr, gateway, dst);
*/ // hand them to the IPv4 path rather than mangling them as if they were v6.
private fun fqdn(value: String): String { if (value.count { it == ':' } < 2) return ip4(value)
if (value.isEmpty()) return value if (value.contains('/')) {
val trailing = value.endsWith(".") return ip6(value.substringBefore('/')) + "/" + value.substringAfter('/')
val labels = value.trimEnd('.').split(".") }
if (labels.size == 1) return pseudo("fqdn", value) { "host-" + it.take(6) } val v = value.lowercase(Locale.ROOT)
val keep = if (labels.last() in publicSuffixes) 1 else 0 // The unspecified address and the default route are not identities; mangling them would
val head = labels.dropLast(keep).map { l -> pseudo("label", l) { "l-" + it.take(6) } } // make a routing table unreadable for no privacy gain.
return (head + labels.takeLast(keep)).joinToString(".") + if (trailing) "." else "" if (v == "::1" || v == "::" || v.startsWith("fe80:") || v.startsWith("ff")) return v
} val groups = v.substringBefore('%').split(":")
if (groups.size < 3) return v
// ---- STRICT --------------------------------------------------------------------------- val h = pseudo("ip6", value) { it }
return "${groups[0]}:${groups[1]}:${h.substring(0, 4)}:${h.substring(4, 8)}::${h.substring(8, 12)}"
/** }
* STRICT keeps the shape of the document and the numbers, and nothing that quotes the
* network back. Evidence goes (trains carry addresses and hostnames), finding prose goes /**
* (it interpolates real names), networks go entirely. * Per-label pseudonyms with the public suffix kept, so "it resolved somewhere under .local"
*/ * or "…under example.com" survives without naming the host. The suffix list is deliberately
private fun strip(doc: JsonObject): JsonObject = buildJsonObject { * short: guessing wrong keeps *more* pseudonymized, never less.
for ((k, v) in doc) { */
when (k) { private fun fqdn(value: String): String {
"networks", "server_sessions" -> Unit if (value.isEmpty()) return value
"tests" -> put(k, JsonArray((v as? JsonArray ?: JsonArray(emptyList())).map { t -> val trailing = value.endsWith(".")
val o = t.jsonObject val labels = value.trimEnd('.').split(".")
JsonObject(o.filterKeys { it != "evidence" && it != "params" }) if (labels.size == 1) return pseudo("fqdn", value) { "host-" + it.take(6) }
})) val keep = if (labels.last() in publicSuffixes) 1 else 0
"findings" -> put(k, JsonArray((v as? JsonArray ?: JsonArray(emptyList())).map { f -> val head = labels.dropLast(keep).map { l -> pseudo("label", l) { "l-" + it.take(6) } }
val o = f.jsonObject return (head + labels.takeLast(keep)).joinToString(".") + if (trailing) "." else ""
JsonObject(o.filterKeys { it != "description" && it != "title" && it != "evidence_refs" }) }
}))
"run" -> put(k, JsonObject(v.jsonObject.filterKeys { it != "notes" })) // ---- STRICT ---------------------------------------------------------------------------
else -> put(k, v)
} /**
} * STRICT keeps the shape of the document and the numbers, and nothing that quotes the
} * network back. Evidence goes (trains carry addresses and hostnames), finding prose goes
* (it interpolates real names), networks go entirely.
// ---- pseudonym machinery --------------------------------------------------------------- */
private fun strip(doc: JsonObject): JsonObject = buildJsonObject {
/** Deterministic per (domain, value, salt); memoized so one value maps to one pseudonym. */ for ((k, v) in doc) {
private fun pseudo(domain: String, value: String, shape: (String) -> String): String = when (k) {
cache.getOrPut("$domain$value") { "networks", "server_sessions" -> Unit
val md = MessageDigest.getInstance("SHA-256") "tests" -> put(k, JsonArray((v as? JsonArray ?: JsonArray(emptyList())).map { t ->
md.update(salt.bytes) val o = t.jsonObject
md.update(domain.toByteArray()) JsonObject(o.filterKeys { it != "evidence" && it != "params" })
md.update(0) }))
md.update(value.lowercase(Locale.ROOT).toByteArray()) "findings" -> put(k, JsonArray((v as? JsonArray ?: JsonArray(emptyList())).map { f ->
shape(md.digest().joinToString("") { "%02x".format(it) }) val o = f.jsonObject
} JsonObject(o.filterKeys { it != "description" && it != "title" && it != "evidence_refs" })
}))
private companion object { "run" -> put(k, JsonObject(v.jsonObject.filterKeys { it != "notes" }))
val publicSuffixes = setOf( else -> put(k, v)
"local", "lan", "home", "internal", "arpa", }
"com", "net", "org", "io", "app", "dev", "at", "de", "eu", "uk", }
) }
}
} // ---- pseudonym machinery ---------------------------------------------------------------
/** Deterministic per (domain, value, salt); memoized so one value maps to one pseudonym. */
private fun pseudo(domain: String, value: String, shape: (String) -> String): String =
cache.getOrPut("$domain$value") {
val md = MessageDigest.getInstance("SHA-256")
md.update(salt.bytes)
md.update(domain.toByteArray())
md.update(0)
md.update(value.lowercase(Locale.ROOT).toByteArray())
shape(md.digest().joinToString("") { "%02x".format(it) })
}
private companion object {
val publicSuffixes = setOf(
"local", "lan", "home", "internal", "arpa",
"com", "net", "org", "io", "app", "dev", "at", "de", "eu", "uk",
)
}
}
echolot-app/core-privacy/src/main/kotlin/app/echo_lot/privacy/Classification.kt
+55
View File
@@ -32,6 +32,16 @@ object Classification {
"link_local", "ra_source", "prefix", "link_local", "ra_source", "prefix",
).forEach { put(it, LogicalType.IP6) } ).forEach { put(it, LogicalType.IP6) }
// Family-agnostic address fields — the names the models actually use (Address.addr,
// Route.gateway, Route.dst, DnsConfig.servers). Their absence here was a real leak: the
// device's own global IPv6 address went out verbatim at the level whose description
// promises addresses are pseudonymized. Typed IP6 because the transform detects the
// family from the value, falling through to the IPv4 path for a dotted quad.
listOf(
"addr", "address", "gateway", "dst", "src", "servers", "server", "resolver",
"next_hop", "via", "public_ip", "observed_ip",
).forEach { put(it, LogicalType.IP6) }
listOf("mac", "hw_addr", "gateway_mac", "router_mac", "sender_mac", "peer_mac") listOf("mac", "hw_addr", "gateway_mac", "router_mac", "sender_mac", "peer_mac")
.forEach { put(it, LogicalType.MAC) } .forEach { put(it, LogicalType.MAC) }
listOf("bssid", "ap_mac").forEach { put(it, LogicalType.BSSID) } listOf("bssid", "ap_mac").forEach { put(it, LogicalType.BSSID) }
@@ -40,6 +50,8 @@ object Classification {
listOf( listOf(
"fqdn", "hostname", "host", "name", "reverse_dns", "ptr", "domain", "query_name", "fqdn", "hostname", "host", "name", "reverse_dns", "ptr", "domain", "query_name",
"friendly_name", "server_name", "sni", "cname", "search_domain", "device_name", "friendly_name", "server_name", "sni", "cname", "search_domain", "device_name",
// Plural and prefixed variants the models actually use.
"search_domains", "private_dns_hostname", "domains", "hostnames",
).forEach { put(it, LogicalType.FQDN) } ).forEach { put(it, LogicalType.FQDN) }
listOf("session_id", "credential", "token", "device_id", "android_id", "serial", "imsi", "iccid") listOf("session_id", "credential", "token", "device_id", "android_id", "serial", "imsi", "iccid")
@@ -78,6 +90,49 @@ object Classification {
return null return null
} }
/**
* Last-resort classification from the *value*, when the field name is unrecognised.
*
* A name table can only protect fields somebody remembered to add, which is the wrong
* property for a privacy control: the dangerous field is the one nobody thought of. This
* exists because that failed once already — `addresses[].addr` holds the device's own global
* IPv6 address, the table had never heard of the name, and it went out verbatim.
*
* Only addresses and MACs are inferred, because only those have shapes that cannot be
* mistaken for something else. Hostnames deliberately are not: `train.udp_updown` is
* indistinguishable from a domain by shape, and mangling a test type would corrupt the
* document to protect nothing.
*/
fun inferFromValue(value: String): LogicalType? {
val v = value.trim()
if (v.isEmpty() || v.length > 64) return null
if (looksLikeMac(v)) return LogicalType.MAC
if (looksLikeIp6(v)) return LogicalType.IP6
if (looksLikeIp4(v)) return LogicalType.IP4
return null
}
private fun isHex(c: Char) = c in '0'..'9' || c in 'a'..'f' || c in 'A'..'F'
private fun looksLikeMac(v: String): Boolean {
val parts = v.split(':', '-')
return parts.size == 6 && parts.all { p -> p.length == 2 && p.all(::isHex) }
}
private fun looksLikeIp4(v: String): Boolean {
val parts = v.substringBefore('/').split('.')
return parts.size == 4 && parts.all { p ->
p.isNotEmpty() && p.length <= 3 && p.all(Char::isDigit) && p.toInt() <= 255
}
}
private fun looksLikeIp6(v: String): Boolean {
val core = v.substringBefore('/').substringBefore('%')
// Two colons minimum, so a time or a MAC fragment does not qualify, and nothing but the
// characters an address may contain.
return core.count { it == ':' } >= 2 && core.all { it == ':' || isHex(it) }
}
fun dropAtBalanced(path: List<String>): Boolean { fun dropAtBalanced(path: List<String>): Boolean {
if (path.isNotEmpty() && path.last() in droppedKeys) return true if (path.isNotEmpty() && path.last() in droppedKeys) return true
return droppedPaths.any { dropped -> dropped.all { path.contains(it) } } return droppedPaths.any { dropped -> dropped.all { path.contains(it) } }
echolot-app/core-privacy/src/test/kotlin/app/echo_lot/privacy/LeakTest.kt
+113
View File
@@ -0,0 +1,113 @@
// SPDX-FileCopyrightText: 2026 Echolot contributors
// SPDX-License-Identifier: GPL-3.0-or-later
package app.echo_lot.privacy
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.jsonObject
import kotlin.test.Test
import kotlin.test.assertTrue
/**
* The blunt instrument: build a document with identifying values in every place one can actually
* occur, anonymize it, and assert none of them survive.
*
* [AnonymizerTest] checks that the fields the classification table knows about are handled
* correctly. This checks the other half — the fields it does *not* know about. A per-field test
* can only fail for a field someone remembered to write a case for, which is exactly the wrong
* property for a privacy check: the dangerous field is the one nobody thought of.
*
* Concretely, this is written the way it is because the schema's own field names disagree with
* the classifier's. `Address.addr` carries an IP and is documented as such in
* measurement-schema.md §8, but the classifier keys on names like `ip4` and `gateway_ip4` and had
* never heard of `addr`.
*/
class LeakTest {
private val json = Json { prettyPrint = false }
private val salt = Salt.perRun(ByteArray(32) { 3 })
/**
* Every string here is something that identifies a person, a household or a device, placed
* where the real models actually put it (`core-measurement`'s Network/Link/Address/DnsConfig).
*/
private val secrets = listOf(
"Rambossek WLAN", // ssid
"78:9a:18:aa:bb:cc", // bssid
"aa:bb:cc:dd:ee:11", // gateway mac
"2001:1ad0:c4fe:6767::150", // global v6 address on the interface
"2a02:1748:dead:beef::1", // v6 default gateway
"203.0.113.77", // public v4
"nas.rambossek.lan", // private-dns hostname
"rambossek.lan", // search domain
"Anna's Chromecast", // neighbour name
"kitchen table", // free-text note
)
private fun document(): String = """
{
"schema": "echolot/measurement",
"run": {
"id": "run-1", "trigger": "manual", "notes": "${secrets[9]}",
"device": {"manufacturer": "OnePlus", "model": "CPH2747"}
},
"networks": [{
"id": "net-1", "transport": "wifi",
"link": {
"mtu": 1500,
"addresses": [
{"addr": "${secrets[3]}", "prefix_len": 64, "scope": "global"},
{"addr": "192.168.1.44", "prefix_len": 24, "scope": "global"}
],
"routes": [
{"dst": "::/0", "gateway": "${secrets[4]}", "iface": "wlan0"},
{"dst": "0.0.0.0/0", "gateway": "192.168.1.1", "iface": "wlan0"}
],
"dns": {
"servers": ["${secrets[5]}", "192.168.1.1"],
"private_dns_hostname": "${secrets[6]}",
"search_domains": ["${secrets[7]}"]
}
},
"wifi": {"ssid": "${secrets[0]}", "bssid": "${secrets[1]}"},
"neighbors": [{"name": "${secrets[8]}", "mac": "${secrets[2]}"}]
}],
"tests": [{"id": "t1", "type": "train.udp_updown", "status": "ok",
"metrics": {"rtt_ms_avg": 12.4}}],
"findings": [],
"summary": {"verdict": "ok"}
}
""".trimIndent()
private fun anonymized(level: PrivacyLevel): String =
json.encodeToString(
kotlinx.serialization.json.JsonObject.serializer(),
Anonymizer(level, salt).anonymize(json.parseToJsonElement(document()).jsonObject),
)
@Test
fun nothingIdentifyingSurvivesBalanced() {
val out = anonymized(PrivacyLevel.BALANCED)
val leaked = secrets.filter { out.contains(it) }
assertTrue(
leaked.isEmpty(),
"these identifying values were uploaded verbatim at BALANCED: $leaked\n\n$out",
)
}
@Test
fun nothingIdentifyingSurvivesStrict() {
val out = anonymized(PrivacyLevel.STRICT)
val leaked = secrets.filter { out.contains(it) }
assertTrue(leaked.isEmpty(), "leaked at STRICT: $leaked\n\n$out")
}
// Private addresses are kept on purpose — they describe the topology and not the person — so
// this pins that the leak test above is not passing by accident of over-redaction.
@Test
fun privateAddressesAreStillReadable() {
val out = anonymized(PrivacyLevel.BALANCED)
assertTrue(out.contains("192.168.1.1"), "RFC1918 gateway should survive: $out")
assertTrue(out.contains("192.168.1.44"), "RFC1918 interface address should survive: $out")
}
}