privacy: fix a real leak - global IPv6 addresses were uploaded verbatim
Setting out to build the machine-readable schema, the first step was checking whether the anonymizer covers the fields the schema declares sensitive. It did not, and five identifying values were going out at the `balanced` level: networks[].link.addresses[].addr the device's own global IPv6 address networks[].link.routes[].gateway the ISP allocation networks[].link.dns.servers[] the configured resolver private_dns_hostname an internal hostname search_domains[] the internal domain The settings screen describes that level as pseudonymizing addresses. Root cause: classification keyed on field names, and the schema's actual names were never added to the table. Every existing test passed, because each checked a field somebody had remembered to write a case for - an unfalsifiable design for a privacy control. So beyond adding the names, classification now falls back to the *value* when the name is unknown: anything shaped like an IPv4/IPv6 address or a MAC is treated as one. Hostnames deliberately are not inferred by shape, since train.udp_updown is indistinguishable from a domain and mangling a test type would corrupt the document to protect nothing. LeakTest is the guard, and is written to fail for fields nobody thought of: it plants identifying values wherever one can occur and asserts none survive. It also pins that RFC1918 addresses stay readable, so it cannot pass by over-redacting. Route prefixes and :: needed care - 0.0.0.0/0 must stay itself or a routing table becomes unreadable for no privacy gain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
mrambossek
co-authored by
Claude Fable 5
Claude Fable 5
parent
e7afc2210f
commit
172afb421d
@@ -1,237 +1,256 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
// Package privacy implements the anonymization contract of measurement-schema.md §8.
|
||||
//
|
||||
// The threat model is specific. An engineer running their own server wants the full document —
|
||||
// SSIDs and MACs are what make a run useful a week later. Someone measuring against a stranger's
|
||||
// server wants the numbers to survive and the identifiers not to. So this is a *transform*, not a
|
||||
// filter: the output is still a valid measurement document with the same tests, metrics and
|
||||
// findings; only the identifying scalars change, and consistently, so "same SSID as last run" is
|
||||
// still answerable from pseudonyms alone.
|
||||
//
|
||||
// Two properties are load-bearing and are what the tests pin:
|
||||
// - Consistency within a document: one input value always maps to one pseudonym, so
|
||||
// correlations inside a run survive.
|
||||
// - No consistency *across* documents unless the user asks for it: the salt is per-run by
|
||||
// default, so pseudonyms cannot be used to track a device between uploads. A stable salt is
|
||||
// opt-in (`Salt.stable`) for people diffing their own history on their own server.
|
||||
package app.echo_lot.privacy
|
||||
|
||||
import kotlinx.serialization.json.*
|
||||
import java.security.MessageDigest
|
||||
import java.util.Locale
|
||||
|
||||
/** How much to strip. Ordered: FULL < BALANCED < STRICT. Wire values match the server's. */
|
||||
enum class PrivacyLevel(val wire: String) {
|
||||
/** Nothing removed. The right choice for your own server. */
|
||||
FULL("full"),
|
||||
|
||||
/**
|
||||
* Identifiers pseudonymized, neighbour inventory dropped. Topology and timing survive:
|
||||
* you can still see that the gateway is a MikroTik at a /24 boundary with 3 % loss, but not
|
||||
* which MikroTik, on which SSID, next to whose Chromecast.
|
||||
*/
|
||||
BALANCED("balanced"),
|
||||
|
||||
/**
|
||||
* Numbers only: tests keep their metrics and status, evidence is dropped, findings keep their
|
||||
* codes and severities but lose descriptions (which quote real names). What is left cannot
|
||||
* identify a network, and is still enough for aggregate "how common is this fault" work.
|
||||
*/
|
||||
STRICT("strict");
|
||||
|
||||
companion object {
|
||||
fun fromWire(s: String?): PrivacyLevel =
|
||||
entries.firstOrNull { it.wire == s?.lowercase(Locale.ROOT) } ?: FULL
|
||||
|
||||
/** The stricter of two levels — used to honour a server's minimum. */
|
||||
fun max(a: PrivacyLevel, b: PrivacyLevel): PrivacyLevel = if (a.ordinal >= b.ordinal) a else b
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The pseudonymization salt. Per-run by default: a fresh random salt means the same SSID uploaded
|
||||
* twice yields two different pseudonyms, so an upload endpoint cannot link runs to a device.
|
||||
* A stable salt trades that away for cross-run diffing and is only appropriate on a server you
|
||||
* own — the app makes that an explicit choice, not a default.
|
||||
*/
|
||||
class Salt private constructor(internal val bytes: ByteArray, val stable: Boolean) {
|
||||
companion object {
|
||||
fun perRun(random: ByteArray): Salt = Salt(random.copyOf(), stable = false)
|
||||
fun stable(secret: ByteArray): Salt = Salt(secret.copyOf(), stable = true)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Transforms a measurement document to [level].
|
||||
*
|
||||
* Field classification is by JSON key name, because the schema names things consistently
|
||||
* (`ssid`, `bssid`, `mac`, `ip4`, `ip6`, `fqdn`, …) and a name-driven pass is auditable by
|
||||
* reading one table. Anything unrecognized is treated as identifying when it is a string inside
|
||||
* a known-sensitive container, and left alone otherwise — see [Classification].
|
||||
*/
|
||||
class Anonymizer(private val level: PrivacyLevel, private val salt: Salt) {
|
||||
|
||||
private val cache = HashMap<String, String>()
|
||||
|
||||
fun anonymize(doc: JsonObject): JsonObject {
|
||||
if (level == PrivacyLevel.FULL) return stamp(doc)
|
||||
val walked = walkObject(doc, path = emptyList())
|
||||
val out = if (level == PrivacyLevel.STRICT) strip(walked) else walked
|
||||
return stamp(out)
|
||||
}
|
||||
|
||||
/** Records what was done, so a reader of the archived/uploaded document is never guessing. */
|
||||
private fun stamp(doc: JsonObject): JsonObject {
|
||||
val run = doc["run"]?.jsonObject ?: return doc
|
||||
val privacy = buildJsonObject {
|
||||
put("anonymization", level.wire)
|
||||
put("salt", if (salt.stable) "stable" else "per_run")
|
||||
}
|
||||
return JsonObject(doc + ("run" to JsonObject(run + ("privacy" to privacy))))
|
||||
}
|
||||
|
||||
// ---- the tree walk -------------------------------------------------------------------
|
||||
|
||||
private fun walkObject(obj: JsonObject, path: List<String>): JsonObject = buildJsonObject {
|
||||
for ((k, v) in obj) {
|
||||
val childPath = path + k
|
||||
when {
|
||||
Classification.dropAtBalanced(childPath) -> Unit // omit entirely
|
||||
else -> put(k, walk(k, v, childPath))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun walk(key: String, v: JsonElement, path: List<String>): JsonElement = when (v) {
|
||||
is JsonObject -> walkObject(v, path)
|
||||
is JsonArray -> JsonArray(v.map { walk(key, it, path) })
|
||||
is JsonPrimitive ->
|
||||
if (v.isString) transform(Classification.typeOf(key, path), v.content).let(::JsonPrimitive)
|
||||
else v
|
||||
}
|
||||
|
||||
private fun transform(type: LogicalType?, value: String): String = when (type) {
|
||||
null -> value
|
||||
LogicalType.SSID -> pseudo("ssid", value) { "net-" + it.take(6) }
|
||||
LogicalType.MAC, LogicalType.BSSID -> macPreservingOui(value)
|
||||
LogicalType.IP4 -> ip4(value)
|
||||
LogicalType.IP6 -> ip6(value)
|
||||
LogicalType.FQDN -> fqdn(value)
|
||||
LogicalType.OPAQUE_ID -> "redacted"
|
||||
LogicalType.FREETEXT -> "[removed: may contain identifying text]"
|
||||
}
|
||||
|
||||
// ---- per-type transforms -------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Keeps the OUI (which vendor) and pseudonymizes the NIC part (which unit). Vendor is the
|
||||
* diagnostically valuable half — "the RA comes from a MikroTik" survives, "…from THAT
|
||||
* MikroTik" does not.
|
||||
*/
|
||||
private fun macPreservingOui(value: String): String {
|
||||
val sep = if (value.contains('-')) '-' else ':'
|
||||
val parts = value.split(sep)
|
||||
if (parts.size != 6 || parts.any { it.length != 2 }) return pseudo("mac", value) { "mac-" + it.take(8) }
|
||||
val nic = pseudo("mac", value) { it }
|
||||
return (parts.take(3) + listOf(nic.substring(0, 2), nic.substring(2, 4), nic.substring(4, 6)))
|
||||
.joinToString(sep.toString())
|
||||
.lowercase(Locale.ROOT)
|
||||
}
|
||||
|
||||
/**
|
||||
* Prefix-preserving within the same class, with reserved ranges kept verbatim: RFC1918 and
|
||||
* CGNAT addresses say something about the topology and nothing about the person, and a run
|
||||
* where 192.168.1.1 became a random public address would be actively misleading to read.
|
||||
* Public addresses keep only their /16 so the network is still locatable at ISP granularity.
|
||||
*/
|
||||
private fun ip4(value: String): String {
|
||||
val o = value.split(".")
|
||||
if (o.size != 4 || o.any { it.toIntOrNull() == null }) return value
|
||||
val n = o.map { it.toInt() }
|
||||
val reserved = n[0] == 10 ||
|
||||
(n[0] == 172 && n[1] in 16..31) ||
|
||||
(n[0] == 192 && n[1] == 168) ||
|
||||
(n[0] == 169 && n[1] == 254) ||
|
||||
(n[0] == 100 && n[1] in 64..127) ||
|
||||
n[0] == 127 || n[0] == 0 || n[0] >= 224
|
||||
if (reserved) return value
|
||||
val h = pseudo("ip4", value) { it }
|
||||
return "${n[0]}.${n[1]}.${h.substring(0, 2).toInt(16)}.${h.substring(2, 4).toInt(16)}"
|
||||
}
|
||||
|
||||
/**
|
||||
* IPv6 keeps the scope and the first 32 bits (so 2001:db8:… still reads as global unicast in
|
||||
* the same allocation) and pseudonymizes the rest — the interface identifier is the part that
|
||||
* is a device fingerprint, especially with EUI-64.
|
||||
*/
|
||||
private fun ip6(value: String): String {
|
||||
val v = value.lowercase(Locale.ROOT)
|
||||
if (v == "::1" || v == "::" || v.startsWith("fe80:") || v.startsWith("ff")) return v
|
||||
val groups = v.substringBefore('%').split(":")
|
||||
if (groups.size < 3) return v
|
||||
val h = pseudo("ip6", value) { it }
|
||||
return "${groups[0]}:${groups[1]}:${h.substring(0, 4)}:${h.substring(4, 8)}::${h.substring(8, 12)}"
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-label pseudonyms with the public suffix kept, so "it resolved somewhere under .local"
|
||||
* or "…under example.com" survives without naming the host. The suffix list is deliberately
|
||||
* short: guessing wrong keeps *more* pseudonymized, never less.
|
||||
*/
|
||||
private fun fqdn(value: String): String {
|
||||
if (value.isEmpty()) return value
|
||||
val trailing = value.endsWith(".")
|
||||
val labels = value.trimEnd('.').split(".")
|
||||
if (labels.size == 1) return pseudo("fqdn", value) { "host-" + it.take(6) }
|
||||
val keep = if (labels.last() in publicSuffixes) 1 else 0
|
||||
val head = labels.dropLast(keep).map { l -> pseudo("label", l) { "l-" + it.take(6) } }
|
||||
return (head + labels.takeLast(keep)).joinToString(".") + if (trailing) "." else ""
|
||||
}
|
||||
|
||||
// ---- STRICT ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* STRICT keeps the shape of the document and the numbers, and nothing that quotes the
|
||||
* network back. Evidence goes (trains carry addresses and hostnames), finding prose goes
|
||||
* (it interpolates real names), networks go entirely.
|
||||
*/
|
||||
private fun strip(doc: JsonObject): JsonObject = buildJsonObject {
|
||||
for ((k, v) in doc) {
|
||||
when (k) {
|
||||
"networks", "server_sessions" -> Unit
|
||||
"tests" -> put(k, JsonArray((v as? JsonArray ?: JsonArray(emptyList())).map { t ->
|
||||
val o = t.jsonObject
|
||||
JsonObject(o.filterKeys { it != "evidence" && it != "params" })
|
||||
}))
|
||||
"findings" -> put(k, JsonArray((v as? JsonArray ?: JsonArray(emptyList())).map { f ->
|
||||
val o = f.jsonObject
|
||||
JsonObject(o.filterKeys { it != "description" && it != "title" && it != "evidence_refs" })
|
||||
}))
|
||||
"run" -> put(k, JsonObject(v.jsonObject.filterKeys { it != "notes" }))
|
||||
else -> put(k, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- pseudonym machinery ---------------------------------------------------------------
|
||||
|
||||
/** Deterministic per (domain, value, salt); memoized so one value maps to one pseudonym. */
|
||||
private fun pseudo(domain: String, value: String, shape: (String) -> String): String =
|
||||
cache.getOrPut("$domain $value") {
|
||||
val md = MessageDigest.getInstance("SHA-256")
|
||||
md.update(salt.bytes)
|
||||
md.update(domain.toByteArray())
|
||||
md.update(0)
|
||||
md.update(value.lowercase(Locale.ROOT).toByteArray())
|
||||
shape(md.digest().joinToString("") { "%02x".format(it) })
|
||||
}
|
||||
|
||||
private companion object {
|
||||
val publicSuffixes = setOf(
|
||||
"local", "lan", "home", "internal", "arpa",
|
||||
"com", "net", "org", "io", "app", "dev", "at", "de", "eu", "uk",
|
||||
)
|
||||
}
|
||||
}
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
// Package privacy implements the anonymization contract of measurement-schema.md §8.
|
||||
//
|
||||
// The threat model is specific. An engineer running their own server wants the full document —
|
||||
// SSIDs and MACs are what make a run useful a week later. Someone measuring against a stranger's
|
||||
// server wants the numbers to survive and the identifiers not to. So this is a *transform*, not a
|
||||
// filter: the output is still a valid measurement document with the same tests, metrics and
|
||||
// findings; only the identifying scalars change, and consistently, so "same SSID as last run" is
|
||||
// still answerable from pseudonyms alone.
|
||||
//
|
||||
// Two properties are load-bearing and are what the tests pin:
|
||||
// - Consistency within a document: one input value always maps to one pseudonym, so
|
||||
// correlations inside a run survive.
|
||||
// - No consistency *across* documents unless the user asks for it: the salt is per-run by
|
||||
// default, so pseudonyms cannot be used to track a device between uploads. A stable salt is
|
||||
// opt-in (`Salt.stable`) for people diffing their own history on their own server.
|
||||
package app.echo_lot.privacy
|
||||
|
||||
import kotlinx.serialization.json.*
|
||||
import java.security.MessageDigest
|
||||
import java.util.Locale
|
||||
|
||||
/** How much to strip. Ordered: FULL < BALANCED < STRICT. Wire values match the server's. */
|
||||
enum class PrivacyLevel(val wire: String) {
|
||||
/** Nothing removed. The right choice for your own server. */
|
||||
FULL("full"),
|
||||
|
||||
/**
|
||||
* Identifiers pseudonymized, neighbour inventory dropped. Topology and timing survive:
|
||||
* you can still see that the gateway is a MikroTik at a /24 boundary with 3 % loss, but not
|
||||
* which MikroTik, on which SSID, next to whose Chromecast.
|
||||
*/
|
||||
BALANCED("balanced"),
|
||||
|
||||
/**
|
||||
* Numbers only: tests keep their metrics and status, evidence is dropped, findings keep their
|
||||
* codes and severities but lose descriptions (which quote real names). What is left cannot
|
||||
* identify a network, and is still enough for aggregate "how common is this fault" work.
|
||||
*/
|
||||
STRICT("strict");
|
||||
|
||||
companion object {
|
||||
fun fromWire(s: String?): PrivacyLevel =
|
||||
entries.firstOrNull { it.wire == s?.lowercase(Locale.ROOT) } ?: FULL
|
||||
|
||||
/** The stricter of two levels — used to honour a server's minimum. */
|
||||
fun max(a: PrivacyLevel, b: PrivacyLevel): PrivacyLevel = if (a.ordinal >= b.ordinal) a else b
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The pseudonymization salt. Per-run by default: a fresh random salt means the same SSID uploaded
|
||||
* twice yields two different pseudonyms, so an upload endpoint cannot link runs to a device.
|
||||
* A stable salt trades that away for cross-run diffing and is only appropriate on a server you
|
||||
* own — the app makes that an explicit choice, not a default.
|
||||
*/
|
||||
class Salt private constructor(internal val bytes: ByteArray, val stable: Boolean) {
|
||||
companion object {
|
||||
fun perRun(random: ByteArray): Salt = Salt(random.copyOf(), stable = false)
|
||||
fun stable(secret: ByteArray): Salt = Salt(secret.copyOf(), stable = true)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Transforms a measurement document to [level].
|
||||
*
|
||||
* Field classification is by JSON key name, because the schema names things consistently
|
||||
* (`ssid`, `bssid`, `mac`, `ip4`, `ip6`, `fqdn`, …) and a name-driven pass is auditable by
|
||||
* reading one table. Anything unrecognized is treated as identifying when it is a string inside
|
||||
* a known-sensitive container, and left alone otherwise — see [Classification].
|
||||
*/
|
||||
class Anonymizer(private val level: PrivacyLevel, private val salt: Salt) {
|
||||
|
||||
private val cache = HashMap<String, String>()
|
||||
|
||||
fun anonymize(doc: JsonObject): JsonObject {
|
||||
if (level == PrivacyLevel.FULL) return stamp(doc)
|
||||
val walked = walkObject(doc, path = emptyList())
|
||||
val out = if (level == PrivacyLevel.STRICT) strip(walked) else walked
|
||||
return stamp(out)
|
||||
}
|
||||
|
||||
/** Records what was done, so a reader of the archived/uploaded document is never guessing. */
|
||||
private fun stamp(doc: JsonObject): JsonObject {
|
||||
val run = doc["run"]?.jsonObject ?: return doc
|
||||
val privacy = buildJsonObject {
|
||||
put("anonymization", level.wire)
|
||||
put("salt", if (salt.stable) "stable" else "per_run")
|
||||
}
|
||||
return JsonObject(doc + ("run" to JsonObject(run + ("privacy" to privacy))))
|
||||
}
|
||||
|
||||
// ---- the tree walk -------------------------------------------------------------------
|
||||
|
||||
private fun walkObject(obj: JsonObject, path: List<String>): JsonObject = buildJsonObject {
|
||||
for ((k, v) in obj) {
|
||||
val childPath = path + k
|
||||
when {
|
||||
Classification.dropAtBalanced(childPath) -> Unit // omit entirely
|
||||
else -> put(k, walk(k, v, childPath))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun walk(key: String, v: JsonElement, path: List<String>): JsonElement = when (v) {
|
||||
is JsonObject -> walkObject(v, path)
|
||||
is JsonArray -> JsonArray(v.map { walk(key, it, path) })
|
||||
is JsonPrimitive ->
|
||||
if (v.isString) {
|
||||
// Name first (it is precise), then shape (it is exhaustive). A field nobody
|
||||
// classified must not be a field that leaks.
|
||||
val type = Classification.typeOf(key, path) ?: Classification.inferFromValue(v.content)
|
||||
JsonPrimitive(transform(type, v.content))
|
||||
} else {
|
||||
v
|
||||
}
|
||||
}
|
||||
|
||||
private fun transform(type: LogicalType?, value: String): String = when (type) {
|
||||
null -> value
|
||||
LogicalType.SSID -> pseudo("ssid", value) { "net-" + it.take(6) }
|
||||
LogicalType.MAC, LogicalType.BSSID -> macPreservingOui(value)
|
||||
LogicalType.IP4 -> ip4(value)
|
||||
LogicalType.IP6 -> ip6(value)
|
||||
LogicalType.FQDN -> fqdn(value)
|
||||
LogicalType.OPAQUE_ID -> "redacted"
|
||||
LogicalType.FREETEXT -> "[removed: may contain identifying text]"
|
||||
}
|
||||
|
||||
// ---- per-type transforms -------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Keeps the OUI (which vendor) and pseudonymizes the NIC part (which unit). Vendor is the
|
||||
* diagnostically valuable half — "the RA comes from a MikroTik" survives, "…from THAT
|
||||
* MikroTik" does not.
|
||||
*/
|
||||
private fun macPreservingOui(value: String): String {
|
||||
val sep = if (value.contains('-')) '-' else ':'
|
||||
val parts = value.split(sep)
|
||||
if (parts.size != 6 || parts.any { it.length != 2 }) return pseudo("mac", value) { "mac-" + it.take(8) }
|
||||
val nic = pseudo("mac", value) { it }
|
||||
return (parts.take(3) + listOf(nic.substring(0, 2), nic.substring(2, 4), nic.substring(4, 6)))
|
||||
.joinToString(sep.toString())
|
||||
.lowercase(Locale.ROOT)
|
||||
}
|
||||
|
||||
/**
|
||||
* Prefix-preserving within the same class, with reserved ranges kept verbatim: RFC1918 and
|
||||
* CGNAT addresses say something about the topology and nothing about the person, and a run
|
||||
* where 192.168.1.1 became a random public address would be actively misleading to read.
|
||||
* Public addresses keep only their /16 so the network is still locatable at ISP granularity.
|
||||
*/
|
||||
private fun ip4(value: String): String {
|
||||
// A route destination carries a prefix length; pseudonymize the address and put it back,
|
||||
// or "0.0.0.0/0" turns into nonsense and the routing table becomes unreadable.
|
||||
value.substringAfter('/', "").takeIf { it.isNotEmpty() && value.contains('/') }?.let { len ->
|
||||
return ip4(value.substringBefore('/')) + "/" + len
|
||||
}
|
||||
val o = value.split(".")
|
||||
if (o.size != 4 || o.any { it.toIntOrNull() == null }) return value
|
||||
val n = o.map { it.toInt() }
|
||||
val reserved = n[0] == 10 ||
|
||||
(n[0] == 172 && n[1] in 16..31) ||
|
||||
(n[0] == 192 && n[1] == 168) ||
|
||||
(n[0] == 169 && n[1] == 254) ||
|
||||
(n[0] == 100 && n[1] in 64..127) ||
|
||||
n[0] == 127 || n[0] == 0 || n[0] >= 224
|
||||
if (reserved) return value
|
||||
val h = pseudo("ip4", value) { it }
|
||||
return "${n[0]}.${n[1]}.${h.substring(0, 2).toInt(16)}.${h.substring(2, 4).toInt(16)}"
|
||||
}
|
||||
|
||||
/**
|
||||
* IPv6 keeps the scope and the first 32 bits (so 2001:db8:… still reads as global unicast in
|
||||
* the same allocation) and pseudonymizes the rest — the interface identifier is the part that
|
||||
* is a device fingerprint, especially with EUI-64.
|
||||
*/
|
||||
private fun ip6(value: String): String {
|
||||
// Dotted quads reach here through the family-agnostic field names (addr, gateway, dst);
|
||||
// hand them to the IPv4 path rather than mangling them as if they were v6.
|
||||
if (value.count { it == ':' } < 2) return ip4(value)
|
||||
if (value.contains('/')) {
|
||||
return ip6(value.substringBefore('/')) + "/" + value.substringAfter('/')
|
||||
}
|
||||
val v = value.lowercase(Locale.ROOT)
|
||||
// The unspecified address and the default route are not identities; mangling them would
|
||||
// make a routing table unreadable for no privacy gain.
|
||||
if (v == "::1" || v == "::" || v.startsWith("fe80:") || v.startsWith("ff")) return v
|
||||
val groups = v.substringBefore('%').split(":")
|
||||
if (groups.size < 3) return v
|
||||
val h = pseudo("ip6", value) { it }
|
||||
return "${groups[0]}:${groups[1]}:${h.substring(0, 4)}:${h.substring(4, 8)}::${h.substring(8, 12)}"
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-label pseudonyms with the public suffix kept, so "it resolved somewhere under .local"
|
||||
* or "…under example.com" survives without naming the host. The suffix list is deliberately
|
||||
* short: guessing wrong keeps *more* pseudonymized, never less.
|
||||
*/
|
||||
private fun fqdn(value: String): String {
|
||||
if (value.isEmpty()) return value
|
||||
val trailing = value.endsWith(".")
|
||||
val labels = value.trimEnd('.').split(".")
|
||||
if (labels.size == 1) return pseudo("fqdn", value) { "host-" + it.take(6) }
|
||||
val keep = if (labels.last() in publicSuffixes) 1 else 0
|
||||
val head = labels.dropLast(keep).map { l -> pseudo("label", l) { "l-" + it.take(6) } }
|
||||
return (head + labels.takeLast(keep)).joinToString(".") + if (trailing) "." else ""
|
||||
}
|
||||
|
||||
// ---- STRICT ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* STRICT keeps the shape of the document and the numbers, and nothing that quotes the
|
||||
* network back. Evidence goes (trains carry addresses and hostnames), finding prose goes
|
||||
* (it interpolates real names), networks go entirely.
|
||||
*/
|
||||
private fun strip(doc: JsonObject): JsonObject = buildJsonObject {
|
||||
for ((k, v) in doc) {
|
||||
when (k) {
|
||||
"networks", "server_sessions" -> Unit
|
||||
"tests" -> put(k, JsonArray((v as? JsonArray ?: JsonArray(emptyList())).map { t ->
|
||||
val o = t.jsonObject
|
||||
JsonObject(o.filterKeys { it != "evidence" && it != "params" })
|
||||
}))
|
||||
"findings" -> put(k, JsonArray((v as? JsonArray ?: JsonArray(emptyList())).map { f ->
|
||||
val o = f.jsonObject
|
||||
JsonObject(o.filterKeys { it != "description" && it != "title" && it != "evidence_refs" })
|
||||
}))
|
||||
"run" -> put(k, JsonObject(v.jsonObject.filterKeys { it != "notes" }))
|
||||
else -> put(k, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- pseudonym machinery ---------------------------------------------------------------
|
||||
|
||||
/** Deterministic per (domain, value, salt); memoized so one value maps to one pseudonym. */
|
||||
private fun pseudo(domain: String, value: String, shape: (String) -> String): String =
|
||||
cache.getOrPut("$domain $value") {
|
||||
val md = MessageDigest.getInstance("SHA-256")
|
||||
md.update(salt.bytes)
|
||||
md.update(domain.toByteArray())
|
||||
md.update(0)
|
||||
md.update(value.lowercase(Locale.ROOT).toByteArray())
|
||||
shape(md.digest().joinToString("") { "%02x".format(it) })
|
||||
}
|
||||
|
||||
private companion object {
|
||||
val publicSuffixes = setOf(
|
||||
"local", "lan", "home", "internal", "arpa",
|
||||
"com", "net", "org", "io", "app", "dev", "at", "de", "eu", "uk",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -32,6 +32,16 @@ object Classification {
|
||||
"link_local", "ra_source", "prefix",
|
||||
).forEach { put(it, LogicalType.IP6) }
|
||||
|
||||
// Family-agnostic address fields — the names the models actually use (Address.addr,
|
||||
// Route.gateway, Route.dst, DnsConfig.servers). Their absence here was a real leak: the
|
||||
// device's own global IPv6 address went out verbatim at the level whose description
|
||||
// promises addresses are pseudonymized. Typed IP6 because the transform detects the
|
||||
// family from the value, falling through to the IPv4 path for a dotted quad.
|
||||
listOf(
|
||||
"addr", "address", "gateway", "dst", "src", "servers", "server", "resolver",
|
||||
"next_hop", "via", "public_ip", "observed_ip",
|
||||
).forEach { put(it, LogicalType.IP6) }
|
||||
|
||||
listOf("mac", "hw_addr", "gateway_mac", "router_mac", "sender_mac", "peer_mac")
|
||||
.forEach { put(it, LogicalType.MAC) }
|
||||
listOf("bssid", "ap_mac").forEach { put(it, LogicalType.BSSID) }
|
||||
@@ -40,6 +50,8 @@ object Classification {
|
||||
listOf(
|
||||
"fqdn", "hostname", "host", "name", "reverse_dns", "ptr", "domain", "query_name",
|
||||
"friendly_name", "server_name", "sni", "cname", "search_domain", "device_name",
|
||||
// Plural and prefixed variants the models actually use.
|
||||
"search_domains", "private_dns_hostname", "domains", "hostnames",
|
||||
).forEach { put(it, LogicalType.FQDN) }
|
||||
|
||||
listOf("session_id", "credential", "token", "device_id", "android_id", "serial", "imsi", "iccid")
|
||||
@@ -78,6 +90,49 @@ object Classification {
|
||||
return null
|
||||
}
|
||||
|
||||
/**
|
||||
* Last-resort classification from the *value*, when the field name is unrecognised.
|
||||
*
|
||||
* A name table can only protect fields somebody remembered to add, which is the wrong
|
||||
* property for a privacy control: the dangerous field is the one nobody thought of. This
|
||||
* exists because that failed once already — `addresses[].addr` holds the device's own global
|
||||
* IPv6 address, the table had never heard of the name, and it went out verbatim.
|
||||
*
|
||||
* Only addresses and MACs are inferred, because only those have shapes that cannot be
|
||||
* mistaken for something else. Hostnames deliberately are not: `train.udp_updown` is
|
||||
* indistinguishable from a domain by shape, and mangling a test type would corrupt the
|
||||
* document to protect nothing.
|
||||
*/
|
||||
fun inferFromValue(value: String): LogicalType? {
|
||||
val v = value.trim()
|
||||
if (v.isEmpty() || v.length > 64) return null
|
||||
if (looksLikeMac(v)) return LogicalType.MAC
|
||||
if (looksLikeIp6(v)) return LogicalType.IP6
|
||||
if (looksLikeIp4(v)) return LogicalType.IP4
|
||||
return null
|
||||
}
|
||||
|
||||
private fun isHex(c: Char) = c in '0'..'9' || c in 'a'..'f' || c in 'A'..'F'
|
||||
|
||||
private fun looksLikeMac(v: String): Boolean {
|
||||
val parts = v.split(':', '-')
|
||||
return parts.size == 6 && parts.all { p -> p.length == 2 && p.all(::isHex) }
|
||||
}
|
||||
|
||||
private fun looksLikeIp4(v: String): Boolean {
|
||||
val parts = v.substringBefore('/').split('.')
|
||||
return parts.size == 4 && parts.all { p ->
|
||||
p.isNotEmpty() && p.length <= 3 && p.all(Char::isDigit) && p.toInt() <= 255
|
||||
}
|
||||
}
|
||||
|
||||
private fun looksLikeIp6(v: String): Boolean {
|
||||
val core = v.substringBefore('/').substringBefore('%')
|
||||
// Two colons minimum, so a time or a MAC fragment does not qualify, and nothing but the
|
||||
// characters an address may contain.
|
||||
return core.count { it == ':' } >= 2 && core.all { it == ':' || isHex(it) }
|
||||
}
|
||||
|
||||
fun dropAtBalanced(path: List<String>): Boolean {
|
||||
if (path.isNotEmpty() && path.last() in droppedKeys) return true
|
||||
return droppedPaths.any { dropped -> dropped.all { path.contains(it) } }
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
package app.echo_lot.privacy
|
||||
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.jsonObject
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* The blunt instrument: build a document with identifying values in every place one can actually
|
||||
* occur, anonymize it, and assert none of them survive.
|
||||
*
|
||||
* [AnonymizerTest] checks that the fields the classification table knows about are handled
|
||||
* correctly. This checks the other half — the fields it does *not* know about. A per-field test
|
||||
* can only fail for a field someone remembered to write a case for, which is exactly the wrong
|
||||
* property for a privacy check: the dangerous field is the one nobody thought of.
|
||||
*
|
||||
* Concretely, this is written the way it is because the schema's own field names disagree with
|
||||
* the classifier's. `Address.addr` carries an IP and is documented as such in
|
||||
* measurement-schema.md §8, but the classifier keys on names like `ip4` and `gateway_ip4` and had
|
||||
* never heard of `addr`.
|
||||
*/
|
||||
class LeakTest {
|
||||
|
||||
private val json = Json { prettyPrint = false }
|
||||
private val salt = Salt.perRun(ByteArray(32) { 3 })
|
||||
|
||||
/**
|
||||
* Every string here is something that identifies a person, a household or a device, placed
|
||||
* where the real models actually put it (`core-measurement`'s Network/Link/Address/DnsConfig).
|
||||
*/
|
||||
private val secrets = listOf(
|
||||
"Rambossek WLAN", // ssid
|
||||
"78:9a:18:aa:bb:cc", // bssid
|
||||
"aa:bb:cc:dd:ee:11", // gateway mac
|
||||
"2001:1ad0:c4fe:6767::150", // global v6 address on the interface
|
||||
"2a02:1748:dead:beef::1", // v6 default gateway
|
||||
"203.0.113.77", // public v4
|
||||
"nas.rambossek.lan", // private-dns hostname
|
||||
"rambossek.lan", // search domain
|
||||
"Anna's Chromecast", // neighbour name
|
||||
"kitchen table", // free-text note
|
||||
)
|
||||
|
||||
private fun document(): String = """
|
||||
{
|
||||
"schema": "echolot/measurement",
|
||||
"run": {
|
||||
"id": "run-1", "trigger": "manual", "notes": "${secrets[9]}",
|
||||
"device": {"manufacturer": "OnePlus", "model": "CPH2747"}
|
||||
},
|
||||
"networks": [{
|
||||
"id": "net-1", "transport": "wifi",
|
||||
"link": {
|
||||
"mtu": 1500,
|
||||
"addresses": [
|
||||
{"addr": "${secrets[3]}", "prefix_len": 64, "scope": "global"},
|
||||
{"addr": "192.168.1.44", "prefix_len": 24, "scope": "global"}
|
||||
],
|
||||
"routes": [
|
||||
{"dst": "::/0", "gateway": "${secrets[4]}", "iface": "wlan0"},
|
||||
{"dst": "0.0.0.0/0", "gateway": "192.168.1.1", "iface": "wlan0"}
|
||||
],
|
||||
"dns": {
|
||||
"servers": ["${secrets[5]}", "192.168.1.1"],
|
||||
"private_dns_hostname": "${secrets[6]}",
|
||||
"search_domains": ["${secrets[7]}"]
|
||||
}
|
||||
},
|
||||
"wifi": {"ssid": "${secrets[0]}", "bssid": "${secrets[1]}"},
|
||||
"neighbors": [{"name": "${secrets[8]}", "mac": "${secrets[2]}"}]
|
||||
}],
|
||||
"tests": [{"id": "t1", "type": "train.udp_updown", "status": "ok",
|
||||
"metrics": {"rtt_ms_avg": 12.4}}],
|
||||
"findings": [],
|
||||
"summary": {"verdict": "ok"}
|
||||
}
|
||||
""".trimIndent()
|
||||
|
||||
private fun anonymized(level: PrivacyLevel): String =
|
||||
json.encodeToString(
|
||||
kotlinx.serialization.json.JsonObject.serializer(),
|
||||
Anonymizer(level, salt).anonymize(json.parseToJsonElement(document()).jsonObject),
|
||||
)
|
||||
|
||||
@Test
|
||||
fun nothingIdentifyingSurvivesBalanced() {
|
||||
val out = anonymized(PrivacyLevel.BALANCED)
|
||||
val leaked = secrets.filter { out.contains(it) }
|
||||
assertTrue(
|
||||
leaked.isEmpty(),
|
||||
"these identifying values were uploaded verbatim at BALANCED: $leaked\n\n$out",
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun nothingIdentifyingSurvivesStrict() {
|
||||
val out = anonymized(PrivacyLevel.STRICT)
|
||||
val leaked = secrets.filter { out.contains(it) }
|
||||
assertTrue(leaked.isEmpty(), "leaked at STRICT: $leaked\n\n$out")
|
||||
}
|
||||
|
||||
// Private addresses are kept on purpose — they describe the topology and not the person — so
|
||||
// this pins that the leak test above is not passing by accident of over-redaction.
|
||||
@Test
|
||||
fun privateAddressesAreStillReadable() {
|
||||
val out = anonymized(PrivacyLevel.BALANCED)
|
||||
assertTrue(out.contains("192.168.1.1"), "RFC1918 gateway should survive: $out")
|
||||
assertTrue(out.contains("192.168.1.44"), "RFC1918 interface address should survive: $out")
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user