EchoLot/echolot
1
0
Fork 0
Code Pull Requests Actions Packages Projects Releases 31 Wiki Activity

privacy: scrub identifiers inside raw shell output

Browse Source 
Running the Shizuku tier for the first time uploaded every MAC address on the
local network to the server at the balanced level - fourteen of them, router and
all. The probes embed raw command output verbatim (ip neigh, ip route), which is
good evidence and also a complete household device inventory, and the anonymizer
could not see it: classification is by field name and whole-value shape, and
ip_neigh is one long string that is itself neither a MAC nor an address.

measurement-schema.md flagged raw dumps as hard to anonymize and proposed
dropping them from exports. Scrubbing is better: identifiers inside unclassified
strings are replaced in place with the same pseudonyms used elsewhere, so a MAC
appearing in both a parsed field and a raw dump still reads as one device, and
the dump stays readable - neighbour-table shape, host count, RFC1918 addresses
and vendor prefixes all survive. Dropping it would have protected the same data
by destroying the reason for collecting it.

One pass, not three: sequential passes re-process their own output. Once a MAC
became 78:9a:18:xx:yy:zz the IPv6 pattern matched it - six hex groups separated
by colons is an address - and destroyed the vendor prefix the MAC rule had just
preserved. Ordered alternation resolves each position once, MAC first.

RealDocumentTest runs the anonymizer over a captured run when ECHOLOT_REAL_RUN
points at one and fails on any surviving MAC; it self-skips otherwise so no
one's network lands in the repo. Against the document that leaked: 14 in, 0 out.

Also: the Settings preview button did nothing, reading UiState.history which is
empty until the History screen has been opened - same root cause as the "0
run(s)" count. It reads the archive now, and says when there is nothing to show.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
mrambossek
2026-08-01 16:12:13 +02:00
co-authored by Claude Fable 5
parent d04babff51
commit c19f382640
7 changed files with 501 additions and 283 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/build-status.md
+37
View File
@@ -965,3 +965,40 @@ one session measures itself rather than inheriting the first one's packets. The
Live against fmr: **3125 sent, 3125 counted, 0 % loss, 10.0 Mbit/s** at a 10 Mbit/s request, with Live against fmr: **3125 sent, 3125 counted, 0 % loss, 10.0 Mbit/s** at a 10 Mbit/s request, with
`measures_network: false` — correct, since what arrived matched what was offered, so the path was `measures_network: false` — correct, since what arrived matched what was offered, so the path was
never the constraint. never the constraint.
### Raw shell dumps leaked the whole LAN (2026-08-01)
Found by running the Shizuku shell tier for the first time. The tier works — `tiers.shizuku: true`,
`exec_path: UserService` (so the UserService binds on the OnePlus, as recorded), `runs_as
shell(2000)`, 7/7 commands — and the run promptly uploaded **every MAC address on the local
network** to fmr at the `balanced` level: router, phones, whatever else was on the wifi. Fourteen
of them.
The probes embed raw command output verbatim (`ip neigh`, `ip route`, `id`), which is genuinely
good evidence and also a complete household device inventory. The anonymizer could not see it:
classification is by field name and by whole-value shape, and `ip_neigh` is one long string that is
itself neither a MAC nor an address. measurement-schema.md §9 item 2 had flagged raw dumps as "hard
to anonymize" and proposed dropping them from exports; nothing enforced either.
**Scrubbing beats dropping.** Identifiers inside any unclassified string are now replaced in place,
using the same pseudonyms as everywhere else — so a MAC that appears both in a parsed field and in
a raw dump still reads as one device. The dump stays readable and auditable: you can still see the
neighbour table's shape, the host count, RFC1918 addresses and vendor prefixes. Dropping the
evidence would have protected the same data while destroying the reason for collecting it.
Two implementation notes worth keeping:
- **One pass, not three.** Sequential passes re-process their own output: once a MAC became
`78:9a:18:xx:yy:zz`, the IPv6 pattern matched it — six hex groups separated by colons *is* an
address — and destroyed the vendor prefix the MAC rule had just preserved. Ordered alternation
resolves each position once, MAC first.
- The patterns are conservative on purpose. A missed address gets caught by another rule or not at
all; an over-eager one mangles timestamps and version strings, corrupting evidence to protect
nothing.
`RealDocumentTest` runs the anonymizer over a captured run when `ECHOLOT_REAL_RUN` points at one,
and fails on any MAC that survives. It self-skips otherwise, so no one's network is committed to the
repo. Against the actual leaked document: **14 MACs in, 0 surviving.**
Also fixed: the Settings *Preview what an upload would send* button did nothing. It read
`UiState.history`, which is empty until the History screen has been opened — the same root cause as
the "0 run(s)" count. It now reads the archive directly, and says so when there is nothing to
preview rather than silently ignoring the tap.
echolot-app/app/src/main/kotlin/app/echo_lot/app/MainActivity.kt
+4 -5
View File
@@ -101,11 +101,10 @@ class MainActivity : ComponentActivity() {
onApplyRetention = vm::applyRetention, onApplyRetention = vm::applyRetention,
onDeleteAll = vm::deleteAllRuns, onDeleteAll = vm::deleteAllRuns,
onPreviewUpload = { onPreviewUpload = {
// Preview the newest run, since that is the one the user just made // Straight from the archive: the newest run is the one the user just
// and the one they are deciding about. // made and the one they are deciding about. Always shows something,
vm.state.history.firstOrNull()?.let { r -> // even when there is nothing to preview yet.
lifecycleScope.launch { preview = vm.uploadPreview(r.id) } lifecycleScope.launch { preview = vm.previewNewestRun() }
}
}, },
onCheckServer = vm::checkServer, onCheckServer = vm::checkServer,
onEnroll = vm::enroll, onEnroll = vm::enroll,
echolot-app/app/src/main/kotlin/app/echo_lot/app/RunViewModel.kt
+15
View File
@@ -194,6 +194,21 @@ class RunViewModel(app: Application) : AndroidViewModel(app) {
store.read(id)?.let { store.redactedForUpload(it) } store.read(id)?.let { store.redactedForUpload(it) }
} }
/**
* Preview of the most recent run, read from the archive rather than from [UiState.history].
*
* The history list is only populated once the History screen has been opened, so a preview
* driven from it did nothing at all on a freshly-opened Settings screen — a button that
* silently does nothing is worse than one that says why.
*/
suspend fun previewNewestRun(): String = withContext(Dispatchers.IO) {
val newest = store.list().firstOrNull()
?: return@withContext "No archived runs yet. Run a measurement first, then this will " +
"show exactly what an upload would send."
store.read(newest.id)?.let { store.redactedForUpload(it) }
?: "That run could not be read back from the archive."
}
fun archivedBytes(): Long = store.totalBytes() fun archivedBytes(): Long = store.totalBytes()
/** Counted from the archive itself, not from [UiState.history], which is empty until the /** Counted from the archive itself, not from [UiState.history], which is empty until the
echolot-app/core-privacy/build.gradle.kts
+5 -1
View File
@@ -20,4 +20,8 @@ kotlin {
} }
java { sourceCompatibility = JavaVersion.VERSION_17; targetCompatibility = JavaVersion.VERSION_17 } java { sourceCompatibility = JavaVersion.VERSION_17; targetCompatibility = JavaVersion.VERSION_17 }
tasks.test { useJUnitPlatform() } tasks.test {
useJUnitPlatform()
// Opt-in: point this at a captured run to check the anonymizer against real data.
System.getenv("ECHOLOT_REAL_RUN")?.let { environment("ECHOLOT_REAL_RUN", it) }
}
echolot-app/core-privacy/src/main/kotlin/app/echo_lot/privacy/Anonymizer.kt
+327 -276
View File
@@ -1,276 +1,327 @@
// SPDX-FileCopyrightText: 2026 Echolot contributors // SPDX-FileCopyrightText: 2026 Echolot contributors
// SPDX-License-Identifier: GPL-3.0-or-later // SPDX-License-Identifier: GPL-3.0-or-later
// Package privacy implements the anonymization contract of measurement-schema.md §8. // Package privacy implements the anonymization contract of measurement-schema.md §8.
// //
// The threat model is specific. An engineer running their own server wants the full document — // The threat model is specific. An engineer running their own server wants the full document —
// SSIDs and MACs are what make a run useful a week later. Someone measuring against a stranger's // SSIDs and MACs are what make a run useful a week later. Someone measuring against a stranger's
// server wants the numbers to survive and the identifiers not to. So this is a *transform*, not a // server wants the numbers to survive and the identifiers not to. So this is a *transform*, not a
// filter: the output is still a valid measurement document with the same tests, metrics and // filter: the output is still a valid measurement document with the same tests, metrics and
// findings; only the identifying scalars change, and consistently, so "same SSID as last run" is // findings; only the identifying scalars change, and consistently, so "same SSID as last run" is
// still answerable from pseudonyms alone. // still answerable from pseudonyms alone.
// //
// Two properties are load-bearing and are what the tests pin: // Two properties are load-bearing and are what the tests pin:
// - Consistency within a document: one input value always maps to one pseudonym, so // - Consistency within a document: one input value always maps to one pseudonym, so
// correlations inside a run survive. // correlations inside a run survive.
// - No consistency *across* documents unless the user asks for it: the salt is per-run by // - No consistency *across* documents unless the user asks for it: the salt is per-run by
// default, so pseudonyms cannot be used to track a device between uploads. A stable salt is // default, so pseudonyms cannot be used to track a device between uploads. A stable salt is
// opt-in (`Salt.stable`) for people diffing their own history on their own server. // opt-in (`Salt.stable`) for people diffing their own history on their own server.
package app.echo_lot.privacy package app.echo_lot.privacy
import kotlinx.serialization.json.* import kotlinx.serialization.json.*
import java.security.MessageDigest import java.security.MessageDigest
import java.util.Locale import java.util.Locale
/** How much to strip. Ordered: FULL < BALANCED < STRICT. Wire values match the server's. */ /** How much to strip. Ordered: FULL < BALANCED < STRICT. Wire values match the server's. */
enum class PrivacyLevel(val wire: String) { enum class PrivacyLevel(val wire: String) {
/** Nothing removed. The right choice for your own server. */ /** Nothing removed. The right choice for your own server. */
FULL("full"), FULL("full"),
/** /**
* Identifiers pseudonymized, neighbour inventory dropped. Topology and timing survive: * Identifiers pseudonymized, neighbour inventory dropped. Topology and timing survive:
* you can still see that the gateway is a MikroTik at a /24 boundary with 3 % loss, but not * you can still see that the gateway is a MikroTik at a /24 boundary with 3 % loss, but not
* which MikroTik, on which SSID, next to whose Chromecast. * which MikroTik, on which SSID, next to whose Chromecast.
*/ */
BALANCED("balanced"), BALANCED("balanced"),
/** /**
* Numbers only: tests keep their metrics and status, evidence is dropped, findings keep their * Numbers only: tests keep their metrics and status, evidence is dropped, findings keep their
* codes and severities but lose descriptions (which quote real names). What is left cannot * codes and severities but lose descriptions (which quote real names). What is left cannot
* identify a network, and is still enough for aggregate "how common is this fault" work. * identify a network, and is still enough for aggregate "how common is this fault" work.
*/ */
STRICT("strict"); STRICT("strict");
companion object { companion object {
fun fromWire(s: String?): PrivacyLevel = fun fromWire(s: String?): PrivacyLevel =
entries.firstOrNull { it.wire == s?.lowercase(Locale.ROOT) } ?: FULL entries.firstOrNull { it.wire == s?.lowercase(Locale.ROOT) } ?: FULL
/** The stricter of two levels — used to honour a server's minimum. */ /** The stricter of two levels — used to honour a server's minimum. */
fun max(a: PrivacyLevel, b: PrivacyLevel): PrivacyLevel = if (a.ordinal >= b.ordinal) a else b fun max(a: PrivacyLevel, b: PrivacyLevel): PrivacyLevel = if (a.ordinal >= b.ordinal) a else b
} }
} }
/** /**
* The pseudonymization salt. Per-run by default: a fresh random salt means the same SSID uploaded * The pseudonymization salt. Per-run by default: a fresh random salt means the same SSID uploaded
* twice yields two different pseudonyms, so an upload endpoint cannot link runs to a device. * twice yields two different pseudonyms, so an upload endpoint cannot link runs to a device.
* A stable salt trades that away for cross-run diffing and is only appropriate on a server you * A stable salt trades that away for cross-run diffing and is only appropriate on a server you
* own — the app makes that an explicit choice, not a default. * own — the app makes that an explicit choice, not a default.
*/ */
class Salt private constructor(internal val bytes: ByteArray, val stable: Boolean) { class Salt private constructor(internal val bytes: ByteArray, val stable: Boolean) {
companion object { companion object {
fun perRun(random: ByteArray): Salt = Salt(random.copyOf(), stable = false) fun perRun(random: ByteArray): Salt = Salt(random.copyOf(), stable = false)
fun stable(secret: ByteArray): Salt = Salt(secret.copyOf(), stable = true) fun stable(secret: ByteArray): Salt = Salt(secret.copyOf(), stable = true)
} }
} }
/** /**
* Transforms a measurement document to [level]. * Transforms a measurement document to [level].
* *
* Field classification is by JSON key name, because the schema names things consistently * Field classification is by JSON key name, because the schema names things consistently
* (`ssid`, `bssid`, `mac`, `ip4`, `ip6`, `fqdn`, …) and a name-driven pass is auditable by * (`ssid`, `bssid`, `mac`, `ip4`, `ip6`, `fqdn`, …) and a name-driven pass is auditable by
* reading one table. Anything unrecognized is treated as identifying when it is a string inside * reading one table. Anything unrecognized is treated as identifying when it is a string inside
* a known-sensitive container, and left alone otherwise — see [Classification]. * a known-sensitive container, and left alone otherwise — see [Classification].
*/ */
class Anonymizer(private val level: PrivacyLevel, private val salt: Salt) { class Anonymizer(private val level: PrivacyLevel, private val salt: Salt) {
private val cache = HashMap<String, String>() private val cache = HashMap<String, String>()
fun anonymize(doc: JsonObject): JsonObject { fun anonymize(doc: JsonObject): JsonObject {
if (level == PrivacyLevel.FULL) return stamp(doc) if (level == PrivacyLevel.FULL) return stamp(doc)
val walked = walkObject(doc, path = emptyList()) val walked = walkObject(doc, path = emptyList())
val out = if (level == PrivacyLevel.STRICT) strip(walked) else walked val out = if (level == PrivacyLevel.STRICT) strip(walked) else walked
return stamp(out) return stamp(out)
} }
/** Records what was done, so a reader of the archived/uploaded document is never guessing. */ /** Records what was done, so a reader of the archived/uploaded document is never guessing. */
private fun stamp(doc: JsonObject): JsonObject { private fun stamp(doc: JsonObject): JsonObject {
val run = doc["run"]?.jsonObject ?: return doc val run = doc["run"]?.jsonObject ?: return doc
val privacy = buildJsonObject { val privacy = buildJsonObject {
put("anonymization", level.wire) put("anonymization", level.wire)
put("salt", if (salt.stable) "stable" else "per_run") put("salt", if (salt.stable) "stable" else "per_run")
} }
return JsonObject(doc + ("run" to JsonObject(run + ("privacy" to privacy)))) return JsonObject(doc + ("run" to JsonObject(run + ("privacy" to privacy))))
} }
// ---- the tree walk ------------------------------------------------------------------- // ---- the tree walk -------------------------------------------------------------------
private fun walkObject(obj: JsonObject, path: List<String>): JsonObject = buildJsonObject { private fun walkObject(obj: JsonObject, path: List<String>): JsonObject = buildJsonObject {
for ((k, v) in obj) { for ((k, v) in obj) {
val childPath = path + k val childPath = path + k
when { when {
Classification.dropAtBalanced(childPath) -> Unit // omit entirely Classification.dropAtBalanced(childPath) -> Unit // omit entirely
else -> put(k, walk(k, v, childPath)) else -> put(k, walk(k, v, childPath))
} }
} }
} }
private fun walk(key: String, v: JsonElement, path: List<String>): JsonElement = when (v) { private fun walk(key: String, v: JsonElement, path: List<String>): JsonElement = when (v) {
is JsonObject -> walkObject(v, path) is JsonObject -> walkObject(v, path)
is JsonArray -> JsonArray(v.map { walk(key, it, path) }) is JsonArray -> JsonArray(v.map { walk(key, it, path) })
is JsonPrimitive -> is JsonPrimitive ->
if (v.isString) { if (v.isString) {
// Name first (it is precise), then shape (it is exhaustive). A field nobody // Name first (it is precise), then shape (it is exhaustive). A field nobody
// classified must not be a field that leaks. // classified must not be a field that leaks.
val type = Classification.typeOf(key, path) ?: Classification.inferFromValue(v.content) val type = Classification.typeOf(key, path) ?: Classification.inferFromValue(v.content)
JsonPrimitive(transform(type, v.content)) JsonPrimitive(transform(type, v.content))
} else { } else {
v v
} }
} }
private fun transform(type: LogicalType?, value: String): String = when (type) { private fun transform(type: LogicalType?, value: String): String = when (type) {
null -> value // Unclassified strings still get their *embedded* identifiers scrubbed. A whole-value
LogicalType.SSID -> pseudo("ssid", value) { "net-" + it.take(6) } // check cannot see them: raw shell output is one long string that is neither a MAC nor an
LogicalType.MAC, LogicalType.BSSID -> macPreservingOui(value) // address, so it sailed through both the name table and the shape check carrying every
LogicalType.IP4 -> ip4(value) // MAC on the user's LAN.
LogicalType.IP6 -> ip6(value) null -> scrubEmbedded(value)
LogicalType.FQDN -> fqdn(value) LogicalType.SSID -> pseudo("ssid", value) { "net-" + it.take(6) }
LogicalType.OPAQUE_ID -> "redacted" LogicalType.MAC, LogicalType.BSSID -> macPreservingOui(value)
LogicalType.FREETEXT -> "[removed: may contain identifying text]" LogicalType.IP4 -> ip4(value)
} LogicalType.IP6 -> ip6(value)
LogicalType.FQDN -> fqdn(value)
// ---- per-type transforms ------------------------------------------------------------- LogicalType.OPAQUE_ID -> "redacted"
LogicalType.FREETEXT -> "[removed: may contain identifying text]"
/** }
* Keeps the OUI (which vendor) and pseudonymizes the NIC part (which unit). Vendor is the
* diagnostically valuable half — "the RA comes from a MikroTik" survives, "…from THAT /**
* MikroTik" does not. * Replaces addresses and MACs found *inside* a longer string.
*/ *
private fun macPreservingOui(value: String): String { * Shizuku probes embed raw command output verbatim — `ip neigh`, `ip route`, `dumpsys` — which
val sep = if (value.contains('-')) '-' else ':' * is genuinely valuable evidence and also a complete inventory of every device on the user's
val parts = value.split(sep) * network, with hardware addresses. measurement-schema.md §9 flagged these as "hard to
if (parts.size != 6 || parts.any { it.length != 2 }) return pseudo("mac", value) { "mac-" + it.take(8) } * anonymize" and proposed dropping them from exports.
val nic = pseudo("mac", value) { it } *
return (parts.take(3) + listOf(nic.substring(0, 2), nic.substring(2, 4), nic.substring(4, 6))) * Scrubbing beats dropping: the output stays readable and auditable — you can still see the
.joinToString(sep.toString()) * shape of the neighbour table and how many hosts there were — while the identifiers become
.lowercase(Locale.ROOT) * the same pseudonyms used everywhere else in the document. So a MAC appearing both in a
} * parsed field and in a raw dump still reads as one device.
*
/** * Only addresses and MACs are touched, for the same reason as [Classification.inferFromValue]:
* Prefix-preserving within the same class, with reserved ranges kept verbatim: RFC1918 and * they are the patterns that cannot be mistaken for something else in free text.
* CGNAT addresses say something about the topology and nothing about the person, and a run */
* where 192.168.1.1 became a random public address would be actively misleading to read. private fun scrubEmbedded(value: String): String {
* Public addresses keep only their /16 so the network is still locatable at ISP granularity. // Cheap bail-out: the overwhelming majority of strings are short and contain neither.
*/ if (value.length < 7 || (!value.contains(':') && !value.contains('.'))) return value
private fun ip4(value: String): String { // One pass, not three. Sequential passes re-process their own output: after a MAC became
// A route destination carries a prefix length; pseudonymize the address and put it back, // 78:9a:18:xx:yy:zz the IPv6 pattern matched it — six hex groups separated by colons is
// or "0.0.0.0/0" turns into nonsense and the routing table becomes unreadable. // exactly an address — and mangled the vendor prefix that the MAC rule had just taken
value.substringAfter('/', "").takeIf { it.isNotEmpty() && value.contains('/') }?.let { len -> // care to preserve. Ordered alternation resolves each position once, MAC first.
return ip4(value.substringBefore('/')) + "/" + len return EMBEDDED.replace(value) { m ->
} when {
val o = value.split(".") m.groups[1] != null -> macPreservingOui(m.value)
if (o.size != 4 || o.any { it.toIntOrNull() == null }) return value m.groups[2] != null -> ip6(m.value)
val n = o.map { it.toInt() } else -> ip4(m.value)
val reserved = n[0] == 10 || }
(n[0] == 172 && n[1] in 16..31) || }
(n[0] == 192 && n[1] == 168) || }
(n[0] == 169 && n[1] == 254) ||
(n[0] == 100 && n[1] in 64..127) || // ---- per-type transforms -------------------------------------------------------------
n[0] == 127 || n[0] == 0 || n[0] >= 224
if (reserved) return value /**
val h = pseudo("ip4", value) { it } * Keeps the OUI (which vendor) and pseudonymizes the NIC part (which unit). Vendor is the
return "${n[0]}.${n[1]}.${h.substring(0, 2).toInt(16)}.${h.substring(2, 4).toInt(16)}" * diagnostically valuable half — "the RA comes from a MikroTik" survives, "…from THAT
} * MikroTik" does not.
*/
/** private fun macPreservingOui(value: String): String {
* IPv6 keeps the scope and the first 32 bits (so 2001:db8:… still reads as global unicast in val sep = if (value.contains('-')) '-' else ':'
* the same allocation) and pseudonymizes the rest — the interface identifier is the part that val parts = value.split(sep)
* is a device fingerprint, especially with EUI-64. if (parts.size != 6 || parts.any { it.length != 2 }) return pseudo("mac", value) { "mac-" + it.take(8) }
*/ val nic = pseudo("mac", value) { it }
private fun ip6(value: String): String { return (parts.take(3) + listOf(nic.substring(0, 2), nic.substring(2, 4), nic.substring(4, 6)))
// Dotted quads reach here through the family-agnostic field names (addr, gateway, dst); .joinToString(sep.toString())
// hand them to the IPv4 path rather than mangling them as if they were v6. .lowercase(Locale.ROOT)
if (value.count { it == ':' } < 2) return ip4(value) }
if (value.contains('/')) {
return ip6(value.substringBefore('/')) + "/" + value.substringAfter('/') /**
} * Prefix-preserving within the same class, with reserved ranges kept verbatim: RFC1918 and
val v = value.lowercase(Locale.ROOT) * CGNAT addresses say something about the topology and nothing about the person, and a run
// The unspecified address and the default route are not identities; mangling them would * where 192.168.1.1 became a random public address would be actively misleading to read.
// make a routing table unreadable for no privacy gain. * Public addresses keep only their /16 so the network is still locatable at ISP granularity.
if (v == "::1" || v == "::" || v.startsWith("fe80:") || v.startsWith("ff")) return v */
private fun ip4(value: String): String {
// Unique local addresses (fc00::/7) need the *whole* prefix replaced, not the tail. // A route destination carries a prefix length; pseudonymize the address and put it back,
// // or "0.0.0.0/0" turns into nonsense and the routing table becomes unreadable.
// They look like the v6 equivalent of RFC1918, and the first instinct is to keep them for value.substringAfter('/', "").takeIf { it.isNotEmpty() && value.contains('/') }?.let { len ->
// the same reason: private, topological, says nothing about anyone. That reasoning does return ip4(value.substringBefore('/')) + "/" + len
// not carry over. An RFC1918 prefix is shared by millions of networks and identifies }
// none of them; a ULA global ID is 40 *random* bits, unique to one network by val o = value.split(".")
// construction (RFC 4193). It is a network fingerprint. Passing the leading groups if (o.size != 4 || o.any { it.toIntOrNull() == null }) return value
// through - which is what the general path does - leaked 32 of those 40 bits. val n = o.map { it.toInt() }
// val reserved = n[0] == 10 ||
// The prefix is pseudonymized as a unit, so two addresses on the same ULA subnet still (n[0] == 172 && n[1] in 16..31) ||
// land on the same pseudonymous prefix. "These hosts are on one network" survives; (n[0] == 192 && n[1] == 168) ||
// "this is *that* network" does not. (n[0] == 169 && n[1] == 254) ||
if (v.startsWith("fc") || v.startsWith("fd")) { (n[0] == 100 && n[1] in 64..127) ||
val groups = v.substringBefore('%').split(":") n[0] == 127 || n[0] == 0 || n[0] >= 224
val prefix = pseudo("ula-prefix", groups.take(3).joinToString(":")) { it } if (reserved) return value
val host = pseudo("ula-host", v) { it } val h = pseudo("ip4", value) { it }
return "fd${prefix.substring(0, 2)}:${prefix.substring(2, 6)}:${prefix.substring(6, 10)}" + return "${n[0]}.${n[1]}.${h.substring(0, 2).toInt(16)}.${h.substring(2, 4).toInt(16)}"
"::${host.substring(0, 4)}" }
}
val groups = v.substringBefore('%').split(":") /**
if (groups.size < 3) return v * IPv6 keeps the scope and the first 32 bits (so 2001:db8:… still reads as global unicast in
val h = pseudo("ip6", value) { it } * the same allocation) and pseudonymizes the rest — the interface identifier is the part that
return "${groups[0]}:${groups[1]}:${h.substring(0, 4)}:${h.substring(4, 8)}::${h.substring(8, 12)}" * is a device fingerprint, especially with EUI-64.
} */
private fun ip6(value: String): String {
/** // Dotted quads reach here through the family-agnostic field names (addr, gateway, dst);
* Per-label pseudonyms with the public suffix kept, so "it resolved somewhere under .local" // hand them to the IPv4 path rather than mangling them as if they were v6.
* or "…under example.com" survives without naming the host. The suffix list is deliberately if (value.count { it == ':' } < 2) return ip4(value)
* short: guessing wrong keeps *more* pseudonymized, never less. if (value.contains('/')) {
*/ return ip6(value.substringBefore('/')) + "/" + value.substringAfter('/')
private fun fqdn(value: String): String { }
if (value.isEmpty()) return value val v = value.lowercase(Locale.ROOT)
val trailing = value.endsWith(".") // The unspecified address and the default route are not identities; mangling them would
val labels = value.trimEnd('.').split(".") // make a routing table unreadable for no privacy gain.
if (labels.size == 1) return pseudo("fqdn", value) { "host-" + it.take(6) } if (v == "::1" || v == "::" || v.startsWith("fe80:") || v.startsWith("ff")) return v
val keep = if (labels.last() in publicSuffixes) 1 else 0
val head = labels.dropLast(keep).map { l -> pseudo("label", l) { "l-" + it.take(6) } } // Unique local addresses (fc00::/7) need the *whole* prefix replaced, not the tail.
return (head + labels.takeLast(keep)).joinToString(".") + if (trailing) "." else "" //
} // They look like the v6 equivalent of RFC1918, and the first instinct is to keep them for
// the same reason: private, topological, says nothing about anyone. That reasoning does
// ---- STRICT --------------------------------------------------------------------------- // not carry over. An RFC1918 prefix is shared by millions of networks and identifies
// none of them; a ULA global ID is 40 *random* bits, unique to one network by
/** // construction (RFC 4193). It is a network fingerprint. Passing the leading groups
* STRICT keeps the shape of the document and the numbers, and nothing that quotes the // through - which is what the general path does - leaked 32 of those 40 bits.
* network back. Evidence goes (trains carry addresses and hostnames), finding prose goes //
* (it interpolates real names), networks go entirely. // The prefix is pseudonymized as a unit, so two addresses on the same ULA subnet still
*/ // land on the same pseudonymous prefix. "These hosts are on one network" survives;
private fun strip(doc: JsonObject): JsonObject = buildJsonObject { // "this is *that* network" does not.
for ((k, v) in doc) { if (v.startsWith("fc") || v.startsWith("fd")) {
when (k) { val groups = v.substringBefore('%').split(":")
"networks", "server_sessions" -> Unit val prefix = pseudo("ula-prefix", groups.take(3).joinToString(":")) { it }
"tests" -> put(k, JsonArray((v as? JsonArray ?: JsonArray(emptyList())).map { t -> val host = pseudo("ula-host", v) { it }
val o = t.jsonObject return "fd${prefix.substring(0, 2)}:${prefix.substring(2, 6)}:${prefix.substring(6, 10)}" +
JsonObject(o.filterKeys { it != "evidence" && it != "params" }) "::${host.substring(0, 4)}"
})) }
"findings" -> put(k, JsonArray((v as? JsonArray ?: JsonArray(emptyList())).map { f -> val groups = v.substringBefore('%').split(":")
val o = f.jsonObject if (groups.size < 3) return v
JsonObject(o.filterKeys { it != "description" && it != "title" && it != "evidence_refs" }) val h = pseudo("ip6", value) { it }
})) return "${groups[0]}:${groups[1]}:${h.substring(0, 4)}:${h.substring(4, 8)}::${h.substring(8, 12)}"
"run" -> put(k, JsonObject(v.jsonObject.filterKeys { it != "notes" })) }
else -> put(k, v)
} /**
} * Per-label pseudonyms with the public suffix kept, so "it resolved somewhere under .local"
} * or "…under example.com" survives without naming the host. The suffix list is deliberately
* short: guessing wrong keeps *more* pseudonymized, never less.
// ---- pseudonym machinery --------------------------------------------------------------- */
private fun fqdn(value: String): String {
/** Deterministic per (domain, value, salt); memoized so one value maps to one pseudonym. */ if (value.isEmpty()) return value
private fun pseudo(domain: String, value: String, shape: (String) -> String): String = val trailing = value.endsWith(".")
cache.getOrPut("$domain$value") { val labels = value.trimEnd('.').split(".")
val md = MessageDigest.getInstance("SHA-256") if (labels.size == 1) return pseudo("fqdn", value) { "host-" + it.take(6) }
md.update(salt.bytes) val keep = if (labels.last() in publicSuffixes) 1 else 0
md.update(domain.toByteArray()) val head = labels.dropLast(keep).map { l -> pseudo("label", l) { "l-" + it.take(6) } }
md.update(0) return (head + labels.takeLast(keep)).joinToString(".") + if (trailing) "." else ""
md.update(value.lowercase(Locale.ROOT).toByteArray()) }
shape(md.digest().joinToString("") { "%02x".format(it) })
} // ---- STRICT ---------------------------------------------------------------------------
private companion object { /**
val publicSuffixes = setOf( * STRICT keeps the shape of the document and the numbers, and nothing that quotes the
"local", "lan", "home", "internal", "arpa", * network back. Evidence goes (trains carry addresses and hostnames), finding prose goes
"com", "net", "org", "io", "app", "dev", "at", "de", "eu", "uk", * (it interpolates real names), networks go entirely.
) */
} private fun strip(doc: JsonObject): JsonObject = buildJsonObject {
} for ((k, v) in doc) {
when (k) {
"networks", "server_sessions" -> Unit
"tests" -> put(k, JsonArray((v as? JsonArray ?: JsonArray(emptyList())).map { t ->
val o = t.jsonObject
JsonObject(o.filterKeys { it != "evidence" && it != "params" })
}))
"findings" -> put(k, JsonArray((v as? JsonArray ?: JsonArray(emptyList())).map { f ->
val o = f.jsonObject
JsonObject(o.filterKeys { it != "description" && it != "title" && it != "evidence_refs" })
}))
"run" -> put(k, JsonObject(v.jsonObject.filterKeys { it != "notes" }))
else -> put(k, v)
}
}
}
// ---- pseudonym machinery ---------------------------------------------------------------
/** Deterministic per (domain, value, salt); memoized so one value maps to one pseudonym. */
private fun pseudo(domain: String, value: String, shape: (String) -> String): String =
cache.getOrPut("$domain\u0000$value") {
val md = MessageDigest.getInstance("SHA-256")
md.update(salt.bytes)
md.update(domain.toByteArray())
md.update(0)
md.update(value.lowercase(Locale.ROOT).toByteArray())
shape(md.digest().joinToString("") { "%02x".format(it) })
}
private companion object {
/**
* MAC | IPv6 | IPv4, in that order — alternation is ordered, so a MAC-shaped token is
* claimed by the MAC rule before the IPv6 rule can see it.
*
* The patterns are deliberately conservative. A missed address is scrubbed by another
* rule or not at all; an over-eager one mangles timestamps, version strings and log
* prefixes, corrupting evidence to protect nothing.
*/
val EMBEDDED = Regex(
// Raw strings: a regex written with escaped escapes is a regex nobody can check.
"""(\b[0-9a-fA-F]{2}(?:[:-][0-9a-fA-F]{2}){5}\b)""" +
"""|(\b(?:[0-9a-fA-F]{1,4}:){2,7}(?::|[0-9a-fA-F]{1,4})(?:[0-9a-fA-F:]*))""" +
"""|(\b(?:\d{1,3}\.){3}\d{1,3}\b)"""
)
val publicSuffixes = setOf(
"local", "lan", "home", "internal", "arpa",
"com", "net", "org", "io", "app", "dev", "at", "de", "eu", "uk",
)
}
}
echolot-app/core-privacy/src/test/kotlin/app/echo_lot/privacy/LeakTest.kt
+65 -1
View File
@@ -4,8 +4,12 @@
package app.echo_lot.privacy package app.echo_lot.privacy
import kotlinx.serialization.json.Json import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.jsonArray
import kotlinx.serialization.json.jsonObject import kotlinx.serialization.json.jsonObject
import kotlinx.serialization.json.jsonPrimitive
import kotlin.test.Test import kotlin.test.Test
import kotlin.test.assertFalse
import kotlin.test.assertTrue import kotlin.test.assertTrue
/** /**
@@ -110,4 +114,64 @@ class LeakTest {
assertTrue(out.contains("192.168.1.1"), "RFC1918 gateway should survive: $out") assertTrue(out.contains("192.168.1.1"), "RFC1918 gateway should survive: $out")
assertTrue(out.contains("192.168.1.44"), "RFC1918 interface address should survive: $out") assertTrue(out.contains("192.168.1.44"), "RFC1918 interface address should survive: $out")
} }
}
/**
* Raw shell output embeds a complete inventory of the local network, and neither the field-name
* table nor the whole-value shape check can see it: `ip_neigh` is one long string that is
* itself neither a MAC nor an address.
*
* This is not hypothetical. The blob below is (abridged) real output that reached the server
* at the `balanced` level from a test device, carrying the hardware address of every host on
* the network. measurement-schema.md §9 had flagged raw dumps as "hard to anonymize"; nothing
* enforced it.
*/
@Test
fun identifiersInsideRawShellOutputAreScrubbed() {
// Joined rather than written with escapes, so the fixture stays readable and there is no
// chance of an escape being mangled on its way into the JSON below.
val dump = listOf(
"uid=2000",
"10.13.102.5 dev wlan0 lladdr 90:09:d0:1a:83:e4 REACHABLE",
"10.13.102.1 dev wlan0 lladdr 78:9a:18:54:b8:f9 REACHABLE",
"10.13.102.111 dev wlan0 lladdr dc:a2:66:08:69:95 STALE",
"2001:4bb8:46a:e724:289d:87ff:feb6:ebd3 dev wlan0 lladdr b8:be:f4:bc:ca:cf STALE",
).joinToString(" | ")
val doc = json.parseToJsonElement(
"""{"run":{"id":"r"},"tests":[{"id":"t","type":"link.ip_monitor",
"evidence":{"ip_neigh":"$dump"}}]}"""
).jsonObject
val out = json.encodeToString(
kotlinx.serialization.json.JsonObject.serializer(),
Anonymizer(PrivacyLevel.BALANCED, salt).anonymize(doc),
)
for (mac in listOf("90:09:d0:1a:83:e4", "78:9a:18:54:b8:f9", "dc:a2:66:08:69:95", "b8:be:f4:bc:ca:cf")) {
assertFalse(out.contains(mac), "a neighbour's MAC survived inside the raw dump: $mac")
}
assertFalse(out.contains("2001:4bb8:46a:e724:289d:87ff:feb6:ebd3"),
"a global IPv6 survived inside the raw dump")
// Scrubbed, not dropped: the evidence must still be readable, or the raw dump stops being
// evidence at all. Structure, hostnames of the fields, and RFC1918 addresses stay.
assertTrue(out.contains("REACHABLE") && out.contains("STALE"), "the dump lost its structure")
assertTrue(out.contains("10.13.102.1"), "RFC1918 addresses should stay readable: $out")
assertTrue(out.contains("78:9a:18"), "the vendor prefix should survive for identification")
}
// A MAC in a raw dump and the same MAC in a parsed field must land on the same pseudonym, or
// the document stops being internally consistent and one device reads as two.
@Test
fun theSameIdentifierMatchesAcrossParsedAndRawFields() {
val doc = json.parseToJsonElement(
"""{"run":{"id":"r"},
"networks":[{"wifi":{"bssid":"78:9a:18:54:b8:f9"}}],
"tests":[{"id":"t","evidence":{"ip_neigh":"gw dev wlan0 lladdr 78:9a:18:54:b8:f9 REACHABLE"}}]}"""
).jsonObject
val out = Anonymizer(PrivacyLevel.BALANCED, salt).anonymize(doc)
val parsed = out["networks"]!!.jsonArray[0].jsonObject["wifi"]!!.jsonObject["bssid"]!!
.jsonPrimitive.content
val raw = json.encodeToString(kotlinx.serialization.json.JsonObject.serializer(), out)
assertTrue(raw.contains(parsed),
"the parsed BSSID pseudonym ($parsed) does not appear in the scrubbed dump")
}
}
echolot-app/core-privacy/src/test/kotlin/app/echo_lot/privacy/RealDocumentTest.kt
+48
View File
@@ -0,0 +1,48 @@
// SPDX-FileCopyrightText: 2026 Echolot contributors
// SPDX-License-Identifier: GPL-3.0-or-later
package app.echo_lot.privacy
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.jsonObject
import java.io.File
import kotlin.test.Test
import kotlin.test.assertTrue
/**
* Runs the anonymizer over a real captured document when one is supplied via ECHOLOT_REAL_RUN,
* and reports every MAC and public address that survives.
*
* Fixtures only contain the identifiers somebody thought to put in them. A real run off a real
* phone contains whatever the probes actually produce — which is how the raw-shell-output leak was
* found in the first place. Self-skips when no document is supplied, so nobody's network ends up
* committed to the repository.
*/
class RealDocumentTest {
@Test
fun noIdentifiersSurviveInARealDocument() {
val path = System.getenv("ECHOLOT_REAL_RUN")
if (path.isNullOrBlank() || !File(path).isFile) {
println("RealDocumentTest skipped (set ECHOLOT_REAL_RUN to a captured run)"); return
}
val json = Json { prettyPrint = false }
val doc = json.parseToJsonElement(File(path).readText()).jsonObject
val out = json.encodeToString(
JsonObject.serializer(),
Anonymizer(PrivacyLevel.BALANCED, Salt.perRun(ByteArray(32) { 5 })).anonymize(doc),
)
val macs = Regex("""\b[0-9a-fA-F]{2}(?::[0-9a-fA-F]{2}){5}\b""").findAll(out)
.map { it.value.lowercase() }
.filter { it != "00:00:00:00:00:00" }
.toSet()
val original = Regex("""\b[0-9a-fA-F]{2}(?::[0-9a-fA-F]{2}){5}\b""")
.findAll(File(path).readText()).map { it.value.lowercase() }.toSet()
val survived = macs intersect original
println("MACs in the original: ${original.size}; unchanged after anonymizing: ${survived.size}")
assertTrue(survived.isEmpty(), "these real MAC addresses survived anonymization: $survived")
}
}