Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ce6d0c2f64 | ||
|
|
57a5ef8796 | ||
|
|
c19f382640 | ||
|
|
d04babff51 |
@@ -941,3 +941,64 @@ Also: a `BackHandler` now returns from Settings/History to the run screen. The s
|
|||||||
state variable with nothing connecting it to the back stack, so the system Back gesture left the
|
state variable with nothing connecting it to the back stack, so the system Back gesture left the
|
||||||
app entirely. Enabled only when there is somewhere to go back to, so Back still exits from the run
|
app entirely. Enabled only when there is somewhere to go back to, so Back still exits from the run
|
||||||
screen.
|
screen.
|
||||||
|
|
||||||
|
### Upstream throughput (server-v0.6.3, 2026-08-01)
|
||||||
|
The mirror of the downstream case: the client generates the traffic and the server counts it. No
|
||||||
|
grant is involved — the client is sending its own packets, so there is nothing to amplify — but it
|
||||||
|
does need the server's tally, because **only the far end knows how much arrived**. Without that
|
||||||
|
number a sender measures how fast it can *transmit*, which is usually just the speed of the local
|
||||||
|
NIC and is a different question from the one being asked.
|
||||||
|
|
||||||
|
`TYPE_THROUGHPUT_UP` (0x0F) is counted and deliberately **never answered**: a reply would double
|
||||||
|
the traffic and drag the return path into a measurement that is specifically about the outbound
|
||||||
|
one.
|
||||||
|
|
||||||
|
The tally is a counter, not a list, and short-circuits **before** the observation log. A
|
||||||
|
five-second run at 20 Mbps is around ten thousand packets; one struct each would turn a
|
||||||
|
measurement into an allocation storm on a shared server, and nothing needs the per-packet detail
|
||||||
|
since the client holds the send-side record. The gap between the two counts is the loss.
|
||||||
|
|
||||||
|
`direction=up` on the throughput action sends nothing — it zeroes the counter, so a second run in
|
||||||
|
one session measures itself rather than inheriting the first one's packets. The live test asserts
|
||||||
|
`received <= sent`, which is what catches a counter that was never reset.
|
||||||
|
|
||||||
|
Live against fmr: **3125 sent, 3125 counted, 0 % loss, 10.0 Mbit/s** at a 10 Mbit/s request, with
|
||||||
|
`measures_network: false` — correct, since what arrived matched what was offered, so the path was
|
||||||
|
never the constraint.
|
||||||
|
|
||||||
|
### Raw shell dumps leaked the whole LAN (2026-08-01)
|
||||||
|
Found by running the Shizuku shell tier for the first time. The tier works — `tiers.shizuku: true`,
|
||||||
|
`exec_path: UserService` (so the UserService binds on the OnePlus, as recorded), `runs_as
|
||||||
|
shell(2000)`, 7/7 commands — and the run promptly uploaded **every MAC address on the local
|
||||||
|
network** to fmr at the `balanced` level: router, phones, whatever else was on the wifi. Fourteen
|
||||||
|
of them.
|
||||||
|
|
||||||
|
The probes embed raw command output verbatim (`ip neigh`, `ip route`, `id`), which is genuinely
|
||||||
|
good evidence and also a complete household device inventory. The anonymizer could not see it:
|
||||||
|
classification is by field name and by whole-value shape, and `ip_neigh` is one long string that is
|
||||||
|
itself neither a MAC nor an address. measurement-schema.md §9 item 2 had flagged raw dumps as "hard
|
||||||
|
to anonymize" and proposed dropping them from exports; nothing enforced either.
|
||||||
|
|
||||||
|
**Scrubbing beats dropping.** Identifiers inside any unclassified string are now replaced in place,
|
||||||
|
using the same pseudonyms as everywhere else — so a MAC that appears both in a parsed field and in
|
||||||
|
a raw dump still reads as one device. The dump stays readable and auditable: you can still see the
|
||||||
|
neighbour table's shape, the host count, RFC1918 addresses and vendor prefixes. Dropping the
|
||||||
|
evidence would have protected the same data while destroying the reason for collecting it.
|
||||||
|
|
||||||
|
Two implementation notes worth keeping:
|
||||||
|
- **One pass, not three.** Sequential passes re-process their own output: once a MAC became
|
||||||
|
`78:9a:18:xx:yy:zz`, the IPv6 pattern matched it — six hex groups separated by colons *is* an
|
||||||
|
address — and destroyed the vendor prefix the MAC rule had just preserved. Ordered alternation
|
||||||
|
resolves each position once, MAC first.
|
||||||
|
- The patterns are conservative on purpose. A missed address gets caught by another rule or not at
|
||||||
|
all; an over-eager one mangles timestamps and version strings, corrupting evidence to protect
|
||||||
|
nothing.
|
||||||
|
|
||||||
|
`RealDocumentTest` runs the anonymizer over a captured run when `ECHOLOT_REAL_RUN` points at one,
|
||||||
|
and fails on any MAC that survives. It self-skips otherwise, so no one's network is committed to the
|
||||||
|
repo. Against the actual leaked document: **14 MACs in, 0 surviving.**
|
||||||
|
|
||||||
|
Also fixed: the Settings *Preview what an upload would send* button did nothing. It read
|
||||||
|
`UiState.history`, which is empty until the History screen has been opened — the same root cause as
|
||||||
|
the "0 run(s)" count. It now reads the archive directly, and says so when there is nothing to
|
||||||
|
preview rather than silently ignoring the tap.
|
||||||
|
|||||||
@@ -101,11 +101,10 @@ class MainActivity : ComponentActivity() {
|
|||||||
onApplyRetention = vm::applyRetention,
|
onApplyRetention = vm::applyRetention,
|
||||||
onDeleteAll = vm::deleteAllRuns,
|
onDeleteAll = vm::deleteAllRuns,
|
||||||
onPreviewUpload = {
|
onPreviewUpload = {
|
||||||
// Preview the newest run, since that is the one the user just made
|
// Straight from the archive: the newest run is the one the user just
|
||||||
// and the one they are deciding about.
|
// made and the one they are deciding about. Always shows something,
|
||||||
vm.state.history.firstOrNull()?.let { r ->
|
// even when there is nothing to preview yet.
|
||||||
lifecycleScope.launch { preview = vm.uploadPreview(r.id) }
|
lifecycleScope.launch { preview = vm.previewNewestRun() }
|
||||||
}
|
|
||||||
},
|
},
|
||||||
onCheckServer = vm::checkServer,
|
onCheckServer = vm::checkServer,
|
||||||
onEnroll = vm::enroll,
|
onEnroll = vm::enroll,
|
||||||
|
|||||||
@@ -194,6 +194,21 @@ class RunViewModel(app: Application) : AndroidViewModel(app) {
|
|||||||
store.read(id)?.let { store.redactedForUpload(it) }
|
store.read(id)?.let { store.redactedForUpload(it) }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Preview of the most recent run, read from the archive rather than from [UiState.history].
|
||||||
|
*
|
||||||
|
* The history list is only populated once the History screen has been opened, so a preview
|
||||||
|
* driven from it did nothing at all on a freshly-opened Settings screen — a button that
|
||||||
|
* silently does nothing is worse than one that says why.
|
||||||
|
*/
|
||||||
|
suspend fun previewNewestRun(): String = withContext(Dispatchers.IO) {
|
||||||
|
val newest = store.list().firstOrNull()
|
||||||
|
?: return@withContext "No archived runs yet. Run a measurement first, then this will " +
|
||||||
|
"show exactly what an upload would send."
|
||||||
|
store.read(newest.id)?.let { store.redactedForUpload(it) }
|
||||||
|
?: "That run could not be read back from the archive."
|
||||||
|
}
|
||||||
|
|
||||||
fun archivedBytes(): Long = store.totalBytes()
|
fun archivedBytes(): Long = store.totalBytes()
|
||||||
|
|
||||||
/** Counted from the archive itself, not from [UiState.history], which is empty until the
|
/** Counted from the archive itself, not from [UiState.history], which is empty until the
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ import androidx.compose.foundation.verticalScroll
|
|||||||
import androidx.compose.material3.Button
|
import androidx.compose.material3.Button
|
||||||
import androidx.compose.material3.Card
|
import androidx.compose.material3.Card
|
||||||
import androidx.compose.material3.FilterChip
|
import androidx.compose.material3.FilterChip
|
||||||
|
import androidx.compose.material3.LocalContentColor
|
||||||
import androidx.compose.material3.MaterialTheme
|
import androidx.compose.material3.MaterialTheme
|
||||||
import androidx.compose.material3.OutlinedTextField
|
import androidx.compose.material3.OutlinedTextField
|
||||||
import androidx.compose.material3.Switch
|
import androidx.compose.material3.Switch
|
||||||
@@ -130,12 +131,21 @@ fun SettingsScreen(
|
|||||||
}
|
}
|
||||||
Text(privacyExplanation(privacy), style = MaterialTheme.typography.bodySmall)
|
Text(privacyExplanation(privacy), style = MaterialTheme.typography.bodySmall)
|
||||||
|
|
||||||
|
// At FULL nothing is pseudonymized, so a salt has nothing to act on. Shown
|
||||||
|
// disabled rather than hidden: the setting is still stored and still applies the
|
||||||
|
// moment the level changes, and a control that vanishes hides that fact.
|
||||||
Toggle(
|
Toggle(
|
||||||
label = "Stable pseudonyms across runs",
|
label = "Stable pseudonyms across runs",
|
||||||
detail = "Lets you compare uploaded runs over time (same SSID reads the same " +
|
detail = if (privacy == PrivacyLevel.FULL) {
|
||||||
"each time). It also links your uploads together, so leave it off on a " +
|
"Not used at this level — nothing is pseudonymized, so there is nothing " +
|
||||||
"server you don't run yourself.",
|
"to keep stable. Choose balanced or strict to use this."
|
||||||
checked = stableSalt,
|
} else {
|
||||||
|
"Lets you compare uploaded runs over time (same SSID reads the same " +
|
||||||
|
"each time). It also links your uploads together, so leave it off on " +
|
||||||
|
"a server you don't run yourself."
|
||||||
|
},
|
||||||
|
checked = stableSalt && privacy != PrivacyLevel.FULL,
|
||||||
|
enabled = privacy != PrivacyLevel.FULL,
|
||||||
) { stableSalt = it; settings.stableSalt = it }
|
) { stableSalt = it; settings.stableSalt = it }
|
||||||
|
|
||||||
TextButton(onClick = onPreviewUpload) { Text("Preview what an upload would send") }
|
TextButton(onClick = onPreviewUpload) { Text("Preview what an upload would send") }
|
||||||
@@ -237,13 +247,22 @@ private fun privacyExplanation(level: PrivacyLevel): String = when (level) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
@Composable
|
@Composable
|
||||||
private fun Toggle(label: String, detail: String, checked: Boolean, onChange: (Boolean) -> Unit) {
|
private fun Toggle(
|
||||||
|
label: String,
|
||||||
|
detail: String,
|
||||||
|
checked: Boolean,
|
||||||
|
enabled: Boolean = true,
|
||||||
|
onChange: (Boolean) -> Unit,
|
||||||
|
) {
|
||||||
Row(Modifier.fillMaxWidth(), verticalAlignment = Alignment.Top) {
|
Row(Modifier.fillMaxWidth(), verticalAlignment = Alignment.Top) {
|
||||||
Column(Modifier.weight(1f)) {
|
Column(Modifier.weight(1f)) {
|
||||||
Text(label, style = MaterialTheme.typography.bodyMedium)
|
// Dimmed together with the switch, so "this does nothing right now" reads at a glance
|
||||||
Text(detail, style = MaterialTheme.typography.bodySmall)
|
// instead of only on close inspection.
|
||||||
|
val alpha = if (enabled) 1f else 0.5f
|
||||||
|
Text(label, style = MaterialTheme.typography.bodyMedium, color = LocalContentColor.current.copy(alpha = alpha))
|
||||||
|
Text(detail, style = MaterialTheme.typography.bodySmall, color = LocalContentColor.current.copy(alpha = alpha))
|
||||||
}
|
}
|
||||||
Switch(checked = checked, onCheckedChange = onChange)
|
Switch(checked = checked, onCheckedChange = onChange, enabled = enabled)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -65,4 +65,41 @@ class LiveThroughputTest {
|
|||||||
assertTrue(m.contains(""""limited_by":"duration""""),
|
assertTrue(m.contains(""""limited_by":"duration""""),
|
||||||
"the run did not end on the clock, so the rate measures the server, not the path: $m")
|
"the run did not end on the clock, so the rate measures the server, not the path: $m")
|
||||||
}
|
}
|
||||||
}
|
|
||||||
|
// Upstream is the direction only the far end can measure. The assertion that matters is that
|
||||||
|
// the server's count is present and plausible against what we sent — a test that only checked
|
||||||
|
// "we transmitted some Mbps" would pass against a server that counted nothing at all.
|
||||||
|
@Test
|
||||||
|
fun measuresUpstreamAgainstTheServersCount() {
|
||||||
|
if (url == null || pin == null || cred == null || udp == null) {
|
||||||
|
println("LiveThroughputTest(up) skipped"); return
|
||||||
|
}
|
||||||
|
val control = ControlClient(url, setOf(pin), "0.2.0")
|
||||||
|
val session = control.createSession(cred, target)
|
||||||
|
val (host, port) = udp.split(":").let { it[0] to it[1].toInt() }
|
||||||
|
|
||||||
|
val (test, findings) = ProbeSession(cred, session, host, port).use { ps ->
|
||||||
|
ps.echo()
|
||||||
|
ThroughputMeasurement(SystemIdSource()).runUpstream(
|
||||||
|
cred, session.sessionId, control, ps, sessionRef = "sess-1",
|
||||||
|
durationS = 3, kbps = 10_000,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
control.deleteSession(cred, session.sessionId)
|
||||||
|
|
||||||
|
val m = assertNotNull(test.metrics).toString()
|
||||||
|
println("upstream: ${test.status} $m")
|
||||||
|
for (f in findings) println("finding ${f.code} [${f.severity}] ${f.title}")
|
||||||
|
|
||||||
|
assertEquals(TestStatus.OK, test.status, "the server counted nothing: $m")
|
||||||
|
val recv = Regex(""""received_packets":(\d+)""").find(m)?.groupValues?.get(1)?.toInt()
|
||||||
|
val sent = Regex(""""sent_packets":(\d+)""").find(m)?.groupValues?.get(1)?.toInt()
|
||||||
|
assertNotNull(recv); assertNotNull(sent)
|
||||||
|
assertTrue(sent > 100, "barely anything was sent, so the rate means nothing: $m")
|
||||||
|
assertTrue(recv > 0, "the server received none of $sent packets: $m")
|
||||||
|
// The counts should be close on a healthy path; wildly different means the two sides are
|
||||||
|
// counting different things rather than the network losing packets.
|
||||||
|
assertTrue(recv <= sent, "the server counted MORE than we sent — the counter is not being reset")
|
||||||
|
println("sent $sent, server saw $recv")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -20,4 +20,8 @@ kotlin {
|
|||||||
}
|
}
|
||||||
java { sourceCompatibility = JavaVersion.VERSION_17; targetCompatibility = JavaVersion.VERSION_17 }
|
java { sourceCompatibility = JavaVersion.VERSION_17; targetCompatibility = JavaVersion.VERSION_17 }
|
||||||
|
|
||||||
tasks.test { useJUnitPlatform() }
|
tasks.test {
|
||||||
|
useJUnitPlatform()
|
||||||
|
// Opt-in: point this at a captured run to check the anonymizer against real data.
|
||||||
|
System.getenv("ECHOLOT_REAL_RUN")?.let { environment("ECHOLOT_REAL_RUN", it) }
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,276 +1,327 @@
|
|||||||
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
// SPDX-FileCopyrightText: 2026 Echolot contributors
|
||||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||||
|
|
||||||
// Package privacy implements the anonymization contract of measurement-schema.md §8.
|
// Package privacy implements the anonymization contract of measurement-schema.md §8.
|
||||||
//
|
//
|
||||||
// The threat model is specific. An engineer running their own server wants the full document —
|
// The threat model is specific. An engineer running their own server wants the full document —
|
||||||
// SSIDs and MACs are what make a run useful a week later. Someone measuring against a stranger's
|
// SSIDs and MACs are what make a run useful a week later. Someone measuring against a stranger's
|
||||||
// server wants the numbers to survive and the identifiers not to. So this is a *transform*, not a
|
// server wants the numbers to survive and the identifiers not to. So this is a *transform*, not a
|
||||||
// filter: the output is still a valid measurement document with the same tests, metrics and
|
// filter: the output is still a valid measurement document with the same tests, metrics and
|
||||||
// findings; only the identifying scalars change, and consistently, so "same SSID as last run" is
|
// findings; only the identifying scalars change, and consistently, so "same SSID as last run" is
|
||||||
// still answerable from pseudonyms alone.
|
// still answerable from pseudonyms alone.
|
||||||
//
|
//
|
||||||
// Two properties are load-bearing and are what the tests pin:
|
// Two properties are load-bearing and are what the tests pin:
|
||||||
// - Consistency within a document: one input value always maps to one pseudonym, so
|
// - Consistency within a document: one input value always maps to one pseudonym, so
|
||||||
// correlations inside a run survive.
|
// correlations inside a run survive.
|
||||||
// - No consistency *across* documents unless the user asks for it: the salt is per-run by
|
// - No consistency *across* documents unless the user asks for it: the salt is per-run by
|
||||||
// default, so pseudonyms cannot be used to track a device between uploads. A stable salt is
|
// default, so pseudonyms cannot be used to track a device between uploads. A stable salt is
|
||||||
// opt-in (`Salt.stable`) for people diffing their own history on their own server.
|
// opt-in (`Salt.stable`) for people diffing their own history on their own server.
|
||||||
package app.echo_lot.privacy
|
package app.echo_lot.privacy
|
||||||
|
|
||||||
import kotlinx.serialization.json.*
|
import kotlinx.serialization.json.*
|
||||||
import java.security.MessageDigest
|
import java.security.MessageDigest
|
||||||
import java.util.Locale
|
import java.util.Locale
|
||||||
|
|
||||||
/** How much to strip. Ordered: FULL < BALANCED < STRICT. Wire values match the server's. */
|
/** How much to strip. Ordered: FULL < BALANCED < STRICT. Wire values match the server's. */
|
||||||
enum class PrivacyLevel(val wire: String) {
|
enum class PrivacyLevel(val wire: String) {
|
||||||
/** Nothing removed. The right choice for your own server. */
|
/** Nothing removed. The right choice for your own server. */
|
||||||
FULL("full"),
|
FULL("full"),
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Identifiers pseudonymized, neighbour inventory dropped. Topology and timing survive:
|
* Identifiers pseudonymized, neighbour inventory dropped. Topology and timing survive:
|
||||||
* you can still see that the gateway is a MikroTik at a /24 boundary with 3 % loss, but not
|
* you can still see that the gateway is a MikroTik at a /24 boundary with 3 % loss, but not
|
||||||
* which MikroTik, on which SSID, next to whose Chromecast.
|
* which MikroTik, on which SSID, next to whose Chromecast.
|
||||||
*/
|
*/
|
||||||
BALANCED("balanced"),
|
BALANCED("balanced"),
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Numbers only: tests keep their metrics and status, evidence is dropped, findings keep their
|
* Numbers only: tests keep their metrics and status, evidence is dropped, findings keep their
|
||||||
* codes and severities but lose descriptions (which quote real names). What is left cannot
|
* codes and severities but lose descriptions (which quote real names). What is left cannot
|
||||||
* identify a network, and is still enough for aggregate "how common is this fault" work.
|
* identify a network, and is still enough for aggregate "how common is this fault" work.
|
||||||
*/
|
*/
|
||||||
STRICT("strict");
|
STRICT("strict");
|
||||||
|
|
||||||
companion object {
|
companion object {
|
||||||
fun fromWire(s: String?): PrivacyLevel =
|
fun fromWire(s: String?): PrivacyLevel =
|
||||||
entries.firstOrNull { it.wire == s?.lowercase(Locale.ROOT) } ?: FULL
|
entries.firstOrNull { it.wire == s?.lowercase(Locale.ROOT) } ?: FULL
|
||||||
|
|
||||||
/** The stricter of two levels — used to honour a server's minimum. */
|
/** The stricter of two levels — used to honour a server's minimum. */
|
||||||
fun max(a: PrivacyLevel, b: PrivacyLevel): PrivacyLevel = if (a.ordinal >= b.ordinal) a else b
|
fun max(a: PrivacyLevel, b: PrivacyLevel): PrivacyLevel = if (a.ordinal >= b.ordinal) a else b
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The pseudonymization salt. Per-run by default: a fresh random salt means the same SSID uploaded
|
* The pseudonymization salt. Per-run by default: a fresh random salt means the same SSID uploaded
|
||||||
* twice yields two different pseudonyms, so an upload endpoint cannot link runs to a device.
|
* twice yields two different pseudonyms, so an upload endpoint cannot link runs to a device.
|
||||||
* A stable salt trades that away for cross-run diffing and is only appropriate on a server you
|
* A stable salt trades that away for cross-run diffing and is only appropriate on a server you
|
||||||
* own — the app makes that an explicit choice, not a default.
|
* own — the app makes that an explicit choice, not a default.
|
||||||
*/
|
*/
|
||||||
class Salt private constructor(internal val bytes: ByteArray, val stable: Boolean) {
|
class Salt private constructor(internal val bytes: ByteArray, val stable: Boolean) {
|
||||||
companion object {
|
companion object {
|
||||||
fun perRun(random: ByteArray): Salt = Salt(random.copyOf(), stable = false)
|
fun perRun(random: ByteArray): Salt = Salt(random.copyOf(), stable = false)
|
||||||
fun stable(secret: ByteArray): Salt = Salt(secret.copyOf(), stable = true)
|
fun stable(secret: ByteArray): Salt = Salt(secret.copyOf(), stable = true)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Transforms a measurement document to [level].
|
* Transforms a measurement document to [level].
|
||||||
*
|
*
|
||||||
* Field classification is by JSON key name, because the schema names things consistently
|
* Field classification is by JSON key name, because the schema names things consistently
|
||||||
* (`ssid`, `bssid`, `mac`, `ip4`, `ip6`, `fqdn`, …) and a name-driven pass is auditable by
|
* (`ssid`, `bssid`, `mac`, `ip4`, `ip6`, `fqdn`, …) and a name-driven pass is auditable by
|
||||||
* reading one table. Anything unrecognized is treated as identifying when it is a string inside
|
* reading one table. Anything unrecognized is treated as identifying when it is a string inside
|
||||||
* a known-sensitive container, and left alone otherwise — see [Classification].
|
* a known-sensitive container, and left alone otherwise — see [Classification].
|
||||||
*/
|
*/
|
||||||
class Anonymizer(private val level: PrivacyLevel, private val salt: Salt) {
|
class Anonymizer(private val level: PrivacyLevel, private val salt: Salt) {
|
||||||
|
|
||||||
private val cache = HashMap<String, String>()
|
private val cache = HashMap<String, String>()
|
||||||
|
|
||||||
fun anonymize(doc: JsonObject): JsonObject {
|
fun anonymize(doc: JsonObject): JsonObject {
|
||||||
if (level == PrivacyLevel.FULL) return stamp(doc)
|
if (level == PrivacyLevel.FULL) return stamp(doc)
|
||||||
val walked = walkObject(doc, path = emptyList())
|
val walked = walkObject(doc, path = emptyList())
|
||||||
val out = if (level == PrivacyLevel.STRICT) strip(walked) else walked
|
val out = if (level == PrivacyLevel.STRICT) strip(walked) else walked
|
||||||
return stamp(out)
|
return stamp(out)
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Records what was done, so a reader of the archived/uploaded document is never guessing. */
|
/** Records what was done, so a reader of the archived/uploaded document is never guessing. */
|
||||||
private fun stamp(doc: JsonObject): JsonObject {
|
private fun stamp(doc: JsonObject): JsonObject {
|
||||||
val run = doc["run"]?.jsonObject ?: return doc
|
val run = doc["run"]?.jsonObject ?: return doc
|
||||||
val privacy = buildJsonObject {
|
val privacy = buildJsonObject {
|
||||||
put("anonymization", level.wire)
|
put("anonymization", level.wire)
|
||||||
put("salt", if (salt.stable) "stable" else "per_run")
|
put("salt", if (salt.stable) "stable" else "per_run")
|
||||||
}
|
}
|
||||||
return JsonObject(doc + ("run" to JsonObject(run + ("privacy" to privacy))))
|
return JsonObject(doc + ("run" to JsonObject(run + ("privacy" to privacy))))
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- the tree walk -------------------------------------------------------------------
|
// ---- the tree walk -------------------------------------------------------------------
|
||||||
|
|
||||||
private fun walkObject(obj: JsonObject, path: List<String>): JsonObject = buildJsonObject {
|
private fun walkObject(obj: JsonObject, path: List<String>): JsonObject = buildJsonObject {
|
||||||
for ((k, v) in obj) {
|
for ((k, v) in obj) {
|
||||||
val childPath = path + k
|
val childPath = path + k
|
||||||
when {
|
when {
|
||||||
Classification.dropAtBalanced(childPath) -> Unit // omit entirely
|
Classification.dropAtBalanced(childPath) -> Unit // omit entirely
|
||||||
else -> put(k, walk(k, v, childPath))
|
else -> put(k, walk(k, v, childPath))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private fun walk(key: String, v: JsonElement, path: List<String>): JsonElement = when (v) {
|
private fun walk(key: String, v: JsonElement, path: List<String>): JsonElement = when (v) {
|
||||||
is JsonObject -> walkObject(v, path)
|
is JsonObject -> walkObject(v, path)
|
||||||
is JsonArray -> JsonArray(v.map { walk(key, it, path) })
|
is JsonArray -> JsonArray(v.map { walk(key, it, path) })
|
||||||
is JsonPrimitive ->
|
is JsonPrimitive ->
|
||||||
if (v.isString) {
|
if (v.isString) {
|
||||||
// Name first (it is precise), then shape (it is exhaustive). A field nobody
|
// Name first (it is precise), then shape (it is exhaustive). A field nobody
|
||||||
// classified must not be a field that leaks.
|
// classified must not be a field that leaks.
|
||||||
val type = Classification.typeOf(key, path) ?: Classification.inferFromValue(v.content)
|
val type = Classification.typeOf(key, path) ?: Classification.inferFromValue(v.content)
|
||||||
JsonPrimitive(transform(type, v.content))
|
JsonPrimitive(transform(type, v.content))
|
||||||
} else {
|
} else {
|
||||||
v
|
v
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private fun transform(type: LogicalType?, value: String): String = when (type) {
|
private fun transform(type: LogicalType?, value: String): String = when (type) {
|
||||||
null -> value
|
// Unclassified strings still get their *embedded* identifiers scrubbed. A whole-value
|
||||||
LogicalType.SSID -> pseudo("ssid", value) { "net-" + it.take(6) }
|
// check cannot see them: raw shell output is one long string that is neither a MAC nor an
|
||||||
LogicalType.MAC, LogicalType.BSSID -> macPreservingOui(value)
|
// address, so it sailed through both the name table and the shape check carrying every
|
||||||
LogicalType.IP4 -> ip4(value)
|
// MAC on the user's LAN.
|
||||||
LogicalType.IP6 -> ip6(value)
|
null -> scrubEmbedded(value)
|
||||||
LogicalType.FQDN -> fqdn(value)
|
LogicalType.SSID -> pseudo("ssid", value) { "net-" + it.take(6) }
|
||||||
LogicalType.OPAQUE_ID -> "redacted"
|
LogicalType.MAC, LogicalType.BSSID -> macPreservingOui(value)
|
||||||
LogicalType.FREETEXT -> "[removed: may contain identifying text]"
|
LogicalType.IP4 -> ip4(value)
|
||||||
}
|
LogicalType.IP6 -> ip6(value)
|
||||||
|
LogicalType.FQDN -> fqdn(value)
|
||||||
// ---- per-type transforms -------------------------------------------------------------
|
LogicalType.OPAQUE_ID -> "redacted"
|
||||||
|
LogicalType.FREETEXT -> "[removed: may contain identifying text]"
|
||||||
/**
|
}
|
||||||
* Keeps the OUI (which vendor) and pseudonymizes the NIC part (which unit). Vendor is the
|
|
||||||
* diagnostically valuable half — "the RA comes from a MikroTik" survives, "…from THAT
|
/**
|
||||||
* MikroTik" does not.
|
* Replaces addresses and MACs found *inside* a longer string.
|
||||||
*/
|
*
|
||||||
private fun macPreservingOui(value: String): String {
|
* Shizuku probes embed raw command output verbatim — `ip neigh`, `ip route`, `dumpsys` — which
|
||||||
val sep = if (value.contains('-')) '-' else ':'
|
* is genuinely valuable evidence and also a complete inventory of every device on the user's
|
||||||
val parts = value.split(sep)
|
* network, with hardware addresses. measurement-schema.md §9 flagged these as "hard to
|
||||||
if (parts.size != 6 || parts.any { it.length != 2 }) return pseudo("mac", value) { "mac-" + it.take(8) }
|
* anonymize" and proposed dropping them from exports.
|
||||||
val nic = pseudo("mac", value) { it }
|
*
|
||||||
return (parts.take(3) + listOf(nic.substring(0, 2), nic.substring(2, 4), nic.substring(4, 6)))
|
* Scrubbing beats dropping: the output stays readable and auditable — you can still see the
|
||||||
.joinToString(sep.toString())
|
* shape of the neighbour table and how many hosts there were — while the identifiers become
|
||||||
.lowercase(Locale.ROOT)
|
* the same pseudonyms used everywhere else in the document. So a MAC appearing both in a
|
||||||
}
|
* parsed field and in a raw dump still reads as one device.
|
||||||
|
*
|
||||||
/**
|
* Only addresses and MACs are touched, for the same reason as [Classification.inferFromValue]:
|
||||||
* Prefix-preserving within the same class, with reserved ranges kept verbatim: RFC1918 and
|
* they are the patterns that cannot be mistaken for something else in free text.
|
||||||
* CGNAT addresses say something about the topology and nothing about the person, and a run
|
*/
|
||||||
* where 192.168.1.1 became a random public address would be actively misleading to read.
|
private fun scrubEmbedded(value: String): String {
|
||||||
* Public addresses keep only their /16 so the network is still locatable at ISP granularity.
|
// Cheap bail-out: the overwhelming majority of strings are short and contain neither.
|
||||||
*/
|
if (value.length < 7 || (!value.contains(':') && !value.contains('.'))) return value
|
||||||
private fun ip4(value: String): String {
|
// One pass, not three. Sequential passes re-process their own output: after a MAC became
|
||||||
// A route destination carries a prefix length; pseudonymize the address and put it back,
|
// 78:9a:18:xx:yy:zz the IPv6 pattern matched it — six hex groups separated by colons is
|
||||||
// or "0.0.0.0/0" turns into nonsense and the routing table becomes unreadable.
|
// exactly an address — and mangled the vendor prefix that the MAC rule had just taken
|
||||||
value.substringAfter('/', "").takeIf { it.isNotEmpty() && value.contains('/') }?.let { len ->
|
// care to preserve. Ordered alternation resolves each position once, MAC first.
|
||||||
return ip4(value.substringBefore('/')) + "/" + len
|
return EMBEDDED.replace(value) { m ->
|
||||||
}
|
when {
|
||||||
val o = value.split(".")
|
m.groups[1] != null -> macPreservingOui(m.value)
|
||||||
if (o.size != 4 || o.any { it.toIntOrNull() == null }) return value
|
m.groups[2] != null -> ip6(m.value)
|
||||||
val n = o.map { it.toInt() }
|
else -> ip4(m.value)
|
||||||
val reserved = n[0] == 10 ||
|
}
|
||||||
(n[0] == 172 && n[1] in 16..31) ||
|
}
|
||||||
(n[0] == 192 && n[1] == 168) ||
|
}
|
||||||
(n[0] == 169 && n[1] == 254) ||
|
|
||||||
(n[0] == 100 && n[1] in 64..127) ||
|
// ---- per-type transforms -------------------------------------------------------------
|
||||||
n[0] == 127 || n[0] == 0 || n[0] >= 224
|
|
||||||
if (reserved) return value
|
/**
|
||||||
val h = pseudo("ip4", value) { it }
|
* Keeps the OUI (which vendor) and pseudonymizes the NIC part (which unit). Vendor is the
|
||||||
return "${n[0]}.${n[1]}.${h.substring(0, 2).toInt(16)}.${h.substring(2, 4).toInt(16)}"
|
* diagnostically valuable half — "the RA comes from a MikroTik" survives, "…from THAT
|
||||||
}
|
* MikroTik" does not.
|
||||||
|
*/
|
||||||
/**
|
private fun macPreservingOui(value: String): String {
|
||||||
* IPv6 keeps the scope and the first 32 bits (so 2001:db8:… still reads as global unicast in
|
val sep = if (value.contains('-')) '-' else ':'
|
||||||
* the same allocation) and pseudonymizes the rest — the interface identifier is the part that
|
val parts = value.split(sep)
|
||||||
* is a device fingerprint, especially with EUI-64.
|
if (parts.size != 6 || parts.any { it.length != 2 }) return pseudo("mac", value) { "mac-" + it.take(8) }
|
||||||
*/
|
val nic = pseudo("mac", value) { it }
|
||||||
private fun ip6(value: String): String {
|
return (parts.take(3) + listOf(nic.substring(0, 2), nic.substring(2, 4), nic.substring(4, 6)))
|
||||||
// Dotted quads reach here through the family-agnostic field names (addr, gateway, dst);
|
.joinToString(sep.toString())
|
||||||
// hand them to the IPv4 path rather than mangling them as if they were v6.
|
.lowercase(Locale.ROOT)
|
||||||
if (value.count { it == ':' } < 2) return ip4(value)
|
}
|
||||||
if (value.contains('/')) {
|
|
||||||
return ip6(value.substringBefore('/')) + "/" + value.substringAfter('/')
|
/**
|
||||||
}
|
* Prefix-preserving within the same class, with reserved ranges kept verbatim: RFC1918 and
|
||||||
val v = value.lowercase(Locale.ROOT)
|
* CGNAT addresses say something about the topology and nothing about the person, and a run
|
||||||
// The unspecified address and the default route are not identities; mangling them would
|
* where 192.168.1.1 became a random public address would be actively misleading to read.
|
||||||
// make a routing table unreadable for no privacy gain.
|
* Public addresses keep only their /16 so the network is still locatable at ISP granularity.
|
||||||
if (v == "::1" || v == "::" || v.startsWith("fe80:") || v.startsWith("ff")) return v
|
*/
|
||||||
|
private fun ip4(value: String): String {
|
||||||
// Unique local addresses (fc00::/7) need the *whole* prefix replaced, not the tail.
|
// A route destination carries a prefix length; pseudonymize the address and put it back,
|
||||||
//
|
// or "0.0.0.0/0" turns into nonsense and the routing table becomes unreadable.
|
||||||
// They look like the v6 equivalent of RFC1918, and the first instinct is to keep them for
|
value.substringAfter('/', "").takeIf { it.isNotEmpty() && value.contains('/') }?.let { len ->
|
||||||
// the same reason: private, topological, says nothing about anyone. That reasoning does
|
return ip4(value.substringBefore('/')) + "/" + len
|
||||||
// not carry over. An RFC1918 prefix is shared by millions of networks and identifies
|
}
|
||||||
// none of them; a ULA global ID is 40 *random* bits, unique to one network by
|
val o = value.split(".")
|
||||||
// construction (RFC 4193). It is a network fingerprint. Passing the leading groups
|
if (o.size != 4 || o.any { it.toIntOrNull() == null }) return value
|
||||||
// through - which is what the general path does - leaked 32 of those 40 bits.
|
val n = o.map { it.toInt() }
|
||||||
//
|
val reserved = n[0] == 10 ||
|
||||||
// The prefix is pseudonymized as a unit, so two addresses on the same ULA subnet still
|
(n[0] == 172 && n[1] in 16..31) ||
|
||||||
// land on the same pseudonymous prefix. "These hosts are on one network" survives;
|
(n[0] == 192 && n[1] == 168) ||
|
||||||
// "this is *that* network" does not.
|
(n[0] == 169 && n[1] == 254) ||
|
||||||
if (v.startsWith("fc") || v.startsWith("fd")) {
|
(n[0] == 100 && n[1] in 64..127) ||
|
||||||
val groups = v.substringBefore('%').split(":")
|
n[0] == 127 || n[0] == 0 || n[0] >= 224
|
||||||
val prefix = pseudo("ula-prefix", groups.take(3).joinToString(":")) { it }
|
if (reserved) return value
|
||||||
val host = pseudo("ula-host", v) { it }
|
val h = pseudo("ip4", value) { it }
|
||||||
return "fd${prefix.substring(0, 2)}:${prefix.substring(2, 6)}:${prefix.substring(6, 10)}" +
|
return "${n[0]}.${n[1]}.${h.substring(0, 2).toInt(16)}.${h.substring(2, 4).toInt(16)}"
|
||||||
"::${host.substring(0, 4)}"
|
}
|
||||||
}
|
|
||||||
val groups = v.substringBefore('%').split(":")
|
/**
|
||||||
if (groups.size < 3) return v
|
* IPv6 keeps the scope and the first 32 bits (so 2001:db8:… still reads as global unicast in
|
||||||
val h = pseudo("ip6", value) { it }
|
* the same allocation) and pseudonymizes the rest — the interface identifier is the part that
|
||||||
return "${groups[0]}:${groups[1]}:${h.substring(0, 4)}:${h.substring(4, 8)}::${h.substring(8, 12)}"
|
* is a device fingerprint, especially with EUI-64.
|
||||||
}
|
*/
|
||||||
|
private fun ip6(value: String): String {
|
||||||
/**
|
// Dotted quads reach here through the family-agnostic field names (addr, gateway, dst);
|
||||||
* Per-label pseudonyms with the public suffix kept, so "it resolved somewhere under .local"
|
// hand them to the IPv4 path rather than mangling them as if they were v6.
|
||||||
* or "…under example.com" survives without naming the host. The suffix list is deliberately
|
if (value.count { it == ':' } < 2) return ip4(value)
|
||||||
* short: guessing wrong keeps *more* pseudonymized, never less.
|
if (value.contains('/')) {
|
||||||
*/
|
return ip6(value.substringBefore('/')) + "/" + value.substringAfter('/')
|
||||||
private fun fqdn(value: String): String {
|
}
|
||||||
if (value.isEmpty()) return value
|
val v = value.lowercase(Locale.ROOT)
|
||||||
val trailing = value.endsWith(".")
|
// The unspecified address and the default route are not identities; mangling them would
|
||||||
val labels = value.trimEnd('.').split(".")
|
// make a routing table unreadable for no privacy gain.
|
||||||
if (labels.size == 1) return pseudo("fqdn", value) { "host-" + it.take(6) }
|
if (v == "::1" || v == "::" || v.startsWith("fe80:") || v.startsWith("ff")) return v
|
||||||
val keep = if (labels.last() in publicSuffixes) 1 else 0
|
|
||||||
val head = labels.dropLast(keep).map { l -> pseudo("label", l) { "l-" + it.take(6) } }
|
// Unique local addresses (fc00::/7) need the *whole* prefix replaced, not the tail.
|
||||||
return (head + labels.takeLast(keep)).joinToString(".") + if (trailing) "." else ""
|
//
|
||||||
}
|
// They look like the v6 equivalent of RFC1918, and the first instinct is to keep them for
|
||||||
|
// the same reason: private, topological, says nothing about anyone. That reasoning does
|
||||||
// ---- STRICT ---------------------------------------------------------------------------
|
// not carry over. An RFC1918 prefix is shared by millions of networks and identifies
|
||||||
|
// none of them; a ULA global ID is 40 *random* bits, unique to one network by
|
||||||
/**
|
// construction (RFC 4193). It is a network fingerprint. Passing the leading groups
|
||||||
* STRICT keeps the shape of the document and the numbers, and nothing that quotes the
|
// through - which is what the general path does - leaked 32 of those 40 bits.
|
||||||
* network back. Evidence goes (trains carry addresses and hostnames), finding prose goes
|
//
|
||||||
* (it interpolates real names), networks go entirely.
|
// The prefix is pseudonymized as a unit, so two addresses on the same ULA subnet still
|
||||||
*/
|
// land on the same pseudonymous prefix. "These hosts are on one network" survives;
|
||||||
private fun strip(doc: JsonObject): JsonObject = buildJsonObject {
|
// "this is *that* network" does not.
|
||||||
for ((k, v) in doc) {
|
if (v.startsWith("fc") || v.startsWith("fd")) {
|
||||||
when (k) {
|
val groups = v.substringBefore('%').split(":")
|
||||||
"networks", "server_sessions" -> Unit
|
val prefix = pseudo("ula-prefix", groups.take(3).joinToString(":")) { it }
|
||||||
"tests" -> put(k, JsonArray((v as? JsonArray ?: JsonArray(emptyList())).map { t ->
|
val host = pseudo("ula-host", v) { it }
|
||||||
val o = t.jsonObject
|
return "fd${prefix.substring(0, 2)}:${prefix.substring(2, 6)}:${prefix.substring(6, 10)}" +
|
||||||
JsonObject(o.filterKeys { it != "evidence" && it != "params" })
|
"::${host.substring(0, 4)}"
|
||||||
}))
|
}
|
||||||
"findings" -> put(k, JsonArray((v as? JsonArray ?: JsonArray(emptyList())).map { f ->
|
val groups = v.substringBefore('%').split(":")
|
||||||
val o = f.jsonObject
|
if (groups.size < 3) return v
|
||||||
JsonObject(o.filterKeys { it != "description" && it != "title" && it != "evidence_refs" })
|
val h = pseudo("ip6", value) { it }
|
||||||
}))
|
return "${groups[0]}:${groups[1]}:${h.substring(0, 4)}:${h.substring(4, 8)}::${h.substring(8, 12)}"
|
||||||
"run" -> put(k, JsonObject(v.jsonObject.filterKeys { it != "notes" }))
|
}
|
||||||
else -> put(k, v)
|
|
||||||
}
|
/**
|
||||||
}
|
* Per-label pseudonyms with the public suffix kept, so "it resolved somewhere under .local"
|
||||||
}
|
* or "…under example.com" survives without naming the host. The suffix list is deliberately
|
||||||
|
* short: guessing wrong keeps *more* pseudonymized, never less.
|
||||||
// ---- pseudonym machinery ---------------------------------------------------------------
|
*/
|
||||||
|
private fun fqdn(value: String): String {
|
||||||
/** Deterministic per (domain, value, salt); memoized so one value maps to one pseudonym. */
|
if (value.isEmpty()) return value
|
||||||
private fun pseudo(domain: String, value: String, shape: (String) -> String): String =
|
val trailing = value.endsWith(".")
|
||||||
cache.getOrPut("$domain | |||||||